{"notes":[{"content":{"venue":{"value":"CoRR 2023"},"pdf":{"value":"http://arxiv.org/pdf/2312.00752v2"},"venueid":{"value":"dblp.org/journals/CORR/2023"},"paperhash":{"value":"gu|mamba_lineartime_sequence_modeling_with_selective_state_spaces"},"authorids":{"value":["~Albert_Gu1","https://dblp.org/search/pid/api?q=author:Tri_Dao:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2312.00752"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2312-00752,\n  publtype={informal},\n  author={Albert Gu and Tri Dao},\n  title={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},\n  year={2023},\n  cdate={1672531200000},\n  journal={CoRR},\n  volume={abs/2312.00752},\n  url={https://doi.org/10.48550/arXiv.2312.00752}\n}\n"},"abstract":{"value":"Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5$\\times$ higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation."},"title":{"value":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces"},"authors":{"value":["Albert Gu","Tri Dao"]}},"tmdate":1737249477667,"pdate":1672531200000,"tcdate":1737249467645,"writers":["~"],"signatures":["~Albert_Gu1"],"forum":"u7TcGRvHEr","license":"CC BY-SA 4.0","number":268108,"cdate":1672531200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1737249477667,"domain":"DBLP.org","id":"u7TcGRvHEr","version":2},{"content":{"TLDR":{"value":"This paper introduces the Mamba architecture."},"venue":{"value":"COLM"},"code_of_ethics":{"value":["I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"]},"keywords":{"value":"sequence model, deep learning, state space model, S4, Mamba"},"abstract":{"value":"Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5x higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation."},"_bibtex":{"value":"@inproceedings{\ngu2024mamba,\ntitle={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},\nauthor={Albert Gu and Tri Dao},\nbooktitle={First Conference on Language Modeling},\nyear={2024},\nurl={https://openreview.net/forum?id=tEYskw1VY2}\n}"},"title":{"value":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces"},"research_area":{"value":["Compute efficient LMs","Learning algorithms for LMs"]},"pdf":{"value":"/pdf/05ba820a837c2f704f5b5bb1ccdb66c503b0e180.pdf"},"venueid":{"value":"colmweb.org/COLM/2024/Conference"},"paperhash":{"value":"gu|mamba_lineartime_sequence_modeling_with_selective_state_spaces"},"authorids":{"value":["~Albert_Gu1","~Tri_Dao1"]},"author_guide":{"value":["I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"]},"authors":{"value":["Albert Gu","Tri Dao"]}},"tmdate":1724633538480,"pdate":1720608922685,"tcdate":1711143523490,"writers":["colmweb.org/COLM/2024/Conference","colmweb.org/COLM/2024/Conference/Submission1039/Authors"],"signatures":["colmweb.org/COLM/2024/Conference/Submission1039/Authors"],"forum":"tEYskw1VY2","license":"CC BY 4.0","number":1039,"cdate":1711143523490,"readers":["everyone"],"invitations":["colmweb.org/COLM/2024/Conference/-/Submission","colmweb.org/COLM/2024/Conference/-/Post_Submission","colmweb.org/COLM/2024/Conference/Submission1039/-/Revision","colmweb.org/COLM/2024/Conference/-/Edit","colmweb.org/COLM/2024/Conference/Submission1039/-/Camera_Ready"],"mdate":1724633538480,"odate":1724633538465,"domain":"colmweb.org/COLM/2024/Conference","id":"tEYskw1VY2","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2024"},"TLDR":{"value":"We introduce a selection mechanism into state space models, leading to state-of-the-art results on general sequence modeling including language."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Sequence model","language model","state space model","RNN","SSM","S4","Mamba"]},"primary_area":{"value":"representation learning for computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language.  We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to *selectively* propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (**Mamba**). Mamba enjoys fast inference (5$\\times$ higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-1.4B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation."},"_bibtex":{"value":"@misc{\ngu2024mamba,\ntitle={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},\nauthor={Albert Gu and Tri Dao},\nyear={2024},\nurl={https://openreview.net/forum?id=AL1fq05o7H}\n}"},"title":{"value":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces"},"pdf":{"value":"/pdf/27144ad57139872b668e5c522eeed631b23f5dce.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"gu|mamba_lineartime_sequence_modeling_with_selective_state_spaces"},"authorids":{"value":["~Albert_Gu1","~Tri_Dao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Albert Gu","Tri Dao"]}},"tmdate":1707625706736,"tcdate":1695337863403,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission4202/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission4202/Authors"],"forum":"AL1fq05o7H","number":4202,"cdate":1695337863403,"mdate":1707625706736,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/-/Submission","ICLR.cc/2024/Conference/-/Post_Submission","ICLR.cc/2024/Conference/Submission4202/-/Revision","ICLR.cc/2024/Conference/-/Edit"],"odate":1697213872796,"domain":"ICLR.cc/2024/Conference","id":"AL1fq05o7H","version":2},{"content":{"summary":{"value":"This paper proposes MRConv, a new way to parameterize global convolutional kernels for long sequence modeling. MRConv pads all sub-kernels to the same length and aggregate outputs of sub-kernels with batchnorm and linear rescaling. Three different kernel initializations are explored. Experiments on Long Range Arena and speech and image classification tasks show that MRConv is more computationally efficient than SSM baselines and slightly better than SGConv in accuracy."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"1. Line 107-110 says BatchNorm is added since output statistics are different for different kernel size. But BatchNorm is normalizing over instances in the same batch, the difference of kernel size still remains. Am I understanding this correctly or is there other reason behind BatchNorm?\n2. Line 159, what is this $k$ ? The number of parameters in the kernel?\n3. Is it possible to have all three kernel parameterization together using gating mechanism to choose which kernel to use?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. MRConv achieves state-of-the-art performance on LRA benchmark and several classification tasks compared to prior long sequence modeling methods. \n2. MRConv maintains inference speed advantage over SSM baselines with added modeling complexity compared to SGConv.\n3. Ablations show that Fourier kernels are the most robust across different tasks, saving users' time to choose kernels themselves."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Lack experiments on generative tasks like language modeling, speech and image synthesis. These are more challenging tasks where many SSMs like Mamba [1] are already tested on. \n2. Multi resolutional convolution methods including SGConv and MRConv lack theoretical analysis on its expressivity. Is it more powerful than SSM or are they equivalent? A clarification is needed.\n3. Table 8 shows that MRConv needs additional modification to work on Path-X task. Task-specific modification on the model diminishes its general applicability to a wider range of tasks. \n\n[1] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023)."},"limitations":{"value":"The authors have adequately addressed the limitations."}},"nonreaders":[],"tmdate":1730879322208,"tcdate":1720830093841,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission9552/Reviewer_pbnE"],"signatures":["NeurIPS.cc/2024/Conference/Submission9552/Reviewer_pbnE"],"forum":"RwgNbIpCpk","number":3,"license":"CC BY 4.0","cdate":1720830093841,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission9552/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879322208,"domain":"NeurIPS.cc/2024/Conference","replyto":"RwgNbIpCpk","id":"Mv2Od1gVvu","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Sequence Modelling","Convolutions","Structural Reparameterization","Self Attention","State Space Models"]},"supplementary_material":{"value":"/attachment/b31a082bbd7b4c4b945895ba10a0c90591d33abb.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ($\\texttt{MRConv}$), a novel approach to parameterizing global convolutional kernels for long-sequence modeling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, $\\texttt{MRConv}$ learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D $\\texttt{MRConv}$ layers."},"_bibtex":{"value":"@inproceedings{\ncunningham2024reparameterized,\ntitle={Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling},\nauthor={Harry Jake Cunningham and Giorgio Giannone and Mingtian Zhang and Marc Peter Deisenroth},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=RwgNbIpCpk}\n}"},"title":{"value":"Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling"},"pdf":{"value":"/pdf/99fdd8c02bfd174bfdb9bf7e36f2b9016b370ad4.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"cunningham|reparameterized_multiresolution_convolutions_for_long_sequence_modelling"},"authorids":{"value":["~Harry_Jake_Cunningham1","~Giorgio_Giannone1","~Mingtian_Zhang1","~Marc_Peter_Deisenroth1"]},"authors":{"value":["Harry Jake Cunningham","Giorgio Giannone","Mingtian Zhang","Marc Peter Deisenroth"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Mamba-Branching, a novel method for the Branch and Bound (B&B) algorithm in Mixed Integer Linear Programming (MILP) solvers. The authors' key insight is that existing neural branching methods (e.g., TreeGate and T-BranT) fail to effectively exploit the sequential nature of B&B tree expansion. To address this, they propose using the Mamba architecture (a selective state-space model) to model the complete branching path from the root to the current node. The method treats the historical decisions and tree states along the path as a long sequence. Experiments on the MILP-S and MILP-L datasets show that Mamba-Branching outperforms existing neural branching methods (TreeGate, T-BranT) in solving time, primal-dual integral (PD Integral), and number of nodes, and significantly surpasses Transformer-based models in computational efficiency."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"+ Are Mamba-Branching and baselines (e.g., TreeGate) compared using identical input features? It is recommended to disclose the feature list to ensure a fair comparison.\n+ Has the method's generalization been validated on datasets beyond MILP-S/L, such as ML4CO 2021?\n+ How does model performance vary with the length of the B&B path? Are there any observed performance bottlenecks for very long sequences?\n+ Could the authors compare their approach with recent learning to branch methods [1, 2] to better position their contribution?\n+ What is the specific selection logic for the instances used in the experiments?\n\n**If the authors can convincingly demonstrate the method's generalization capability and the effectiveness of the contrastive learning component through additional experiments, I would be willing to update my score accordingly**.\n\n[1] Zhang C, Ouyang W, Yuan H, et al. Towards imitation learning to branch for mip: A hybrid reinforcement learning based sample augmentation approach[C]// ICLR 2024.\n\n[2] Parsonson C W F, Laterre A, Barrett T D. Reinforcement learning for branch-and-bound optimisation using retrospective trajectories[C]// AAAI 2023."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"+ Insightful problem formulation: The observation that traditional neural methods \"generally overlook the sequential nature of B&B tree expansion\" is a key insight. Modeling the \"branching path\" as a sequence is a novel and well-motivated idea that aligns with the intrinsic logic of the B&B process.\n+ Outstanding computational efficiency: The paper clearly demonstrates the memory and speed advantages of Mamba-Branching in training and inference through Table 13. Compared to Transformer-Branching, it consumes significantly less memory and achieves an 80x faster inference speed, a remarkable advantage crucial for practical applications."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"+ **Insufficient validation of the contrastive learning module**: The effectiveness of the contrastive learning (CL) module is inadequately validated. Although the paper introduces CL for embedding layer training in Section 4.1 and presents its effect via t-SNE visualization in Figure 6, the argument remains weak. The authors only compare with the \"without CL\" setting, which only proves that CL itself is helpful, but fails to demonstrate that their specific CL design is superior or necessary. To more comprehensively assess the value of their approach, the authors should compare their CL framework against several standard and widely-used contrastive learning loss functions and paradigms, such as InfoNCE.\n+ **Ambiguous innovation attribution of Mamba**: The paper's demonstration of computational efficiency (Table 13, Figure 3) is excellent, strongly proving Mamba's significant advantage over Transformers in handling long sequential branching histories. However, these advantages primarily stem from the inherent properties of the Mamba architecture as an existing advanced model (e.g., linear complexity, selective state spaces), and the paper does not introduce any structural or algorithmic modifications or optimizations to the Mamba model itself. Therefore, the core innovation of the work leans more towards \"successfully applying the cutting-edge sequence model Mamba to B&B.\""}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923003289,"tcdate":1761653413291,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12016/Reviewer_7iig"],"signatures":["ICLR.cc/2026/Conference/Submission12016/Reviewer_7iig"],"forum":"iQKzHeLDrN","number":2,"license":"CC BY 4.0","cdate":1761653413291,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12016/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923003289,"domain":"ICLR.cc/2026/Conference","replyto":"iQKzHeLDrN","id":"o0Ell8alir","forumContent":{"TLDR":{"value":"We propose a branching policy that utilizes the sequential branching path of the branch-and-bound tree to guide current decision."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Mixed-Integer Linear Programming","Branch-and-Bound","Branching Variable Selection Policy","Generalization","Mamba"]},"supplementary_material":{"value":"/attachment/92e991cc82f9ad41529abd7aef31e00b1f3c69c9.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"The branch-and-bound (B\\&B) method is a dominant exact algorithm for solving Mixed-Integer Linear Programming problems (MILPs). While recent deep learning approaches have shown promise in learning branching policies using instance-independent features, they often struggle to capture the sequential decision-making nature of B\\&B, particularly over long horizons with complex inter-step dependencies and intra-step variable interactions. To address these challenges, we propose Mamba-Branching, a novel learning-based branching policy that leverages the Mamba architecture for efficient long-sequence modeling, enabling effective capture of temporal dynamics across B\\&B steps. Additionally, we introduce a contrastive learning strategy to pre-train discriminative embeddings for candidate branching variables, significantly enhancing Mamba's performance. Experimental results demonstrate that Mamba-Branching outperforms all previous neural branching policies on real-world MILP instances and achieves superior computational efficiency compared to the advanced open-source solver SCIP. The source code can be accessed at https://github.com/doctor-watson626/Mamba-Branching/."},"_bibtex":{"value":"@inproceedings{\nzhang2026towards,\ntitle={Towards Better Branching Policies: Leveraging the Sequential Nature of Branch-and-Bound Tree},\nauthor={Ce Zhang and Bin Zhang and Guoliang Fan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=iQKzHeLDrN}\n}"},"title":{"value":"Towards Better Branching Policies: Leveraging the Sequential Nature of Branch-and-Bound Tree"},"pdf":{"value":"/pdf/252a128e120299bcb68a697f9200fb14e8927006.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|towards_better_branching_policies_leveraging_the_sequential_nature_of_branchandbound_tree"},"authorids":{"value":["~Ce_Zhang10","~Bin_Zhang12","~Guoliang_Fan3"]},"authors":{"value":["Ce Zhang","Bin Zhang","Guoliang Fan"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Mamba, which is a linear-time sequence model with selective state spaces. The authors propose to modify conventional state space models (SSMs) such that the modified models are input-dependent. The authors further propose engineering techniques for performance optimization. Experiments are conducted to demonstrate the effectiveness of the proposed method. In particular, several flavors of pre-trained models are provided."},"presentation":{"value":"3 good"},"contribution":{"value":"2 fair"},"soundness":{"value":"2 fair"},"strengths":{"value":"* The proposed Mamba method includes a simple modification to the conventional SSM model: add additional models to make SSM models dependent on the inputs. SSMs are known for their computational difficulties, and the authors address this issue by several performance optimization techniques.\n\n* The authors pre-train several variants of Mamba, ranging from 130M parameters to 1.4B parameters. These pre-trained models show performance improvements compared with the baselines in the paper."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Concerns about model design:\n\n* The motivation of Mamba is to address the drawbacks of recurrent models while improving the efficiency of attention-based models. There are many works following the same direction: S4-diagonal [1], SGConv [2], MEGA [3], SPADE [4], and many efficient Transformer models (e.g., [5]). All of these models achieve near linear complexity, and the authors need to compare Mamba with these works in terms of both model performance and efficiency. For model performance, some simple experiments such as language modeling on Wikitext-103 should suffice.\n\n* Many attention-based Transformer models show length generalization ability, i.e., models can be trained on a shorter sequence length and tested on a longer sequence length. Some examples include relative positional encoding (T5) and Alibi [6]. Because SSMs are in general sequential, does Mamba have this length generalization ability?\n\n\nConcerns about experiments:\n\n* The authors need to compare with stronger baselines. The authors acknowledge that H3 was used as a motivation for the model architecture. However, they did not compare with H3 in the experiments. From Table 4 in [7], ppl of H3 is 8.8 (125M), 7.1 (355M), and 6.0 (1.3B) on the Pile dataset, which are considerably better than Mamba. The authors need to show comparisons with H3.\n\n* For the pre-trained models, the authors only show results on zero-shot inference. This setting is quite limited and the results cannot support the effectiveness of Mamba well. I suggest the authors run more long-sequence experiments such as document summarization, where the input sequence is naturally long (e.g., the average sequence length of the arXiv dataset is greater than 8k).\n\n* One of the main contributions that the authors claim is long sequence modeling. The authors should compare with more baselines on LRA (Long Range Arena), which is essentially the standard benchmark for long sequence understanding.\n\n* Memory benchmarking is missing. Even though Section 4.5 is titled “speed and memory benchmark”, only speed comparisons are presented. Also, the authors should provide more detailed setups of Figure 8 left, e.g., model layers, model sizes, details of the convolution, etc. Could the authors provide some intuitions why FlashAttention is the slowest when the sequence length is very large (Figure 8 left)?\n\n[1] https://arxiv.org/pdf/2203.14343.pdf \\\n[2] https://arxiv.org/pdf/2210.09298.pdf \\\n[3] https://arxiv.org/pdf/2209.10655.pdf \\\n[4] https://arxiv.org/pdf/2212.08136.pdf \\\n[5] https://arxiv.org/pdf/2202.10447.pdf \\\n[6] https://arxiv.org/pdf/2108.12409.pdf \\\n[7] https://arxiv.org/pdf/2212.14052.pdf"},"confidence":{"value":"5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully."},"questions":{"value":"See above"},"rating":{"value":"3: reject, not good enough"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636386876,"tcdate":1699240330960,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission4202/Reviewer_du8a"],"signatures":["ICLR.cc/2024/Conference/Submission4202/Reviewer_du8a"],"forum":"AL1fq05o7H","number":4,"license":"CC BY 4.0","cdate":1699240330960,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission4202/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636386876,"domain":"ICLR.cc/2024/Conference","replyto":"AL1fq05o7H","id":"eIarHE3vGs","forumContent":{"venue":{"value":"Submitted to ICLR 2024"},"TLDR":{"value":"We introduce a selection mechanism into state space models, leading to state-of-the-art results on general sequence modeling including language."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Sequence model","language model","state space model","RNN","SSM","S4","Mamba"]},"primary_area":{"value":"representation learning for computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language.  We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to *selectively* propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (**Mamba**). Mamba enjoys fast inference (5$\\times$ higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-1.4B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation."},"_bibtex":{"value":"@misc{\ngu2024mamba,\ntitle={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},\nauthor={Albert Gu and Tri Dao},\nyear={2024},\nurl={https://openreview.net/forum?id=AL1fq05o7H}\n}"},"title":{"value":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces"},"pdf":{"value":"/pdf/27144ad57139872b668e5c522eeed631b23f5dce.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"gu|mamba_lineartime_sequence_modeling_with_selective_state_spaces"},"authorids":{"value":["~Albert_Gu1","~Tri_Dao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Albert Gu","Tri Dao"]}},"version":2},{"content":{"summary":{"value":"The paper introduces mGRADE which is a minimal linear GRU with convolutions that support parallel training using scan algorithm and RNN-style inference of State Space Models and Linear Transformers. The main motivation behind design is to build a device for embedded systems and resource constraint environments. The experiments are done in LRA benchmark and Lorenz attractor.  moreover, the paper highlights that mGRADE enjoys selective gating which helps for long-context modeling (which is shown in LRA benchmark)."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1) What is the training time of mGRADE compared to Mamba2 specifically since it is using SSD?\n\n2) How does the model perform in time-seires tasks such as **Table 13** of S4 paper [1]?\n\n3) How does mGRADE works without causal convoulution on LRA to see how performant is the sequence mixing of the mGRADE?\n\n4) What is the difference of mGRADE's minimal GRU compared to GRFA's or Mamba2's sequence mixing?\n\n------\n\n### References\n\n[1] Efficiently Modeling Long Sequences with Structured State Spaces: Albert Gu, Karan Goel, and Christopher R´e"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The major strengths of paper are:\n\n- **Direction of the paper** Paper build upon a current important direction of efficient sequence modeling with sub-quadratic models such as linear transformers and SSMs.\n\n- **Background analysis** Paper nicely present a very sufficient background material on SSMs and progress of literature specially in its related work section and introduction.\n\n- **Loranz and LRA Experiments** The design of these experiments are indeed relevant and sound.\n\n- **Presentation and writing** The paper is well-written and clearly presented its approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"However, while the paper is generally well written and the overall design appears sound, there are several important gray areas regarding comparisons with SSMs—particularly Mamba, Mamba2, and GLA [2], as well as some outdated training strategies.\n\n* **Model differences with Mamba2 and Gated RFA:**\n  The main architectural components appear very similar to existing SSMs. In particular, the sequence mixing mechanism based on the minimal GRU closely resembles the recurrence structure of the Gated RFA. Moreover, the role and necessity of short-range causal Conv1D layers have already been explored in modern Linear Transformers such as Gated DeltaNet [1] and in prior analyses (see [this post](https://kexue.fm/archives/11320)). This raises the question: what is the key difference and contribution of mGRADE compared to existing models, given that both causal Conv1D and GRU-style recurrence have been well studied in SSMs?\n\n* **Lack of time-series experiments:**\n  Although the paper claims to target sensory data and signal processing, it does not include any experiments on time-series tasks. The only experimental setup is the LRA benchmark, which is both outdated and largely synthetic. For a model motivated by real-world signals, it would be important to demonstrate performance on datasets such as *Speech Commands* (as used in S4 models), or other representative time-series benchmarks.\n\n* **Training efficiency and scan algorithm:**\n  The paper relies on the parallel scan algorithm for training, which has been shown to be significantly slower and less efficient than chunkwise parallel training strategies such as those used in SSD and GLA (see Figure 6 of the paper). It would be useful to clarify how the training time of mGRADE compares to Mamba2 and GLA, and whether mGRADE also supports chunkwise parallel training.\n\n* **Selectivity and language modeling:**\n  If the model is designed to be selective, how does it perform on language modeling tasks? Selective SSMs like Mamba were specifically developed for language tasks and are known to perform poorly on LRA benchmarks while excelling on linguistic datasets (as highlighted by the authors [here](https://github.com/state-spaces/mamba/issues/282#issuecomment-2221135197)). It would be important to see how mGRADE performs in comparison, and whether its selectivity mechanism aligns with those used in language-oriented SSMs.\n\n\n\n\n-------\n\n\n### References\n\n[1] Gated Delta Networks: Improving Mamba2 with Delta Rule: Songlin Yang, Jan Kautz, Ali Hatamizadeh\n\n\n[2] Gated Linear Attention Transformers with Hardware-Efficient Training: Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931393541,"tcdate":1761909289989,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19489/Reviewer_7yyH"],"signatures":["ICLR.cc/2026/Conference/Submission19489/Reviewer_7yyH"],"forum":"nqUwo7R2vv","number":2,"license":"CC BY 4.0","cdate":1761909289989,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19489/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931393541,"domain":"ICLR.cc/2026/Conference","replyto":"nqUwo7R2vv","id":"ZYR6ImcDD2","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["RNN","sequence processing","model architecture","embedded systems"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Processing temporal data directly at the sensor source demands models that capture both short- and long-range dynamics under tight memory constraints. While State-of-the-Art (SotA) sequence models such as Transformers excel at these tasks, their quadratic memory scaling with sequence length makes them impractical for edge settings. Recurrent Neural Networks (RNNs) offer constant memory scaling, but train sequentially and slowly, and Temporal Convolutional Networks (TCNs), though efficiently trainable, also scale memory with kernel length. For more memory-efficient sequence modeling, we propose mGRADE (minimally Gated Recurrent Architecture with Delay Embedding), a hybrid-memory system that integrates a temporal convolution with learnable spacings with a gated recurrent component equivalent to a minimal Gated Recurrent Unit (minGRU). The convolution with learnable spacings can express a flexible delay embedding that captures rapid temporal variations, while the recurrent component efficiently maintains global context with minimal memory overhead. We theoretically ground and empirically validate our approach on two types of synthetic tasks, demonstrating that mGRADE effectively separates and preserves temporal features across multiple timescales. Furthermore, on the challenging Long-Range Arena (LRA) benchmark, mGRADE reduces the memory footprint by up to a factor of 8, while maintaining competitive performance compared to SotA models."},"_bibtex":{"value":"@misc{\ntorchet2026mgrade,\ntitle={m{GRADE}: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling},\nauthor={Tristan Torchet and Christian Metzner and Karthik Charan Raghunathan and Jimmy Weber and Laura Kriener and Melika Payvand},\nyear={2026},\nurl={https://openreview.net/forum?id=nqUwo7R2vv}\n}"},"title":{"value":"mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling"},"pdf":{"value":"/pdf/5c41faee958abf75002def8814a54ee2bdafa10b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"torchet|mgrade_minimal_recurrent_gating_meets_delay_convolutions_for_lightweight_sequence_modeling"},"authorids":{"value":["~Tristan_Torchet1","~Christian_Metzner1","~Karthik_Charan_Raghunathan1","~Jimmy_Weber1","~Laura_Kriener1","~Melika_Payvand1"]},"authors":{"value":["Tristan Torchet","Christian Metzner","Karthik Charan Raghunathan","Jimmy Weber","Laura Kriener","Melika Payvand"]}},"version":2},{"content":{"summary":{"value":"The paper \"Demystifying the Token Dynamics of Deep Selective State Space Models\" explores the theoretical behavior of tokens in the Mamba model, a type of Selective State Space Model (SSM) used in sequence tasks like language modeling and image processing. Despite Mamba's empirical success, its internal dynamics are not well understood.\nThe authors show that in a one-dimensional case, tokens either converge to zero or diverge to infinity, depending on model parameters. Convergence harms performance, while divergence leads to unequal contributions from tokens during training. To address this, the paper proposes two improvements: eliminating the convergent scenario and reordering tokens by importance. Experiments confirm these refinements improve accuracy and speed, particularly in image classification tasks.\nThis study offers new theoretical insights into token dynamics and suggests practical adjustments to improve the Mamba model's performance."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"**Summary of Actionable Improvements**:\n- Expand the theoretical analysis to higher dimensions or at least provide empirical evidence for higher-dimensional token dynamics.\n- Broaden the experimental validation to include diverse tasks beyond vision (e.g., language modeling and reinforcement learning).\n- Discuss the computational costs and trade-offs of the proposed refinements in real-world applications.\n- Increase the novelty by exploring more complex dynamical behaviors or offering a deeper comparison with existing models like transformers.\n- Provide a more comprehensive experimental comparison to other recent SSMs and transformers, including computational efficiency."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- **Originality**: The paper makes a meaningful contribution by exploring the often overlooked internal dynamics of tokens in Selective State Space Models (SSMs), specifically the Mamba model. While SSMs are well-known for their efficiency in sequence modeling, this work tackles a fresh problem—understanding how token behavior impacts model performance. By analyzing the continuous-time limit of token dynamics, the authors open up a new line of inquiry that could lead to more informed model designs and optimizations. The refinements proposed—eliminating the convergent token scenario and reordering tokens by their importance—are practical ideas that could lead to significant performance gains.\n- **Quality**: The paper is technically solid, with rigorous theoretical analysis that provides clear criteria for determining whether tokens in the Mamba model will converge to zero or diverge to infinity. This kind of formal, mathematical foundation is a strong aspect of the paper, ensuring that its claims are well-supported. The empirical results on large-scale tasks such as ImageNet classification offer clear evidence that the proposed refinements are effective in improving model accuracy and convergence speed. The experiments directly tie into the theoretical findings, showcasing a strong link between the analysis and practical outcomes.\n- **Clarity**: The paper is clearly structured, with a logical flow from the problem statement, through the theoretical analysis, to the empirical validation. The technical content, while dense in some areas, is well-explained, and the use of diagrams and tables helps to clarify key findings. The authors do a good job of contextualizing their work within existing research, making it clear how their contributions build on prior work in SSMs and neural ODEs. Overall, the clarity of the presentation allows readers to follow the complex theoretical insights without becoming lost in technical details.\n- **Significance**: The significance of the work lies in its potential to impact both the theoretical understanding and practical application of SSMs. By uncovering how token dynamics influence model performance, the paper provides a foundation for improving not only Mamba but other models that rely on similar state-space mechanisms. The proposed refinements—removing harmful convergent behaviors and reordering tokens based on importance—offer actionable insights that can directly improve model training efficiency and accuracy, as shown by the experimental results. Given the growing use of SSMs in fields such as language modeling, computer vision, and reinforcement learning, these contributions could lead to broader advancements in how such models are designed and optimized."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Theoretical Scope is Too Narrow**: \nWhile the paper makes a valuable contribution by exploring token dynamics in the one-dimensional case, the focus is too limited. The analysis is restricted to this specific setting, which weakens the generalizability of the conclusions. The authors mention extending their findings to higher-dimensional cases, but without any concrete exploration of these cases, the scope of the theoretical contribution feels incomplete. Actionable Insight: A more comprehensive exploration of higher-dimensional token dynamics should be included. If full proofs are too complex for this paper, at least preliminary results or conjectures, backed by empirical evidence, would make the work stronger. Additionally, comparisons with more sophisticated state-space approaches (such as diagonal vs. non-diagonal state space models) could show how the proposed ideas generalize.\n- **Insufficient Diversity of Experiments**: \nThe empirical validation is focused on tasks like ImageNet classification, which is a strong benchmark, but does not provide sufficient diversity to substantiate the claims made in the paper. The experiments demonstrate the refinements' effectiveness in vision tasks, but the proposed improvements—such as reordering tokens by importance—are presented as potentially broadly applicable to sequence models across various domains. The paper could have benefited from testing the refinements on different types of data and tasks, such as language modeling or reinforcement learning, where SSMs are also commonly applied. Actionable Insight: Expanding the experiments to include tasks like language modeling(e.g., WIKITEXT or other sequence-based datasets) or sequence-based tasks in reinforcement learning would provide stronger evidence that the proposed refinements generalize beyond vision tasks. This would substantiate the claim that these refinements are broadly beneficial across domains.\n- **Shallow Investigation of Practical Impact**: \nThe paper claims that eliminating the convergence scenario and reordering tokens significantly improves the performance of Mamba. While the empirical results show improvements in accuracy and convergence speed, there is little discussion on the practical implications of these refinements in real-world applications. For instance, the computational cost of implementing token reordering or of checking for convergence scenarios during training is not discussed. Additionally, the analysis of why these changes work, especially in larger-scale, practical deployments, is somewhat superficial. Actionable Insight: Including an analysis of the computational overhead or potential trade-offs of implementing these refinements in large-scale models is crucial for practical relevance. Furthermore, deeper explanations of why these adjustments improve performance, with specific references to token dynamics, would enhance the practical significance of the work. For instance, analyzing the impact of token reordering on different architectures or showing how it interacts with other model components would provide more actionable insights for model developers.\n- **Limited Novelty of Some Contributions**: \nWhile the paper presents the analysis of token convergence and divergence as a central contribution, this analysis—though useful—builds heavily on well-known concepts from continuous-time neural networks (e.g., neural ODEs) and does not introduce significantly new ideas. The observation that token dynamics can either converge to zero or diverge to infinity, depending on model parameters, is mathematically rigorous but not particularly surprising in the context of dynamical systems. This reduces the novelty of the contribution. Actionable Insight: To strengthen the originality of the work, the authors could extend the analysis beyond what is currently expected in dynamical systems and neural ODEs. For instance, they could explore more complex behaviors in the interaction between tokens, such as the emergence of periodic behavior, chaotic dynamics, or the impact of noise on the stability of the token trajectories. Alternatively, a deeper comparison with transformer-based models (such as interpreting token dynamics in transformers vs. Mamba) would increase the novelty of the work.\n- **Lack of Comprehensive Comparison to Prior Work**: \nThe paper briefly acknowledges prior work on SSMs and transformers but lacks a comprehensive comparison to alternative models. Given the ongoing research in state space models and their applications, a more thorough evaluation against recent models like GSS (Generalized State Spaces) or HiPPO-based models would provide a stronger benchmark for the proposed refinements. Additionally, since the paper suggests that Mamba can be a more efficient alternative to transformers, it should compare the computational efficiency and performance against contemporary transformer models on long-range sequence tasks. Actionable Insight: The authors should include a more detailed experimental comparison against state-of-the-art models in SSMs and transformers. They should also explicitly compare their proposed refinements with other model optimizations, such as low-rank approximations or efficient attention mechanisms in transformers. This would give the community a clearer picture of where Mamba, with its refinements, stands in the current landscape of sequence modeling."}},"nonreaders":[],"tmdate":1732609998335,"tcdate":1730588072420,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5608/Reviewer_3W4C"],"signatures":["ICLR.cc/2025/Conference/Submission5608/Reviewer_3W4C"],"forum":"qtTIP5Gjc5","number":3,"license":"CC BY 4.0","cdate":1730588072420,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5608/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732609998335,"domain":"ICLR.cc/2025/Conference","replyto":"qtTIP5Gjc5","id":"JrrOsu3fFK","forumContent":{"TLDR":{"value":"We describe the dynamical properties of tokens in a deep selective state-space model, and based on that, we suggest improvements to the model by excluding negatively impactful scenarios and reordering tokens using importance scores."},"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Selective state-space model","continuous-time limit","dynamical system","asymptotic behavior","token reordering"]},"supplementary_material":{"value":"/attachment/f01b2b77a31c531086f0fefd1143d89ad8415a39.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity.  We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance.  For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training.  Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance.  Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications."},"_bibtex":{"value":"@inproceedings{\nvo2025demystifying,\ntitle={Demystifying the Token Dynamics of Deep Selective State Space Models},\nauthor={Thieu Vo and Duy-Tung Pham and Xin T. Tong and Tan Minh Nguyen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=qtTIP5Gjc5}\n}"},"title":{"value":"Demystifying the Token Dynamics of Deep Selective State Space Models"},"pdf":{"value":"/pdf/abc1efe1f910ee48b743f7ba26000b1d5e622f85.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"vo|demystifying_the_token_dynamics_of_deep_selective_state_space_models"},"authorids":{"value":["~Thieu_Vo1","~Duy-Tung_Pham1","~Xin_T._Tong2","~Tan_Minh_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Thieu Vo","Duy-Tung Pham","Xin T. Tong","Tan Minh Nguyen"]}},"version":2},{"content":{"summary":{"value":"RLS-RFF is a self-supervised RF fingerprinting framework that learns channel-invariant, device-specific representations from raw baseband signals.\n A dual-stream Mamba encoder processes each waveform: one time-domain branch models transient I/Q features directly, while a time–frequency branch encodes CWT scalograms capturing steady-state spectral cues. The two streams are fused through stacked selective state-space (Mamba) blocks, enabling efficient long-context sequence modeling.\nDuring pretraining, the system generates two physics-consistent augmentations of each signal (e.g., SNR scaling, phase or carrier drift) and passes both through the shared encoder. Their latent embeddings are aligned via a multi-view consistency loss—encouraging invariance to channel and receiver variation while preserving device identity. Gradients update the encoder and projection head jointly.\nThe authors pair this self-supervised training with an additional final layer for downstream classification / rejection to complete their method.\nExperiments were conducted on the UAV RF fingerprint dataset and on  UV RF and  UCI HAR for  proving open-set classification robustness."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1.How do you handle synchronization and normalization between the two branches?\n\n2.What was the memory footprint of the model and what was the sample size used?\n\n3.How impactful was normalization on training stability and generalization on your unseen data?\n\n4.The work is definitely an interesting concept, my worry is that the data was gathered in relatively similar conditions. How can we be certain that it would generalize well on say, open air data vs in-doors data?\n\n5.THE UIC HAR dataset is used as proof-of-concept for larger open set identification. How are you handling it? What are the I/Q components here? Is the pipeline the same across both of the datasets?\n\n6.How extensive was hyperparameter tuning? How were the parameters chosen?\n\n7.The UC RF dataset is relatively small and in very controlled conditions. How can we be sure that the accuracy presented here can generalize well in open-air, multipath heavy conditions?\n\n8.It would be quite more motivating to see per class confusion breakdowns than simple accuracy, especially in the open-set case.\n\nWhy is the green cluster on Figure 3b so much more discernible?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1) Physically grounded self-supervision\n The method ties representation learning directly to the physics of RF propagation. Its augmentations (SNR scaling, phase shifts, CFO, fading) are realistic and target true channel and receiver variations, giving the learned embeddings meaningful, reality-rooted invariance.\n\n2) Efficient long-context modeling with Mamba\n The selective state-space (Mamba) backbone can process entire 92 k-sample RF bursts in linear time, capturing both transient and steady-state patterns that CNNs or short RNNs typically miss. This provides genuine architectural novelty with computational efficiency.\n\n3) Empirically strong domain robustness\nMotivating results in their selected datasets."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Limited dataset diversity and scale. The UAV dataset has only seven nearly identical devices collected in controlled settings (6–15 ft, single receiver). Reported 98–99 % accuracies may reflect dataset saturation rather than real-world robustness.\n\n2. Unclear tuning and training cost. The model requires 1000 epochs of pretraining and uses Mamba blocks that can be sensitive to state dimension, initialization, and learning rate. The paper does not detail hyperparameter stability or fairness of tuning against baselines.\n\n3. Cross-domain generalization partly untested. The second dataset (UCI HAR) is non-RF and lacks channel effects, so results there demonstrate open-set mechanics, not RF-specific invariance. Broader testing on heterogeneous RF environments or different hardware would better substantiate generalization claims.\n\n\nThe above ultimately limit the scope of the author's contribution. As it stands it is a very interesting step towards physics-aware representation learning but the pragmatic efficacy is not strongly proven.\n\nPrior Work\n\nThere are early prior contributions, examining the existence and identification potential of physical artifacts in a signal that stem from manufacturing imperfections. Chameleons oblivion comes to mind, as an example. Since the authors explicitly state “RF fingerprints (RFFs), unintentional physical artifacts embedded in baseband waveforms, originate in the physical layer and remain tightly coupled to individual transmitter”, that body of work and perhaps others  need to be acknoeldged among the foundational evidence base. \n\nFollowing that direction the authors should elaborate on the contribution of their approach which I believe is indeed quite interesting: they encode physical layer physical constraints to learn novel representational schemes that can reliably be matched to a signal.\nA sound juxtaposition of of their approach could also highlight their contributions which lay in the robustness and invariance of their representations rather that simple efficacy of hardware artifacts as identifiers.\n\n\nFormatting\n\nA plain conceptual overview followed by an earlier presentation of figure 1 would go, I believe, a long way towards understanding the paper.\n\nThe overall pipeline presentation is very confusing. It is hard for the reader to understand the interplay between the mamba backbone and the constraints enforcing architecture. Which happens first? What is the loss for the backbone of figure 1? Is it the loss of the constraints network of figure 2? A flowchart explicitly outlining exactly what happens is absolutely necessary."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918168112,"tcdate":1761937921503,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5634/Reviewer_mELh"],"signatures":["ICLR.cc/2026/Conference/Submission5634/Reviewer_mELh"],"forum":"N5gNhO2OPB","number":3,"license":"CC BY 4.0","cdate":1761937921503,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5634/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918168112,"domain":"ICLR.cc/2026/Conference","replyto":"N5gNhO2OPB","id":"N9WVjB6Mrd","forumContent":{"TLDR":{"value":"Physics-consistent multi-view self-supervised contrastive pretraining for a Mamba dual-stream RF model, learning hardware-invariant fingerprints and improving cross-domain classification & open-world detection with fewer labels."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Selective State-Space Models (Mamba);Radio-Frequency Fingerprinting (RFF);Cross-Domain Generalization; Open-World Device Recognition; Wireless Security"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Under domain shifts and open-world conditions, reliably re-identifying radio frequency (RF) devices is essential for wireless security. As a hardware-rooted physical-layer signature, the RF fingerprint has been widely used for device re-identification. However, supervised RF fingerprint identification (RFFI) models often overfit acquisition artifacts and rely on extensive supervision, leading to sharp cross-domain performance drops and weak open-world behavior. To address these limitations, we introduce PRLS-RFF, which targets physically consistent RF fingerprint representations. We design a dual-stream Mamba-based backbone with physics-consistent, multi-view perturbations to encode representation-level invariances. Additionally, to better capture the structure of RF fingerprints (RFFs) across transient and steady-state regimes, the backbone fuses time-domain and time--frequency features using efficient long-context modeling. As a result, the learned representations are domain-robust, which can support reliable open-set recognition. To validate its robustness, we conduct extensive experiments on several public datasets and observe the performance surpassing state-of-the-art models in both cross-domain identification and open-set recognition tasks."},"_bibtex":{"value":"@misc{\nzhang2026prlsrff,\ntitle={{PRLS}-{RFF}: Physically Consistent Representation Learning with Self-Supervised Pretraining for {RF} Fingerprinting},\nauthor={Tao Zhang and Siya Mi and Yu Zhang and Zihua Wang and chunlin wen and Xiu-Shen Wei},\nyear={2026},\nurl={https://openreview.net/forum?id=N5gNhO2OPB}\n}"},"title":{"value":"PRLS-RFF: Physically Consistent Representation Learning with Self-Supervised Pretraining for RF Fingerprinting"},"pdf":{"value":"/pdf/61f1ddb3a683159bfdc1cadc6df2970af594f6a7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|prlsrff_physically_consistent_representation_learning_with_selfsupervised_pretraining_for_rf_fingerprinting"},"authorids":{"value":["~Tao_Zhang74","~Siya_Mi1","~Yu_Zhang21","~Zihua_Wang1","~chunlin_wen1","~Xiu-Shen_Wei1"]},"authors":{"value":["Tao Zhang","Siya Mi","Yu Zhang","Zihua Wang","chunlin wen","Xiu-Shen Wei"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2409.18654v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"gao|speechmamba_longcontext_speech_recognition_with_selective_state_spaces_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xiaoxue_Gao:","~Nancy_F._Chen1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2409.18654"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2409-18654,\n  publtype={informal},\n  author={Xiaoxue Gao and Nancy F. Chen},\n  title={Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2409.18654},\n  url={https://doi.org/10.48550/arXiv.2409.18654}\n}\n"},"abstract":{"value":"Current automatic speech recognition systems struggle with modeling long speech sequences due to high quadratic complexity of Transformer-based models. Selective state space models such as Mamba has performed well on long-sequence modeling in natural language processing and computer vision tasks. However, research endeavors in speech technology tasks has been under-explored. We propose Speech-Mamba, which incorporates selective state space modeling in Transformer neural architectures. Long sequence representations with selective state space models in Speech-Mamba is complemented with lower-level representations from Transformer-based modeling. Speech-mamba achieves better capacity to model long-range dependencies, as it scales near-linearly with sequence length."},"title":{"value":"Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models"},"authors":{"value":["Xiaoxue Gao","Nancy F. Chen"]}},"tmdate":1759849066765,"pdate":1704067200000,"externalIds":["dblp:journals/corr/abs-2409-18654"],"tcdate":1759849050850,"writers":["~"],"signatures":["~Nancy_F._Chen1"],"forum":"0TuE5a7xtw","license":"CC BY-SA 4.0","number":640067,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1759849066765,"domain":"DBLP.org","id":"0TuE5a7xtw","version":2},{"content":{"venue":{"value":"SLT 2024"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/10830790/10830793/10832137.pdf"},"venueid":{"value":"dblp.org/conf/SLT/2024"},"paperhash":{"value":"gao|speechmamba_longcontext_speech_recognition_with_selective_state_spaces_models"},"authorids":{"value":["~Xiaoxue_Gao1","~Nancy_F._Chen1"]},"html":{"value":"https://doi.org/10.1109/SLT61566.2024.10832137"},"_bibtex":{"value":"@inproceedings{DBLP:conf/slt/GaoC24,\n  author={Xiaoxue Gao and Nancy F. Chen},\n  title={Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models},\n  year={2024},\n  cdate={1704067200000},\n  pages={1-8},\n  url={https://doi.org/10.1109/SLT61566.2024.10832137},\n  booktitle={SLT},\n  crossref={conf/slt/2024}\n}\n"},"abstract":{"value":"Current automatic speech recognition systems struggle with modeling long speech sequences due to high quadratic complexity of Transformer-based models. Selective state space models such as Mamba has performed well on long-sequence modeling in natural language processing and computer vision tasks. However, research endeavors in speech technology tasks has been under-explored. We propose Speech-Mamba, which incorporates selective state space modeling in Transformer neural architectures. Long sequence representations with selective state space models in Speech-Mamba is complemented with lower-level representations from Transformer-based modeling. Speech-mamba achieves better capacity to model long-range dependencies, as it scales near-linearly with sequence length."},"title":{"value":"Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models"},"authors":{"value":["Xiaoxue Gao","Nancy F. Chen"]}},"tmdate":1759849051891,"pdate":1704067200000,"tcdate":1744876595789,"writers":["~"],"signatures":["~Xiaoxue_Gao1"],"forum":"oe0YHjcKlh","license":"CC BY-SA 4.0","number":402676,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1759849051891,"domain":"DBLP.org","id":"oe0YHjcKlh","version":2},{"content":{"summary":{"value":"The paper presents *LION*, a bidirectional sequence modeling framework based on State Space-like Models (SSMs) such as Linear Attention and Mamba. Supplementing the standard parallel vectorized implementation, LION enables output computation through two recurrences, designed to preserve output fidelity with the parallelized approach. The main contributions are as follows:\n\n1. Design linear recurrences that correspond to Linear Attention (LA) aka LION-LIT\n2. Add Mamba's selectivity to those recurrences aka LION-S. \n3. Empirically test the model on the LRA task, GLUE, and image classification\n4. The resulting model does not require position embeddings and can be extrapolated beyond the training length."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"N/A - see weaknesses"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":2},"strengths":{"value":"1. The paper is very well written and quite illustrative which makes it an easy read. \n2. The goals and the methodology proposed in the paper are easy to understand.\n3. The paper shows that bidirectional linear attention (and its selective variant) can be written in a recurrent form."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**1: Recurrence Form of Linear Attention is a straightforward extension**  \nThe claim that Linear Attention (LA) can be expressed as a recurrence seems like a straightforward calculation, given it is already known that causal LA can be written as a recurrence. To extend this to bidirectional settings, the numerator is simply a forward recurrence, $\\sum_{j \\leq i} q_i^Tk_j$, combined with a backward recurrence, $\\sum_{j \\geq i} q_i^Tk_j$, and a correction term $-q_i^Tk_j$ to account for double-counting. The same approach works with the denominator. While this is mathematically valid, the recurrence structure derived here seems like a straightforward extension of known results.\n\n\n**2: LION's Continuous-Domain Selectivity is derived in Mamba**\n\nThe continuous selectivity mechanism in LION seems to be derived in Mamba’s formulation (see Eq. (4) in [1]). Mamba originally derived this expression into the form used here and then simplified it, i.e. they start with $\\mathbf{\\bar{B}} = (\\Delta \\mathbf{A})^{-1} (\\exp(\\Delta \\mathbf{A}) - \\mathbf{I}) \\Delta \\mathbf{B}$, and drop $(\\Delta \\mathbf{A})^{-1} (\\exp(\\Delta \\mathbf{A}) - \\mathbf{I})$ to simplify the term and notice no drop in performance. LION chooses the $\\mathbf{A}$ matrix as identity which reduces the above expression to LION's (27a)\n\n\n**3: No Position Encodings required is a consequence of using SSMs**  \nThe claim that this model achieves independence from position encoding and can hence be extrapolated to longer sequences is a standard characteristic of all State Space Models, which implicitly capture positional information through recurrence. This has been recognized and documented within the community (see [GitHub issue #29](https://github.com/state-spaces/mamba/issues/29)). I feel that is not a contribution of LION but SSMs at large.\n\n**4: Practicality of Recurrent Inference**\n\nI am unsure where the recurrence form in the paper may have practical utility. For language modeling using recurrence makes imminent sense because it helps in streaming out the output data to the user in real time. However, in case of bidirectional modeling the parallel implementation already achieves $O(L)$ space and time complexity which is the same as LION\n\nThe only limited application I can see this being used is if the memory constraints are such that we have $O(L)$ memory but the constant is probably not big enough, then instead of computing it in a purely recurrent fashion which would be very slow, it is standard practice in SSMs to processes input in chunks, using:\n  - **Input:** [SSM's State, subset of contiguous tokens]\n  - **Output:** [Updated SSM's State, corresponding output tokens]\n\n Each chunk can be computed using the parallel form under available memory without needing to fully unroll the recurrence which would be very slow on GPUs. I would like to note that the above approach is well-established in the SSM community.\n\n**5: Experimental Validity Concerns**\n\n- **GLUE Scores:** The reported GLUE scores seem low; prior works using selective recurrences and adding them without correction factors at 70M parameters has been shown to yield a GLUE score of 80.6 which is higher than LION's 78.5. (See [2])\n  \n- **Vision Task Comparisons:** In the vision tasks, the authors only compare their model with ViT-small, which was originally designed for distillation purposes. Training a 22M model directly for vision tasks is impractical; comparisons with a larger model, such as ViT-base (86M parameters), would be more convincing and relevant.\n\n[2]: Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers. Sukjun Hwang, Aakash Lahoti, Tri Dao, Albert Gu"}},"nonreaders":[],"tmdate":1731428746849,"tcdate":1730717601575,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13771/Reviewer_tj5X"],"signatures":["ICLR.cc/2025/Conference/Submission13771/Reviewer_tj5X"],"forum":"VxJjw52dZu","number":4,"license":"CC BY 4.0","cdate":1730717601575,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13771/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428746849,"domain":"ICLR.cc/2025/Conference","replyto":"VxJjw52dZu","id":"JuT21G6yPS","forumContent":{"TLDR":{"value":"We provide a bidirectional selective recurrent form of full kernelized attention with learnable mask allowing scalability in context length and superior inference efficiency"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Self-attention","transformer","bidirectionality","mamba","ssm","rnn","inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We introduce LION, a novel sequence-to-sequence framework that unifies the bidirectionality and parallelized training of Transformers with the fast inference of recurrent neural networks. LION is built upon a mathematical formulation where full kernelized attention with a learnable mask is efficiently computed using a bidirectional selective recurrent model, matching the effectiveness of softmax-based attention with constant-time inference. Our framework naturally accounts for spatial and temporal relationships within input sequences, reducing reliance on heuristic positional embeddings and facilitating straightforward scalability in context length and resolution. Using our framework and inspired by the recent state-space models, we propose three main running examples LIOn-LIT, LION-RETNET, and LION-S, a transformer with selective mask and recurrent inference. Numerical evaluations on tasks such as language modeling, the Long-Range Arena, and image classification show that LION framework achieves performance on par with state-of-the-art models while delivering fast training and inference efficiency."},"_bibtex":{"value":"@misc{\nafzal2025lion,\ntitle={{LION}: A bidirectional framework that trains like a Transformer and infers like an {RNN}},\nauthor={Arshia Afzal and Elias Abad Rocamora and Leyla Naz Candogan and Pol Puigdemont and Francesco Tonin and Yongtao Wu and Mahsa Shoaran and Volkan Cevher},\nyear={2025},\nurl={https://openreview.net/forum?id=VxJjw52dZu}\n}"},"title":{"value":"LION: A bidirectional framework that trains like a Transformer and infers like an RNN"},"pdf":{"value":"/pdf/6e8b8585752d6f27da24ffe876859459a5b56efe.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"afzal|lion_a_bidirectional_framework_that_trains_like_a_transformer_and_infers_like_an_rnn"},"authorids":{"value":["~Arshia_Afzal1","~Elias_Abad_Rocamora1","~Leyla_Naz_Candogan1","~Pol_Puigdemont1","~Francesco_Tonin1","~Yongtao_Wu1","~Mahsa_Shoaran1","~Volkan_Cevher1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Arshia Afzal","Elias Abad Rocamora","Leyla Naz Candogan","Pol Puigdemont","Francesco Tonin","Yongtao Wu","Mahsa Shoaran","Volkan Cevher"]}},"version":2},{"content":{"summary":{"value":"In summary, PointMamba, an innovative state space model tailored for point cloud analysis, successfully harnesses the global modeling prowess of Mamba, a representative SSM from the NLP domain. By adopting a linear complexity algorithm, PointMamba addresses the computational challenges posed by traditional Transformer-based methods while maintaining their global modeling capabilities. Its key techniques include utilizing space-filling curves for effective point tokenization and employing a non-hierarchical Mamba encoder as the foundation. Extensive experiments across multiple datasets validate the superior performance of PointMamba while significantly reducing GPU memory usage and FLOPs. This work not only demonstrates the vast potential of SSMs in 3D vision tasks but also provides a simple yet effective baseline for future research in this domain."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. Based on weakness 4, is it possible that PointMamba actually only works well in Hiber conditions? As this very much affects the final results and conclusions of the paper.\n\n2. Intuitively, PointMamba seems to be just a replacement of the Mamba block (Equation 4) for the previous Transformer block, and even if this results in good performance (Figure 1), I am not convinced. The paper is theoretically and experimentally fleshed out, but I still don't find an obvious innovative contribution. Therefore, I would like to confirm whether the authors have designed a specialized Mamba technique for 3D point clouds?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. **Linear Complexity with Global Modeling**: PointMamba leverages state space models to achieve linear complexity while maintaining global modeling capabilities, overcoming the computational challenges of traditional Transformers.\n\n2. **Efficient Point Cloud Representation**: The use of space-filling curves for point tokenization enables efficient representation of point clouds, capturing spatial structure while facilitating global feature extraction.\n\n3. **Simple and Effective Mamba Encoder**: The non-hierarchical Mamba encoder provides a simple yet powerful backbone for PointMamba, enabling fast and accurate global feature modeling.\n\n4. **Superior Performance**: Comprehensive evaluations show that PointMamba achieves state-of-the-art performance across multiple datasets, demonstrating its effectiveness for point cloud analysis tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Although PointMamba borrows structurally from Mamba, it may not take full advantage of the unique characteristics of point cloud data, such as spatial distribution, density variations, and local geometry.\n\n2. PointMamba, while inheriting the strengths of Mamba, may have inherited some of the limitations of its design for natural language processing tasks. These limitations may not be applicable to point cloud analysis tasks, such as the lack of specific preprocessing steps, feature extraction methods, or post-processing techniques for point cloud data. The feature extraction and processing steps in the paper are largely similar to those of the previous PointMAE.\n\n3. Although PointMamba achieves global modeling of linear complexity by introducing state-space models, there may be trade-offs between model complexity and performance in real-world applications. Because according to the experimental results of the thesis, the effect of partially using PointMamba becomes worse instead.\n\n4. According to Table 5 of the ablation experiment, the serialization operations Hiber and Trans-Hiber have the most obvious impact on the experimental results. Meanwhile, the framework diagram at the core of the paper, i.e., Figure 4, shows that the only change is Hiber, from the directly cited literature [27], and there is almost no additional description or setting in the paper."},"limitations":{"value":"Content-wise, the paper has no obvious limitations, and its core technology builds on Mamba, which is popular in other fields. Formatting-wise, the paper shows the mamba icon several times; please confirm whether this representation is appropriate in an academic paper."}},"nonreaders":[],"tmdate":1730878669311,"tcdate":1718545763643,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission940/Reviewer_MyQv"],"signatures":["NeurIPS.cc/2024/Conference/Submission940/Reviewer_MyQv"],"forum":"Kc37srXvan","number":1,"license":"CC BY 4.0","cdate":1718545763643,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission940/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878669311,"domain":"NeurIPS.cc/2024/Conference","replyto":"Kc37srXvan","id":"x0UDLSfNPJ","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Point cloud analysis; State space model; Mamba; PointMamba"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Transformers have become one of the foundational architectures in point cloud analysis tasks due to their excellent global modeling ability. However, the attention mechanism has quadratic complexity, making the design of a linear complexity method with global modeling appealing. In this paper, we propose PointMamba, transferring the success of Mamba, a recent representative state space model (SSM), from NLP to point cloud analysis tasks. Unlike traditional Transformers, PointMamba employs a linear complexity algorithm, presenting global modeling capacity while significantly reducing computational costs. Specifically, our method leverages space-filling curves for effective point tokenization and adopts an extremely simple, non-hierarchical Mamba encoder as the backbone. Comprehensive evaluations demonstrate that PointMamba achieves superior performance across multiple datasets while significantly reducing GPU memory usage and FLOPs. This work underscores the potential of SSMs in 3D vision-related tasks and presents a simple yet effective Mamba-based baseline for future research. The code is available at https://github.com/LMD0311/PointMamba."},"_bibtex":{"value":"@inproceedings{\nliang2024pointmamba,\ntitle={PointMamba: A Simple State Space Model for Point Cloud Analysis},\nauthor={Dingkang Liang and Xin Zhou and Wei Xu and Xingkui Zhu and Zhikang Zou and Xiaoqing Ye and Xiao Tan and Xiang Bai},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=Kc37srXvan}\n}"},"title":{"value":"PointMamba: A Simple State Space Model for Point Cloud Analysis"},"pdf":{"value":"/pdf/799eef8a02512c60290e1c8180f95b4799a27ec9.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"liang|pointmamba_a_simple_state_space_model_for_point_cloud_analysis"},"authorids":{"value":["~Dingkang_Liang2","~Xin_Zhou17","~Wei_Xu15","~Xingkui_Zhu1","~Zhikang_Zou2","~Xiaoqing_Ye1","~Xiao_Tan3","~Xiang_Bai1"]},"authors":{"value":["Dingkang Liang","Xin Zhou","Wei Xu","Xingkui Zhu","Zhikang Zou","Xiaoqing Ye","Xiao Tan","Xiang Bai"]}},"version":2},{"content":{"summary":{"value":"This paper focuses on online video super-resolution (VSR) and proposes a new model called TS-Mamba, which is built upon State Space Models (SSMs). The method introduces long-term trajectory modeling combined with the lightweight Mamba framework to improve spatio-temporal information aggregation efficiency in video sequences."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"N/A"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The integration of Mamba-based spatial modeling with trajectory-aware design is conceptually clear and implemented in a well-structured manner.\n\nThe proposed Trajectory-aware Shifted Mamba Aggregation (TSMA) module is technically sound and engineering-wise elegant.\n\nThe model achieves reasonable complexity-efficiency trade-offs — it incorporates long-term temporal modeling while maintaining relatively low MACs and parameter counts compared to prior online VSR baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Biased motivation.\n\nThe paper claims that “most existing VSR methods solely employ one neighboring previous frame”, yet this statement overlooks numerous recent works that already explore long-range temporal modeling. For instance:\n\n[1] Wang, Xijun, et al. LiftVSR: Lifting Image Diffusion to Video Super-Resolution via Hybrid Temporal Modeling with Only 4×RTX 4090s. arXiv:2506.08529 (2025).\n\n[2] Liu, Yong, et al. UltraVSR: Achieving Ultra-Realistic Video Super-Resolution with Efficient One-Step Diffusion Space. arXiv:2505.19958 (2025).\n\n[3] Zhuang, Junhao, et al. FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolution. arXiv:2510.12747 (2025).\nThese studies demonstrate substantial progress in long-term temporal modeling, undermining the paper’s claim that such research is scarce. This weakens the motivation and positioning of the work.\n\n2. Lack of analysis on real-time VSR methods.\n\nIn lines 109–117, the authors argue that optical-flow or deformable-convolution-based methods are computationally expensive. However, multiple studies have achieved real-time performance with efficient architectures, such as:\n\n[1] Cao, Yanpeng, et al. Real-Time Super-Resolution System of 4K-Video Based on Deep Learning. IEEE ASAP 2021.\n\n[2] Jiang, Y., Nawała, J., Feng, C., et al. RTSR: A Real-Time Super-Resolution Model for AV1 Compressed Content. IEEE ISCAS 2025.\nThe paper fails to include any quantitative or empirical comparison with these real-time approaches and instead relies on generic qualitative claims. This lack of evidence weakens its argument regarding computational superiority.\n\n3. Outdated comparison on standard benchmarks.\n\nThe experiments are conducted on REDS4, Vid4, and Vimeo-90K-T datasets, but do not include recent state-of-the-art models that have achieved higher PSNR and SSIM. For example:\n\nLin, Jing, et al. Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration. ICML 2022 — achieving 31.96 dB and 37.63 PSNR on REDS4 and Vimeo-90K-T, outperforming the proposed method.\n\nDong, Shuting, et al. DFVSR: Directional Frequency Video Super-Resolution via Asymmetric and Enhancement Alignment Network. IJCAI 2023.\nWithout comparisons to these recent methods, the experimental evaluation is incomplete and fails to convincingly establish the method’s competitiveness.\n\nLimited novelty in applying Mamba to VSR.\nThe use of Mamba-based State Space Models for video super-resolution is not new.\n\nEvent-based Video Super-Resolution via State Space Models (CVPR 2025) already introduced a similar Mamba framework for VSR.\nThe authors do not clarify how TS-Mamba fundamentally differs from or advances beyond these prior Mamba-based designs. Thus, the methodological contribution appears incremental rather than novel."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919223141,"tcdate":1761880604701,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7012/Reviewer_zoqq"],"signatures":["ICLR.cc/2026/Conference/Submission7012/Reviewer_zoqq"],"forum":"RygnSGcV49","number":2,"license":"CC BY 4.0","cdate":1761880604701,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7012/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919223141,"domain":"ICLR.cc/2026/Conference","replyto":"RygnSGcV49","id":"OcqYdZk5P3","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Video Super-resolution","Online","Mamba","Trajectory"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Online video super-resolution (VSR) is an important technique for many real-world video processing applications, which aims to restore the current high-resolution video frame based on temporally previous frames. Most of the existing online VSR methods solely employ one neighboring previous frame to achieve temporal alignment, which limits long-range temporal modeling of videos. Recently, state space models (SSMs) have been proposed with linear computational complexity and a global receptive field, which significantly improve computational efficiency and performance. In this context, this paper presents a novel online VSR method based on Trajectory-aware Shifted SSMs (TS-Mamba), leveraging both long-term trajectory modeling and low-complexity Mamba to achieve efficient spatio-temporal information aggregation. Specifically, TS-Mamba first constructs the trajectories within a video to select the most similar tokens from the previous frames. Then, a Trajectory-aware Shifted Mamba Aggregation (TSMA) module consisting of proposed shifted SSMs blocks is employed to aggregate the selected tokens. The shifted SSMs blocks are designed based on Hilbert scannings and corresponding shift operations to compensate for scanning losses and strengthen the spatial continuity of Mamba. Additionally, we propose a trajectory-aware loss function to supervise the trajectory generation, ensuring the accuracy of token selection when training our model. Extensive experiments on three widely used VSR test datasets demonstrate that compared with six online VSR benchmark models, our TS-Mamba achieves state-of-the-art performance in most cases and over 22.7% complexity reduction (in MACs). The source code for TS-Mamba is available at https://github.com/QZ1-boy/TS-Mamba."},"_bibtex":{"value":"@inproceedings{\nzhu2026trajectoryaware,\ntitle={Trajectory-aware Shifted State Space Models for Online Video Super-Resolution},\nauthor={Qiang Zhu and Xiandong MENG and Yuxuan Jiang and Fan Zhang and David Bull and Shuyuan Zhu and Bing Zeng and Ronggang Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=RygnSGcV49}\n}"},"title":{"value":"Trajectory-aware Shifted State Space Models for Online Video Super-Resolution"},"pdf":{"value":"/pdf/63207ba1dfa93f1f2594d2cd3d80db2064dc0722.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhu|trajectoryaware_shifted_state_space_models_for_online_video_superresolution"},"authorids":{"value":["~Qiang_Zhu6","~Xiandong_MENG1","~Yuxuan_Jiang8","~Fan_Zhang6","~David_Bull1","~Shuyuan_Zhu1","~Bing_Zeng1","~Ronggang_Wang1"]},"authors":{"value":["Qiang Zhu","Xiandong MENG","Yuxuan Jiang","Fan Zhang","David Bull","Shuyuan Zhu","Bing Zeng","Ronggang Wang"]}},"version":2},{"content":{"summary":{"value":"This paper investigates to utilize the Mamba [1] architecture for In-Context RL task. Addressing this task with Transformer architecture is effective while it is very inefficient due to the quadratic computation overhead of Transformer. The Mamba can reduce this overhead dramatically while sustain the performance somewhat. The application of State-Space Models (SSMs) to In-Context RL task is studied in [2], but different from [2], they combinationally utilize Mamba and Transformer as high-level memory and low-level (short-term) memory. Additionally, as Mamba predicts the sub-goal for the Transformer short-term memory, they improved the performance. Through this modeling, they can achieve better performance than previous works while improving the efficiency.\n\n[1] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n\n[2] Lu, Chris, et al. \"Structured state space models for in-context reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2024)."},"soundness":{"value":4},"confidence":{"value":5},"questions":{"value":"- I am curious why AD (transformer) shows worse performance than HM. I thought AD (transformer) performance would be the upper bound of HM while HM is more efficient. However, AD (transformer) performance is generally worse than HM in your tests, especially for Grid World in Figure 2. Why is AD (transformer) performance poor in Grid World? Did you use a smaller context size for the Grid World test? If not (using the same context size), what could be the reasons for the significant performance gap?"},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"- Appropriate modeling is applied in this study. While the effectiveness of hybrid modeling of SSMs and local Attention has been previously explored in [1], the authors effectively implement this concept for the In-Context RL task with new functionalities, such as predicting high-value sub-goals.\n- The introduction and methodology sections are well written. The motivation is clearly articulated, and the logical flow of their method proposal is coherent. The empirical analysis comparing Mamba and Transformer in RL tasks convincingly demonstrates the need for more advanced modeling.\n- The paper provides extensive empirical analyses. It shares experimental results on multiple benchmarks, including ablation studies and performance changes with varying hyperparameter values.\n\n[1] De, Soham, et al. \"Griffin: Mixing gated linear recurrences with local attention for efficient language models.\" arXiv preprint arXiv:2402.19427 (2024)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The high-level encoding is done by encoding the intervalled trajectories (e.g., every $c$ -th trajectory), which might miss important information in the middle of the interval.\n- The section on Hybrid Mamba with Valuable Sub-goals is initially confusing, especially regarding the relationship between Mamba’s sub-goal prediction and the collected valuable sub-goals. Discussing this relationship at the beginning of the Valuable Sub-goal section could help readers understand the content more easily.\n- One of the experimental results differs from my expectations, but the paper does not provide an analysis for this. I will address this in the Questions section."},"limitations":{"value":"The authors properly addressed their limitations."}},"nonreaders":[],"tmdate":1730879032928,"tcdate":1720494686806,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission5823/Reviewer_wVw1"],"signatures":["NeurIPS.cc/2024/Conference/Submission5823/Reviewer_wVw1"],"forum":"wFzIMbTsY7","number":3,"license":"CC BY 4.0","cdate":1720494686806,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission5823/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879032928,"domain":"NeurIPS.cc/2024/Conference","replyto":"wFzIMbTsY7","id":"RXOiXhJg9a","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["In-context Reinforcement Learning","Mamba"]},"primary_area":{"value":"reinforcement_learning"},"abstract":{"value":"Recent works have shown the remarkable superiority of transformer models in reinforcement learning (RL), where the decision-making problem is formulated as sequential generation. Transformer-based agents could emerge with self-improvement in online environments by providing task contexts, such as multiple trajectories, called in-context RL. However, due to the quadratic computation complexity of attention in transformers, current in-context RL methods suffer from huge computational costs as the task horizon increases. In contrast, the Mamba model is renowned for its efficient ability to process long-term dependencies, which provides an opportunity for in-context RL to solve tasks that require long-term memory. To this end, we first implement Decision Mamba (DM) by replacing the backbone of Decision Transformer (DT). Then, we propose a Decision Mamba-Hybrid (DM-H) with the merits of transformers and Mamba in high-quality prediction and long-term memory. Specifically, DM-H first generates high-value sub-goals from long-term memory through the Mamba model. Then, we use sub-goals to prompt the transformer, establishing high-quality predictions. Experimental results demonstrate that DM-H achieves state-of-the-art in long and short-term tasks, such as D4RL, Grid World, and Tmaze benchmarks. Regarding efficiency, the online testing of DM-H in the long-term task is 28$\\times$ times faster than the transformer-based baselines."},"_bibtex":{"value":"@inproceedings{\nhuang2024decision,\ntitle={Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling},\nauthor={Sili Huang and Jifeng Hu and Zhejian Yang and Liwei Yang and Tao Luo and Hechang Chen and Lichao Sun and Bo Yang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=wFzIMbTsY7}\n}"},"title":{"value":"Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling"},"pdf":{"value":"/pdf/f79cbc369ef6968176c7cc958c79839cb99e59b0.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"huang|decision_mamba_reinforcement_learning_via_hybrid_selective_sequence_modeling"},"authorids":{"value":["~Sili_Huang1","~Jifeng_Hu1","~Zhejian_Yang1","~Liwei_Yang2","~Tao_Luo2","~Hechang_Chen2","~Lichao_Sun1","~Bo_Yang6"]},"authors":{"value":["Sili Huang","Jifeng Hu","Zhejian Yang","Liwei Yang","Tao Luo","Hechang Chen","Lichao Sun","Bo Yang"]}},"version":2},{"content":{"venue":{"value":"CLAWAR (1) 2024"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-031-70722-3_6.pdf"},"venueid":{"value":"dblp.org/conf/CLAWAR/2024"},"paperhash":{"value":"wilinski|hapmamba_lineartime_sequence_modeling_for_terrain_classification_by_legged_robots"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Michal_Wilinski:","https://dblp.org/search/pid/api?q=author:Mikolaj_Lysakowski:","~Krzysztof_Walas2"]},"html":{"value":"https://doi.org/10.1007/978-3-031-70722-3_6"},"_bibtex":{"value":"@inproceedings{DBLP:conf/clawar/WilinskiLW24,\n  author={Michal Wilinski and Mikolaj Lysakowski and Krzysztof Walas},\n  title={HAPmamba: Linear-Time Sequence Modeling for Terrain Classification by Legged Robots},\n  year={2024},\n  cdate={1704067200000},\n  pages={34-44},\n  url={https://doi.org/10.1007/978-3-031-70722-3_6},\n  booktitle={CLAWAR (1)},\n  crossref={conf/clawar/2024-1}\n}\n"},"abstract":{"value":"This paper proposes a neural network model for terrain classification using force/torque signals registered with sensors mounted on robot feet. The proposed model is based on the latest architecture for Linear-Time Sequence Modeling using Selective State Spaces, called Mamba. We obtained lightweight models with very low inference times, which are two times faster than comparable transformer-based solutions. We evaluated HAPmamba alongside other state-of-the-art approaches, and while the classification measures are comparable, HAPmamba is the fastest among all evaluated models."},"title":{"value":"HAPmamba: Linear-Time Sequence Modeling for Terrain Classification by Legged Robots"},"authors":{"value":["Michal Wilinski","Mikolaj Lysakowski","Krzysztof Walas"]}},"tmdate":1762277187486,"pdate":1704067200000,"externalIds":["dblp:conf/clawar/WilinskiLW24"],"tcdate":1762277182225,"writers":["~"],"signatures":["~Krzysztof_Walas2"],"forum":"ltP7XWD6ct","license":"CC BY-SA 4.0","number":654609,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1762277187486,"domain":"DBLP.org","id":"ltP7XWD6ct","version":2},{"content":{"summary":{"value":"This work jointly models temporal dynamics and graph structure for dynamic graphs. It integrates the Mamba state-space architecture with HiPPO-based memory to compress historical information, enabling long-term and long-range sequence modeling. Extensive experiments across multiple benchmarks demonstrate state-of-the-art performance."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"Please refer to the above-mentioned weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This paper introduces high-order topological information into dynamic graph representation learning and achieves strong performance on long-sequence classification tasks.\n\n2. This paper extends the SSM framework to dynamic graph modeling with a solid theoretical foundation.\n\n3. This paper conducts extensive experiments, showing strong results on dynamic link prediction, dynamic node classification, and long-sequence classification."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Time complexity is our primary concern. For example, matrix inversion costs O(N^3); each batch requires constructing/updating the Laplacian; and a K-order polynomial filter entails K matrix multiplications. The paper does not analyze time complexity or provide comparative runtime experiments. We believe the computational cost is substantial and may hinder real-world deployment. Moreover, the absence of experiments on large-scale graphs further undermines confidence in practical applicability.\n\n2. The proposed HiPPO matrix appears very close to JinTang Li et al. (NeurIPS 2024), seemingly as a direct extension to dynamic graphs. In addition, the adopted Mamba structure looks like a straightforward application of Mamba, without a detailed comparison to existing Mamba-based frameworks. This weakens the claimed architectural novelty.\n\n3. The paper lacks essential ablations. For instance: How do we know the proposed high-order graph filters are effective? How do we verify that the model truly handles and benefits from long-range dependencies? How is robustness demonstrated?\n\n4. The dynamic link prediction benchmarks are selectively chosen. Common datasets such as Flights, Can. Parl., US Legis., UN Trade, UN Vote, and Contact are missing. Comparisons with the latest baselines are also absent—for example: [1] DyG-Mamba: Continuous State Space Model for Dynamic Graphs; [2] Towards Better Evaluation for Dynamic Link Prediction; [3] FreeDyG: Frequency-Enhanced Continuous-Time Dynamic Graph Model for Link Prediction.\n\n5. There is no clear investigation of input sequence length or of higher-order graph structure. It remains unclear whether gains come from long-sequence modeling or from high-order structural information. More experiments are needed to disentangle these factors. Prior studies (e.g., GraphMixer) have reported that longer sequences may not help, which the authors should address explicitly.\n\n6. The results on MOOC are surprisingly high, and our reproduction raises a potential data-leakage issue. Specifically, the code calls ssm_utlis.py::get_delta_t(...) with the default parameter default=1e+11. This may leak information for negative samples. The authors should clarify this choice and whether it preserves fairness with DyGLib and other baselines."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762943076817,"tcdate":1761732315352,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission24420/Reviewer_Ks8e"],"signatures":["ICLR.cc/2026/Conference/Submission24420/Reviewer_Ks8e"],"forum":"ihlRtPqPDL","number":2,"license":"CC BY 4.0","cdate":1761732315352,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission24420/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762943076817,"domain":"ICLR.cc/2026/Conference","replyto":"ihlRtPqPDL","id":"6nGQcLfLgv","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Continuous time dynamic graphs","State space models","Long range propogation","Higher-order polynomial projection operator"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Continuous-time dynamic graphs (CTDGs) provide a richer framework to capture fine-grained temporal patterns in evolving relational data. Long-range information propagation is a key challenge in learning representations for CTDGs, wherein it is important to retain and update information over long temporal horizons. Existing approaches restrict models to capture one-hop or local temporal neighborhoods and fail to capture multi-hop or global structural patterns. To mitigate limitations of the current approaches, we derive the state-space modelling framework for continuous-time dynamic graphs $\\texttt{(CTDG-SSM)}$ from first principles. We first introduce continuous-time Topology-Aware higher order polynomial projection operator ($\\texttt{CTT-HiPPO})$, a novel memory-based reformulation of $\\texttt{HiPPO}$ to jointly encode temporal dynamics and graph structure, where solution for memory representations from $\\texttt{CTT-HiPPO}$ are obtained by projecting the classical HiPPO solution through a polynomial of the Laplacian matrix, yielding topology-aware memory updates that admit an equivalent state-space formulation for CTDGs ($\\texttt{CTDG-SSM}$). This is then discretized (e.g., using the zero-order hold method) for practical implementation. We further provide theoretical guarantees demonstrating the robustness of memory representations under graph structure perturbations. Across benchmarks on dynamic link prediction, dynamic node classification, and sequence classification, $\\texttt{CTDG-SSM}$ achieves state-of-the-art performance. \nNotably, it achieves large performance gains on dynamic link prediction and sequence classification tasks, specifically on datasets that require long range temporal (LRT) and spatial reasoning."},"_bibtex":{"value":"@misc{\nraghuvanshi2026ctdgssm,\ntitle={{CTDG}-{SSM}: Continuous-time Dynamic Graph State Space Models for Long Range Propagation},\nauthor={Ayushman Raghuvanshi and Thummaluru Siddartha Reddy and Sundeep Prabhakar Chepuri and Mahesh Chandran},\nyear={2026},\nurl={https://openreview.net/forum?id=ihlRtPqPDL}\n}"},"title":{"value":"CTDG-SSM: Continuous-time Dynamic Graph State Space Models for Long Range Propagation"},"pdf":{"value":"/pdf/fb584f34040081715648ab80bef51f39321dbfa9.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"raghuvanshi|ctdgssm_continuoustime_dynamic_graph_state_space_models_for_long_range_propagation"},"authorids":{"value":["~Ayushman_Raghuvanshi1","~Thummaluru_Siddartha_Reddy1","~Sundeep_Prabhakar_Chepuri1","~Mahesh_Chandran1"]},"authors":{"value":["Ayushman Raghuvanshi","Thummaluru Siddartha Reddy","Sundeep Prabhakar Chepuri","Mahesh Chandran"]}},"version":2},{"content":{"summary":{"value":"MambaMatcher combines efficient high-dimensional correlation modeling with a selective scan mechanism, achieving superior performance on semantic correspondence benchmarks with lower computational costs.\n\nWhile traditional feature-metric methods often fail to capture complex relationships between images, newer correlation-metric approaches address this but are hindered by the high computational cost of processing 4D correlation maps. MambaMatcher overcomes these issues by modeling high-dimensional correlations efficiently through selective state-space models (SSMs). Using a similarity-aware selective scan mechanism inspired by Mamba’s linear-complexity algorithm, MambaMatcher refines the 4D correlation tensor without compromising feature map resolution or receptive field. Experiments on standard semantic correspondence benchmarks demonstrate that MambaMatcher achieves state-of-the-art performance, capturing rich inter-image correlations while avoiding large input images and costly diffusion-based features."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Ambiguity in Feature and Correlation Aggregations. Section 4.1 introduces Multi-level Feature Aggregation using a lightweight 2D convolution network, while Section 4.2 proposes MULTI-LEVEL CORRELATION COMPUTATION AND AGGREGATION using Mamba. Both aggregation modules require training, but throughout the text, the authors often refer to “the aggregation” without specifying which aggregation they mean. This lack of clarity can easily lead to confusion, as in Lines 251-352: “We freeze the visual feature extractor during training to focus on learning the aggregation layers.” Here, does “the aggregation layers” refer to the feature aggregation in Section 4.1, the correlation aggregation in Section 4.2, or both?\n\n2.Effects of selective scan order. Table 5 indicates that the relative order of correlation elements significantly impacts performance, with ascending order notably reducing effectiveness compared to descending order. This may be because, in causal inference, later tokens pass information to all preceding tokens. Placing high-similarity tokens at the beginning can help reduce interference from other correlation elements. The paper briefly claims that \"Early processing of strong matches helps resolve ambiguities in these regions,\" in Sec. 4.2, but a more in-depth analysis is needed."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Efficient Modeling of High-Dimensional Correlations: By using selective state-space models (SSMs), MambaMatcher models complex inter-image relationships effectively while avoiding the high computational cost typically associated with 4D correlation maps.\n\n2. Novel Similarity-Aware Selective Scan Mechanism: The paper introduces a unique scan mechanism inspired by Mamba’s linear-complexity algorithm. This mechanism refines the 4D correlation tensor accurately, enabling high-resolution processing of inter-image correlations."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The content in Sec 4.2 MULTI-LEVEL CORRELATION COMPUTATION AND AGGREGATION is somewhat ambiguous. The overall process flow should be: Shape the correlation map from 4D to 1D -> order -> reorder -> reshape  the correlation map  from 1D to 4D. Is the sorting performed along the $H \\times W \\times H \\times W$ dimension or along the $2L$ dimension? If it is along the $H \\times W \\times H \\times W$ dimension, is the sorting directly based on similarity in descending order? How are the potential spatial relationships between correlation elements encoded?\n\n2. Similarity-aware Selective Scanning: Mamba relies on its inherent order to capture the relative positional relationships between sequence tokens. When  correlation elements are sorted by similarity before scanning, does this mean  no longer need to consider the spatial locations of these elements? \n\n3. Parameters of Feature Aggregation: Line 215 states that \"The feature aggregators share the same weights across all levels to maintain efficiency.\" However, Lines 320-322 describe the feature aggregation layer as consisting of two layers of 2D convolution with a kernel size of 5, with output channel dimensions of 64 and 14, respectively, and a ReLU activation function in between. This implies that feature aggregation includes four 2D convolutional layers and two ReLU activations. However, in Table 6, feature aggregation is reported to have 42.5M parameters, which appears inconsistent. Please provide a more detailed explanation."}},"nonreaders":[],"tmdate":1731428303610,"tcdate":1730519057485,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5072/Reviewer_Fxfi"],"signatures":["ICLR.cc/2025/Conference/Submission5072/Reviewer_Fxfi"],"forum":"GmE8ovvXaJ","number":2,"license":"CC BY 4.0","cdate":1730519057485,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5072/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428303610,"domain":"ICLR.cc/2025/Conference","replyto":"GmE8ovvXaJ","id":"eOb9HsWdMk","forumContent":{"TLDR":{"value":"We introduce MambaMatcher, which formulates Mamba layers to process the correlation map between image pairs to establish accurate semantic correspondences with improved performance and efficiency."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["semantic correspondence","state-space models","correlation aggregation","Mamba","feature aggregation","Similarity-aware Selective Scan"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Establishing semantic correspondences between images is a fundamental yet challenging task in computer vision. Traditional feature-metric methods enhance visual features but may miss complex inter-image relationships, while recent correlation-metric approaches attempt to model these relationships but are hindered by high computational costs due to processing 4D correlation maps. We introduce MambaMatcher, a novel method that overcomes these limitations by efficiently modeling high-dimensional correlations using selective state-space models (SSMs), treating multi-level correlation scores as states. By implementing a similarity-aware selective scan mechanism adapted from Mamba’s linear-complexity algorithm, MambaMatcher refines the 4D correlation tensor effectively without compromising feature map resolution or receptive field. Experiments on standard semantic correspondence benchmarks demonstrate that MambaMatcher achieves state-of-the-art performance without relying on large input images or computationally expensive diffusion-based feature extractors, effectively capturing rich inter-image correlations while maintaining computational efficiency."},"_bibtex":{"value":"@misc{\nkim2025selective,\ntitle={Selective State-Space Modeling of Correlation Maps for Semantic Correspondence},\nauthor={Seungwook Kim and Minsu Cho},\nyear={2025},\nurl={https://openreview.net/forum?id=GmE8ovvXaJ}\n}"},"title":{"value":"Selective State-Space Modeling of Correlation Maps for Semantic Correspondence"},"pdf":{"value":"/pdf/904fad05fa77d7d08d5defc1a4f7abaade7973d6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"kim|selective_statespace_modeling_of_correlation_maps_for_semantic_correspondence"},"authorids":{"value":["~Seungwook_Kim2","~Minsu_Cho1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seungwook Kim","Minsu Cho"]}},"version":2},{"content":{"summary":{"value":"This paper introduces MambaSL, a framework that minimally modifies the selective state space models and projection layers of a single-layer Mamba architecture for time series classification. Experimental evaluations conducted on 30 datasets from the UEA benchmark demonstrate that MambaSL consistently outperforms 20 competitive baseline methods."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. In univariate time series classification, the original **InceptionTime** paper evaluates models based on the checkpoint with the lowest training loss, and the same evaluation strategy is adopted in **[2]**. Under a comparable setting, how does **MambaSL** perform relative to **HC2** and **MultiRocket+Hydra** on the 30 UEA multivariate time series datasets?  \n\n2. On the 128 UCR univariate time series datasets, following the **InceptionTime evaluation protocol**—where the model with the lowest training loss is used for testing—how does **MambaSL** compare in classification performance with **HC2** and **MultiRocket+Hydra**?  \n\n3. Beyond the evaluation setups in **InceptionTime** and **[2]**, some studies (e.g., **TSLANet**) adopt a different experimental protocol that uses **20% of the training set as a validation subset** for model selection on both UCR and UEA datasets. Under this widely used setting, how does **MambaSL** perform compared with **HC2**, **MultiRocket+Hydra**, and other competitive baselines?  \n\n4. Compared to 20 baseline methods, what are the advantages and disadvantages of the proposed **MambaSL** method in terms of runtime?\n\n**Reference**\n\n[2] *Inherently Interpretable Time Series Classification via Multiple Instance Learning*, ICLR, 2024."},"rating":{"value":4},"details_of_ethics_concerns":{"value":"None."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This paper applies the Mamba architecture to time series classification tasks and achieves superior performance compared to Transformer, MLP, and CNN models on multivariate time series classification benchmarks. This demonstrates the strong potential of Mamba for time series classification modeling.  \n2. The authors provide open-source code for the proposed model, along with implementations of the baseline methods, ensuring the reproducibility and transparency of the experimental results."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper’s title, introduction, and model design focus primarily on time series classification without providing an in-depth discussion of how variable relationships are modeled in multivariate time series. In this case, the experiments are limited to the 30 datasets from the UEA multivariate time series classification archive, while excluding the 128 univariate datasets. This omission weakens the paper’s motivation and makes it difficult to assess whether the proposed method achieves state-of-the-art performance under univariate settings.  \n\n2. Although the application of Mamba to time series classification is commendable, the paper does not clearly explain how the proposed MambaSL framework captures intrinsic temporal properties—such as temporal dependencies, inter-variable relationships, and short- or long-term sequence characteristics. The introduction and contribution sections also fail to clarify how MambaSL learns discriminative feature patterns beneficial for classification.  \n\n3. The model section contains extensive background on basic Mamba concepts, making it difficult for readers unfamiliar with Mamba to follow. At the same time, readers experienced in time series classification may still find it unclear how the model effectively learns task-relevant features for classification.  \n\n4. Despite the considerable experimental effort and the inclusion of reproducible scripts, all deep learning results in the paper suffer from a **test data leakage issue**. As shown in the provided code (`MambaSL/exp/exp_classification.py`, lines 22–25), the test set is incorrectly used as the validation set:\n\n   ```python\n   self.train_data, self.train_loader = self._get_data(flag='TRAIN')\n   if self.args.is_training:\n       self.vali_data, self.vali_loader = self._get_data(flag='TEST')\n   self.test_data, self.test_loader = self._get_data(flag='TEST')\n\nThis indicates that the test set is used for model validation and hyperparameter selection, resulting in a significant evaluation bias.\n\n5. In Appendix A, the authors mention that the experimental framework is based on the **Time-Series Library (TSLib)** (Wu et al., 2023). While TSLib is widely recognized for fair benchmarking in forecasting tasks, prior research ([1], Section 4.4 “Leaky Baselines”) has shown that TSLib introduces **test data leakage** in classification experiments. Furthermore, several non-deep learning baselines (e.g., Rocket, HC2, Hydra) do not rely on validation sets during training and for model selection. Consequently, using the maximum test performance as the final evaluation metric for MambaSL constitutes an **unfair comparison** with these baselines.\n\n**Reference**\n\n[1] *TOTEM: Tokenized Time Series Embeddings for General Time Series Analysis*, TMLR, 2024."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922392124,"tcdate":1762697069241,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11231/Reviewer_wsQb"],"signatures":["ICLR.cc/2026/Conference/Submission11231/Reviewer_wsQb"],"forum":"YDl4vqQqGP","number":3,"license":"CC BY 4.0","cdate":1762697069241,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11231/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922392124,"domain":"ICLR.cc/2026/Conference","replyto":"YDl4vqQqGP","id":"pog7d1pEPA","forumContent":{"TLDR":{"value":"We introduce MambaSL, a minimally redesigned single-layer Mamba that achieves state-of-the-art accuracy on the UEA30 benchmark, with reproducible evaluation covering all baselines."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["time series classification","single-layer Mamba","modular selective SSM","multi-head adaptive pooling","skip connection"]},"supplementary_material":{"value":"/attachment/dfc093493a30882d173d5eedf1107e0c42d4d9fc.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Despite recent advances in state space models (SSMs) such as Mamba across various sequence domains, research on their standalone capacity for time series classification (TSC) has remained limited. \nWe propose MambaSL, a framework that minimally redesigns the selective SSM and projection layers of a single-layer Mamba, guided by four TSC-specific hypotheses.\nTo address benchmarking limitations—restricted configurations, partial University of East Anglia (UEA) dataset coverage, and insufficiently reproducible setups—we re-evaluate 20 strong baselines across all 30 UEA datasets under a unified protocol. \nAs a result, MambaSL achieves state-of-the-art performance with statistically significant average improvements, while ensuring reproducibility via public checkpoints for all evaluated models.\nTogether with visualizations, these results demonstrate the potential of Mamba-based architectures as a TSC backbone."},"_bibtex":{"value":"@inproceedings{\njung2026mambasl,\ntitle={Mamba{SL}: Exploring Single-Layer Mamba for Time Series Classification},\nauthor={Yoo-Min Jung and Leekyung Kim},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=YDl4vqQqGP}\n}"},"title":{"value":"MambaSL: Exploring Single-Layer Mamba for Time Series Classification"},"pdf":{"value":"/pdf/36a22e78ae778d70b99d6fc145e8d5a9be7e0c1d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jung|mambasl_exploring_singlelayer_mamba_for_time_series_classification"},"authorids":{"value":["~Yoo-Min_Jung1","~Leekyung_Kim1"]},"authors":{"value":["Yoo-Min Jung","Leekyung Kim"]}},"version":2},{"content":{"summary":{"value":"This paper identifies the fixed state size of currently best performing linear attention variants with gating, such as Mamba-2 and Gated DeltaNet as the main limitation to handle information in a long context. \nBy introducing log-linear attention - a framework with a logarithmically growing set of hidden states that can be applied to existing linear attention variants - it provides a middle ground between standard attention with linear growing memory and linear attention with a fixed state size, which is independent of sequence length.\nLog-linear attention is based on the insight that efficient attention variants depend on the structure of the structure of the masking matrix in the attention operation, and replaces existing masking structures with a hierarchical one. \nWith log-linear variants of Mamba-2 and Gated Delta Net, the authors demonstrate the general applicability of the log-linear attention framework. \nIn their experiments on small scale language modeling setups and synthetic tasks the log-linear variants show mild but consistent improvements over the standard linear RNN variants. \nIn training throughput and runtime benchmarks the authors demonstrate benefits of the log-linear Mamba-2 variant over Flash-attention 2 at longer context and only small runtime overheads compared to the default Mamba-2 implementation."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"- L.392-393: Why does the linear layer on top of the hidden states add 3% additional parameters to Mamba-2 and only 0.4% to Gated Delta Net?\n- Description of the P matrix L.236 - 245 would help understandability\n- L. 337, 997: Could the authors elaborate on the MVA pattern for Mamba-2 and/or provide references to descriptions of this?\n- Does there exist an optimized kernel implementation for the log-linear variant of Gated Delta Net?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"- Clear motivation\n- Well written\n- The overview of related variants and the view of efficient attention mechanisms as different parametrizations of structured masking matrices is great. It shows how this naturally results in the idea & implementation for log-linear attention.\n- To the best of my knowledge log-linear attention is a novel method for expanding the state size.\n- The paper provides simple pure PyTorch implementations and shows experiments with optimized kernels (even though code for these is missing) achieving runtime benefits over existing methods"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Only small performance improvements over linear counter parts / base methods across several tasks (admitted by authors)\n- The authors place log-linear attention as middle ground between standard attention and linear attention in terms of memory state size: Hence I would expect an exemplary calculation of the memory consumption of log-linear attention, standard linear attention and KV-cache for various sequence lengths and reasonable model sizes\n- No code provided for Mamba2 and Gated Delta Net log linear attention variants\n- The paper would further benefit from more details on the efficient kernel implementations (including code) for Mamba-2 \n\n\nDespite these weaknesses, the paper has a clear motivation, is very well written, contributes new insights on efficient attention variants and their implementation, as well as a novel method for expanding the state size of linear attention variants, which outweighs the weaknesses. Therefore I recommend acceptance."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942109126,"tcdate":1761828368561,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22191/Reviewer_Zys3"],"signatures":["ICLR.cc/2026/Conference/Submission22191/Reviewer_Zys3"],"forum":"mOJgZWkXKW","number":2,"license":"CC BY 4.0","cdate":1761828368561,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22191/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942109126,"domain":"ICLR.cc/2026/Conference","replyto":"mOJgZWkXKW","id":"TzBFbDKerL","forumContent":{"TLDR":{"value":"We introduce a tensor attention framework and propose log-linear attention, which expands beyond fixed-size hidden states to achieve log-linear complexity."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["subquadratic architecture","triton kernel","structured matrices"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The attention mechanism in Transformers is an important primitive for accurate and scalable sequence modeling. Its quadratic-compute and linear-memory complexity however remain significant bottlenecks. Linear attention and state-space models enable linear-time, constant-memory sequence modeling and can moreover be trained efficiently through matmul-rich parallelization across sequence length. However, at their core these models are still RNNs, and thus their use of a fixed-size hidden state to model the context is a fundamental limitation. This paper develops log-linear attention, an attention mechanism that balances linear attention's efficiency and the expressiveness of softmax attention. Log-linear attention replaces the fixed-size hidden state with a logarithmically growing set of hidden states. We show that with a particular growth function, log-linear attention admits a similarly matmul-rich parallel form whose compute cost is log-linear in sequence length. Log-linear attention is a general framework and can be applied on top of existing linear attention variants. As case studies, we instantiate log-linear variants of two recent architectures---Mamba-2 and Gated DeltaNet---and find they perform well compared to their linear-time variants."},"_bibtex":{"value":"@inproceedings{\nguo2026loglinear,\ntitle={Log-Linear Attention},\nauthor={Han Guo and Songlin Yang and Tarushii Goel and Eric P. Xing and Tri Dao and Yoon Kim},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=mOJgZWkXKW}\n}"},"title":{"value":"Log-Linear Attention"},"pdf":{"value":"/pdf/481db6dd1f1973193f58822e77e56402f31473e9.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"guo|loglinear_attention"},"authorids":{"value":["~Han_Guo1","~Songlin_Yang1","~Tarushii_Goel1","~Eric_P._Xing1","~Tri_Dao1","~Yoon_Kim1"]},"authors":{"value":["Han Guo","Songlin Yang","Tarushii Goel","Eric P. Xing","Tri Dao","Yoon Kim"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a tracking-by-propagation framework, SambaMOTR, which wanna tackles the question of how to model long-range dependencies within tracklets, interdependencies among tracklets, and the associated temporal occlusions. The core function of the  SambaMOTR is the Samba module, a novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet.  SambaMOTR is evaluated on DanceTrack, BFT, and SportsMOT datasets and achieves the state-of-the-art performance."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"see Weaknesses."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. SambaMOTR introduces the novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet.\n2. SambaMOTR achieves good tracking results on DanceTrack, BFT, and SportsMOT datasets."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The authors are advised to analyze the computational complexity.\n2. The authors are advised to analyze why Samba is suitable for trajectory modeling and compare it with xLSTM, other SSMs, and RRNs.\n3. The authors are advised to provide some tracking cases of SambaMOTR to demonstrate its superiority in trajectory modeling."}},"nonreaders":[],"tmdate":1732567739404,"tcdate":1730388989144,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1948/Reviewer_WHQW"],"signatures":["ICLR.cc/2025/Conference/Submission1948/Reviewer_WHQW"],"forum":"OeBY9XqiTz","number":1,"license":"CC BY 4.0","cdate":1730388989144,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1948/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732567739404,"domain":"ICLR.cc/2025/Conference","replyto":"OeBY9XqiTz","id":"UWR5fTNROP","forumContent":{"TLDR":{"value":"SambaMOTR is a tracking-by-propagation method that (i) leverages long-range dependencies and (ii) synchronizes memory across object trajectories to better estimate future track queries and handle occlusions."},"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Synchronized sequence modeling; multiple object tracking"]},"supplementary_material":{"value":"/attachment/fa9520ca056ddd85a7a4643d378716d3643d0ef6.pdf"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Multiple object tracking in complex scenarios - such as coordinated dance performances, team sports, or dynamic animal groups - presents unique challenges. In these settings, objects frequently move in coordinated patterns, occlude each other, and exhibit long-term dependencies in their trajectories. However, it remains a key open research question on how to model long-range dependencies within tracklets, interdependencies among tracklets, and the associated temporal occlusions. To this end, we introduce Samba, a novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet. Samba autoregressively predicts the future track query for each sequence while maintaining synchronized long-term memory representations across tracklets. By integrating Samba into a tracking-by-propagation framework, we propose SambaMOTR, the first tracker effectively addressing the aforementioned issues, including long-range dependencies, tracklet interdependencies, and temporal occlusions. Additionally, we introduce an effective technique for dealing with uncertain observations (MaskObs) and an efficient training recipe to scale SambaMOTR to longer sequences. By modeling long-range dependencies and interactions among tracked objects, SambaMOTR implicitly learns to track objects accurately through occlusions without any hand-crafted heuristics. Our approach significantly surpasses prior state-of-the-art on the DanceTrack, BFT, and SportsMOT datasets."},"_bibtex":{"value":"@inproceedings{\nsegu2025samba,\ntitle={Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking},\nauthor={Mattia Segu and Luigi Piccinelli and Siyuan Li and Yung-Hsu Yang and Luc Van Gool and Bernt Schiele},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=OeBY9XqiTz}\n}"},"title":{"value":"Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking"},"pdf":{"value":"/pdf/890e8eb3a35d82db2064c3693269d0856cab26fe.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"segu|samba_synchronized_setofsequences_modeling_for_multiple_object_tracking"},"authorids":{"value":["~Mattia_Segu1","~Luigi_Piccinelli1","~Siyuan_Li9","~Yung-Hsu_Yang1","~Luc_Van_Gool1","~Bernt_Schiele1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Mattia Segu","Luigi Piccinelli","Siyuan Li","Yung-Hsu Yang","Luc Van Gool","Bernt Schiele"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.19925v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"ota|decision_mamba_reinforcement_learning_via_sequence_modeling_with_selective_state_spaces"},"authorids":{"value":[""]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.19925"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-19925,\n  publtype={informal},\n  author={Toshihiro Ota},\n  title={Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.19925},\n  url={https://doi.org/10.48550/arXiv.2403.19925}\n}\n"},"abstract":{"value":"Decision Transformer, a promising approach that applies Transformer architectures to reinforcement learning, relies on causal self-attention to model sequences of states, actions, and rewards. While this method has shown competitive results, this paper investigates the integration of the Mamba framework, known for its advanced capabilities in efficient and effective sequence modeling, into the Decision Transformer architecture, focusing on the potential performance enhancements in sequential decision-making tasks. Our study systematically evaluates this integration by conducting a series of experiments across various decision-making environments, comparing the modified Decision Transformer, Decision Mamba, with its traditional counterpart. This work contributes to the advancement of sequential decision-making models, suggesting that the architecture and training methodology of neural networks can significantly impact their performance in complex tasks, and highlighting the potential of Mamba as a valuable tool for improving the efficacy of Transformer-based models in reinforcement learning scenarios."},"title":{"value":"Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces"},"authors":{"value":["Toshihiro Ota"]}},"tmdate":1745541800368,"pdate":1704067200000,"tcdate":1745541545028,"writers":["~"],"signatures":["~Toshihiro_Ota1"],"forum":"F5BY6Xb6gE","license":"CC BY-SA 4.0","number":406086,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1745541800368,"domain":"DBLP.org","id":"F5BY6Xb6gE","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.19925v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"ota|decision_mamba_reinforcement_learning_via_sequence_modeling_with_selective_state_spaces"},"authorids":{"value":["~Toshihiro_Ota1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.19925"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-19925,\n  publtype={informal},\n  author={Toshihiro Ota},\n  title={Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.19925},\n  url={https://doi.org/10.48550/arXiv.2403.19925}\n}\n"},"abstract":{"value":"Decision Transformer, a promising approach that applies Transformer architectures to reinforcement learning, relies on causal self-attention to model sequences of states, actions, and rewards. While this method has shown competitive results, this paper investigates the integration of the Mamba framework, known for its advanced capabilities in efficient and effective sequence modeling, into the Decision Transformer architecture, focusing on the potential performance enhancements in sequential decision-making tasks. Our study systematically evaluates this integration by conducting a series of experiments across various decision-making environments, comparing the modified Decision Transformer, Decision Mamba, with its traditional counterpart. This work contributes to the advancement of sequential decision-making models, suggesting that the architecture and training methodology of neural networks can significantly impact their performance in complex tasks, and highlighting the potential of Mamba as a valuable tool for improving the efficacy of Transformer-based models in reinforcement learning scenarios."},"title":{"value":"Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces"},"authors":{"value":["Toshihiro Ota"]}},"tmdate":1745541485800,"pdate":1704067200000,"tcdate":1745541476364,"writers":["~"],"signatures":["~Toshihiro_Ota1"],"forum":"Ovpf9LWbxg","license":"CC BY-SA 4.0","number":406085,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1745541485800,"domain":"DBLP.org","id":"Ovpf9LWbxg","version":2},{"content":{"summary":{"value":"The paper claims that it addresses the challenge of long-sequence prediction in time series forecasting by enhancing Mamba’s sequence selection capability. It introduces Repetitive Contrastive Learning with sequence augmentation and noise to improve performance, which is shown to universally benefit Mamba-based models without extra memory overhead."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. The proposed method seems not to be specific to the Mamba structure. Applying it to more structures like Transformer could help the community better understand its significance.\n\n2. The paper lacks comparisons with existing time series pretraining and representation learning methods. Since the approach introduces a pre-training / data-augmentation stage rather than being end-to-end, fair comparisons with previous time series representation learning / data augmentation + downstream forecasting methods are needed.\n\n3. Mamba is inherently a sequential SSM for ordered input tokens. It seems the authors used vanilla Mamba rather than bidirectional Mamba and did not mention whether input shuffling was applied. Without shuffling during the repeat process, as described in Figure 1, sequential SSM characteristics could lead to information leakage in the noise data generation process. Also, when treating the entire series as an embedding $X^I$, a structure like bidirectional Mamba, which is order-independent, may be more appropriate.\n\n4. What are the basic settings for the forecasting problem in the experiments, such as input lookback length and forecasting horizon? These details are crucial for analyzing the model’s characteristics but are not provided. Additional training settings, like the train-test split, are also missing.\n\n5. Additionally, the term “selective scan mechanism” used in the paper seems somewhat misleading. The “scan” in Mamba (associative scan) refers to hardware parallelization and is unrelated to the algorithm’s selection ability."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"1. The proposed repeating sequence augmentation and repetitive contrastive learning methods are simple, easy to implement, and transferable to various TSF model architectures.\n\n2. Extensive experiments demonstrate that Mamba-based TSF models show significant performance improvement after applying the proposed pretraining and augmentation methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper contains some formatting errors, such as many mathematical symbols in Figures 1 and 2 not rendering correctly. The citation format is also inconsistent, with a lot of textual citations used in places where parenthetical citations would be more appropriate, making the paper harder to read.\n\n2. About selection ability: \n\nThe paper claims that Mamba is chosen to be used in the paper because of its stronger temporal selection ability compared to previous architectures. However, vanilla attention, with its temporal-adaptive dynamical parameters, is clearly more suitable for selection task than Mamba. The improvement of Mamba's selective SSM in selection ability is compared against the fixed temporal weights of S4, not against the more flexible softmax attention.\n\nMamba’s selective SSM was designed to introduce selection capabilities, like the induction heads learned in Transformer-based LLMs, to fixed SSM-weighted models like S4. Due to time complexity constraints, Mamba is less likely to surpass standard attention, which has $O(L^2)$ complexity because it calculates unique softmax temporal weights for each token, maximizing selection and matching abilities. In contrast, near-linear complexity methods (like Mamba and linear attention variants) avoid computing unique weights for each token, inheriting attention weights from previous steps, making adjustments through methods like gating mechanisms rather than recalculating all the weights independently for each step, to decrease the time complexity. This temporal propagation of weights limits their ability to achieve the distinct selection ability of softmax attention for each token.\n\nThe paper’s argument that “Mamba's better performance in TSF comes from its selection ability, thus requiring better pretraining methods to enhance this ability for the performance improvement of downstream forecasting” lacks sufficient support. While the proposed contrastive pretraining method does improve performance, is this really due to enhanced selection ability? Contrastive learning literature more often emphasizes the maintenance of a well-structured latent space to distinguish samples. The performance improvement might stem from learning more distinguishable representations in the latent space rather than an unquantified enhancement in selection ability.\n\nIn summary, the paper would benefit from more discussion, analysis, and experiments to demonstrate that the proposed methods enhance the model's selective ability. For instance, the authors could adopt the analysis techniques from LLM literature that show when and how induction heads are learned during training to compare the selective ability before and after applying the methods in this paper. Otherwise, the paper may need rebranding to explain the performance improvement from alternative perspectives, such as emphasizing how contrastive pretraining enhances the geometric properties of the latent space to improve the representation and discrimination of time series, as suggested in prior contrastive learning literature."}},"nonreaders":[],"tmdate":1733222116300,"tcdate":1730611779965,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9847/Reviewer_NrLR"],"signatures":["ICLR.cc/2025/Conference/Submission9847/Reviewer_NrLR"],"forum":"tIURLNBTPx","number":3,"license":"CC BY 4.0","cdate":1730611779965,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9847/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733222116300,"domain":"ICLR.cc/2025/Conference","replyto":"tIURLNBTPx","id":"Awj7FZGVco","forumContent":{"TLDR":{"value":"We propose a Repetitive Contrastive Learning method to enhance the sequence selection capability of Mamba SSM, which can bring improvements to multiple Mamba-based models in time series prediction problems."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Self-supervised learning; Time Series Prediction"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The prediction of long sequences has always been a challenge in time series forecasting tasks. Due to Mamba's sequence selection capability, many Mamba-based models have been proposed, achieving state-of-the-art results in long sequence prediction problems. However, much research has focused on integrating mamba-ssm into specific model structures for better performance, while the core of mamba-ssm, its sequence selection capability, has not been deeply explored. We believe there is significant potential in Mamba's sequence selection capability and propose a Repetitive Contrastive Learning (RCL) method to enhance it. Specifically, we utilize Repeating Sequence Augmentation to expand the sequence while introducing Gaussian noise, thereby enhancing the Mamba block's sequence selection capability through both inter-sequence and intra-sequence contrastive techniques. Then our methods transfer parameters directly from a single pretrained Mamba block to a variety of Mamba-based models. This approach provides superior initialization for forecasting tasks. Our experiments consistently demonstrate that this technique improves the forecasting performance of many Mamba-based models, without imposing additional memory requirements."},"_bibtex":{"value":"@misc{\nyan2025repetitive,\ntitle={Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction},\nauthor={WenBo Yan and Hanzhong Cao and Ying Tan},\nyear={2025},\nurl={https://openreview.net/forum?id=tIURLNBTPx}\n}"},"title":{"value":"Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction"},"pdf":{"value":"/pdf/25ba192abf3d52ee17ddd05bad92e932a9f5e0c0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"yan|repetitive_contrastive_learning_enhances_mambas_selectivity_in_time_series_prediction"},"authorids":{"value":["~WenBo_Yan1","~Hanzhong_Cao1","~Ying_Tan1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["WenBo Yan","Hanzhong Cao","Ying Tan"]}},"version":2},{"content":{"summary":{"value":"This work proposes a method to improve the modeling performance of modern SSMs/ Linear RNNs such as Mamba and Gated DeltaNet, whose recurrent state can be formulated as $S_{t} = \\alpha_t * S_{t-1} + (KV)_t$. To get final output, these algorithms multiply query $Q_t$ by a single state matrix $S_t$, which aggregates information about all previous KV states up to step $t$. Log-linear attention instead disaggregates $S_t$ into $O(\\log T)$ states of the same size, each containing information about a disjoint contiguous subsequence of  $\\{0, …, t\\}$. Then the query gets multiplied independently by each state and also by an input-dependent scalar coefficient $\\lambda_i, \\; i \\in \\{0, …, O(\\log T)\\}$, corresponding to that state. Finally, the results sum up. A pure linear attention variant is equivalent to log-linear attention where $\\lambda_i=1$ for all i.\n\nThe partitioning of the sequence proceeds according to Fenwick tree scheme, where each subsequence can have at most $2^i$ consecutive timesteps, and the shortest subsequences is located at the latest step t, while longer subsequences’s boundaries go toward the sequence start.\n\nI feel excited about this novel method (see strengths) and vote for its acceptance."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"1. Can you come up with a theoretical explanation why log-linear attention offers performance improvements in comparison with pure linear variants?\n\n2. Did you try any other partition schemes besides Fenwick Tree partitioning? I could think of other schemes, with overlapping and disjoined partitions. There could even be some schemes that recover $O(T)$ complexity (e.g., proceed as usual until sequence length is X, then keep placing the oldest tokens into the outermost bucket instead of creating new levels of hierarchy). As such, why did you choose this specific scheme?\n\n3. A follow-up question: could there be some trade-off, where this algorithm could run in linear time at the expense of an arbitrary higher memory consumption?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":4},"strengths":{"value":"* Potent sub-quadratic runtime alternatives to Transformers is an important open area of research, and this work provides a promising way to improve modeling quality of such architectures, hopefully bringing us closer to an algorithm capable of fully replacing Transformer in autoregressive language modeling.\n\n* The proposed method is coherent and intuitive: if we want to increase long-range performance in comparison with pure linear-time algorithms such as Mamba and DeltaNet, it is plausible that we have to execute a higher relative amount of computations, than for short sequences.\n\n* Log-linear attention is a meta-algorithm, which is compatible with many linear-time alternatives to softmax attention.\n\n* I believe the proposed approach has a potential to be extended and generalized by using other partitioning schemes which could bring further performance gains.\n\n* I’d like to specifically praise comprehensive and fair comparisons which don’t shy away from presenting both positive and negative results. They help to build an honest and complete picture of the method’s strong and weak sides and possible areas of application.\n\n* The validation shows meaningful improvements relative to pure linear counterparts on a large subset of tested benchmarks, effectuated by log-linear extension."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"My judgement is that the paper doesn't have major problems. There are some minor issues mostly related to exposition/ formatting which I listed below. \n\n1. Did you perform any measurements of the memory footprint of the algorithm during inference (prefill, decode) and training workloads? A comparison for different sequence lengths with vanilla Mamba-2 and Gated DeltaNet, as well as with FlashAttention would be helpful. I understand that it’s likely to be $O(\\log(T))$ times greater than aforementioned algorithms, but that’s an expected trade-off, which is easily tolerated given log-linear attention’s superior modeling quality. Nonetheless, these numbers would be important for finding out the most fitting conditions to use the algo.\n\n2. It’s not clear to me from the paper how the log-linear part of the algorithm modifies the underlying linear part, both in chunk-wise parallel and recurrent algos. For example, what happens when two states for neighboring filled buckets get merged after a recurrent step? Since the states are created independently, $S_{[t_1:t_2)} + S_{[t_2:t_3)} \\neq S_{[t_1:t_3)}$, although the underlying algorithm requires precisely $S_{[t_1:t_3)}$. Similar ambiguities emerge when considering chunk-wise form. \n\n3. There is no formal definition of the exact functional form of $\\lambda_i$s. Are they simply linear projections of input vector query $q_t$? Or are they calculated using the same laws as alphas (i.e. in a different manner for each underlying architecture)? How does the algorithm handle that the number of lambdas $max(i)$ is not bounded from above and can extrapolate beyond the maximal value during pre-training?\n\n4. Minor typos/ formatting problems:\n* Line 189 – why is the right bound $t$ open? From the text it follows that the t-th token itself is a part of the partition.\n\n* Line 213 – it would be clearer to mention explicitly that $b_t^{(i)}$ denotes the starting position of partition i, it was somewhat hard to infer for me at first glance.\n\n* Lines 237-246 – there’s no caption of this figure.\n\n* Line 291 – I believe it’s $\\lceil T/C \\rceil$.\n\n* Line 763 – image and table captions are overlaid.\n\n* Table 3 – There’s no mention of the table in the text, and it takes an attentive reader to understand that it’s the summary of Table 6."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942108953,"tcdate":1761971023501,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22191/Reviewer_xdQ7"],"signatures":["ICLR.cc/2026/Conference/Submission22191/Reviewer_xdQ7"],"forum":"mOJgZWkXKW","number":3,"license":"CC BY 4.0","cdate":1761971023501,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22191/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942108953,"domain":"ICLR.cc/2026/Conference","replyto":"mOJgZWkXKW","id":"NeYEPIrrLZ","forumContent":{"TLDR":{"value":"We introduce a tensor attention framework and propose log-linear attention, which expands beyond fixed-size hidden states to achieve log-linear complexity."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["subquadratic architecture","triton kernel","structured matrices"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The attention mechanism in Transformers is an important primitive for accurate and scalable sequence modeling. Its quadratic-compute and linear-memory complexity however remain significant bottlenecks. Linear attention and state-space models enable linear-time, constant-memory sequence modeling and can moreover be trained efficiently through matmul-rich parallelization across sequence length. However, at their core these models are still RNNs, and thus their use of a fixed-size hidden state to model the context is a fundamental limitation. This paper develops log-linear attention, an attention mechanism that balances linear attention's efficiency and the expressiveness of softmax attention. Log-linear attention replaces the fixed-size hidden state with a logarithmically growing set of hidden states. We show that with a particular growth function, log-linear attention admits a similarly matmul-rich parallel form whose compute cost is log-linear in sequence length. Log-linear attention is a general framework and can be applied on top of existing linear attention variants. As case studies, we instantiate log-linear variants of two recent architectures---Mamba-2 and Gated DeltaNet---and find they perform well compared to their linear-time variants."},"_bibtex":{"value":"@inproceedings{\nguo2026loglinear,\ntitle={Log-Linear Attention},\nauthor={Han Guo and Songlin Yang and Tarushii Goel and Eric P. Xing and Tri Dao and Yoon Kim},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=mOJgZWkXKW}\n}"},"title":{"value":"Log-Linear Attention"},"pdf":{"value":"/pdf/481db6dd1f1973193f58822e77e56402f31473e9.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"guo|loglinear_attention"},"authorids":{"value":["~Han_Guo1","~Songlin_Yang1","~Tarushii_Goel1","~Eric_P._Xing1","~Tri_Dao1","~Yoon_Kim1"]},"authors":{"value":["Han Guo","Songlin Yang","Tarushii Goel","Eric P. Xing","Tri Dao","Yoon Kim"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Test-Time Training (TTT) layers, which make the hidden state a model updated via self-supervised learning, enabling efficient long-context processing with linear complexity. Two implementations, TTT-Linear and TTT-MLP, outperform Transformers and Mamba on long-context tasks."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Q1. Could you demonstrate statistically significant performance improvement by adding error bars to the experimental results? While TTT-Linear shows similar computational efficiency to Mamba, its perplexity appears nearly identical. Without error bars in the results, it is difficult to determine how effective this method truly is.\n\nQ2. Is it possible to include evaluations on downstream tasks as part of the experiment?\nIn Mamba’s original paper, experiments were conducted on downstream tasks to demonstrate performance beyond language modeling alone. At a minimum, it seems necessary to include experiments similar to those in Section 4.2.2 of the following paper:\nhttps://arxiv.org/abs/2312.00752\n\nQ3. Should comparative experiments with the following studies also be conducted? Also,one seems to be a missing citation.\n\n1. Reconsidering the Past: Optimizing Hidden States in Language Models, Davis Yoshida, Kevin Gimpel, EMNLP 2021 Findings\n2. Meta-Learning Fast Weight Language Models, Kevin Clark, Kelvin Guu, Ming-Wei Chang, Panupong Pasupat, Geoffrey Hinton, Mohammad Norouzi, EMNLP 2022"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"TTT layers offer efficient long-context processing with linear complexity, outperforming traditional Transformers and Mamba in terms of scalability and performance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper shows limited improvement over Mamba, and its evaluation is restricted to language modeling tasks. Furthermore, it lacks comparative experiments with related methods like Fast Weight Layers and Hidden-State Optimization. \n\nReconsidering the Past: Optimizing Hidden States in Language Models\nDavis Yoshida, Kevin Gimpel, EMNLP2021 findings\nMeta-Learning Fast Weight Language Models\nKevin Clark, Kelvin Guu, Ming-Wei Chang, Panupong Pasupat, Geoffrey Hinton, Mohammad Norouzi, EMNLP2022"}},"nonreaders":[],"tmdate":1732926434619,"tcdate":1730906527999,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5310/Reviewer_eEfe"],"signatures":["ICLR.cc/2025/Conference/Submission5310/Reviewer_eEfe"],"forum":"eifW0W0xgt","number":3,"license":"CC BY 4.0","cdate":1730906527999,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5310/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732926434619,"domain":"ICLR.cc/2025/Conference","replyto":"eifW0W0xgt","id":"2aixVrNvVK","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["test-time training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state. Inspired by prior work, we present a practical framework to instantiate sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Both TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer, they can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. With preliminary systems optimization, TTT-Linear is already faster than Transformer at 8k context and matches Mamba in wall-clock time. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research."},"_bibtex":{"value":"@misc{\nsun2025learning,\ntitle={Learning to (Learn at Test Time): {RNN}s with Expressive Hidden States},\nauthor={Yu Sun and Xinhao Li and Karan Dalal and Jiarui Xu and Arjun Vikram and Genghan Zhang and Yann Dubois and Xinlei Chen and Xiaolong Wang and Sanmi Koyejo and Tatsunori Hashimoto and Carlos Guestrin},\nyear={2025},\nurl={https://openreview.net/forum?id=eifW0W0xgt}\n}"},"title":{"value":"Learning to (Learn at Test Time): RNNs with Expressive Hidden States"},"pdf":{"value":"/pdf/be1b3f1cf37ec5812fb041e8bbdd1569d92a9992.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sun|learning_to_learn_at_test_time_rnns_with_expressive_hidden_states"},"authorids":{"value":["~Yu_Sun1","~Xinhao_Li3","~Karan_Dalal1","~Jiarui_Xu1","~Arjun_Vikram1","~Genghan_Zhang1","~Yann_Dubois1","~Xinlei_Chen1","~Xiaolong_Wang3","~Sanmi_Koyejo1","~Tatsunori_Hashimoto1","~Carlos_Guestrin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yu Sun","Xinhao Li","Karan Dalal","Jiarui Xu","Arjun Vikram","Genghan Zhang","Yann Dubois","Xinlei Chen","Xiaolong Wang","Sanmi Koyejo","Tatsunori Hashimoto","Carlos Guestrin"]}},"version":2},{"content":{"summary":{"value":"This paper investigates the adaptation of a recently emerged  Mamba architecture to long-term time-series forecasting. It highlights three key aspects: order, semantic, and cross-variate dependencies. The derived Samba architecture encompasses two branches for temporal and variable dependency encoding, of which the encoded representations are concatenated to produce forecasts."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See weaknesses."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":1},"strengths":{"value":"It is interesting to investigate the pros and cons of the Mamba architecture, or more broadly, structured state-space models (SSMs), on long-term time-series forecasting."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Concerns Regarding the Soundness and Contributions of the Paper\n\n1. Overlooking the Unique Advantage of Mamba: Modeling Long Sequences\n\nMamba, a recent and popular instantiation of state-space models (SSMs), is renowned for its efficiency in processing long sequences. It significantly reduces the computational overheads of self-attention in Transformers while seemingly maintaining long-term dependencies in various real-world datasets. However, this paper primarily compares time-series representation learning among MLP, Transformer, and Mamba, fixing the lookback length at T=96. When modeling relatively shorter sequences, the necessity of introducing SSMs diminishes significantly, which in turn reduces the impact of this research.\n\n2. Inappropriate and Biased Expressions/Claims\n\nLines 014-016: The claim, \"However, most approaches still struggle to comprehensively capture reliable and informative dependencies inherent in time series data,\" lacks rigor. Existing time-series forecasting models have made significant progress in learning effective time-series representations and deliver commendable forecasts in many scenarios. The paper's experiments do not show a substantial difference in model performance to support this claim.\n\nLines 052-054: The assertion that \"they struggle with perceiving temporal order due to the permutation-invariant nature of self-attention, even with positional encodings,\" is debatable. It is not clear how sensitivity to sequence order impacts the capability of learning effective temporal representations. Extensive research on position encoding for Transformers has already led to the success of modern large language models. The experiments in Table 1 only demonstrate that Transformers are less sensitive to order permutation than Linear and Mamba, which is expected. However, it is unclear how this capability affects temporal representation learning. In fact, being less sensitive to order permutation might be advantageous in sequence modeling. Furthermore, the use of Mamba to model cross-variate dependency seems flawed, as time-series variates lack a ground-truth order. Following the author's logic, using an order-sensitive model like Mamba for cross-variate dependency might not be appropriate.\n\nLines 058-061: The statement, \"existing approaches that utilize cross-variate dependency (Channel Dependent, CD) frequently underperform compared to methods that treat each variate independently (Channel-Independent, CI),\" is not universally applicable. The performance of CD and CI approaches highly depends on specific cases. Studies, such as iTransformer, have demonstrated the benefits of cross-variate modeling.\n\n3. Insufficient Experiments Leading to Unreliable Conclusions\n\nThe analyses in Section 4, which inform the design of Samba, are primarily based on experiments conducted on the small ETTm1 dataset and compared across basic neural architectures like Linear, vanilla MLP, and Transformer. These results can be interpreted in multiple ways, not solely as the authors suggest. For instance, Table 1 shows Transformers' reduced sensitivity to order perturbation, which may actually be beneficial for encoding effective temporal representations. Table 2 investigates patching effectiveness, originally introduced in PatchTST, yet PatchTST itself is not included in the analysis. Additionally, the impact of patch size on results and whether all datasets lead to the same conclusion are unexplored. Other factors, such as TSMixer extending MLP-based architectures on time-series and PatchTST's designs to prevent overfitting, are not considered in Table 3. Consequently, the analyses in Section 4 are unconvincing regarding the necessity of introducing Mamba.\n\n4. Seemingly Downgraded Performance of Baselines\n\nFor example, in Table 4, PatchTST on ETTh1 produces an MSE of 0.469 and an MAE of 0.454. However, the original paper reported much lower errors (https://arxiv.org/pdf/2211.14730).\n\n5. Lack of Unique Contributions\n\nIn addition to studying Mamba for time series, this paper appears to offer few unique contributions or novel insights. Concepts such as patching, cross-variate and cross-time dependency, and normalization to prevent overfitting have been extensively explored in prior research. Furthermore, Transformer variants in time series have effectively addressed challenges in these areas. Therefore, without the focus on modeling very long sequences, the necessity of adapting Mamba for time series remains questionable."}},"nonreaders":[],"tmdate":1731427314262,"tcdate":1730704912565,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission767/Reviewer_qdrT"],"signatures":["ICLR.cc/2025/Conference/Submission767/Reviewer_qdrT"],"forum":"9VRFPC29nb","number":4,"license":"CC BY 4.0","cdate":1730704912565,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission767/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427314262,"domain":"ICLR.cc/2025/Conference","replyto":"9VRFPC29nb","id":"nF1cN5dDvR","forumContent":{"TLDR":{"value":"propose a simplifed mamba and a theoretically sound disentangled encoding strategy for long-term time series forecasting"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["long-term time series forecasting","time series modeling","mamba"]},"supplementary_material":{"value":"/attachment/f88a207f308111754eeb7cbc865e432708675aab.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent advances in deep learning have led to the development of numerous models for Long-term Time Series Forecasting (LTSF). However, most approaches still struggle to comprehensively capture reliable and informative dependencies inherent in time series data. In this paper, we identify and formally define three critical dependencies essential for improving forecasting accuracy: the order dependency and semantic dependency in the time dimension as well as cross-variate dependency in the variate dimension. Despite their significance, these dependencies are rarely considered holistically in existing models. Moreover, improper handling of these dependencies can introduce harmful noise that significantly impairs forecasting performance. To address these challenges, we explore the potential of Mamba for LTSF, highlighting its three key advantages to capture three dependencies, respectively. We further empirically observe that nonlinear activation functions used in vanilla Mamba are redundant for semantically sparse time series data. Therefore, we propose SAMBA, a Simplified Mamba with disentangled dependency encoding. Specifically, we first eliminate the nonlinearity of vanilla Mamba to make it more suitable for LTSF. Along this line, we propose a disentangled dependency encoding strategy to endow Mamba with efficient cross-variate dependency modeling capability while minimizing the interference between time and variate dimensions. We also provide rigorous theory as a justification for our design. Extensive experiments on nine real-world datasets demonstrate the effectiveness of SAMBA over state-of-the-art forecasting models."},"_bibtex":{"value":"@misc{\nweng2025simplified,\ntitle={Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting},\nauthor={Zixuan Weng and Jindong Han and Wenzhao Jiang and Hao Liu},\nyear={2025},\nurl={https://openreview.net/forum?id=9VRFPC29nb}\n}"},"title":{"value":"Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting"},"pdf":{"value":"/pdf/cb12f563a9e3d09396284efb314db4d48c961f42.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"weng|simplified_mamba_with_disentangled_dependency_encoding_for_longterm_time_series_forecasting"},"authorids":{"value":["~Zixuan_Weng1","~Jindong_Han1","~Wenzhao_Jiang1","~Hao_Liu17"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zixuan Weng","Jindong Han","Wenzhao Jiang","Hao Liu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a method to predict the oligomeric state of fluorescent proteins using a deep learning model based on the Mamba architecture. The authors apply different data augmentation methods to enhance the model’s generalizability and compare it to other sequence-based models like RNN, LSTM, and Transformer. The Mamba-based model achieves high accuracy while reducing computational costs compared to other models."},"strengths":{"value":"The paper contributes Mamba architecture scheme for predicting the oligomeric state of fluorescent proteins, which improves accuracy and computational efficiency compared to traditional models. The introduction of protein-specific data augmentation techniques enhances the model's generalizability, offering a new direction for handling limited datasets in protein classification tasks."},"weaknesses":{"value":"The introduction lacks relevant background on the Mamba architecture and its prior applications, as well as the role of data augmentation in this context. Expanding on these topics would provide a clearer foundation for understanding the paper's contributions.\n\nWhile the paper highlights the advantages of GPU-based methods, it would benefit from additional experiments comparing the Mamba-based model with CPU-based approaches discussed in the literature review section. This comparison could further emphasize the performance improvements, and the literature review could be condensed to make room for these experiments.\n\nIn section 3.1, the model implementation details could be moved to a separate \"Implementation\" section, making the paper more organized.\n\nTable 2 is not explicitly referenced in the main body, which diminishes its usefulness.\n\nLastly, the paper does not discuss specific sequence patterns or features in a protein's amino acid sequence that may influence its oligomeric state. Addressing this could provide deeper biological insights and strengthen the interpretability of the model's predictions."},"confidence":{"value":4},"justification":{"value":"The paper presents a promising method using the Mamba architecture to predict the oligomeric state of fluorescent proteins, improving computational efficiency and accuracy compared to traditional models. Its use of data augmentation techniques is a strong contribution.\n\nHowever, the paper lacks sufficient background on Mamba and data augmentation, which are crucial to understanding the approach. Additionally, the absence of direct comparisons with CPU-based methods weakens the evidence for the model's improvements. Organizational issues, such as missing references to key tables, also detract from clarity. While the contributions are notable, these limitations reduce the overall impact of the work."},"title":{"value":"Promising Approach with Mamba Architecture for Predicting Protein Oligomeric States, but Lacks Key Comparisons and Clarity"}},"nonreaders":[],"tmdate":1731586640578,"tcdate":1728992720349,"writers":["NLDL.org/2025/Conference","NLDL.org/2025/Conference/Submission50/Reviewer_m2uG"],"signatures":["NLDL.org/2025/Conference/Submission50/Reviewer_m2uG"],"forum":"KaZzDtUeJY","number":4,"license":"CC BY 4.0","cdate":1728992720349,"readers":["everyone"],"invitations":["NLDL.org/2025/Conference/Submission50/-/Official_Review","NLDL.org/2025/Conference/-/Edit"],"mdate":1731586640578,"domain":"NLDL.org/2025/Conference","replyto":"KaZzDtUeJY","id":"ycIbYbIl5N","forumContent":{"TLDR":{"value":"Use of deep learning model Mamba to predict the Oligomeric state of a protein from its amino acid sequence"},"venue":{"value":"NLDL 2025 Oral"},"keywords":{"value":["Mamba","Fluorescent proteins","Oligomerization state"]},"abstract":{"value":"Fluorescent proteins (FPs) are essential tools in biomedical imaging, known for their ability to absorb and emit light, thereby allowing visualization of biological processes. Understanding the oligomeric state is crucial, as monomeric forms are often preferred in applications to minimize potential artifacts and prevent interference with cellular functions. Experimental methods to find the oligomeric state can be time-consuming and expensive. Most of the current computational model is CPU-based, limiting their speed and scalability. This paper studies the effectiveness of GPU-based deep-learning models in predicting the oligomeric states of fluorescent proteins directly from their amino acid sequences, specifically focusing on the Mamba architecture. Various protein-specific augmentations were also employed to enhance the model's generalizability. Our results indicate that the mamba-based model achieves accuracy and F1 score close to 90\\% and an MCC value of 0.8 with in predicting the oligomeric states of fluorescent proteins directly from its amino acid sequence. The code used in this study is available at [GitHub repository](https://github.com/smlab-niser/FluorMamba)."},"_bibtex":{"value":"@inproceedings{\nrajeev2024predicting,\ntitle={Predicting Oligomeric states of Fluorescent Proteins using Mamba},\nauthor={Agney K Rajeev and Joel Joseph K B and Subhankar Mishra},\nbooktitle={Northern Lights Deep Learning Conference 2025},\nyear={2024},\nurl={https://openreview.net/forum?id=KaZzDtUeJY}\n}"},"title":{"value":"Predicting Oligomeric states of Fluorescent Proteins using Mamba"},"pdf":{"value":"/pdf/80c5dc499c363959a7ab7f01e18b2d69fd297da3.pdf"},"git":{"value":"https://github.com/smlab-niser/FluorMamba"},"venueid":{"value":"NLDL.org/2025/Conference"},"paperhash":{"value":"rajeev|predicting_oligomeric_states_of_fluorescent_proteins_using_mamba"},"authorids":{"value":["~Agney_K_Rajeev1","~Joel_Joseph_K_B2","~Subhankar_Mishra1"]},"authors":{"value":["Agney K Rajeev","Joel Joseph K B","Subhankar Mishra"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Spatial-Mamba, a  state space models visual backbone designed for 2D visual tasks. It mainly introduce a multi-scale depthwise conv layer after the mamba operators to fuse the contextual information from different spatial locations. The author conducts theoretical analysis. Extensive experiements like image classification, dense prediction tasks are conducted.  The results show it outperforms the baselines and achieve competitive resutls."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please see weakness."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. The motivation is reasonbal and good. The vision mamba models suffers from processing  2D visual data due to the complexity of spatial structures. It heavily relys  on the 1D scanning strategy to transform 2D format into 1D sequence. The author apply conv-based layers to fuse the contextual information  in spatial domain, which is complemenary to the Mamba based operator. \n\n2. The experimetanl results are good. The accuracy in imageNet shows the improvement. The speed is still competitive as the conv operators are optimized, thus bringing less latency. \n\n3. The overall method is simple and easy to understand. Applying conv layers as a compelmentary module to improve the spatial modeling capability of the mamba is simple and direct."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The method seems to be a combination of Mamba and a classical module (depthwise conv).  It is also mentioned in No.3 point in Strengths. I would recommned author to validate/explore the spatial modeling enhancement methods more throughly. For example, applying the multi-scale conv layer to the original bi-directional Mamba (Vim) and cross-scan mamba (VMamba) to see the improvements. THough the one-direction scan and conv layer can aggregate enough spatial contextual information. \n\n2. The analysis of linear attention, mamba and the proposed method in section 4.3 is good. But the method proposed is rather straightforward. It seems the method doesnot improve the weakness of Mamba from the in-depth analysis. It first uses the Selective scan in Mamba to calculate the hidden states (one direction), following a depthwise conv to fuse the hidden states, then calculate the output y. It is not known the improvements over the method that adding more direcitons into the selective scan and computing the output. \n\n3. The modules/operators to fuse the contextual information can have mutiple choices. The paper chooses depthwise conv. Conv operator may have inductive bias, which aggregate the spatial information by a fixed pattern. I would see more explorations, which could improve the paper."}},"nonreaders":[],"tmdate":1732523496508,"tcdate":1730451330713,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2673/Reviewer_Q4w4"],"signatures":["ICLR.cc/2025/Conference/Submission2673/Reviewer_Q4w4"],"forum":"iDe1mtxqK5","number":3,"license":"CC BY 4.0","cdate":1730451330713,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2673/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732523496508,"domain":"ICLR.cc/2025/Conference","replyto":"iDe1mtxqK5","id":"F3y25660Dr","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Representation learning","Visual state space models","Structure-aware state fusion"]},"supplementary_material":{"value":"/attachment/bfe4420ea6a429fc96daffcd3ec6b05d08ace381.pdf"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSMs), such as Mamba, highly excel at capturing long-range dependencies in 1D sequential data, while their applications to 2D vision tasks still face challenges. Current visual SSMs often convert images into 1D sequences and employ various scanning patterns to incorporate local spatial dependencies. However, these methods are limited in effectively capturing the complex image spatial structures and the increased computational cost caused by the lengthened scanning paths. To address these limitations, we propose Spatial-Mamba, a novel approach that establishes neighborhood connectivity directly in the state space. Instead of relying solely on sequential state transitions, we introduce a structure-aware state fusion equation, which leverages dilated convolutions to capture image spatial structural dependencies, significantly enhancing the flow of visual contextual information. Spatial-Mamba proceeds in three stages: initial state computation in a unidirectional scan, spatial context acquisition through structure-aware state fusion, and final state computation using the observation equation. Our theoretical analysis shows that Spatial-Mamba unifies the original Mamba and linear attention under the same matrix multiplication framework, providing a deeper understanding of our method. Experimental results demonstrate that Spatial-Mamba, even with a single scan, attains or surpasses the state-of-the-art SSM-based models in image classification, detection and segmentation. Source codes and trained models can be found at \\url{ https://github.com/EdwardChasel/Spatial-Mamba }."},"_bibtex":{"value":"@inproceedings{\nxiao2025spatialmamba,\ntitle={Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion},\nauthor={Chaodong Xiao and Minghan Li and Zhengqiang ZHANG and Deyu Meng and Lei Zhang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=iDe1mtxqK5}\n}"},"title":{"value":"Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion"},"pdf":{"value":"/pdf/8219b6919d5e23d90a43ddcdec43ba242eac39c4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"xiao|spatialmamba_effective_visual_state_space_models_via_structureaware_state_fusion"},"authorids":{"value":["~Chaodong_Xiao2","~Minghan_Li1","~Zhengqiang_ZHANG1","~Deyu_Meng1","~Lei_Zhang2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chaodong Xiao","Minghan Li","Zhengqiang ZHANG","Deyu Meng","Lei Zhang"]}},"version":2},{"content":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Explainable AI","explainability","interpretability","state space models","Mamba","long-range dependency"]},"primary_area":{"value":"interpretability_and_explainability"},"abstract":{"value":"Recent sequence modeling approaches using selective state space sequence models, referred to as Mamba models, have seen a surge of interest. These models allow efficient processing of long sequences in linear time and are rapidly being adopted in a wide range of applications such as language modeling, demonstrating promising performance. To foster their reliable use in real-world scenarios, it is crucial to augment their transparency. Our work bridges this critical gap by bringing explainability, particularly Layer-wise Relevance Propagation (LRP), to the Mamba architecture. Guided by the axiom of relevance conservation, we identify specific components in the Mamba architecture, which cause unfaithful explanations. To remedy this issue, we propose MambaLRP, a novel algorithm within the LRP framework, which ensures a more stable and reliable relevance propagation through these components. Our proposed method is theoretically sound and excels in achieving state-of-the-art explanation performance across a diverse range of models and datasets. Moreover, MambaLRP facilitates a deeper inspection of Mamba architectures, uncovering various biases and evaluating their significance. It also enables the analysis of previous speculations regarding the long-range capabilities of Mamba models."},"_bibtex":{"value":"@inproceedings{\njafari2024mambalrp,\ntitle={Mamba{LRP}: Explaining Selective State Space Sequence Models},\nauthor={Farnoush Rezaei Jafari and Gr{\\'e}goire Montavon and Klaus Robert Muller and Oliver Eberle},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=2n1Ysn1EDl}\n}"},"title":{"value":"MambaLRP: Explaining Selective State Space Sequence Models"},"pdf":{"value":"/pdf/d253dfe5b75f15bb96b1033361699ed2984da98a.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"jafari|mambalrp_explaining_selective_state_space_sequence_models"},"authorids":{"value":["~Farnoush_Rezaei_Jafari1","~Grégoire_Montavon1","~Klaus_Robert_Muller1","~Oliver_Eberle1"]},"authors":{"value":["Farnoush Rezaei Jafari","Grégoire Montavon","Klaus Robert Muller","Oliver Eberle"]}},"tmdate":1736894074449,"pdate":1727287910412,"tcdate":1715679517738,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission9423/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission9423/Authors"],"forum":"2n1Ysn1EDl","license":"CC BY-NC-ND 4.0","number":9423,"cdate":1715679517738,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/-/Submission","NeurIPS.cc/2024/Conference/-/Post_Submission","NeurIPS.cc/2024/Conference/Submission9423/-/Revision","NeurIPS.cc/2024/Conference/-/Edit","NeurIPS.cc/2024/Conference/Submission9423/-/Camera_Ready_Revision"],"mdate":1736894074449,"odate":1730873920520,"domain":"NeurIPS.cc/2024/Conference","id":"2n1Ysn1EDl","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.07592v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"jafari|mambalrp_explaining_selective_state_space_sequence_models"},"authorids":{"value":["","https://dblp.org/search/pid/api?q=author:Grégoire_Montavon:","https://dblp.org/search/pid/api?q=author:Klaus-Robert_Müller:","https://dblp.org/search/pid/api?q=author:Oliver_Eberle:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.07592"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-07592,\n  publtype={informal},\n  author={Farnoush Rezaei Jafari and Grégoire Montavon and Klaus-Robert Müller and Oliver Eberle},\n  title={MambaLRP: Explaining Selective State Space Sequence Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.07592},\n  url={https://doi.org/10.48550/arXiv.2406.07592}\n}\n"},"abstract":{"value":"Recent sequence modeling approaches using Selective State Space Sequence Models, referred to as Mamba models, have seen a surge of interest. These models allow efficient processing of long sequences in linear time and are rapidly being adopted in a wide range of applications such as language modeling, demonstrating promising performance. To foster their reliable use in real-world scenarios, it is crucial to augment their transparency. Our work bridges this critical gap by bringing explainability, particularly Layer-wise Relevance Propagation (LRP), to the Mamba architecture. Guided by the axiom of relevance conservation, we identify specific components in the Mamba architecture, which cause unfaithful explanations. To remedy this issue, we propose MambaLRP, a novel algorithm within the LRP framework, which ensures a more stable and reliable relevance propagation through these components. Our proposed method is theoretically sound and excels in achieving state-of-the-art explanation performance across a diverse range of models and datasets. Moreover, MambaLRP facilitates a deeper inspection of Mamba architectures, uncovering various biases and evaluating their significance. It also enables the analysis of previous speculations regarding the long-range capabilities of Mamba models."},"title":{"value":"MambaLRP: Explaining Selective State Space Sequence Models"},"authors":{"value":["Farnoush Rezaei Jafari","Grégoire Montavon","Klaus-Robert Müller","Oliver Eberle"]}},"tmdate":1732616644939,"pdate":1704067200000,"tcdate":1727720030192,"writers":["~"],"signatures":["~Farnoush_Rezaei_Jafari1"],"forum":"N3yhQvJfsx","license":"CC BY-SA 4.0","number":122494,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1732616644939,"domain":"DBLP.org","id":"N3yhQvJfsx","version":2},{"content":{"summary":{"value":"Probabilistic time series imputation, capable of estimating imputation uncertainty, is widely used in real-world applications. Denoising diffusion probabilistic models excel in this task due to their ability to model complex distributions. However, current DDPM-based methods face challenges in achieving low time complexity for sequence modeling and effectively handling inter-variable and bidirectional dependencies. To overcome these, the authors integrate the efficient state space model Mamba as the backbone denoising module and design SSM-based blocks for bidirectional modeling and inter-variable relation understanding. Experimental results show that the approach achieves state-of-the-art time series imputation results across multiple datasets, missing scenarios, and missing ratios."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See W1-W9."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"S1. Time series imputation is an important and widely studied problem.\n\nS2. Diffusion model and Mamba are popular models recently.\n\nS3. Complexity analysis is provided for the model."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. As discussed in Section 1, the main contribution is proposing a diffusion-based framework for time series imputation, by integrating Mamba-based blocks into diffusion models. However, diffusion models have been proved to be effective in time series imputation, and Mamba is also an existing technique. Modeling bidirectional dependencies and channel dependencies are common techniques to capture characteristics of time series data. Therefore, the novelty and contributions are limited.\n\nW2. There is no theoretical result to support the proposed techniques.\n\nW3. It is unbelievable that the time cost is totally ignored in experiment results, since the main contribution claimed by authors is that the linear complexity of proposed models.\n\nW4. Only synthetic missing values are considered in experiments. It is suggested to use real-world incomplete datasets to demonstrate the effectiveness and efficiency of proposed methods.\n\nW5. 70%, 80%, 90% missing values are injected into the MuJoCo dataset, and 10%, 30%, 50% missing values are considered into the Electricity dataset. It is wondered why different missing rates are considered in different datasets, why not consider 10%, 30%, 50%, 70%, 80%, 90% for them consistently.\n\nW6. Since all the imputation baselines can impute incomplete time series, it is suggested to consider the comparison with them in all datasets, instead of using a subset of them in different datasets.\n\nW7. Scalability experiments are necessary to demonstrate the linear complexity of this paper. Therefore, it is suggested to conduct this on additional large real-world datasets.\n\nW8. There is no parameter sensitivity analysis in experiments, which is also suggested to show the stability of models and illustrate how to determine parameters.\n\nW9. Although the source code has been released, there is not guideline for how to use the codes and how to get the experimental results for each table and figure, leading to the doubts of reproducibility."}},"nonreaders":[],"tmdate":1731427590511,"tcdate":1730457664132,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2397/Reviewer_tnPK"],"signatures":["ICLR.cc/2025/Conference/Submission2397/Reviewer_tnPK"],"forum":"j1OucVFZMJ","number":3,"license":"CC BY 4.0","cdate":1730457664132,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2397/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427590511,"domain":"ICLR.cc/2025/Conference","replyto":"j1OucVFZMJ","id":"tHnZtjyvIq","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["diffusion","time series impuation","state space model"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Probabilistic time series imputation has been widely applied in real-world scenarios due to its ability to estimate uncertainty of imputation results. Meanwhile, denoising diffusion probabilistic models (DDPMs) have achieved great success in probabilistic time series imputation tasks with its power to model complex distributions. However, current DDPM-based probabilistic time series imputation methodologies are confronted with two types of challenges: 1) \\textit{ The backbone modules of the denoising parts are not capable of achieving sequence modeling with low time complexity.} 2) \\textit{ The architecture of denoising modules can not handle the inter-variable and bidirectional dependencies in the time series imputation problem effectively.} To address the first challenge, we integrate the computational efficient state space model, namely Mamba, as the backbone denosing module for DDPMs. To tackle the second challenge, we carefully devise several SSM-based blocks for bidirectional modeling and inter-variable relation understanding. Experimental results demonstrate that our approach can achieve state-of-the-art time series imputation results on multiple datasets, different missing scenarios and missing ratios."},"_bibtex":{"value":"@misc{\ngao2025diffimp,\ntitle={DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone},\nauthor={Hongfan Gao and Wangmeng Shen and Xiangfei Qiu and Ronghui Xu and Jilin Hu and Bin Yang},\nyear={2025},\nurl={https://openreview.net/forum?id=j1OucVFZMJ}\n}"},"title":{"value":"DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone"},"pdf":{"value":"/pdf/26faf0d7208528fa5a326d36fadf37ed808e0f38.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"gao|diffimp_efficient_diffusion_model_for_probabilistic_time_series_imputation_with_bidirectional_mamba_backbone"},"authorids":{"value":["~Hongfan_Gao1","~Wangmeng_Shen1","~Xiangfei_Qiu1","~Ronghui_Xu2","~Jilin_Hu1","~Bin_Yang4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hongfan Gao","Wangmeng Shen","Xiangfei Qiu","Ronghui Xu","Jilin Hu","Bin Yang"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Q-Mamba, a novel offline reinforcement learning framework for Meta-Black-Box Optimization (MetaBBO). The main idea is to leverage Mamba's long-sequence modeling advantage to represent the Q function in MetaBBO. To do this, they train the Q-Mamba offline, which they claim is more efficient than the online counterpart. The authors constructed a dataset to evaluate the performance of the proposed framework."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. The paper mentions a performance degradation of Q-Mamba in the online learning setting. Does this mean that offline data is always more effective than online data in MetaBBO?\n\n2. For those optimizers with continuous action spaces, how does Q-Mamba deal with the information loss caused by action discretization?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"The paper is clearly written. Several strengths of this work:\n\nThe offline learning strategy is shown to significantly improve training efficiency compared to online baselines, a major advantage in computationally expensive MetaBBO tasks.\n\nThe paper presents extensive benchmarking results demonstrating that Q-Mamba achieves competitive or superior performance compared to various online and offline baselines, including on unseen tasks (zero-shot generalization).\n\nAblation studies provide evidence supporting the effectiveness of the key design choices (decomposed Q-function, Mamba architecture, CQL regularization, E&E dataset)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper only briefly mentions the following two reasons for using Mamba in Section 4.2:\n\n●The sequence length of MetaBBO tasks involves thousands of decision steps. Transformer suffers from inefficiency and performance issues when dealing with very long sequences, while Mamba addresses this problem through data-dependent embedding and hardware-aware design.\n\n●\nMamba allows for the selective extraction of essential information and filtering out of irrelevant noise based on the input sequence, which effectively enhances sequence-to-sequence learning.\n\nHowever, the paper lacks a detailed analysis of the following:\n\n●\nHow Mamba improves the efficiency and performance of extremely long sequence modeling through data-dependent embedding and hardware-aware design.\n\n●\nThe specific performance improvements and efficiency advantages of Mamba in processing extremely long Q-function sequences compared to Transformer.\n\n●\nHow Mamba selectively extracts important information and filters out irrelevant noise based on the input sequence, and how this mechanism enhances sequence-to-sequence learning.\n\n2. The performance heavily relies on the quality and composition of the E&E dataset. Further investigation into the sensitivity to dataset variations would strengthen the claims.\n\n3. While zero-shot generalization is demonstrated, the extent of generalizability to significantly different optimization problems beyond those tested remains unclear. More diverse and challenging test scenarios are needed.\n\n4. Limitation on the Number of Action Bins: The number of action bins (M) used in Q-Mamba cannot be too large, which may be a limitation for some optimizers that require fine-grained control."}},"nonreaders":[],"tmdate":1731427533125,"tcdate":1730695496739,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2058/Reviewer_zkyX"],"signatures":["ICLR.cc/2025/Conference/Submission2058/Reviewer_zkyX"],"forum":"cSgEW7EZ9h","number":4,"license":"CC BY 4.0","cdate":1730695496739,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2058/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427533125,"domain":"ICLR.cc/2025/Conference","replyto":"cSgEW7EZ9h","id":"HIamWI9sNO","forumContent":{"TLDR":{"value":"A novel MetaBBO framework which achieves both effectiveness and efficiency for auto-configuring black-box optimizer."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Black-Box Optimization","Dynamic Algorithm Configuration","Learning to Optimize","Offline Reinforcement Learning","Mamba"]},"primary_area":{"value":"optimization"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent progress in Meta-Black-Box-Optimization (MetaBBO) has demonstrated that meta-training a neural network based meta-level control policy over an optimization task distribution could significantly enhance the optimization performance of the low-level black-box optimizers. However, achieving such performance enhancement requires effective policy optimization/search method to locate optimal control policy within a massive joint-action space. The online learning fashion of existing works further makes the efficiency of MetaBBO problematic. To address these technical challenges, we propose an offline learning framework in this paper, termed Q-Mamba. Concretely, our method uses a Mamba neural network architecture to meta-learn decomposed Q-functions for each configurable component in the low-level optimizer. By decomposing the Q-function of the configuration decisions of all components in an optimizer, we can apply effective sequence modelling to avoid searching the control policy in the massive joint-action space. Furthermore, by leveraging the long-sequence modelling advantage of Mamba and moderate offline trajectory samples, Q-Mamba can be efficiently trained through a synergy of offline Temporal-Difference update and Conservative Q-Learning regularization to achieve competitive performance against the online learning paradigms. Through extensive benchmarking, we observe that Q-Mamba achieves competitive or even superior optimization performance to prior online/offline learning baselines, while significantly improving the training efficiency of existing online learning baselines. Additional ablation studies show that each of the proposed key designs contributes to this good performance."},"_bibtex":{"value":"@misc{\nma2025metablackboxoptimization,\ntitle={Meta-Black-Box-Optimization through Offline Q-function Learning with Mamba Architecture},\nauthor={Zeyuan Ma and Hongshu Guo and Zhou Jiang and Zhiguang Cao and Yue-Jiao Gong},\nyear={2025},\nurl={https://openreview.net/forum?id=cSgEW7EZ9h}\n}"},"title":{"value":"Meta-Black-Box-Optimization through Offline Q-function Learning with Mamba Architecture"},"pdf":{"value":"/pdf/2f9052c209e77207cb87544eb40cb37b2a701261.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"ma|metablackboxoptimization_through_offline_qfunction_learning_with_mamba_architecture"},"authorids":{"value":["~Zeyuan_Ma1","~Hongshu_Guo1","~Zhou_Jiang2","~Zhiguang_Cao1","~Yue-Jiao_Gong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zeyuan Ma","Hongshu Guo","Zhou Jiang","Zhiguang Cao","Yue-Jiao Gong"]}},"version":2},{"content":{"summary":{"value":"This paper provides a diffusion-based imputation method for time series data with missing values. The proposed method integrates Mamba-based models and diffusion-based models to achieve linear complexity for sequence data. The method is evaluated on two real-world datasets, Electricity and MuJoco, and the results show that it outperforms other state-of-the-art methods in terms of accuracy and efficiency."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"For contents:\n\n1. the authors state the mask strategy is utilized for training and in all there are three kinds of mask strategies. It is not clear that how you applied the mask. e.g. if there are no missing values in the original dataset, how to construct the mask for training? and if there are missing values in the original dataset, how to construct the mask for training? Now it seems that the mask is applied similarly to all kinds of datasets.\n\n2. It seems that you designed two kinds of blocks for modeling channel dependency and sequential dependency, why you choose unidirectional channel modelling as shown in Fig 3c.\n\n3. As shown in Fig 3d, it seems not clear how the temporal attention is utilized for the bidirectional modelling, besides, as in Fig 3c, the CMB module has the same sigmoid module as Fig 3d, is it also the temporal attention module and why you applied temporal attention module for channel modeling?\n\nFor experiments\n\n1. Can the authors explain the choice of datasets and provide further experiments results and comparsions on AQI and healthcare dataset?\n\n2. As it is an efficient time series diffusion model, can the authors provide more details, including training memory details or model parameters?\n\nFor details\n\n4. as in table 1, the authors state that the SSM models is partial global dependency, why?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. This paper proposes an effective method for the time series data. The authors explores the availabity of using Mamba models as the diffusion backbones. The intergration of Mamba-based models and diffusion-based models provides linear complexity for sequence data and works fine on time series imputation tasks.\n2. The improvements of deterministic results on high missing ratios are significant and the CRPS-sum improvement is also very significant compared with the other baselines, which means the proprosed method is effective in .\n3. The running time of the proposed method is significantly lower than the other baselines and demonstrates the efficiency of the proposed method.\n4. The bidirectional modeling, temporal attention module and channel modeling module are well designed and can effectively capture the temporal and channel dependencies of the time series data.\n5. The authors provide detailed code for reproducible experiments."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. In your paper, the experiments are conducted on the Electricity and MuJoco dataset. And as in [1], the experiments are conducted on AQI and healthcare dataset. The reason of choosing these datasets seems not clear.\n\n2. for efficiency, the authors only compare the time complexity of the proposed method with the state-of-the-art methods.\n\n3. The training manner of building missing values may need further expalaination\n\n[1] CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation, Tashiro et al(2021)"}},"nonreaders":[],"tmdate":1731427590636,"tcdate":1730355966004,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2397/Reviewer_Aeb7"],"signatures":["ICLR.cc/2025/Conference/Submission2397/Reviewer_Aeb7"],"forum":"j1OucVFZMJ","number":2,"license":"CC BY 4.0","cdate":1730355966004,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2397/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427590636,"domain":"ICLR.cc/2025/Conference","replyto":"j1OucVFZMJ","id":"cz2erD6VoZ","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["diffusion","time series impuation","state space model"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Probabilistic time series imputation has been widely applied in real-world scenarios due to its ability to estimate uncertainty of imputation results. Meanwhile, denoising diffusion probabilistic models (DDPMs) have achieved great success in probabilistic time series imputation tasks with its power to model complex distributions. However, current DDPM-based probabilistic time series imputation methodologies are confronted with two types of challenges: 1) \\textit{ The backbone modules of the denoising parts are not capable of achieving sequence modeling with low time complexity.} 2) \\textit{ The architecture of denoising modules can not handle the inter-variable and bidirectional dependencies in the time series imputation problem effectively.} To address the first challenge, we integrate the computational efficient state space model, namely Mamba, as the backbone denosing module for DDPMs. To tackle the second challenge, we carefully devise several SSM-based blocks for bidirectional modeling and inter-variable relation understanding. Experimental results demonstrate that our approach can achieve state-of-the-art time series imputation results on multiple datasets, different missing scenarios and missing ratios."},"_bibtex":{"value":"@misc{\ngao2025diffimp,\ntitle={DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone},\nauthor={Hongfan Gao and Wangmeng Shen and Xiangfei Qiu and Ronghui Xu and Jilin Hu and Bin Yang},\nyear={2025},\nurl={https://openreview.net/forum?id=j1OucVFZMJ}\n}"},"title":{"value":"DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone"},"pdf":{"value":"/pdf/26faf0d7208528fa5a326d36fadf37ed808e0f38.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"gao|diffimp_efficient_diffusion_model_for_probabilistic_time_series_imputation_with_bidirectional_mamba_backbone"},"authorids":{"value":["~Hongfan_Gao1","~Wangmeng_Shen1","~Xiangfei_Qiu1","~Ronghui_Xu2","~Jilin_Hu1","~Bin_Yang4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hongfan Gao","Wangmeng Shen","Xiangfei Qiu","Ronghui Xu","Jilin Hu","Bin Yang"]}},"version":2},{"content":{"summary":{"value":"The paper proposes training video understanding models directly on compressed video byte streams instead of decoded RGB frames. It introduces a multilevel SSM to shorten sequences, and a sequence-parallel state correction scheme that enables training on very long inputs. The method achieves competitive results on multiple benchmarks.\n\nThe combination of compressed video input and state-space modeling is a promising design for scaling to long video sequences. However, the model design remains insufficiently developed, lacking clear mechanisms tailored to compressed data."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1, In line 240, which parameter is $h_0$ in equation 3? Would the correction still be valid when C is input dependent? How does the correction handle multilevel SSM with pooling?\n\n2, How does the model encode the byte input? How does it handle I, B and P frame differently?\n\n3, What is the model performance on long video benchmarks, such as LVBench, MLVU, VUE-TR or Neptune? (Line 73-74)"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1, Clear motivation. The paper is well motivated. While most VLMs struggle to scale to long videos and rely on compressing image tokens from raw pixels, this work instead processes compressed video streams directly, leveraging mature video compression techniques to achieve efficient and scalable long-sequence modeling.\n\n2, Enabling parallelism for SSMs. The model proposes a technique for parallelism for SSMs. The sequence-parallel + state-correction formulation (with explicit update equations) is effective and lets SSMs scale to 15M tokens in training.\n\n3, Competitive results. The paper evaluates the performance in action recognition and  VQA benchmarks  and achieves competitive performance with strong efficiency scaling, including results on Kinetics-400/600 and long-video QA tasks (e.g., 57.1% on ActivityNet-QA with an LLM head)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1, Insufficient Related work. The Related Works section lacks prior research on video compression techniques (e.g., H.264, DCT, motion vectors), state-space models like S4 and Mamba, and compressed-domain learning from JPEG or H.264 inputs. Including these would better contextualize the proposed method and clarify its novelty.\n\n2, Video Byte-based model design underexplored. The model design shows little difference from a standard image-input long-sequence model and lacks specific adaptations for encoded video byte inputs. While the paper focuses on the SSM architecture and long-sequence processing, it omits critical details on handling compressed video data, such as how temporal compression, quantization, and I/P/B frame dependencies are managed—especially since B-frames require both past and future frames while SSMs operate only on past states. It also remains unclear how performance would compare if the model used image-byte sequence as inputs instead of video bytes.\n\n3, Parallelism assumptions vs. selective SSMs. The state-correction derivation (Eq. 1–4) is presented in linear SSM form; since Mamba is input-selective, it would help to clarify when/why these corrections remain valid and how approximation error behaves across chunks.\n\n\nMinor: \n\n1, In abstract, the abbreviation of SSM is used without introduction.\n\n2, Texts in the figures are too small and hard to read."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916043423,"tcdate":1761873070044,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2135/Reviewer_jVJm"],"signatures":["ICLR.cc/2026/Conference/Submission2135/Reviewer_jVJm"],"forum":"psLcyKiuCp","number":4,"license":"CC BY 4.0","cdate":1761873070044,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2135/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916043423,"domain":"ICLR.cc/2026/Conference","replyto":"psLcyKiuCp","id":"spl56sqGsd","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["video understanding"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"We present an approach to understand video from encoded bytes, e.g., mp4s. These compressed videos are 99\\% smaller than the RGB pixel representations which are currently commonly used for video understanding. Encoded videos are able to compress the pixels by taking advantage of the redundant information across the frames using special encoding, such as key frames and motion residuals to handle this. However, standard video understanding models do not take advantage of this significant compression already available for each video, and instead either heavily subsample the frames or only work on short segments of the video. Here, we present an approach to understanding video from encoded bytes directly. We note that simply applying existing models, e.g., Transformers or State-Space models, to video byte sequences does not work, both due to difficulty in handling very long video byte sequences and easy overfitting. To address these challenges, we design a State-Space model with sequence parallelism to handle very long byte sequences, reaching 15 Million tokens in training, and essentially unlimited tokens in inference. We also propose a multilevel SSM activation fusion that reduces sequence length, which we find also benefits video understanding. We evaluate on common video understanding and natural extension to video + audio understanding tasks and demonstrate competitive performance, illustrating, for the first time, the feasibility of learning from compressed video byte representations."},"_bibtex":{"value":"@misc{\npiergiovanni2025learning,\ntitle={Learning from Encoded Video Bytes},\nauthor={AJ Piergiovanni and Ganesh Satish Mallya and Dahun Kim and Anelia Angelova},\nyear={2025},\nurl={https://openreview.net/forum?id=psLcyKiuCp}\n}"},"title":{"value":"Learning from Encoded Video Bytes"},"pdf":{"value":"/pdf/2ff5bbde8303355e1416ad1c6c4e027210586f69.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"piergiovanni|learning_from_encoded_video_bytes"},"authorids":{"value":["~AJ_Piergiovanni1","~Ganesh_Satish_Mallya1","~Dahun_Kim1","~Anelia_Angelova1"]},"authors":{"value":["AJ Piergiovanni","Ganesh Satish Mallya","Dahun Kim","Anelia Angelova"]}},"version":2},{"content":{"summary":{"value":"Mamba-3 is a state-space model (SSM) designed from an inference-first perspective that addresses the limitations of existing sub-quadratic sequence models (e.g., Mamba-2, Gated DeltaNet) in quality, state-tracking capability, and hardware efficiency. It achieves SOTA performance across language modeling, state-tracking, and retrieval tasks while maintaining linear compute and constant memory, setting a new Pareto frontier for performance-efficiency tradeoffs."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. Complex-Valued SSM & RoPE Equivalence: The paper argues that complex-valued SSMs are equivalent to data-dependent RoPE embeddings and compute-efficient. Could you provide a direct comparison between complex-valued SSMs and *explicitly applying RoPE* (e.g., on B/C projections of Mamba-2) in terms of: (1) training/inference speed, (2) memory usage, and (3) state-tracking performance? \n\n2. MIMO Rank Selection: The MIMO variant uses rank \\( r=4 \\) for 440M models, but there is no analysis of how \\( r \\) impacts performance-efficiency tradeoffs. What is the marginal gain of increasing \\( r \\) (e.g., \\( r=8, 16 \\))? At what point does \\( r \\) become compute-bound (negating efficiency gains)? Additionally, why was \\( r=4 \\) chosen as the default, was it optimized via grid search or heuristic?\n\n\n\n\n3. Ablations. The BC bias synergizes with trapezoidal discretization, but you do not specify its initialization strategy (e.g., zero, Xavier, data-dependent) or whether it is updated during training. Does the bias’s initialization significantly affect performance?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The three core improvements, trapezoidal discretization, complex-valued SSMs, and MIMO formulation, are not isolated tweaks but creative combinations of classical SSM theory and modern LLM needs. Trapezoidal discretization generalizes Euler’s rule (used in Mamba-2) to a second-order accurate recurrence, while the complex-valued SSM recovers rotational dynamics absent in real-valued counterparts.\n\n2. Rigorous Theory & Comprehensive Empirical Validation. The paper includes rigorous proofs for key propositions (e.g., complex-to-real SSM equivalence, trapezoidal discretization error bounds) and formalizes the mathematical underpinnings of each innovation. This theoretical rigor ensures the model’s behavior is well-understood, not just empirically effective.\n\n3. The paper reports detailed experimental settings (training data, tokenization, hyperparameters) and provides clear metrics (perplexity, accuracy, TPS, arithmetic intensity) that enable follow-up work.\nAnd structured presentation & accessible exposition increses the avaliability. \n\n4. Addressing Critical LLM Deployment Pain Points. By pushing the Pareto frontier of performance vs. inference efficiency, Mamba-3 directly addresses a key bottleneck in LLM deployment—scaling inference without prohibitive memory/compute costs. The MIMO variant, in particular, improves hardware utilization without increasing state size, making it viable for real-world applications."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Inherent limitations of retrieval capabilities and insufficient comparison. Mamba-3 significantly lags behind Transformer in information extraction tasks from semi-structured/unstructured data, and the root causes and potential solutions are not explored in depth.\n\nTable 2 of the paper shows that Mamba-3 with 1.5B parameters has poor accuracy in real-world retrieval tasks such as SWDE and FDA; even in the synthetic \"needle-in-a-haystack\" task, when the context length exceeds 2048, the accuracy of Mamba-3 is still far below the theoretical upper limit of Transformer, and it is not compared with linear models specifically optimized for retrieval (such as Arora et al., 2025b's \"Just Read Twice\").\n\n2. Limited experimental scale of the key variant (MIMO) and incomplete hyperparameter ablation. The MIMO variant was only trained on a 440M parameter model (paper 4.2.3). Because the \"computational constraint\" was not extended to a 1.5B scale, its performance-efficiency tradeoff under larger models could not be verified. Furthermore, the MIMO rank r was fixed at 4, and the impact of different values ​​such as r=2 and 8 on arithmetic strength and inference speed was not tested (Figure 2 only shows the theoretical trend of r, without experimental data).\n\n3. The paper's baseline only includes Mamba-2, Gated DeltaNet, and the standard Transformer, omitting similar linear models such as Rwkv-7, DeltaProduct, and Block-Biased Mamba. These models all innovated on the expressiveness or efficiency of linear models, and some performed exceptionally well on state tracking tasks."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924147371,"tcdate":1761957175321,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13549/Reviewer_h5BJ"],"signatures":["ICLR.cc/2026/Conference/Submission13549/Reviewer_h5BJ"],"forum":"HwCvaJOiCj","number":2,"license":"CC BY 4.0","cdate":1761957175321,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13549/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924147371,"domain":"ICLR.cc/2026/Conference","replyto":"HwCvaJOiCj","id":"gUkyzxZCN2","forumContent":{"venue":{"value":"ICLR 2026 Oral"},"TLDR":{"value":"Mamba-3, an inference-first SSM that pushes on core SSM principles: improved discretization for better quality, complex dynamics for new capabilities, and MIMO updates for efficient inference."},"keywords":{"value":["State Space Models","Mamba","LLMs","Subquadratic Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While current Transformer models deliver strong quality, their quadratic compute and linear memory requirements make inference expensive. This has spurred the development of sub-quadratic models with reduced compute and constant memory requirements.\nHowever, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice.\nGuided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule enabling richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation that improves model performance without increasing decode latency.\nTogether with architectural refinements, Mamba-3 achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with the MIMO variant further improving accuracy by an additional 1.2 points, for a total gain of 1.8 points.\nAcross state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half the state size. These results demonstrate that Mamba-3 advances the performance–efficiency frontier."},"_bibtex":{"value":"@inproceedings{\nlahoti2026mamba,\ntitle={Mamba-3: Improved Sequence Modeling using State Space Principles},\nauthor={Aakash Lahoti and Kevin Li and Berlin Chen and Caitlin Wang and Aviv Bick and J Zico Kolter and Tri Dao and Albert Gu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=HwCvaJOiCj}\n}"},"title":{"value":"Mamba-3: Improved Sequence Modeling using State Space Principles"},"pdf":{"value":"/pdf/09c67b76b7d1ad254ea30eeba839b8592de9bd78.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lahoti|mamba3_improved_sequence_modeling_using_state_space_principles"},"authorids":{"value":["~Aakash_Lahoti1","~Kevin_Li3","~Berlin_Chen1","~Caitlin_Wang1","~Aviv_Bick1","~J_Zico_Kolter1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Aakash Lahoti","Kevin Li","Berlin Chen","Caitlin Wang","Aviv Bick","J Zico Kolter","Tri Dao","Albert Gu"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Spatial-Mamba, a novel state space model (SSM) designed for visual tasks, capable of effectively capturing spatial dependencies in images to enhance contextual modeling. Experimental results demonstrate that Spatial-Mamba outperforms existing SSMs and other mainstream models in areas such as image classification and semantic segmentation."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Could you further explain how state variables are modeled in the state space to design the structure-aware state fusion?\n2. Please provide details on the number of hidden states in each SSM layer and include relevant ablation experiments.\n3. Is Spatial-Mamba implemented using the hardware scanning algorithm proposed by Mamba?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. Improved upon existing SSMs by directly integrating spatial neighborhood dependencies into the state transition process.\n\n2. Analyzed the connections between Spatial-Mamba, Mamba, and linear attention, unifying them under a matrix multiplication framework."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper does not mention the number of hidden states used in the SSM, and the ablation study lacks an analysis of the impact of different numbers of hidden states.\n\n2. While the authors analyze the relationship between Spatial-Mamba, Mamba, and linear attention, they do not clarify the framework used to implement Spatial-Mamba, such as Mamba's hardware scanning algorithm or Mamba2's SSD framework.\n\n3. In Mamba’s proposed SSM, hidden states do not inherently have spatial relationships. I suspect the authors intend to model the hidden states as 2D local observation points?"}},"nonreaders":[],"tmdate":1731427635127,"tcdate":1729944436641,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2673/Reviewer_P6PW"],"signatures":["ICLR.cc/2025/Conference/Submission2673/Reviewer_P6PW"],"forum":"iDe1mtxqK5","number":1,"license":"CC BY 4.0","cdate":1729944436641,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2673/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427635127,"domain":"ICLR.cc/2025/Conference","replyto":"iDe1mtxqK5","id":"IbCKuPd1XA","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Representation learning","Visual state space models","Structure-aware state fusion"]},"supplementary_material":{"value":"/attachment/bfe4420ea6a429fc96daffcd3ec6b05d08ace381.pdf"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSMs), such as Mamba, highly excel at capturing long-range dependencies in 1D sequential data, while their applications to 2D vision tasks still face challenges. Current visual SSMs often convert images into 1D sequences and employ various scanning patterns to incorporate local spatial dependencies. However, these methods are limited in effectively capturing the complex image spatial structures and the increased computational cost caused by the lengthened scanning paths. To address these limitations, we propose Spatial-Mamba, a novel approach that establishes neighborhood connectivity directly in the state space. Instead of relying solely on sequential state transitions, we introduce a structure-aware state fusion equation, which leverages dilated convolutions to capture image spatial structural dependencies, significantly enhancing the flow of visual contextual information. Spatial-Mamba proceeds in three stages: initial state computation in a unidirectional scan, spatial context acquisition through structure-aware state fusion, and final state computation using the observation equation. Our theoretical analysis shows that Spatial-Mamba unifies the original Mamba and linear attention under the same matrix multiplication framework, providing a deeper understanding of our method. Experimental results demonstrate that Spatial-Mamba, even with a single scan, attains or surpasses the state-of-the-art SSM-based models in image classification, detection and segmentation. Source codes and trained models can be found at \\url{ https://github.com/EdwardChasel/Spatial-Mamba }."},"_bibtex":{"value":"@inproceedings{\nxiao2025spatialmamba,\ntitle={Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion},\nauthor={Chaodong Xiao and Minghan Li and Zhengqiang ZHANG and Deyu Meng and Lei Zhang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=iDe1mtxqK5}\n}"},"title":{"value":"Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion"},"pdf":{"value":"/pdf/8219b6919d5e23d90a43ddcdec43ba242eac39c4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"xiao|spatialmamba_effective_visual_state_space_models_via_structureaware_state_fusion"},"authorids":{"value":["~Chaodong_Xiao2","~Minghan_Li1","~Zhengqiang_ZHANG1","~Deyu_Meng1","~Lei_Zhang2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chaodong Xiao","Minghan Li","Zhengqiang ZHANG","Deyu Meng","Lei Zhang"]}},"version":2},{"content":{"summary":{"value":"The paper proposes Tramba, a traffic forecasting framework that combines a per-link Mamba backbone for efficient long-range temporal modeling with an adaptive attention branch that retrieves information from non-adjacent road segments under learnable time shifts. A gating module fuses the temporal (Mamba) and non-local (attention) signals to generate multi-step predictions. Evaluations on standard benchmarks (e.g., METR-LA, PEMS-BAY, TOPIS) show that Tramba delivers consistent improvements—especially at medium-to-long horizons—over strong Transformer- and Mamba-based baselines. Ablations indicate that both the adaptive attention and the gating mechanism are critical to the gains, and qualitative analyses visualize how the model prioritizes time-aligned yet spatially distant segments."},"soundness":{"value":3},"confidence":{"value":2},"questions":{"value":"See Weakness."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- Targets a realistic pain point in road networks—non-local dependencies with temporal misalignment—and addresses it by pairing Mamba’s long-sequence modeling with shift-aware attention.\n- Temporal dynamics are handled by the per-link Mamba backbone, while the adaptive attention captures dynamic, non-adjacent relations; a simple learnable gate provides robust fusion.\n- Demonstrates stable gains at longer horizons across multiple datasets and steps, suggesting better robustness for challenging prediction ranges.\n- Removing or altering the adaptive attention, similarity function, or gating leads to notable degradation, clearly attributing improvements to the proposed components."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Please provide a fine-grained computational/parametric complexity breakdown for each module (Mamba branch, adaptive attention over (segments × shifts), and gating), along with GPU memory usage curves under large sequence lengths **T** and large numbers of segments **L** . The article only provides training time, which does not fully reflect efficiency.\n\n- Please add representative GNN models (e.g., GCN/GAT/temporal-graph variants) or explicitly justify their exclusion, and clarify how Tramba’s shift-aware non-local attention differs from GNN-style spatiotemporal modeling in terms of inductive bias, expressivity, and computational trade-offs.\n\n- To strengthen the causal attribution of gains, please replace the Mamba backbone with a Transformer backbone (keeping the rest of the architecture intact) and report the performance and efficiency deltas. This would disentangle the contribution of the proposed adaptive attention/gating from the choice of temporal encoder."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926511817,"tcdate":1761662250670,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16390/Reviewer_NG6K"],"signatures":["ICLR.cc/2026/Conference/Submission16390/Reviewer_NG6K"],"forum":"DjlVwQFRMb","number":2,"license":"CC BY 4.0","cdate":1761662250670,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16390/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926511817,"domain":"ICLR.cc/2026/Conference","replyto":"DjlVwQFRMb","id":"ZVH9PEqrLG","forumContent":{"TLDR":{"value":"We propose Tramba, a novel time-series forecasting framework that integrates a selective state-space model with an adaptive attention mechanism for enhanced predictive accuracy and dynamic spatial reasoning in complex urban networks."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["traffic speed forecasting","adaptive attention","selective state-space model","time series"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"We introduce \\textbf{Tramba}, a novel deep learning model for traffic speed forecasting in complex urban road networks. Unlike conventional methods that rely heavily on short-term trends or local spatial proximity (e.g., upstream and downstream links), Tramba captures dynamic, long-range dependencies across both time and space. It does so by integrating two key components: a Mamba-based temporal encoder that models long-term historical patterns of the target link, and an adaptive attention mechanism that learns temporally similar patterns from non-adjacent road links across the network. We evaluate Tramba on a real-world dataset from Gangnam-gu, Seoul, comprising 5-minute interval speed measurements across 366 road segments. Tramba is tested over forecasting horizons from 1 to 36 steps and compared with six strong baselines. It consistently outperforms all alternatives, achieving an average MAPE of 10.17\\%, MAE of 2.80~km/h, and MSE of 20.50~(km/h)$^2$ on the three datasets for 12-step forecasting. These results highlight Tramba’s ability to model long-range dependencies and detect non-local influences in complex urban networks, reducing prediction lag and improving robustness in dynamic traffic conditions. Code is available at~\\url{https://github.com/tr-anon-users/tramba-code}."},"_bibtex":{"value":"@misc{\nlee2026tramba,\ntitle={Tramba: Mamba with Adaptive Attention for Traffic Speed Forecasting},\nauthor={Eun Hak Lee and Hyunsoo Yun and Zhengzhong Tu},\nyear={2026},\nurl={https://openreview.net/forum?id=DjlVwQFRMb}\n}"},"title":{"value":"Tramba: Mamba with Adaptive Attention for Traffic Speed Forecasting"},"pdf":{"value":"/pdf/3689872be3bd2e79776d2739c2bd1d9e2b46dfa5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"lee|tramba_mamba_with_adaptive_attention_for_traffic_speed_forecasting"},"authorids":{"value":["~Eun_Hak_Lee1","~Hyunsoo_Yun1","~Zhengzhong_Tu1"]},"authors":{"value":["Eun Hak Lee","Hyunsoo Yun","Zhengzhong Tu"]}},"version":2},{"content":{"summary":{"value":"This paper studies the Associative Recall (AR) capabilities of Mamba models (a Selective State Space Model, SSM) from a mechanistic interpretability perspective. The main contributions of this paper includes:\n\n1. Reverse-engineer an underlying structure that enables Mamba to solve MQAR (Multi-Query Associative Recall). The peper first identifies a perfect non-compressive recall circuit with large state and model dimensions (Theorem 1), then relax the requirement for state and model dimensions through claiming a structure that solve MQAR with high probability (Theorem 2). The construction is verified through experiments. The paper further interprets Mamba’s recall as a linear similarity-preserving hash function, linking it to the Johnson–Lindenstrauss (JL) lemma.\n\n2. Develop analytic AR scaling laws characterizing the success rate of a single-layer linear Mamba on solving the MQAR task, as a function of the state and model dimensions (N, D), vocabulary size V and number of facts $N_{\\text{facts}}$ (Theorem 3). Further it derives the dimensions (N, D) required for solving an MQAR task with high probability, which are in the logarithm order of V. These theoretical results are validated empirically through experiments demonstrating matching scaling behavior."},"soundness":{"value":3},"confidence":{"value":2},"questions":{"value":"1. Please refer to the weakness section.\n\n2. In equation (2) (line 133), I assume $h_t, \\hat{x}_t$ are vectors and $\\bar{A_t}, \\bar{B}_t$ are discretized matrices. How did element-wise multiplication applied here?\n\n3. In the scaling law experiments, the authors report the maximum accuracy across trials. Could the authors clarify this choice? Intuitively, averaging results over multiple runs (or mean ± standard deviation) would provide a more robust and statistically meaningful comparison.\n\n4. In this paper, all theoretical results are based on single-layer linear model. It would be valuable to discuss how these results provide insight into multi-layer SSM architectures, or to clarify what key challenges prevent extending the analysis to deeper Mamba models."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"This paper theoretically elucidates how the Mamba model performs associative recall, combining mechanism analysis, probabilistic modeling, and empirical verification.\n\n$\\textbf{Originality}$: The work introduces a novel mechanistic reconstruction of Mamba’s recall circuitry and interprets recall as a linear similarity-preserving hash, grounded in the Johnson–Lindenstrauss lemma. This creates an elegant conceptual link between state-space modeling and random projection theory.\n\n$\\textbf{Quality}$: The theoretical derivations are supported by empirical results. The accompanying simulations closely align with the analytical predictions, reinforcing the validity of the proposed framework.\n\n$\\textbf{Significance}$: This study offers the first theoretical scaling law for associative recall in Mamba models, offering quantitative insight into memory capacity and architectural trade-offs. These contributions advance theoretical understandings on the capability of selective SSMs."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"As SSMs are not my primary area of expertise, my comments focus on readability and clarity. My main concern is that the paper is difficult to follow, even for readers with some familiarity with state-space models. To make the work more accessible to a broader audience, I encourage the authors to provide sufficient details when introducing new concepts or notations. Specifically, the following points could be clarified:\n\n1. It would be beneficial to elaborate the example of MQAR (lines 094-100). What are keys, values, and queries in this context exactly?\n\n2. What are the dimensions of each Mamba components ($\\bar{A}_t,\\bar{B}_t,C_t, \\hat{x}_t, h_t, \\hat{y}_t)$? Generally, when a matrix is first introduced, please specify its dimension. \n\n3. In Figure 1, what is $\\hat{0}$? Also what is the circle product notation? If it is element-wise product, please be consistent with the $\\odot$ notation. Please include sufficient details in the caption to explain how the circuit works.\n\n4. In Theorem 1 and Theorem 2, please be specific what does $\\textsf{expand}$ refer to.\n\n5. If there are any typos in equations (5) and (7), please correct it.\n\n6. In the definition of $\\hat{E}_{in}$ (line 231), what are $W_0, W_1$?\n\n7. In Figure 2, please explain what the axis labels t and $\\tau$ refer to.\n\n8. There appear to be inconsistencies in notation, e.g., $G_{kq}$ and $G_{qk}$ (in page 5), $N_{facts}$ and $N_f$ (in page 7).\n\n9. In Figure 6,  it would be helpful to add a colorbar to indicate the numerical values represented by each color.\n\nOverall, defining all variables clearly, avoiding notation inconsistency, and providing brief explanations when introducing new symbols or concepts would greatly enhance the paper’s readability and accessibility."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762920002822,"tcdate":1762658517067,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8000/Reviewer_dQhk"],"signatures":["ICLR.cc/2026/Conference/Submission8000/Reviewer_dQhk"],"forum":"8cDoHzqDXP","number":3,"license":"CC BY 4.0","cdate":1762658517067,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8000/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762920002822,"domain":"ICLR.cc/2026/Conference","replyto":"8cDoHzqDXP","id":"DXR68WiAi9","forumContent":{"TLDR":{"value":"Using mechanistic interpretability, we show Mamba recalls via hash-like mechanisms and derive scaling laws that predict its recall capacity."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mamba","Memory","Associative Recall","Scaling Laws"]},"supplementary_material":{"value":"/attachment/889fd173211134007994b02141c1466e69caa77b.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Associative Recall (AR) is the cognitive ability to learn and retrieve links between items in memory. In NLP, AR is used as a benchmark for evaluating the in-context memory capacity of architectures such as Mamba, and has been found to strongly correlate with language modeling performance. This paper explores AR from the perspective of mechanistic interpretability, aiming to reverse-engineer the exact internal algorithm used by Mamba to perform recall. Our key insight is that Mamba performs recall by implicitly learning linear hash functions, and we identify the low-level circuit that enables this behavior. Building on these findings and inspired by theoretical tools in similarity-preserving hashing, such as the Johnson–Lindenstrauss (JL) lemma, we develop a theoretical framework for analyzing AR, which we term recall scaling laws. For example, given a model’s state, context length, and embedding dimensions, our theory predicts a lower bound on the largest vocabulary size that can be perfectly recalled in the AR task. Empirical results show that this bound is tight and predictive, offering insights into how AR capacity scales with vocabulary size, state size, embedding size, and model architecture."},"_bibtex":{"value":"@misc{\nkoren2026on,\ntitle={On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing},\nauthor={Yuval Koren and Itamar Zimerman and Assaf Ben-Kish and Raja Giryes and Lior Wolf},\nyear={2026},\nurl={https://openreview.net/forum?id=8cDoHzqDXP}\n}"},"title":{"value":"On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing"},"pdf":{"value":"/pdf/ef768fe2351b73310b384851405050a888cb6835.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"koren|on_the_recall_scaling_laws_in_mamba_a_theoretical_and_mechanistic_study_via_hashing"},"authorids":{"value":["~Yuval_Koren1","~Itamar_Zimerman1","~Assaf_Ben-Kish1","~Raja_Giryes1","~Lior_Wolf1"]},"authors":{"value":["Yuval Koren","Itamar Zimerman","Assaf Ben-Kish","Raja Giryes","Lior Wolf"]}},"version":2},{"content":{"summary":{"value":"This paper addresses problems with copying (and more generally recall) in state-space models by proposing a novel \"mimetic initialization\" scheme based on the following two observations:\n\n1. **State-space models as a generalization of linear attention**: The trainability of the $A_i$ matrices in state-space models allows them to act as a generalization of linear attention. However, this also can give rise to time decay that is too aggressive to learn copying tasks.\n2. **Preserving correlation structure between input tokens**: By initializing the product of the $B$ and $C$ projection matrices to be the identity, the authors aim to maintain the correlation structure between input tokens.\n\nBy leveraging these two observations to construct an initialization scheme, the authors demonstrate that the resulting mimetic Mamba 1 and Mamba 2 architectures outperform the default Mamba 1 and Mamba 2 architectures on both copying and MQAR tasks. The authors extend the results across multiple task and model hyperparameters and conclude by demonstrating how mimetic initialization can be interpreted as an efficient alternative to model pretraining."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Given the weaknesses listed above, I will recommend that this paper be rejected. While the proposed initializer is simple, intuitive, and seems effective, it lacks novelty, and the downstream analyses lack breadth. Furthermore, there is little theoretical justification in this paper (most of which appears to be drawn from prior work). This is acceptable only if a varied empirical analysis compensates for it, which is currently lacking. I think this paper would benefit significantly from another iteration. In particular, I would like to see the following addressed:\n\n1. Explain why the line of work on orthogonal RNNs is not relevant here. If it is, please incorporate it into the paper. In particular, a more extensive related work section in the appendix would add substantial value.\n\n2. Figure 2 is confusing. If the dots correspond to 10 random seeds, why not present the plot as an average with error bars? A more thorough multidimensional display could better reveal the meaningful variations across the ablated dimensions discussed later in the text.\n\n3. The legend in Figure 5 is obscuring half of the plot.\n\n4. Producing results that demonstrate utility beyond synthetic tasks would significantly improve the paper.\n\n5. The authors note that Mamba outperforms linear attention on synthetic tasks but do not offer any intuition for this. Additionally, are the authors using the short convolution from the Mamba block in linear attention? If not, how is this a fair comparison? Please clarify\n\n6. Much of the paper builds on linear attention, as this architecture is intended to serve as a prior in the mimetic initializer. If only one other SSM is to be benchmarked against, why choose Mamba? Isn’t GLA (or even S6) a more appropriate architecture for comparison? Mamba’s discretization step and short convolution are acting as unnecessary confounding factors. More generally, this initializer would be stronger if its effectiveness was demonstrated across a broader range of SSMs rather than just Mamba. Even so, could the authors at least please justify their choice of Mamba as the primary comparison model?\n\n7. To what extent do the models deviate from the initialization prior after training?\n\n8. In which regimes is the initializer most effective? Is it with many tokens to copy, long sequence lengths, etc.? Some plots potentially contain these results, but are hard to parse and there is no discussion in the paper.\n\n9. Can the authors clarify the purpose of the MQAR analysis? Is it intended as a supplemental synthetic analysis, or does it show something distinct from copying? The paper’s claims primarily focus on copying, with MQAR briefly mentioned at the end."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The main strengths of this paper lie in the simplicity of the proposed initializer and the robustness of the empirical results. In particular, I note the following:\n\n1. The empirical findings are strong. Specifically, the mimetic Mamba models achieve significantly higher performance on both copying and recall tasks compared to the default Mamba models. They also reach this performance at a much faster rate.\n\n2. The empirical evaluation is thorough. The authors conduct extensive sweeps across various vocabulary sizes, state sizes, and sequence lengths, demonstrating that mimetic Mamba consistently outperforms default Mamba across these settings.\n\n3. The authors offer insight into how mimetic Mamba functions as a pretraining prior, providing intuition for why the initializer is effective.\n\n4. The work appears reproducible based on the statement provided by the authors, although code has not yet been released."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The weaknesses in this paper primarily lie in lack of presentation, novelty, breadth, and clarity. In particular, I note the following:\n\n1. The paper is not particularly well-written. It seems rushed, and while the overall story makes sense, it lacks the logical flow both within and between sections that would make it easier for readers to follow. Some examples of this include the related works which would benefit from bolded paragraph headers like the authors use throughout the paper, rewriting the \"Correlated tokens should attend to each other\" section which is currently quite confusing to parse, moving the figures so they are actually near the text the author is reading, and more generally adding transitions between the sections that motivate why we need to do the next analysis so the story flows better. Also, I would consider moving the pretraining section after the first set of results since it provides nice intuition as to why the initializer works. \n\n2. The idea is not particularly novel compared to \"Mimetic Initialization of Self-Attention Layers\" (Trockman 2023), which is cited throughout the paper. Specifically, the authors adopt the concept of correlating the query and key projection matrices to maintain correlation structure amongst tokens from this prior work and simply apply it to the analogous components of Mamba.\n\n3. The authors’ use of an identity prior on $A$ aligns with the intuition of orthogonal/unitary non-linear RNNs, a related line of work that is entirely ignored in the paper. Although the focus here is on linear systems, it seems incomplete not to mention this prior work. From my understanding, the approach presented in this work effectively reduces the rate at which the gradient vanishes over time, which is a longstanding idea in RNN research.\n\n4. The authors restrict their results to synthetic tasks and do not demonstrate the applicability of this initializer (or any intuition derived from it) on language tasks, limiting the scope of the findings. Maybe something like the Pile is appropriate here?\n\n5. The authors do not analyze other SSMs outside of Mamba (e.g., GLA, RWKV, Hyena, etc.), despite claiming that this initializer improves performance in SSMs more broadly. Intuitively, mimetic initialization should extend to something like GLA which has a distinct functional form from Mamba in the state-to-state dynamics. This would be nice to show. \n\n6. The authors benchmark this initializer only against the default Mamba initializer, even though the original Mamba paper presents other initializers that could have been included for comparison (these could be included in supplementary materials)."}},"nonreaders":[],"tmdate":1731428295449,"tcdate":1730243899359,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11529/Reviewer_syxM"],"signatures":["ICLR.cc/2025/Conference/Submission11529/Reviewer_syxM"],"forum":"iVy7aRMb0K","number":1,"license":"CC BY 4.0","cdate":1730243899359,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11529/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428295449,"domain":"ICLR.cc/2025/Conference","replyto":"iVy7aRMb0K","id":"bbkyHfHder","forumContent":{"TLDR":{"value":"Mamba learns to recall more easily when we initialize it to be more like linear attention"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","SSM","sequence models","language models","initialization","linear attention"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with respect to their input sequence length. But in practice, state space models have fairly large state sizes, and we conjecture that they should be able to perform much better at these tasks than previously reported. We investigate whether their poor copying and recall performance could be due in part to training difficulties rather than fundamental capacity constraints. Based on observations of their \"attention'' maps, we propose a structured initialization technique that allows state space layers to more readily mimic attention. Across a variety of architecture settings, our initialization makes it substantially easier for Mamba to learn to copy and do associative recall from scratch."},"_bibtex":{"value":"@misc{\ntrockman2025mimetic,\ntitle={Mimetic Initialization Helps State Space Models Learn to Recall},\nauthor={Asher Trockman and Hrayr Harutyunyan and J Zico Kolter and Sanjiv Kumar and Srinadh Bhojanapalli},\nyear={2025},\nurl={https://openreview.net/forum?id=iVy7aRMb0K}\n}"},"title":{"value":"Mimetic Initialization Helps State Space Models Learn to Recall"},"pdf":{"value":"/pdf/5b9624e012b1653fd84db25266eba2a30fcf0588.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"trockman|mimetic_initialization_helps_state_space_models_learn_to_recall"},"authorids":{"value":["~Asher_Trockman1","~Hrayr_Harutyunyan1","~J_Zico_Kolter1","~Sanjiv_Kumar1","~Srinadh_Bhojanapalli1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Asher Trockman","Hrayr Harutyunyan","J Zico Kolter","Sanjiv Kumar","Srinadh Bhojanapalli"]}},"version":2},{"content":{"summary":{"value":"This paper explores post-training quantization (PTQ) for Mamba language models and introduces a new method called Q-Mamba. The authors identify that the bottlenecks in Mamba models arise from the large size of linear projections and state caches, which can be further optimized through model quantization. They observe that state caches contain outliers in both state and channel dimensions. Based on further analyses of this phenomenon, Q-Mamba proposes Decoupled Scale Quantization, utilizing independent scales for different dimensions. Additionally, Q-Mamba introduces a new quantization simulation approach called Efficient Selectivity Reconstruction to facilitate efficient layer-wise reconstruction using the original parallel scan algorithm. Experiments demonstrate the necessity of these methods for effective quantization of Mamba models, achieving W8A8H4 quantization for Mamba models for the first time. Experiments on memory consumption and latency are also presented."},"soundness":{"value":4},"confidence":{"value":5},"questions":{"value":"The appendix provides the results of the GPTQ[1] and SmoothQuant[2] methods on Mamba models. Could you provide more results for different quantization settings (e.g., W3A16, W4A4)? I am curious about the bit precision that can be achieved with Mamba model quantization.\n\n[1] Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023.\n\n[2] Frantar et al., OPTQ: Accurate Quantization for Generative Pre-trained Transformers. ICLR 2023."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":4},"strengths":{"value":"1. The main contribution of this work is the first exploration of quantization for Mamba models. Mamba architecture has recently attracted significant attention in the research community, and this paper offers a fresh perspective on model compression. The authors identify that the bottlenecks in Mamba models stem from the large size of linear projections and state caches, highlighting the potential for further research in model compression for Mamba models.\n\n2. Both the DSQ and ESR methods are well-motivated and specifically tailored for Mamba architectures. The DSQ method is based on the observation that states in SSMs contain outliers in both channel and state dimensions. After a theoretical analysis of this phenomenon, the paper proposes utilizing decoupled (independent) scales for these dimensions. This approach is compelling and demonstrates significant improvements compared to per-channel quantization. ESR is a novel quantization simulation method that approximates quantization noise in layer-wise reconstruction and is necessary when introducing quantization into the parallel scan algorithm. The authors also conduct empirical experiments to select fine-tuning parameters that enhance performance\n\n3. Experiments demonstrate the effectiveness of these methods for quantization of Mamba models. Notably, this paper achieves W8A8H4 (8-bit linear projections and 4bit states) quantization for Mamba models for the first time. For example, Q-Mamba achieves a 50% reduction in memory consumption with only a 2.13% average accuracy degradation on zero-shot tasks for Mamba2-2.7B. Ablation studies also demonstrate the convenience of Q-Mamba, which requires only 128 samples and a few epochs. Experiments on memory consumption and latency are also presented.\n\n4. This paper is clearly written, logically structured, and easy to understand."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The background section could be enhanced for clarity for readers who are not familiar with Mamba. It would be beneficial to include additional information about the differences between the Mamba-1 and Mamba-2 models.\n\n2. Some ablation studies on hyperparameters, such as optimizer settings ( learning rate, and weight decay) should be provided to demonstrate the robustness of ESR.\n\n3. Considering the need for fast deployment, the total time consumption for layer-wise construction should be included, particularly for the largest Mamba models."}},"nonreaders":[],"tmdate":1731427531892,"tcdate":1730638635919,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2045/Reviewer_S8h7"],"signatures":["ICLR.cc/2025/Conference/Submission2045/Reviewer_S8h7"],"forum":"AY1S52vr0a","number":4,"license":"CC BY 4.0","cdate":1730638635919,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2045/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427531892,"domain":"ICLR.cc/2025/Conference","replyto":"AY1S52vr0a","id":"JIThSmzwy4","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Quantization"]},"supplementary_material":{"value":"/attachment/0924a0296e7a1a9ad7158edce07c33777608192e.pdf"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSMs), such as Mamba, have recently demonstrated the potential to match or even surpass Transformers in language understanding tasks, making them a promising alternative for designing Large Language Models (LLMs). \nConcurrently, model quantization, especially Post-Training Quantization (PTQ), has been proven effective in reducing memory usage and inference latency in LLMs.\nIn this paper, we explore post-training quantization for Mamba (\\textbf{Q-Mamba}) by turning both linear projections and state caches into low-bit integers for efficient inference. \nAfter a theoretical analysis of the causes of outliers in states, we propose \\textbf{Decoupled Scale Quantization (DSQ)}, which mitigates outliers in both the state and channel dimensions by applying separate quantization scales.\nTo preserve the selective ability of quantized Mamba, we introduce \\textbf{Efficient Selectivity Reconstruction (ESR)}, a block-wise reconstruction method that involves a novel quantization simulation scheme, enabling fast parallel scan algorithms with the non-linear quantization function.\nWe demonstrate the effectiveness of Q-Mamba across various quantization settings, model sizes, and both generation and zero-shot tasks. \nIn particular, for Mamba2-2.7B with W8A8H4 quantization, Q-Mamba achieves a 50\\% reduction in memory consumption with only a 2.13\\% average accuracy degradation on zero-shot tasks."},"_bibtex":{"value":"@misc{\ntianqi2025qmamba,\ntitle={Q-Mamba: Towards more efficient Mamba models via Post-Training Quantization},\nauthor={Chen Tianqi and Yuanteng Chen and Weixiang Xu and Zeyu Zhu and Peisong Wang and Jian Cheng},\nyear={2025},\nurl={https://openreview.net/forum?id=AY1S52vr0a}\n}"},"title":{"value":"Q-Mamba: Towards more efficient Mamba models via Post-Training Quantization"},"pdf":{"value":"/pdf/1c7e10c8b02e5d50be6957f056d845dde2575456.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"tianqi|qmamba_towards_more_efficient_mamba_models_via_posttraining_quantization"},"authorids":{"value":["~Chen_Tianqi1","~Yuanteng_Chen1","~Weixiang_Xu2","~Zeyu_Zhu1","~Peisong_Wang1","~Jian_Cheng7"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chen Tianqi","Yuanteng Chen","Weixiang Xu","Zeyu Zhu","Peisong Wang","Jian Cheng"]}},"version":2},{"content":{"summary":{"value":"This paper presents MambaSIC, a stereo image compression framework that leverages Mamba-based visual state space blocks (stereo VSSB) to capture inter-view redundancy with linear complexity. The authors further propose a bi-directional multi-reference entropy model with checkerboard partitioning for efficient entropy coding. The method achieves state-of-the-art rate–distortion performance while being faster than existing stereo compression approaches."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please refer to weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The proposed method achieves strong empirical results. It reports a 57.15% BD-rate reduction over BPG on InStereo2K, outperforming BiSIC by 9.08%, while being faster than both BiSIC and CAMSIC. Moreover, it achieves the smallest inter-view PSNR discrepancy (0.057 dB vs. 0.334 dB for BiSIC), indicating that the model preserves cross-view consistency effectively and produces high-quality reconstructions.\n\n2. By leveraging Mamba’s linear complexity and long-range modeling capability, the method provides an efficient way to capture inter-view dependencies.\n\n3. The evaluation is comprehensive. The authors conduct thorough comparisons with over ten baselines, including both traditional codecs (BPG, MV-HEVC, VVC) and learning-based models, and evaluate across multiple benchmarks such as InStereo2K and Cityscapes. These extensive experiments demonstrate consistent improvements in both rate–distortion and runtime performance, lending strong credibility to the empirical claims."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. While the integration of Mamba and checkerboard entropy modeling is effective, the paper’s conceptual novelty is limited. Most of the core components are borrowed or adapted from existing works, including Visual State Space models (VSSL, VMamba), four-directional scanning and checkerboard entropy modeling from MLIC++, and spatial partitioning strategies. As a result, the contribution appears primarily engineering-oriented.\n\n2. The claim of achieving linear complexity over quadratic attention lacks rigorous theoretical or empirical support. Although Mamba inherently scales linearly with sequence length, the paper does not provide a formal FLOPs comparison or complexity analysis between the proposed stereo VSSB and mutual-attention mechanisms used in prior works such as BiSIC. Moreover, existing attention variants (e.g., linear or windowed attention) already achieve comparable efficiency, which weakens the argument that Mamba provides a unique complexity advantage in this context."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916358199,"tcdate":1761994180317,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2750/Reviewer_EvsK"],"signatures":["ICLR.cc/2026/Conference/Submission2750/Reviewer_EvsK"],"forum":"0dHrYUd17W","number":4,"license":"CC BY 4.0","cdate":1761994180317,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2750/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916358199,"domain":"ICLR.cc/2026/Conference","replyto":"0dHrYUd17W","id":"fOq7B79j2F","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Stereo image compression; visual state space model"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Stereo image compression (SIC) has become increasingly vital with its applications surging in fields such as 3D reconstruction and autonomous navigation. Previous methods leverage cross-attention to model inter-view redundancy and employ autoregressive entropy models to predict probability distributions, achieving impressive rate-distortion performance. However, they suffer from slow coding speed due to the quadratic complexity of cross-attention mechanisms and the spatial autoregressive iterations of the entropy models. To address these limitations, we propose MambaSIC, which introduces two key innovations.  First, we propose a Mamba-based stereo visual state space block (stereo VSSB) that leverages its linear complexity and long-range modeling capabilities to more rapidly and efficiently capture redundancy information between the two views. Second, to accelerate the compression process and enhance the accuracy of probability distribution estimation, we introduce a bi-directional multi-reference entropy model that utilizes a checkerboard partitioning strategy and the stereo VSSB to get rich inter-view priors. Experimental results demonstrate that our MambaSIC outperforms the state-of-the-art methods in both rate-distortion performance and coding efficiency. Moreover, it achieves the smallest inter-view PSNR discrepancy, resulting in more balanced reconstruction quality."},"_bibtex":{"value":"@misc{\nqin2025mambasic,\ntitle={Mamba{SIC}: Mamba-based Stereo Image Compression with Bi-directional Multi-reference Entropy Model},\nauthor={Shiyu Qin and Xinjie Zhang and Zhening Liu and Jinpeng Wang and Bin Chen and Jiawei Li and Yifan Ren and Shu-Tao Xia and Jun Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=0dHrYUd17W}\n}"},"title":{"value":"MambaSIC: Mamba-based Stereo Image Compression with Bi-directional Multi-reference Entropy Model"},"pdf":{"value":"/pdf/9496a712bd6ced6d32edf6793dca83982cb63fa4.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"qin|mambasic_mambabased_stereo_image_compression_with_bidirectional_multireference_entropy_model"},"authorids":{"value":["~Shiyu_Qin1","~Xinjie_Zhang2","~Zhening_Liu1","~Jinpeng_Wang3","~Bin_Chen4","~Jiawei_Li4","~Yifan_Ren4","~Shu-Tao_Xia1","~Jun_Zhang25"]},"authors":{"value":["Shiyu Qin","Xinjie Zhang","Zhening Liu","Jinpeng Wang","Bin Chen","Jiawei Li","Yifan Ren","Shu-Tao Xia","Jun Zhang"]}},"version":2},{"content":{"summary":{"value":"The paper presents Taipan, a hybrid model that incorporates attention modules into  Mamba-2 (an SSM). Specifically, it proposes to use Selective Attention Layers (SALs) to manage long-context tasks more efficiently in language modeling, such that only selected tokens are passed to (windowed) attention modules. In that way, Taipan selectively attends to critical tokens within an input, allowing it to capture long-range dependencies while also seeking to maintain computational efficiency."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- Can you clarify which version of Mamba was used in Figure 1b? Can you provide a more detailed breakdown of the computational costs for both Taipan and Mamba-2?\n\n- How does Taipan avoid the risk of all tokens being passed to the attention module at test time if the fixed attention capacity C is exceeded?\n\n- Can you provide a rationale for choosing Gumbel-softmax? What would be potential alternatives? For example, how does Taipan compare with other differentiable strategies, such as gradient surrogates, continuous relaxations, etc, which have been shown to be effective in similar applications? See [1] for a comprehensive overview.\n\n[1] Discrete Latent Structure in Neural Networks (https://arxiv.org/abs/2301.07473)"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"I enjoyed the effort put in this paper towards balancing Mamba-2's efficiency with selective attention mechanisms, an approach that can offer benefits for handling long contexts. I also liked that the experimental setup contains multiple evaluations across various benchmarks and model scales, allowing some insight into Taipan’s potential in extended context scenarios."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I believe the paper has several notable weaknesses that limit its impact. \n\n**Efficiency**: First, the presentation of efficiency gains is potentially misleading in Figure 1b, as Taipan’s backbone, Mamba-2, is slower than Taipan itself. Either that line represents Mamba-1, or the plot should include Mamba-2. To make matters more confusing, line 428 states, \"Notably, Taipan consistently outperforms Mamba-2, primarily due to its selective attention mechanism.\" Therefore, how is it possible for a model that uses Mamba-2 to process the input, along with additional computations, to actually be faster than Mamba-2?  Overall, this discrepancy raises questions about whether computational overheads are fully accounted for. \n\n**Novelty:** Furthermore, the novelty of combining SSMs with attention mechanisms is limited, as previous models, such as Jamba, have explored similar hybrid architectures, while the selective attention mechanism can be seen as an increment over Jamba.\n\n**Gumbel-softmax**: Arbitrary architectural choices, like the selection of Gumbel-softmax without justification or comparison with alternatives, also weaken the paper, especially given that SALs are a primary contribution. The fixed attention capacity $C$ set during training could reduce the model’s flexibility at test time, as the need for attention across tokens may vary, and it is unclear how the model avoids processing all tokens at test time (as $C$ is budget for training). \n\n**Presentation**: Additionally, inconsistencies in result reporting (e.g., bolded Taipan results even where it does not outperform other models) could mislead readers, as could unclear visual elements like Figure 1’s unexplained extrapolation regime and Figure 4’s table format. Moreover, the paper also disregards the proper use of citation styles (citep vs citet). Regarding Figure 1, it is unclear where the extrapolation regime starts, as per section 4.4. Collectively, these issues make the paper feel overly incremental and poorly substantiated. \nTherefore, to improve the paper, I suggest consistently bolding the best results, clearly highlighting the extrapolation regime in Figure 1, improving Figure 4, and fixing the citation format throughout the paper."}},"nonreaders":[],"tmdate":1731429282203,"tcdate":1730779157051,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9385/Reviewer_U9JN"],"signatures":["ICLR.cc/2025/Conference/Submission9385/Reviewer_U9JN"],"forum":"9DnKZbOr4r","number":5,"license":"CC BY 4.0","cdate":1730779157051,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9385/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429282203,"domain":"ICLR.cc/2025/Conference","replyto":"9DnKZbOr4r","id":"GjQIRE3Lmy","forumContent":{"TLDR":{"value":"We introduce Taipan, a novel architecture that combines State Space Models with Selective Attention Layers, achieving superior performance in memory-intensive tasks while enabling efficient language modeling for sequences up to 1 million tokens."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Efficient Language Model","Model Architecture","Long-context Language Model","In-context Retrieval","Hybrid Architecture","Linear Complexity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long sequences due to quadratic computational complexity in training and linearly scaling memory costs during inference. Recent State Space Models (SSMs) such as Mamba offer alternatives with constant memory usage, but they underperform in tasks requiring extensive in-context retrieval. We introduce Taipan, a novel hybrid architecture that combines Mamba-2 with Selective Attention Layers (SALs). These SALs identify tokens requiring long-range interactions, remove less important features, and then augment their representations using the attention module. This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. By constraining the attention budget, Taipan extends accurate predictions to context lengths of up to 1 million tokens while preserving computational efficiency. Our experiments demonstrate Taipan's superior performance across various scales and tasks, offering a promising solution for efficient long-context language modeling."},"_bibtex":{"value":"@misc{\nnguyen2025taipan,\ntitle={Taipan: Efficient and Expressive State Space Language Models with Selective Attention},\nauthor={Chien Van Nguyen and Huy Huu Nguyen and Thang M. Pham and Ruiyi Zhang and Hanieh Deilamsalehy and Puneet Mathur and Ryan A. Rossi and Trung Bui and Viet Dac Lai and Franck Dernoncourt and Thien Huu Nguyen},\nyear={2025},\nurl={https://openreview.net/forum?id=9DnKZbOr4r}\n}"},"title":{"value":"Taipan: Efficient and Expressive State Space Language Models with Selective Attention"},"pdf":{"value":"/pdf/4c569111617737e5532042f859ac730be0ac18ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"nguyen|taipan_efficient_and_expressive_state_space_language_models_with_selective_attention"},"authorids":{"value":["~Chien_Van_Nguyen1","~Huy_Huu_Nguyen1","~Thang_M._Pham1","~Ruiyi_Zhang3","~Hanieh_Deilamsalehy1","~Puneet_Mathur2","~Ryan_A._Rossi2","~Trung_Bui1","~Viet_Dac_Lai1","~Franck_Dernoncourt1","~Thien_Huu_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chien Van Nguyen","Huy Huu Nguyen","Thang M. Pham","Ruiyi Zhang","Hanieh Deilamsalehy","Puneet Mathur","Ryan A. Rossi","Trung Bui","Viet Dac Lai","Franck Dernoncourt","Thien Huu Nguyen"]}},"version":2},{"content":{"summary":{"value":"The paper proposes Geo-Mamba, a deep learning architecture that adapts the selective state-space model (Mamba) to operate directly on the symmetric positive definite (SPD) manifold. This is motivated by the observation that functional connectivity (FC) matrices from fMRI data lie in a non-Euclidean space, making standard Euclidean models geometrically inappropriate. Geo-Mamba leverages Log-Euclidean mappings to move between the manifold and its tangent space, allowing linear operations while preserving intrinsic geometry. The model processes FC matrices through a dual-path design capturing both multi-scale local dependencies and global spatial structure, fused via a geometry-aware mixing operator. The method is evaluated on six fMRI datasets and demonstrates competitive or superior performance to baselines such as Mamba, SPDNet, and STAGIN."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. What is the motivation for including a self-attention layer after the SSM block? Was it ablated, and how did it affect performance or interpretability? \n\n2. How were key hyperparameters (state dimension, window sizes, number of downsamples) chosen? What is the total parameter count and computational overhead relative to Mamba or SPDNet? Is it scalable to higher-resolution atlases? Clarifying these points is essential to understand the true contribution and applicability of the model.\n\n\n3. Did the authors observe any overfitting on smaller datasets?\n\n\n4. In ablation (ii), does removing the “global path” correspond to removing G from $S_{all}$? If so, how critical is this path to multi-scale modeling?\n\n5. How is the attention anchor $Q$ intialized? Was any regularization applied to it during training?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"The core idea of adapting the Mamba (SSM) architecture to the non-Euclidean geometry of SPD manifolds is both novel and technically well-motivated.\nThe use of the Log-Euclidean tangent space for SPD matrices is mathematically justified and enables Euclidean operations while preserving manifold constraints.\nThe design of multi-scale spatial feature extraction through sliding SPD sub-blocks is a creative way to leverage manifold properties for modeling spatial dependencies in functional connectivity data before passing them into the SSM.\nThe method is comprehensively evaluated across six public neuroimaging datasets and demonstrates superior or competitive performance compared to a wide range of relevant baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While the paper presents a compelling and potentially impactful idea, several ambiguities limit the clarity of the contribution and make it difficult to pinpoint the true source of performance gains and whether certain architectural components are genuinely helpful. Some components are not clearly justified or ablated, and those that are seem to have minimal impact. The experimental setup lacks transparency on key implementation details, such as baseline adaptation and hyperparameter tuning. Below are the concerns in order:\n\n1. While the ablation study in Figure 3 is informative, it lacks standard deviations and shows that the four ablated components do not significantly degrade classification performance (based on the * indicators). This makes it difficult to determine whether modules such as the global descriptor G or the “distillation path” meaningfully contribute to the results. Furthermore, several key components including the self-attention layer, the geodesic mixing used in the output mapping, and the learnable geometric anchor $Q$ are not ablated, and their individual impacts remain unclear.\n\n2. The authors mention that FC matrices are typically ordered by brain regions. The model assumes that ROIs adjacent in the n×n functional connectivity matrix correspond to spatially adjacent regions in the brain. However, cortical organization is inherently three-dimensional, and the 1D sliding-window design in the pyramid-based feature extraction cannot completely capture this 3D spatial topology. This weakens the claim of modeling hierarchical spatial information and limits generalization across modalities.\n\n3. The paper does not provide sufficient detail on how the baseline sequence models (e.g., Transformer, Mamba, STAGIN) were adapted for this task. It is unclear if they were fed the vectorized FC matrix, the same \"pseudo-temporal sequence\" as Geo-Mamba, or the time-series fMRI data. Since the authors argue that Euclidean operations are unsuitable for FC matrices, comparing Geo-Mamba to baselines trained on Euclidean FC inputs may be unfair. Clarifying these points is crucial to isolate whether improvements stem from manifold geometry or input design.\n\n4. The number of SSM layers used is unspecified, despite claims about capturing “hierarchical temporal evolution” (line 117). It is not demonstrated how stacking helps capture hierarchical spatial information or affects performance.\n\n5. The paper does not clearly describe how $B_t$, $C_t$, and $\\lambda_t$ are generated from the GRU or how this differs from the selective parameterization in Mamba. The connection between GRU gating and SPD-preserving updates remains unclear.\n\n6. The “pseudo-temporal sequence” is created by stacking spatial SPD sub-blocks from the averaged n×n connectivity matrix, not by modeling real temporal dynamics. Hence, the terms “pseudo-temporal” throughout the text or “time-dependent” (line 449) are misleading, since the model processes only spatial relationships rather than the actual neural time-series.\n\n7. The term \"distillation path\" is confusing. In machine learning, \"distillation\" typically refers to student–teacher knowledge transfer. Using it to mean \"dimensionality reduction\" is non-standard and should be clarified."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922741168,"tcdate":1762249256064,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11691/Reviewer_aJYZ"],"signatures":["ICLR.cc/2026/Conference/Submission11691/Reviewer_aJYZ"],"forum":"7XonwHRkTS","number":4,"license":"CC BY 4.0","cdate":1762249256064,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11691/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922741168,"domain":"ICLR.cc/2026/Conference","replyto":"7XonwHRkTS","id":"PNKlaxX9zU","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Brain functional connectivity","Riemannian geometry","State space model","Geometric deep learning"]},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Functional magnetic resonance imaging (fMRI)–derived functional connectivity (FC) is represented as graphs and as correlation/covariance matrices that live on non-Euclidean spaces—cortical graphs and the Riemannian manifold of symmetric positive-definite (SPD) matrices—so conventional Euclidean sequence models are misspecified. To this end, we introduce *Geo-Mamba*, a geometric variant of Mamba formulated on Riemannian manifolds. *Geo-Mamba* employs a dual-path selective state-space design: *a stacked path* performs hierarchical spatial modeling by aggregating pyramid multi-scale features to capture local and global dependencies, while *a embedding path* combats redundancy in high-dimensional SPD inputs via progressive, geometry-aware dimensionality reduction (operating in the appropriate manifold spaces) to produce compact states without violating Riemannian constraints. Their complementary outputs are fused through the tailored *GeoMix* operator to yield a compact, discriminative SPD representation. *Geo-Mamba* is evaluated on six public fMRI datasets—ADNI, OASIS, PPMI, Taowu, Neurocon, and Mātai—spanning Alzheimer’s and Parkinson’s cohorts as well as multi-site normative populations with diverse acquisition protocols. Across these benchmarks, it delivers consistently competitive accuracy and robustness, supporting the value of dual-path manifold modeling for neuroimaging and its potential for clinical translation."},"_bibtex":{"value":"@misc{\ncao2026geomamba,\ntitle={Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics},\nauthor={Yuwei Cao and Tingting Dan and Yang Yang and Guorong Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=7XonwHRkTS}\n}"},"title":{"value":"Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics"},"pdf":{"value":"/pdf/1ce491d063b410272cb740d9493deafc822b3586.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cao|geomamba_dualpath_riemannian_statespace_models_for_functional_dynamics"},"authorids":{"value":["~Yuwei_Cao3","~Tingting_Dan1","~Yang_Yang12","~Guorong_Wu1"]},"authors":{"value":["Yuwei Cao","Tingting Dan","Yang Yang","Guorong Wu"]}},"version":2},{"content":{"summary":{"value":"The paper proposes Test-Time Training (TTT) layers. TTT outperforms Transformers and Mamba for models up to 1.3B parameters. In terms of runtime, TTT-Linear is more efficient than Transfromers and is competitive with that of Mamba."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"In meta-learning, a common issue of the inner/outer loop formulation with gradient descent is the second-order gradients which are memory intensive. How does TTT resolve this issue?\n\nCould the authors distinguish their usage of (mini-)batch GD from the commonly associated connotation? For example, prefacing the term with \"temporal\" could potentially reduce the initial confusion. \n\nThe training efficiency of TTT appears to be highly dependent on $b$ as it determines the amount of parallelism TTT can leverage. Furthermore, $b$ also significantly affects the performance. In contrast, methods such as Mamba are not dependent on this hyperparameter. Plots comparing the training runtime, memory, and performance of TTT with varying values of $b$ and methods such as Mamba are important. Could the authors include these?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"This paper explores a gradient-based alternative to modern RNNs and self-attention. The approach is uncommon and allows for a more expressive hidden state, which results in better performances. \n\nTTTs achieve competitive or slightly better performance compared to Mamba and Transformers. \n\nConnecting TTT with linear and self attention is a nice touch."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Mamba is parallelized with a parallel scan. However, TTT requires some sequential computations controlled by a hyperparameter $b$, potentially making it significantly slower during training. Unfortunately, the paper lacks memory and runtime analyses during training, making it rather hard to determine the practicality of training a TTT model compared to other modern RNNs that leverage parallel scan (e.g., Mamba, xLSTM, GLA-Transformers, etc...).\n\nMeta-learning is closely related to TTT, but the connection is not properly discussed. The paper briefly mentions a connection on page 3, but the discussion is lacking. For example, TTT using \"online gradient descent\" is reminiscent of MAML, where the task is a sequence and each sample is a timestamp. \n\nWhen initially reading, the reviewer found the usage of the terms \"batch GD\" and \"mini-batch GD\" to be confusing and took some time to get used to. For example, when mentioning \"mini-batch GD\", the reviewer initially thought of its typical use case (per outer-loop training iteration). However, in this work, the mini-batch GD is done temporally with varying timestamps."}},"nonreaders":[],"tmdate":1732644221562,"tcdate":1730701032259,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5310/Reviewer_HodN"],"signatures":["ICLR.cc/2025/Conference/Submission5310/Reviewer_HodN"],"forum":"eifW0W0xgt","number":2,"license":"CC BY 4.0","cdate":1730701032259,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5310/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732644221562,"domain":"ICLR.cc/2025/Conference","replyto":"eifW0W0xgt","id":"7PDPIVPNHN","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["test-time training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state. Inspired by prior work, we present a practical framework to instantiate sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Both TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer, they can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. With preliminary systems optimization, TTT-Linear is already faster than Transformer at 8k context and matches Mamba in wall-clock time. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research."},"_bibtex":{"value":"@misc{\nsun2025learning,\ntitle={Learning to (Learn at Test Time): {RNN}s with Expressive Hidden States},\nauthor={Yu Sun and Xinhao Li and Karan Dalal and Jiarui Xu and Arjun Vikram and Genghan Zhang and Yann Dubois and Xinlei Chen and Xiaolong Wang and Sanmi Koyejo and Tatsunori Hashimoto and Carlos Guestrin},\nyear={2025},\nurl={https://openreview.net/forum?id=eifW0W0xgt}\n}"},"title":{"value":"Learning to (Learn at Test Time): RNNs with Expressive Hidden States"},"pdf":{"value":"/pdf/be1b3f1cf37ec5812fb041e8bbdd1569d92a9992.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sun|learning_to_learn_at_test_time_rnns_with_expressive_hidden_states"},"authorids":{"value":["~Yu_Sun1","~Xinhao_Li3","~Karan_Dalal1","~Jiarui_Xu1","~Arjun_Vikram1","~Genghan_Zhang1","~Yann_Dubois1","~Xinlei_Chen1","~Xiaolong_Wang3","~Sanmi_Koyejo1","~Tatsunori_Hashimoto1","~Carlos_Guestrin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yu Sun","Xinhao Li","Karan Dalal","Jiarui Xu","Arjun Vikram","Genghan Zhang","Yann Dubois","Xinlei Chen","Xiaolong Wang","Sanmi Koyejo","Tatsunori Hashimoto","Carlos Guestrin"]}},"version":2},{"content":{"summary":{"value":"This paper proposes an extension of the SPDNet architecture to the recently emerging Mamba state-space modeling framework, referred to as Geo-Mamba. The method aims to model brain functional connectivity matrices on the SPD manifold while leveraging Mamba’s efficient sequence modeling capacity. The authors conduct experiments on six public fMRI datasets and report consistent performance gains, suggesting that the proposed model effectively captures multi-scale geometric and temporal dependencies.\n\nWhile the idea of integrating geometric deep learning on SPD manifolds with structured state-space models is interesting and timely, the paper lacks clarity in several conceptual, mathematical, and experimental aspects, which significantly weaken its overall rigor and reproducibility."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. Could you elaborate on the motivation for using SPD representations in this task? What specific information from fMRI connectivity are you preserving that would be lost in a Euclidean embedding?\n\n2. How do you reconcile the use of Riemannian Batch Normalization (AIRM-based) with your log-Euclidean metric formulation?\n\n3. Please clarify the exact dataset versions, subject selection criteria, ROI definitions, and cross-validation strategies.\n\n4. How are SPD matrices encoded as tensors in the Mamba architecture? Are manifold operations actually preserved throughout the sequence modeling pipeline? What explains the large performance differences between Geo-Mamba and vanilla Mamba across datasets if the only modification is a logarithmic mapping?\n\n5. Figure 2 appears to show extremely sparse connectivity patterns. Was any thresholding or sparsification applied to produce the visualizations in Figure 2?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The paper addresses a clear architectural gap by bridging SPD-based geometric representation learning and the latest Mamba sequence model, combining two active but previously disconnected research domains (geometric deep learning and state-space modeling).\n\nThe motivation to extend SPDNet to temporal modeling via Mamba is intuitively appealing and potentially impactful for neural signal and fMRI analysis, where both geometric structure and dynamic dependencies are crucial.\n\nThe empirical study across multiple datasets demonstrates that the proposed approach can generalize across diverse fMRI sources, suggesting practical relevance if rigorously validated."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper does not clearly articulate why the SPD manifold is the most appropriate representation for fMRI data in this context. What type of statistical or functional information is being preserved by the SPD structure? Why is the log-Euclidean metric chosen over alternatives such as AIRM or Stein divergence? Without such justification, the methodological grounding feels ad hoc.\n\nThe implementation (as observed in MambaSPD_Attn_then_SSM) employs Riemannian Batch Normalization, which is derived for the AIRM metric, while the paper consistently claims to use the log-Euclidean metric. Mixing these two incompatible metrics within the same model undermines mathematical consistency and raises questions about the validity of the results.\n\nExperimental design lacks important information: the number of subjects used in the experiments does not correspond to any known public version. The category definitions (e.g., diagnostic groups) and cross-validation protocols (5-fold vs. 10-fold) are inconsistent across datasets without explanation. Preprocessing steps, such as confound regression, normalization, or atlas selection, are insufficiently described, making reproducibility difficult, even with access to the code.\n\nIt is not clear how the SPD matrices are represented within the Mamba pipeline in your baseline comparison. From the description, the “Geo-Mamba” variant seems to apply a logarithmic mapping only at the final stage. Conceptually, this should yield behavior similar to standard Mamba, yet large performance differences are reported. Can you explain why?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360252220,"tcdate":1761322439915,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11691/Reviewer_Ai35"],"signatures":["ICLR.cc/2026/Conference/Submission11691/Reviewer_Ai35"],"forum":"7XonwHRkTS","number":1,"license":"CC BY 4.0","cdate":1761322439915,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11691/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360252220,"domain":"ICLR.cc/2026/Conference","replyto":"7XonwHRkTS","id":"FshFNsIFn2","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Brain functional connectivity","Riemannian geometry","State space model","Geometric deep learning"]},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Functional magnetic resonance imaging (fMRI)–derived functional connectivity (FC) is represented as graphs and as correlation/covariance matrices that live on non-Euclidean spaces—cortical graphs and the Riemannian manifold of symmetric positive-definite (SPD) matrices—so conventional Euclidean sequence models are misspecified. To this end, we introduce *Geo-Mamba*, a geometric variant of Mamba formulated on Riemannian manifolds. *Geo-Mamba* employs a dual-path selective state-space design: *a stacked path* performs hierarchical spatial modeling by aggregating pyramid multi-scale features to capture local and global dependencies, while *a embedding path* combats redundancy in high-dimensional SPD inputs via progressive, geometry-aware dimensionality reduction (operating in the appropriate manifold spaces) to produce compact states without violating Riemannian constraints. Their complementary outputs are fused through the tailored *GeoMix* operator to yield a compact, discriminative SPD representation. *Geo-Mamba* is evaluated on six public fMRI datasets—ADNI, OASIS, PPMI, Taowu, Neurocon, and Mātai—spanning Alzheimer’s and Parkinson’s cohorts as well as multi-site normative populations with diverse acquisition protocols. Across these benchmarks, it delivers consistently competitive accuracy and robustness, supporting the value of dual-path manifold modeling for neuroimaging and its potential for clinical translation."},"_bibtex":{"value":"@misc{\ncao2026geomamba,\ntitle={Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics},\nauthor={Yuwei Cao and Tingting Dan and Yang Yang and Guorong Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=7XonwHRkTS}\n}"},"title":{"value":"Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics"},"pdf":{"value":"/pdf/1ce491d063b410272cb740d9493deafc822b3586.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cao|geomamba_dualpath_riemannian_statespace_models_for_functional_dynamics"},"authorids":{"value":["~Yuwei_Cao3","~Tingting_Dan1","~Yang_Yang12","~Guorong_Wu1"]},"authors":{"value":["Yuwei Cao","Tingting Dan","Yang Yang","Guorong Wu"]}},"version":2},{"content":{"comment":{"value":"[R1] Li, Hongkang, et al. “A Theoretical Understanding of Shallow Vision Transformers: Learning,\nGeneralization, and Sample Complexity,” International Conference on Learning Representations\n(ICLR), 2023.\n\n[R2] Zhang, Shuai, et al. “Joint Edge-Model Sparse Learning is Provably Efficient for Graph Neural\nNetworks,” International Conference on Learning Representations (ICLR), 2023.\n\n[R4] Li, Hongkang, et al. “How Do Nonlinear Transformers Learn and Generalize in In-Context\nLearning?” International Conference on Machine Learning (ICML), 2024.\n\n[R8] Zhu, Lianghui, et al. “Vision Mamba: Efficient Visual Representation Learning with Bidirec-\ntional State Space Model,” Proceedings of the 41st International Conference on Machine Learning\n(ICML), PMLR 235, pp. 62429–62442, 2024.\n\n[R9] Wang, Chloe, et al. “Graph-Mamba: Towards Long-Range Graph Sequence Modeling with\nSelective State Spaces,” arXiv preprint arXiv:2402.00789, 2024.\n\n[R10] Liu, Yue, et al. “VMamba: Visual State Space Model,” Advances in Neural Information\nProcessing Systems (NeurIPS), vol. 37, pp. 103031–103063, 2024.\n\n[R11] Behrouz, Ali, et al. “Graph Mamba: Towards Learning on Graphs with State Space Mod-\nels,” Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining\n(KDD), pp. 119–130, 2024.\n\n[R12] Gu, Albert, et al. “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,”\narXiv preprint arXiv:2312.00752, 2023."},"title":{"value":"References"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763749646776,"tcdate":1763749646776,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19648/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission19648/Authors"],"forum":"hvpKqEYJjj","number":9,"license":"CC BY 4.0","cdate":1763749646776,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19648/-/Official_Comment"],"mdate":1763749646776,"domain":"ICLR.cc/2026/Conference","replyto":"Ak28zAMTOW","id":"JW37J9gmS7","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Training Dynamics","Feature Learning","Generalization","Mamba","Learning Theory","Selective State Space Models"]},"supplementary_material":{"value":"/attachment/c9fcb2ad298f51c4efaa9f51d0c97c0844014e27.zip"},"primary_area":{"value":"learning theory"},"abstract":{"value":"The recent empirical success of Mamba and other selective state space models (SSMs) has renewed interest in non-attention architectures for sequence modeling, yet their theoretical foundations remain underexplored. We present a first-step analysis of generalization and learning dynamics for a simplified but representative Mamba block: a single-layer, single-head selective SSM with input-dependent gating, followed by a two-layer MLP trained via gradient descent (GD). Our study adopts a structured data model with tokens that include both class-relevant and class-irrelevant patterns under token-level noise and examines two canonical regimes: majority-voting and locality-structured data sequences. We prove that the model achieves guaranteed generalization by establishing non-asymptotic sample complexity and convergence rate bounds, which improve as the effective signal increases and the noise decreases.\nFurthermore, we show that the gating vector aligns with class-relevant features while ignoring irrelevant ones, thereby formalizing a feature-selection role similar to attention but realized through selective recurrence. Numerical experiments on synthetic data justify our theoretical results. Overall, our results provide principled insight into when and why Mamba-style selective SSMs learn efficiently, offering a theoretical counterpoint to Transformer-centric explanations."},"_bibtex":{"value":"@inproceedings{\nshandirasegaran2026a,\ntitle={A Theoretical Analysis of Mamba{\\textquoteright}s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models},\nauthor={Mugunthan Shandirasegaran and Hongkang Li and Songyang Zhang and Meng Wang and Shuai Zhang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=hvpKqEYJjj}\n}"},"title":{"value":"A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models"},"pdf":{"value":"/pdf/7ff125c33f63bfb0017d10d5e63ca5d19e737a02.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"shandirasegaran|a_theoretical_analysis_of_mambas_training_dynamics_filtering_relevant_features_for_generalization_in_state_space_models"},"authorids":{"value":["~Mugunthan_Shandirasegaran1","~Hongkang_Li1","~Songyang_Zhang2","~Meng_Wang4","~Shuai_Zhang6"]},"authors":{"value":["Mugunthan Shandirasegaran","Hongkang Li","Songyang Zhang","Meng Wang","Shuai Zhang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.07823v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"qu|cloudmamba_grouped_selective_state_spaces_for_point_cloud_analysis"},"html":{"value":"https://doi.org/10.48550/arXiv.2511.07823"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-07823,\n  publtype={informal},\n  author={Kanglin Qu and Pan Gao and Qun Dai and Zhanzhi Ye and Rui Ye and Yuanhao Sun},\n  title={CloudMamba: Grouped Selective State Spaces for Point Cloud Analysis},\n  year={2025},\n  month={November},\n  cdate={1761955200000},\n  journal={CoRR},\n  volume={abs/2511.07823},\n  url={https://doi.org/10.48550/arXiv.2511.07823}\n}\n"},"abstract":{"value":"Due to the long-range modeling ability and linear complexity property, Mamba has attracted considerable attention in point cloud analysis. Despite some interesting progress, related work still suffers from imperfect point cloud serialization, insufficient high-level geometric perception, and overfitting of the selective state space model (S6) at the core of Mamba. To this end, we resort to an SSM-based point cloud network termed CloudMamba to address the above challenges. Specifically, we propose sequence expanding and sequence merging, where the former serializes points along each axis separately and the latter serves to fuse the corresponding higher-order features causally inferred from different sequences, enabling unordered point sets to adapt more stably to the causal nature of Mamba without parameters. Meanwhile, we design chainedMamba that chains the forward and backward processes in the parallel bidirectional Mamba, capturing high-level geometric information during scanning. In addition, we propose a grouped selective state space model (GS6) via parameter sharing on S6, alleviating the overfitting problem caused by the computational mode in S6. Experiments on various point cloud tasks validate CloudMamba's ability to achieve state-of-the-art results with significantly less complexity."},"title":{"value":"CloudMamba: Grouped Selective State Spaces for Point Cloud Analysis"},"authors":{"value":[{"fullname":"Kanglin Qu","username":"~Kanglin_Qu1"},{"fullname":"Pan Gao","username":""},{"fullname":"Qun Dai","username":""},{"fullname":"Zhanzhi Ye","username":""},{"fullname":"Rui Ye","username":""},{"fullname":"Yuanhao Sun","username":""}]}},"tmdate":1791340695550,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2511-07823"],"tcdate":1791340691461,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Kanglin_Qu1"],"forum":"1Dex98BJXX","license":"CC BY-SA 4.0","number":164539,"cdate":1761955200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1791340695550,"domain":"OpenReview.net/Public_Article","id":"1Dex98BJXX","version":2},{"content":{"venue":{"value":"AAAI 2026"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/37818/41780"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"qu|cloudmamba_grouped_selective_state_spaces_for_point_cloud_analysis"},"html":{"value":"https://doi.org/10.1609/aaai.v40i11.37818"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/QuGDYYS26,\n  author={Kanglin Qu and Pan Gao and Qun Dai and Zhanzhi Ye and Rui Ye and Yuanhao Sun},\n  title={CloudMamba: Grouped Selective State Spaces for Point Cloud Analysis},\n  year={2026},\n  cdate={1767225600000},\n  pages={8659-8667},\n  url={https://doi.org/10.1609/aaai.v40i11.37818},\n  booktitle={AAAI},\n  crossref={conf/aaai/2026}\n}\n"},"abstract":{"value":"Due to the long-range modeling ability and linear complexity property, Mamba has attracted considerable attention in point cloud analysis. Despite some interesting progress, related work still suffers from imperfect point cloud serialization, insufficient high-level geometric perception, and overfitting of the selective state space model (S6) at the core of Mamba. To this end, we resort to an SSM-based point cloud network termed CloudMamba to address the above challenges. Specifically, we propose sequence expanding and sequence merging, where the former serializes points along each axis separately and the latter serves to fuse the corresponding higher-order features causally inferred from different sequences, enabling unordered point sets to adapt more stably to the causal nature of Mamba without parameters. Meanwhile, we design chainedMamba that chains the forward and backward processes in the parallel bidirectional Mamba, capturing high-level geometric information during scanning. In addition, we propose a grouped selective state space model (GS6) via parameter sharing on S6, alleviating the overfitting problem caused by the computational mode in S6. Experiments on various point cloud tasks validate CloudMamba's ability to achieve state-of-the-art results with significantly less complexity."},"title":{"value":"CloudMamba: Grouped Selective State Spaces for Point Cloud Analysis"},"authors":{"value":[{"fullname":"Kanglin Qu","username":"~Kanglin_Qu1"},{"fullname":"Pan Gao","username":""},{"fullname":"Qun Dai","username":""},{"fullname":"Zhanzhi Ye","username":""},{"fullname":"Rui Ye","username":""},{"fullname":"Yuanhao Sun","username":""}]}},"tmdate":1791340694792,"pdate":1798675200000,"externalIds":["dblp:conf/aaai/QuGDYYS26"],"tcdate":1791340691446,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Kanglin_Qu1"],"forum":"neP1ywD2it","license":"CC BY-SA 4.0","number":164537,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1791340694792,"domain":"OpenReview.net/Public_Article","id":"neP1ywD2it","version":2},{"content":{"summary":{"value":"This paper introduces Masked Autoregressive Pre-training (MAP) for a Mamba, which is a method to enhance the scalability of Mamba (technically Vision Mamba, Vim) by applying Masked Autoencoder (MAE) in an autoregressive manner for the pre-training method. By applying MAP to a hybrid Mamba-Transformer architecture, the authors achieve state-of-the-art performance on 2D and 3D datasets like ImageNet-1K and MOdelNet40, surpassing existing pre-training strategies. Ablation studies support the efficacy of MAP's design choices."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"See above weaknesses"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"- Exploring MAE-like pre-training methods on state space models like Mamba is still relatively unexplored, making this a valuable research area.\n- The study reveals intriguing findings, such as the effectiveness of autoregressive pre-training for Mamba and the promising approach of integrating mask tokens into autoregressive pre-training."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The paper’s contributions need to be stated more clearly, and some aspects lack novelty. While it proposes both a pre-training method -- Masked Autoregressive Pre-training (MAP) -- and a hybrid architecture (i.e., Mamba-Transformer), their individual contributions are not clearly distinguished, and introducing hybrid architecture itself is not novel. The authors begin by highlighting MAP's effectiveness by showcasing the enhancements to the Mamba-Transformer hybrid architecture (while the pilot study only employed handled Mamba -- ViM --). The reviewer suggests a more logical flow: MAP should first be tested on a standalone Mamba, followed by evaluating it on a hybrid architecture. Furthermore, most evaluations seem to focus on hybrid architectures, which obscures MAP's real aspect and results in a blending of contributions between the method and architecture. For example, there is insufficient reasoning as to why MAP would outperform existing methods like MAE and AR, along with a missing logical link explaining why MAP would be effective specifically for the hybrid architecture. The reviewer recommends that the authors distinctly separate their contributions and provide systematic support for each, suggesting that focusing on MAP would lead to more productive outcomes.\n- As mentioned above, there is a lack of evidence and intuition explaining why MAP is effective for standalone Mamba or Transformer models (actually, we need the intuition as to why MAP does not fall behind MAE for Transformers). Furthermore, the rationale for using random masking (why random masking?) in Mamba-pre-training is also lacking. Without preliminary studies testing random masking in the pilot study (only the last tokens are masked), it is unclear why this approach would work and how it can be effectively combined with Mamba training because detailed explanations are absent.\n- Furthermore, the motivation for introducing a hybrid architecture immediately after MAP is unclear, causing the paper’s main contributions to diverge by adding an architectural focus that may detract from emphasizing the primary contribution.\n- The pilot study does not fully align with the proposed pre-training method. While it successfully indicates that autoregressive pre-training is beneficial for Mamba, it does not clearly explain why random masking is necessary. Additionally, the reviewer feels that details like the optimal scanning direction and masking ratio for tokens at the end of the sequence, while potentially informative, are not prerequisites for the pilot study. Furthermore, the pilot study fails to provide any foundational rationale for a hybrid architecture.\n- In experiments, the effectiveness of MAP on non-hybrid architectures is neither impressive nor fairly compared with competing methods: 1) improvements for ViTs are minimal, and 2) competing methods like AR are not directly compared to MAP (e.g., MambaR-B/L + AR vs. MambaR-B/L + MAP in Table 8).\n- Missing related works:\n   - TokenUnify: Scalable Autoregressive Visual Pre-training with Mixture Token Prediction\n   - MambaMIM: Pre-training Mamba with State Space Token-interpolation\n   - Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations\n   - CMViM: Contrastive Masked Vim Autoencoder for 3D Multi-modal Representation Learning for AD classification\n\n- Many details are missing, including the experimental setup and the baseline used for the experiments:\n  - It is unclear how Mamba can handle random masking (rather than using masks only at the end of the sequence) during pre-training.\n  - The baselines and experimental setups for some experiments are missing, such as the experiments in Tables 2, 3, and others.\n  - The differences between the proposed method and a simple combination of masking strategy and autoregressive modeling are not clearly presented. The reviewer only notes a causal decoder. \n\n- Minors \n  - Ablation studies, such as the full results of the masking ratio of MAP, are not sufficiently presented. \n  - The authors are encouraged to add more spacing after tables and figures to improve readability.\n  - Found duplicated references at lines 554-559, 560-566, 657-664, 696-701.\n  - In Fig. 2(b), the masking position seems somewhat strange."}},"nonreaders":[],"tmdate":1731427595457,"tcdate":1730723152476,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2428/Reviewer_bQeS"],"signatures":["ICLR.cc/2025/Conference/Submission2428/Reviewer_bQeS"],"forum":"hBVywPrw6d","number":3,"license":"CC BY 4.0","cdate":1730723152476,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2428/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427595457,"domain":"ICLR.cc/2025/Conference","replyto":"hBVywPrw6d","id":"IzFKSldPmk","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"TLDR":{"value":"We propose a new pre-training strategy tailored for the Mamba-Transformer backbone."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Hybrid Vision Backbone","Masked Autoregressive","Self-supervised Learning"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has achieved significant advantages in long-context modeling and autoregressive tasks, but its scalability with large parameters remains a major limitation in vision applications. pretraining is a widely used strategy to enhance backbone model performance. Although the success of Masked Autoencoder in Transformer pretraining is well recognized, it does not significantly improve Mamba's visual learning performance. We found that using the correct autoregressive pretraining can significantly boost the performance of the Mamba architecture. Based on this analysis, we propose Masked Autoregressive Pretraining(MAP) to pretrain a hybrid Mamba-Transformer vision backbone network. This strategy combines the strengths of both MAE and Autoregressive pretraining, improving the performance of Mamba and Transformer modules within a unified paradigm. Experimental results show that both the pure Mamba architecture and the hybrid Mamba-Transformer vision backbone network pretrained with MAP significantly outperform other pretraining strategies, achieving state-of-the-art performance. We validate the effectiveness of the method on both 2D and 3D datasets and provide detailed ablation studies to support the design choices for each component."},"_bibtex":{"value":"@misc{\nliu2024map,\ntitle={{MAP}: Unleashing Hybrid Mamba-Transformer Vision Backbone{\\textquoteright}s Potential with Masked Autoregressive Pretraining},\nauthor={Yunze Liu and Li Yi},\nyear={2024},\nurl={https://openreview.net/forum?id=hBVywPrw6d}\n}"},"title":{"value":"MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone’s Potential with Masked Autoregressive Pretraining"},"pdf":{"value":"/pdf/10243949177ada96d6426f1e3d852f7611be953b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"liu|map_unleashing_hybrid_mambatransformer_vision_backbones_potential_with_masked_autoregressive_pretraining"},"authorids":{"value":["~Yunze_Liu2","~Li_Yi2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yunze Liu","Li Yi"]}},"version":2},{"content":{"summary":{"value":"In this work, the authors proposed model, CMamba, incorporates a modified Mamba (M-Mamba) module for temporal dependencies modeling, a global data dependent MLP (GDD-MLP) to effectively capture cross-channel dependencies, and a Channel Mix-up mechanism to mitigate over-fitting."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see weaknesses!"},"rating":{"value":5},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1.  The authors tailor the vanilla Mamba module for multivariate time series forecasting. A modified Mamba (M-Mamba) module is proposed for better cross-time dependencies modeling.\n 2. The authors enable Mamba to capture multivariate correlations using the proposed global data dependent MLP (GDD-MLP) and Channel Mix-up. Coupled with the M-Mamba module, the proposed CMamba captures both cross-time and cross-channel dependencies to learn better representations for multivariate time series forecasting."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The description of the contribution seems to be incremental innovation, and the modeling of dependencies across time variables is the purpose, rather than a clear design motivation. In addition, different exogenous time series have positive or negative effects on the target series. How should this dependence be measured?\n2. It can be seen from the experimental results in Table 1 that the performance improvement is weak. For example, on the ETTm1 dataset, the authors' method improved the MSE metrics by about 2% compared to the SOTA method (i.e., TimeMixer). If the authors claim that the cross-time dependence should produce significant performance improvement, the current experimental results are contrary to the intuition. After all, the dependencies between multiple time variables in electric time series data are much clearer.\n3. Especially in the ablation experiments of Table 2 with standard Mamba, we can find that the performance improvement is close to the test error. In terms of the MSE metric, the standard Mamba is 0.240 and the proposed method is 0.237, which is only 1.25% improvement.\n4. The reviewer is also concerned about the extent to which such modifications impose computational overhead on the several datasets. Especially the GDD-MLP module and CMamba-encoder module, is it because these complex components increase the complexity of the proposed prediction model and thus improve the prediction effect? Therefore, authors are expected to provide training and inference time on multiple time series datasets."}},"nonreaders":[],"tmdate":1731428496314,"tcdate":1730261441291,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5801/Reviewer_Z3ph"],"signatures":["ICLR.cc/2025/Conference/Submission5801/Reviewer_Z3ph"],"forum":"OclHGmt2ZM","number":3,"license":"CC BY 4.0","cdate":1730261441291,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5801/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428496314,"domain":"ICLR.cc/2025/Conference","replyto":"OclHGmt2ZM","id":"0LUdk8PPf9","forumContent":{"TLDR":{"value":"We tailor Mamba for multivariate time series forecasting, equipping it with stronger cross-time and cross-channel modeling capabilities, which achieves the SOTA in real-world datasets."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forecasting","Mamba","Channel-Dependent"]},"supplementary_material":{"value":"/attachment/4810e3ea813ec5d058c32afc9ff6a129c09b060a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent advancements in multivariate time series forecasting have been propelled by Linear-based, Transformer-based, and Convolution-based models, with Transformer-based architectures gaining prominence for their efficacy in temporal and cross-channel mixing.\nMore recently, Mamba, a state space model, has emerged with robust sequence and feature mixing capabilities.\nHowever, the suitability of the vanilla Mamba design for time series forecasting remains an open question, particularly due to its inadequate handling of cross-channel dependencies.\nCapturing cross-channel dependencies is critical in enhancing the performance of multivariate time series prediction.\nRecent findings show that self-attention excels in capturing cross-channel dependencies, whereas other simpler mechanisms, such as MLP, may degrade model performance.\nThis is counterintuitive, as MLP, being a learnable architecture, should theoretically capture both correlations and irrelevances, potentially leading to neutral or improved performance.\nDiving into the self-attention mechanism, we attribute the observed degradation in MLP performance to its lack of data dependence and global receptive field, which result in MLP's lack of generalization ability.\nConsidering the powerful sequence modeling capabilities of Mamba and the high efficiency of MLP, the combination of the two is an effective strategy for solving multivariate time series prediction.\nBased on the above insights, we introduce a refined Mamba variant tailored for time series forecasting.\nOur proposed model, \\textbf{CMamba}, incorporates a modified Mamba (M-Mamba) module for temporal dependencies modeling, a global data-dependent MLP (GDD-MLP) to effectively capture cross-channel dependencies, and a Channel Mixup mechanism to mitigate overfitting.\nComprehensive experiments conducted on seven real-world datasets demonstrate the efficacy of our model in improving forecasting performance."},"_bibtex":{"value":"@misc{\nzeng2025cmamba,\ntitle={{CM}amba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting},\nauthor={Chaolv Zeng and Zhanyu Liu and Guanjie Zheng and Linghe Kong},\nyear={2025},\nurl={https://openreview.net/forum?id=OclHGmt2ZM}\n}"},"title":{"value":"CMamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting"},"pdf":{"value":"/pdf/c6642767e189d0d9628d810af94b4cdc57554e7a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zeng|cmamba_channel_correlation_enhanced_state_space_models_for_multivariate_time_series_forecasting"},"authorids":{"value":["~Chaolv_Zeng2","~Zhanyu_Liu1","~Guanjie_Zheng1","~Linghe_Kong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chaolv Zeng","Zhanyu Liu","Guanjie Zheng","Linghe Kong"]}},"version":2},{"content":{"summary":{"value":"This paper proposes MemMamba, a memory-augmented extension of state-space models for long sequence modeling. The authors systematically analyze memory decay patterns in Mamba with the novel horizontal–vertical memory fidelity metrics. They also introduce the state summarization with cross-layer/cross-token attention mechanisms to mitigate information loss over extended contexts. The evaluation are mainly focused on diverse tasks such as language modeling (PG19), synthetic Passkey Retrieval, and cross-document reasoning, demonstrating improvements over serveral baselines."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See weaknesses"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. This paper provides a systematic analysis of memory decay in Mamba by mathematical derivations and the memory fidelity metrics (ETMF, ECLMF) in both main text and Figure 4.\n2. Paper is clearly written for the most parts, with a good contextualization within the SSM and long-sequence modeling literature."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Although the paper claims to offer a fundamentally “new paradigm” for ultra-long sequence modeling, the MemMamba approach can be interpreted as a synthesis and adaptation of several established ideas (memory summarization, cross-layer attention, and sparsity in attention), rather than introduction of entirely unprecedented architectures. The degree of originality, while respectable, may be somewhat overstated in the positioning.\n2. There are a few areas for improvement in the paper presentation: for example, the clarity of Figure 3 and the font size within the figures could be further adjusted. Also, are lines 456-457 redundant with a previous paragraph? They could be removed.\n3. The description of the thresholding mechanism (e.g., for triggering note-taking and cross-attention) in Section 4 lacks a fully articulated rationale for the chosen thresholds ($\\tau_1$, $\\tau_2$). Moreover, there is no sensitivity analysis or ablation study on their values—an important omission since these could significantly influence empirical outcomes.\n4. The paper lacks a comparison of GPU memory usage between the proposed model and baseline models. Also, regarding Section 5.2 'Efficiency,' why are specific results omitted, with only a comparison between MemMamba and Transformer being presented? \n5. Potential missing related work or baseline models that should be compared in the paper:\n\n[1] Wang, Qianning, He Hu, and Yucheng Zhou. 'Memorymamba: Memory-augmented state space model for defect recognition.' arXiv preprint arXiv:2405.03673 (2024).\n\n[2] Gui, Yiyu, et al. \"EEGMamba: Bidirectional state space model with mixture of experts for EEG multi-task classification.\" _arXiv preprint arXiv:2407.20254_ (2024)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919696889,"tcdate":1761968147459,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7614/Reviewer_bDYZ"],"signatures":["ICLR.cc/2026/Conference/Submission7614/Reviewer_bDYZ"],"forum":"jPYJuvzof2","number":3,"license":"CC BY 4.0","cdate":1761968147459,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7614/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919696889,"domain":"ICLR.cc/2026/Conference","replyto":"jPYJuvzof2","id":"JqSLsNTH4I","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Long-sequence modeling","Mamba","Memory mechanisms"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the explosive growth of data, long-sequence modeling has become increasingly important in tasks such as natural language processing and bioinformatics. However, existing methods face inherent trade-offs between efficiency and memory. Recurrent neural networks suffer from gradient vanishing and explosion, making them hard to scale. Transformers can model global dependencies but are constrained by quadratic complexity. Recently, selective state-space models such as Mamba have demonstrated high efficiency with $O(n)$ time and $O(1)$ recurrent inference, yet their long-range memory decays exponentially. In this work, we conduct mathematical derivations and information-theoretic analysis to systematically uncover the memory decay mechanism of Mamba, answering a fundamental question: what is the nature of Mamba’s long-range memory and how does it retain information? To quantify key information loss, we further introduce horizontal–vertical memory fidelity metrics that capture degradation both within and across layers. Inspired by how humans distill and retain salient information when reading long documents, we propose MemMamba, a novel architectural framework that integrates state summarization mechanism together with cross-layer and cross-token attention, which alleviates long-range forgetting while preserving linear complexity. MemMamba achieves significant improvements over existing Mamba variants and Transformers on long-sequence benchmarks such as PG19-PPL and Passkey Retrieval, while delivering a 48\\% speedup in inference efficiency. Both theoretical analysis and empirical results demonstrate that MemMamba achieves a breakthrough in the complexity–memory trade-off, offering a new paradigm for ultra-long sequence modeling. The code and the pre-trained models will be released upon acceptance."},"_bibtex":{"value":"@misc{\nwang2026memmamba,\ntitle={MemMamba: Rethinking Memory Patterns in State Space Model},\nauthor={Youjin Wang and Yangjingyi Chen and Jiahao yan and Jiaxuan Lu and Xiao Sun},\nyear={2026},\nurl={https://openreview.net/forum?id=jPYJuvzof2}\n}"},"title":{"value":"MemMamba: Rethinking Memory Patterns in State Space Model"},"pdf":{"value":"/pdf/26376d032f685fb35659e4ab241730a5eb60a8ff.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wang|memmamba_rethinking_memory_patterns_in_state_space_model"},"authorids":{"value":["~Youjin_Wang1","~Yangjingyi_Chen1","~Jiahao_yan1","~Jiaxuan_Lu1","~Xiao_Sun8"]},"authors":{"value":["Youjin Wang","Yangjingyi Chen","Jiahao yan","Jiaxuan Lu","Xiao Sun"]}},"version":2},{"content":{"summary":{"value":"Despite the promise of the MetaBBO framework, effective generalization is challenging due to the inefficiency of existing online learning methods. This paper presents Q-Mamba, an innovative offline learning MetaBBO approach. \n\nIt features a decomposed Q-function representation to simplify training in extensive joint-action spaces, a Mamba-based Q-Learner for enhanced long-sequence modeling, and CQL regularization for stability. As a result, Q-Mamba performs competitively against current baselines, requires less training budget, and is easily adaptable to new BBO tasks. \n\nAlong with architectural contribution, the authors contribute an offline MetaBBO dataset, named E&E dataset, for the first time."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- In the Q-value head, the MLP takes $[s^t, token(a^t_{i-1})]$ along with decision information $\\mathbb{O}^t_i$. Can you provide any intuition or empirical evidence for this design choice?\n- Can you provide more details about the mamba_block, such as the number of layers used? If only one block of Mamba is used as I understood from the paper, is it fair to compare it with Q-Transformer in terms of model capability? I wonder if we reduce the layer in Q-transformer the performance drops so significantly.\n- Why does the online Q-Mamba approach take significantly longer to train compared to the Transformer-based online approach? Is this due to the introduction of the decomposed Q function? While the massive training time for Q-Transformer is understandable, what is the primary reason for the difference in training time here?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":2},"strengths":{"value":"- Clear and concise writing style, making the content easy to follow and understand.\n- This work makes a well-motivated and reasonable contribution to the optimization domain by addressing practical problems. Introducing an offline RL framework enhances the efficiency of MetaBBO. Additionally, the architectural contribution of Q-mamba further accelerates this efficiency.\n- The baselines are thoughtfully designed to offer insightful comparisons within the rich context of MetaBBO. Core architectures for meta-level control policy are generally investigated in both online and offline setups to ensure a comprehensive evaluation.\n- As far as I know, this is the first investigation of Decomposed Q Learning with Mamba. Also, this paper’s insights and results can also be generalized to the broader offline RL problem. \n- Valuable contribution to the MetaBBO research by providing offline dataset, with a thoughtful design that balances exploration and exploitation by combining high-quality data with randomly generated data."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Typo:\n   - \"very lone sequence\" should be \"very long sequence\" (line 319).\n   - Equation 6 (line 311) contains a typo: $E^t_1 \\to E^t_i$.\n\n- The approach effectively addresses the practical problem, though the architecture designs are somewhat conventional with limited contributions at the framework level. The experiments yield expected results without any surprising findings.\n- The performance of Q-Mamba and the baselines in Table 1 appear quite similar with large variance. So, this might not demonstrate the true efficiency of models. It would be more informative to compare the time (or steps) taken for the model to reach converged performance. Visualizing performance throughout the optimization process, as depicted in Figure 3, could enhance clarity."}},"nonreaders":[],"tmdate":1731427533328,"tcdate":1730472968938,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2058/Reviewer_6kfW"],"signatures":["ICLR.cc/2025/Conference/Submission2058/Reviewer_6kfW"],"forum":"cSgEW7EZ9h","number":2,"license":"CC BY 4.0","cdate":1730472968938,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2058/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427533328,"domain":"ICLR.cc/2025/Conference","replyto":"cSgEW7EZ9h","id":"YpSRNi7ycy","forumContent":{"TLDR":{"value":"A novel MetaBBO framework which achieves both effectiveness and efficiency for auto-configuring black-box optimizer."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Black-Box Optimization","Dynamic Algorithm Configuration","Learning to Optimize","Offline Reinforcement Learning","Mamba"]},"primary_area":{"value":"optimization"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent progress in Meta-Black-Box-Optimization (MetaBBO) has demonstrated that meta-training a neural network based meta-level control policy over an optimization task distribution could significantly enhance the optimization performance of the low-level black-box optimizers. However, achieving such performance enhancement requires effective policy optimization/search method to locate optimal control policy within a massive joint-action space. The online learning fashion of existing works further makes the efficiency of MetaBBO problematic. To address these technical challenges, we propose an offline learning framework in this paper, termed Q-Mamba. Concretely, our method uses a Mamba neural network architecture to meta-learn decomposed Q-functions for each configurable component in the low-level optimizer. By decomposing the Q-function of the configuration decisions of all components in an optimizer, we can apply effective sequence modelling to avoid searching the control policy in the massive joint-action space. Furthermore, by leveraging the long-sequence modelling advantage of Mamba and moderate offline trajectory samples, Q-Mamba can be efficiently trained through a synergy of offline Temporal-Difference update and Conservative Q-Learning regularization to achieve competitive performance against the online learning paradigms. Through extensive benchmarking, we observe that Q-Mamba achieves competitive or even superior optimization performance to prior online/offline learning baselines, while significantly improving the training efficiency of existing online learning baselines. Additional ablation studies show that each of the proposed key designs contributes to this good performance."},"_bibtex":{"value":"@misc{\nma2025metablackboxoptimization,\ntitle={Meta-Black-Box-Optimization through Offline Q-function Learning with Mamba Architecture},\nauthor={Zeyuan Ma and Hongshu Guo and Zhou Jiang and Zhiguang Cao and Yue-Jiao Gong},\nyear={2025},\nurl={https://openreview.net/forum?id=cSgEW7EZ9h}\n}"},"title":{"value":"Meta-Black-Box-Optimization through Offline Q-function Learning with Mamba Architecture"},"pdf":{"value":"/pdf/2f9052c209e77207cb87544eb40cb37b2a701261.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"ma|metablackboxoptimization_through_offline_qfunction_learning_with_mamba_architecture"},"authorids":{"value":["~Zeyuan_Ma1","~Hongshu_Guo1","~Zhou_Jiang2","~Zhiguang_Cao1","~Yue-Jiao_Gong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zeyuan Ma","Hongshu Guo","Zhou Jiang","Zhiguang Cao","Yue-Jiao Gong"]}},"version":2},{"content":{"summary":{"value":"This paper proposes SparseSSM, the first training-free, one-shot pruning framework designed specifically for Mamba-based state-space models."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"This paper proposes SparseSSM for one-shot pruning of Mamba-based models. While the results demonstrate negligible performance degradation under various sparsity levels, the paper does not clearly quantify the corresponding computational savings. It would be valuable to report metrics such as inference latency, energy efficiency, or throughput speedups to substantiate the claimed efficiency benefits."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"This paper clearly presents its method and provides comprehensive numerical results across multiple Mamba checkpoints, demonstrating consistent performance retention under high sparsity levels. The proposed SparseSSM method is conceptually sound, extending the Optimal Brain Surgeon framework to selective state-space models in a theoretically grounded way. The layer-wise Hessian-based importance estimation and time-weighted aggregation strategy are well-motivated and novel in the context of SSMs. Furthermore, the empirical results convincingly show that SparseSSM can prune up to 50% of weights without fine-tuning or noticeable accuracy degradation, which is a strong practical advantage. The method’s ability to generalize across both Mamba-1 and Mamba-2 architectures also highlights its robustness and general applicability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While the proposed SparseSSM method is well-motivated and empirically effective, several important aspects remain insufficiently addressed. First, since Mamba and other state-space models already benefit from efficient recurrent inference, it is unclear whether pruning further improves real-world inference speed or hardware efficiency. The paper does not provide detailed runtime, memory, or energy analyses to substantiate the claimed computational benefits. Second, evaluating performance solely on language modeling and zero-shot benchmarks may not fully capture the potential trade-offs introduced by sparsity—especially regarding numerical stability and robustness under low-precision or long-sequence settings. A more comprehensive discussion of how sparsity affects runtime performance, stability, and generalization beyond standard benchmarks would greatly strengthen the paper’s empirical rigor and practical relevance."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915754093,"tcdate":1762091417857,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1381/Reviewer_bdWM"],"signatures":["ICLR.cc/2026/Conference/Submission1381/Reviewer_bdWM"],"forum":"bzb9OJVfCP","number":5,"license":"CC BY 4.0","cdate":1762091417857,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1381/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915754093,"domain":"ICLR.cc/2026/Conference","replyto":"bzb9OJVfCP","id":"BVu9OIyUJT","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We introduce an OBS–based pruning methodology tailored to SSM-based architectures such as Mamba, effectively reducing redundancy while preserving model performance."},"keywords":{"value":["Neural network pruning","Mamba","Hessian-based pruning"]},"supplementary_material":{"value":"/attachment/6b5f630754dd0367304e126456137bc23a85f5bf.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"State-space language models such as Mamba match Transformer quality while permitting linear complexity inference, yet still comprise billions of parameters that hinder deployment. Existing one-shot pruning methods are tailored to attention blocks and fail to account for the time-shared and discretized state-transition matrix at the heart of the selective state-space module (SSM). In this paper, we introduce SparseSSM, the first training-free pruning framework that extends the classic optimal brain surgeon (OBS) framework to state space architectures. Our layer-wise algorithm (i) derives an approximate second-order saliency score that aggregates Hessian-trace information across time steps, (ii) incorporates a component sensitivity analysis to guide feed-forward network (FFN) pruning, which also sheds light on where redundancy resides in mamba architecture, (iii) can be easily extended to semi-structured and structured sparsity. Empirically, we prune 50% of SSM weights without fine-tuning and observe no zero-shot accuracy loss, achieving the current state-of-the-art pruning algorithm for Mamba-based LLMs."},"_bibtex":{"value":"@misc{\ntuo2026sparsessm,\ntitle={Sparse{SSM}: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot},\nauthor={Kaiwen Tuo and Huan Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=bzb9OJVfCP}\n}"},"title":{"value":"SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot"},"pdf":{"value":"/pdf/8e219971d001aa872a5a0e48ae02be58f8abb56b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"tuo|sparsessm_efficient_selective_structured_state_space_models_can_be_pruned_in_oneshot"},"authorids":{"value":["~Kaiwen_Tuo1","~Huan_Wang3"]},"authors":{"value":["Kaiwen Tuo","Huan Wang"]}},"version":2},{"content":{"summary":{"value":"This paper presents a novel architecture named MEEPO, designed for point cloud segmentation with a focus on efficient contextual modeling. It introduces Mamba, a state-space model (SSM) that achieves linear complexity compared to traditional Transformers, which have quadratic complexity. The authors identify Mamba's limitations, such as enforced causality and directional bias, and propose solutions through causal-free convolutions and bidirectional strided state-space modeling. MEEPO outperforms previous models, like PTv3, in accuracy, latency, and memory efficiency across several benchmark datasets (e.g., ScanNet, S3DIS, nuScenes)."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"please refer to the weakness"},"rating":{"value":6},"details_of_ethics_concerns":{"value":"No"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1.\tMEEPO introduces a new method by integrating CNN and Mamba components, achieving good contextual modeling with reduced computational costs.\n2.\tThe paper thoroughly explores the limitations of existing architectures (CNN, Transformer, and SSM) and provides comprehensive ablations and comparisons to validate its design choices.\n3.\tThe introduction of causal-free convolutions and bidirectional strided SSM in Mamba addresses some original limitations in Mamba for this task,."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tIn Section 4.2, the authors claim that they propose a 'Bidirectional Strided SSM' method, however, Bidirectional SSMs have already been proposed in works like [1][2] to address the issue of forgetting unidirectional sequences. I believe that not referencing these articles while claiming authorship is not rigorous. Additionally, what baffles me is that in Table 7, a comparison is made for strides ranging from 2 to 16, showing a decreasing trend in performance. So, why not similarly compare and discuss the results for a stride of 1?\n2.\tThe manuscript presents in Fig. 4 the mIoU performance of the Transformer-based PTv3 under different range windows. The advantage of the Mamba model lies in its better memory capabilities for long sequences. The authors emphasize this advantage in line 240, but they do not provide specific data to demonstrate the validation of this conclusion in point cloud data  (the author can show a similar format figure as Fig.4).\n3.\tTable 7 in the article indicates that the parameter volume of CNN+Mamba is smaller than that of pure Mamba. However, based on the author's description, compared to pure Mamba, the author has added a CNN module to Mamba and employed a bidirectional mechanism for computation. Why the parameter count is lower than pure Mamba needs further clarification from the author.\n4.\tIn line 251, the referred figure is not Fig.4, it should be Fig. 5.\n\n\n[1] Bi-Mamba+: Bidirectional Mamba for Time Series Forecasting\n[2] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model"}},"nonreaders":[],"tmdate":1731427394589,"tcdate":1730616961818,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1247/Reviewer_wi14"],"signatures":["ICLR.cc/2025/Conference/Submission1247/Reviewer_wi14"],"forum":"E1ML0nEReb","number":3,"license":"CC BY 4.0","cdate":1730616961818,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1247/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427394589,"domain":"ICLR.cc/2025/Conference","replyto":"E1ML0nEReb","id":"OT3HJXaRek","forumContent":{"TLDR":{"value":"This paper analyzes the key components that make a model architecture efficient and effective for point cloud segmentation and explores various methods to implement these components successfully."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["point cloud segmentation","efficient","contextual modeling"]},"supplementary_material":{"value":"/attachment/62537ba3107759f903cfd3e8f9ea61fd79348a56.zip"},"primary_area":{"value":"applications to robotics, autonomy, planning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Point cloud segmentation is an important topic in 3D understanding that has traditionally been tackled using either the CNN or Transformer. Recently, Mamba has emerged as a promising alternative, offering efficient long-range contextual modeling capabilities without the quadratic complexity associated with Transformer's attention mechanisms. However, despite Mamba's potential, early efforts have all failed to achieve better performance than the best CNN-based and Transformer-based methods. In this work, we address this challenge by identifying the key components of an effective and efficient point cloud segmentation architecture. Specifically, we show that: 1) Spatial locality and robust contextual understanding are critical for strong performance, and 2) Mamba features linear computational complexity, offering superior data and inference efficiency compared to Transformers, while still being capable of delivering strong contextual understanding. Additionally, we further enhance the standard Mamba specifically for point cloud segmentation by identifying its two key shortcomings. First, the enforced causality in the original Mamba is unsuitable for processing point clouds that have no such dependencies. Second, its unidirectional scanning strategy imposes a directional bias, hampering its ability to capture the full context of unordered point clouds in a single pass. To address these issues, we carefully remove the causal convolutions and introduce a novel Bidirectional Strided SSM to enhance the model's capability to capture spatial relationships. Our efforts culminate in a novel architecture named MEEPO that effectively integrates the strengths of CNN and Mamba. MEEPO surpasses the previous state-of-the-art method, PTv3, by up to +0.8 mIoU on multiple key benchmark datasets, while being 42.1\\% faster and 5.53$\\times$ more memory efficient. Our code will be released."},"_bibtex":{"value":"@misc{\nchng2025exploring,\ntitle={Exploring contextual modeling with linear complexity for point cloud segmentation},\nauthor={Yong Xien Chng and Xuchong QIU and Henry Zheng and Yizeng Han and Yifan Pu and Jiewei Cao and Gao Huang},\nyear={2025},\nurl={https://openreview.net/forum?id=E1ML0nEReb}\n}"},"title":{"value":"Exploring contextual modeling with linear complexity for point cloud segmentation"},"pdf":{"value":"/pdf/6670e7aa64624951a356099a0f6c5ec34dd15c15.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chng|exploring_contextual_modeling_with_linear_complexity_for_point_cloud_segmentation"},"authorids":{"value":["~Yong_Xien_Chng1","~Xuchong_QIU1","~Henry_Zheng1","~Yizeng_Han1","~Yifan_Pu1","~Jiewei_Cao2","~Gao_Huang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yong Xien Chng","Xuchong QIU","Henry Zheng","Yizeng Han","Yifan Pu","Jiewei Cao","Gao Huang"]}},"version":2},{"content":{"comment":{"value":"**[Reply 3/3]**\n\n**[Q3]. Do your findings and proposed refinements generalize to other selective state space models beyond Mamba?**\n\n**Answer:** Yes, our findings and proposed refinements generalize to other selective state space models beyond Mamba. Indeed, as described in Algorithms 1 and 2 on page 6 of [Gu2023], S4 and its variants (diagonal, diagonal plus low-rank, HIPPO) can be regarded as special cases of Mamba when the input matrix $B$, output matrix $C$, and step sizes $\\Delta$ are input-independent and are chosen carefully. Consequently, our theoretical analysis and the proposed refinements of Mamba encompasses those for S4 and its variants as special cases.\n\n**References**\n[Gu2023] Gu, A., & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752.\n\n-----\nWe hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience."}},"tmdate":1732161054559,"tcdate":1732161054559,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5608/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5608/Authors"],"forum":"qtTIP5Gjc5","number":3,"license":"CC BY 4.0","cdate":1732161054559,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5608/-/Official_Comment"],"mdate":1732161054559,"domain":"ICLR.cc/2025/Conference","replyto":"I6vBIt5mfc","id":"20Vvg7NwFS","forumContent":{"TLDR":{"value":"We describe the dynamical properties of tokens in a deep selective state-space model, and based on that, we suggest improvements to the model by excluding negatively impactful scenarios and reordering tokens using importance scores."},"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Selective state-space model","continuous-time limit","dynamical system","asymptotic behavior","token reordering"]},"supplementary_material":{"value":"/attachment/f01b2b77a31c531086f0fefd1143d89ad8415a39.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity.  We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance.  For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training.  Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance.  Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications."},"_bibtex":{"value":"@inproceedings{\nvo2025demystifying,\ntitle={Demystifying the Token Dynamics of Deep Selective State Space Models},\nauthor={Thieu Vo and Duy-Tung Pham and Xin T. Tong and Tan Minh Nguyen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=qtTIP5Gjc5}\n}"},"title":{"value":"Demystifying the Token Dynamics of Deep Selective State Space Models"},"pdf":{"value":"/pdf/abc1efe1f910ee48b743f7ba26000b1d5e622f85.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"vo|demystifying_the_token_dynamics_of_deep_selective_state_space_models"},"authorids":{"value":["~Thieu_Vo1","~Duy-Tung_Pham1","~Xin_T._Tong2","~Tan_Minh_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Thieu Vo","Duy-Tung Pham","Xin T. Tong","Tan Minh Nguyen"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Mamba-3, an extension of Mamba-2 with a more inference-focused architecture, which introduces three core improvements to its state-space model (SSM) formulation. First, it replaces Euler discretization with a trapezoidal rule, which can offer offers a lower sequence approximation error.  Second, Mamba-3 empowers a complex-valued SSM via a RoPE-like parameterization. The authors actually show the equivalence of complex-valued SSMs to real SSMs with data-dependent rotations on B and C (analogous to keys and values). Third, Mamba-3 introduces a MIMO (multiple-input multiple-output) SSMs that replace single-vector outer products with matrix-matrix updates in order to increase the arithmetic intensity during decoding. The architecture itself includes other modifications, such as QK-style normalization after B and C projections, and removes the short convolution. The paper evaluates 180M-1.5B models trained on 100B FineWeb-Edu tokens (2K context) across language modeling, retrieval (NIAH), formal languages (parity, modular arithmetic), and inference latency. Mamba-3 generally matches or outperforms Mamba-2 and Gated DeltaNet, with notable gains on state-tracking tasks and inference efficiency, particularly with MIMO."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- What regularity assumptions on $\\lambda_t(x_t)$ guarantee quadratic global error? \n\n- Does the rotation-absorption argument (Eq. 18–19) hold when $A_t$ is diagonal rather than scalar? \n\n- If $\\lambda_t = \\frac{1}{2}$, the scheme is 2nd order; if $\\lambda_t$ is data-dependent but Lipschitz and satisfies $\\lambda_t = \\frac{1}{2} + O(\\Delta_t)$, it is still 2nd order; otherwise the order collapses to 1---Euler's rule. Is this collapse ensured in Mamba-3?\n\n- Inference implementation details: For Table 3, specify: (a) kernel type (Triton/CUDA) and fusion level for each baseline, (b) exact configuration (batch size, sequence length, heads, dstate), (c) whether all models use comparable optimization levels.\n\n- Did you try evaluating the 1.5B model (trained at 2K) on 8K–32K retrieval tasks, not just LM perplexity extrapolation (Fig. 5)? The LM curve is nice, but the claim of better sequence approximation would be more convincing on long retrieval where the state is actually stressed.\n\n- Since your goal is inference improvements, could you report prefill (1K-4K) and decode (1 token) wallclock-times for a strong transformer baseline using FlashDecoding (a comparable open kernel on the same hardware is also valuable)? Even if Mamba-3 still wins on decode, showing the transformer prefill margin would make the paper easier for readers.\n\n- Hammering on this again, but why SISO for main LM runs? If MIMO is superior at equal state size (per the inference section), why use SISO for 820M/1.5B language modeling?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The three modifications are grounded in classical SSM theory and integrate cleanly into Mamba-2's framework. The generalized trapezoid recurrence (Eq. 3–4) elegantly recovers Mamba-2 as a special case ($λ_t=1$).\n\n- Propositions 2-4 clearly derive how complex SSMs reduce to real SSMs with RoPE-style rotations. This is a clean way to say that \"we can have complex dynamics at only RoPE's computational cost.\"\n\n- The arithmetic intensity analysis (SISO =  $\\theta(1)$, MIMO = $\\theta(r)$) convincingly motivates MIMO for IO-bound decoding, and the matrix-matrix update is of course implementation-friendly.\n\n- The ablation study in Table 4 is quite valuable. It shows that convolutions become optional is. In addition, tthe formal-language ablation where disabling RoPE makes the model underperform directly validates the complex/RoPE mechanism. This is insightful and may open new future research ideas into incorporating other forms of positional encoding into SSMs.\n\n- Training four model sizes (180M–1.5B) on identical 100B token budgets enables fair cross-model comparisons.\n\n- Near-perfect accuracy on parity (~100%) and modular arithmetic (~98%), plus improvements on bracketed tasks, demonstrate that Mamba-3 successfully addresses state-tracking limitations of Mamba-2."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- All pretraining uses 2K context, and long-sequence tests mostly stay at 2K (NIAH) or only reach 4K (2x extrapolation). For a model claiming better sequence approximation with the trapezoidal rule, evaluation should stress-test extrapolation at longer scenarios (e.g., 32K), especially since the paper notes performance drops where Gated DeltaNet sometimes wins.\n\n- Table 3 reports single-point latencies without variance. Critically, the authors use \"custom kernels\" for Mamba-3 but \"standard reference\" implementations for baselines—making results difficult to verify. Optimized kernels for competitors could reverse the rankings. The paper must specify kernel implementations, fusion levels, and hardware details. Without this, is very hard to access the real efficiency contributions of Mamba-3.\n\n- Figure 3's missing Mamba-3 MIMO point makes the work seem a bit sloppy. Of course, I think the model will perform well $d_{state}=32$, but still find this unconfortable.\n\n- While the main LM results use SISO, the inference analysis argues for MIMO. In the formal language tables there's only a mention to \"Mamba-3.\", so it is unclear which method was used here. Since MIMO affects FLOPs, parameters, and stability, every benchmark should explicitly state \"SISO\" or \"MIMO (rank=r).\"\n\n- For an \"inference-first\" paper, comparisons should include production-grade transformer inference (FlashDecoding, paged KV cache, vLLM chunked prefill, etc) at realistic batch sizes and sequence lengths. Moreover, only decoding latency is reported, despite prefill being an extremelly important part of inference in LLMs. This omission hides important costs.\n\n- Proposition 1 claims $O(\\Delta_t^2)$ global error with data-dependent $\\lambda_t$ but doesn't state required regularity conditions (e.g., Lipschitz continuity).\n\n- I found the paper to be very condensed. It seems like the distance between letters was reduced to make more things fit into the paper. This make things hard to read."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924147918,"tcdate":1761953991690,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13549/Reviewer_2tn7"],"signatures":["ICLR.cc/2026/Conference/Submission13549/Reviewer_2tn7"],"forum":"HwCvaJOiCj","number":1,"license":"CC BY 4.0","cdate":1761953991690,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13549/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924147918,"domain":"ICLR.cc/2026/Conference","replyto":"HwCvaJOiCj","id":"gcdnsFoQQA","forumContent":{"venue":{"value":"ICLR 2026 Oral"},"TLDR":{"value":"Mamba-3, an inference-first SSM that pushes on core SSM principles: improved discretization for better quality, complex dynamics for new capabilities, and MIMO updates for efficient inference."},"keywords":{"value":["State Space Models","Mamba","LLMs","Subquadratic Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While current Transformer models deliver strong quality, their quadratic compute and linear memory requirements make inference expensive. This has spurred the development of sub-quadratic models with reduced compute and constant memory requirements.\nHowever, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice.\nGuided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule enabling richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation that improves model performance without increasing decode latency.\nTogether with architectural refinements, Mamba-3 achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with the MIMO variant further improving accuracy by an additional 1.2 points, for a total gain of 1.8 points.\nAcross state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half the state size. These results demonstrate that Mamba-3 advances the performance–efficiency frontier."},"_bibtex":{"value":"@inproceedings{\nlahoti2026mamba,\ntitle={Mamba-3: Improved Sequence Modeling using State Space Principles},\nauthor={Aakash Lahoti and Kevin Li and Berlin Chen and Caitlin Wang and Aviv Bick and J Zico Kolter and Tri Dao and Albert Gu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=HwCvaJOiCj}\n}"},"title":{"value":"Mamba-3: Improved Sequence Modeling using State Space Principles"},"pdf":{"value":"/pdf/09c67b76b7d1ad254ea30eeba839b8592de9bd78.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lahoti|mamba3_improved_sequence_modeling_using_state_space_principles"},"authorids":{"value":["~Aakash_Lahoti1","~Kevin_Li3","~Berlin_Chen1","~Caitlin_Wang1","~Aviv_Bick1","~J_Zico_Kolter1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Aakash Lahoti","Kevin Li","Berlin Chen","Caitlin Wang","Aviv Bick","J Zico Kolter","Tri Dao","Albert Gu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes PointESS, a 3D point cloud representation framework built on Mamba2 (selective state-space model) to achieve linear-time sequence modeling over ordered point-patch sequences.\nKey components: \n- Gate Adapter: a lightweight gating module for full fine-tuning that blends pretrained and task-adapted features via a learnable sigmoid gate, aiming to stabilize transfer and improve accuracy without PEFT constraints.\n- KH-Norm (KNN-Hybrid Normalization): a geometry-aware normalization/pooling block. Step 1 performs diffusion-style normalization using standardized feature differences between a patch center and its KNN neighbors. Step 2 aggregates with three complementary pools—FeatPool (feature-magnitude attention), DistPool (distance-weighted), and DirPool (direction-aware)—then fuses them.\nIntroducing Mamba2 to point clouds plus geometry-aware normalization and simple task-adaptive gating yields a compact, efficient, and accurate model for 3D classification."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. The Gate Adapter method retains pretrained parameters while performing comprehensive fine-tuning. Can this be interpreted as doubling the number of model parameters? Will this increase time and memory usage?\n\n2. I'm curious whether KH-Norm can be used as a universally applicable module to replace LayerNorms in other backbones, such as Point-MAE."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. First systematic use of Mamba2 for point-cloud representation, with a full pre-train/fine-tune pipeline designed for linear-time sequence modeling.\n\n2. KH-Norm explicitly encodes local geometry.Multi-geometry aggregation combines FeatPool (feature magnitude), DistPool (distance decay), and DirPool (direction attention), capturing relevance, proximity, and orientation cues.\n\n3. SOTA/competitive accuracy on ModelNet40 and all ScanObjectNN subsets with far fewer parameters and FLOPs than large Transformer-based baselines.\n\n4. Clear gains over Mamba-based baselines (PointMamba, Mamba3D) at similar model sizes."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The experiments primarily focus on synthetic datasets like ShapeNet55 and ModelNet40.  The performance on datasets like ScanObjectNN is promising but could be expanded with more diverse real-world scenarios to validate robustness further.\n\n2. The paper provides a comparison with existing models but lacks a deeper analysis of why certain features or strategies within PointESS outperform others. For instance, a more detailed examination of the contributions of the Gate Adapter and KH-Norm compared to traditional methods would strengthen the argument for their effectiveness.\n\n3. The introduction fails to clearly articulate the specific problems that the Gate Adapter and KH-Norm aim to address, while the paper highlights the linear complexity advantages of the SSM. This lack of clarity in motivation may leave readers unsure about the significance of the proposed methods.\n\n4. The absence of detailed evaluations regarding the time and memory consumption of the Gate Adapter and KH-Norm modules raises concerns. It is difficult to assess the practical feasibility of implementing these modules in real-world applications."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917187221,"tcdate":1761834071218,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4123/Reviewer_qFf7"],"signatures":["ICLR.cc/2026/Conference/Submission4123/Reviewer_qFf7"],"forum":"mgDoHyj4YL","number":2,"license":"CC BY 4.0","cdate":1761834071218,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4123/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917187221,"domain":"ICLR.cc/2026/Conference","replyto":"mgDoHyj4YL","id":"OHxn2ePbAO","forumContent":{"TLDR":{"value":"We propose PointESS, the first point cloud representation learning framework based on Mamba2, achieving strong performance with fewer parameters via a novel Gate Adapter and KH-Norm for better transferability and spatial modeling."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Point Cloud","Mamba2","KH-Norm","Gate Adapter","Recent progress in point cloud modeling highlights the need for efficient pretraining strategies and task adaptability. We introduce \\emph{PointESS}","the first framework to leverage the \\emph{Mamba2} architecture for point cloud representation learning. PointESS balances structural modeling with downstream flexibility and incorporates two key components to enhance local structural awareness and fine-tuning performance: Gate Adapter","which dynamically integrates pretrained and task-specific features for improved transferability","and KH-Norm (KNN-Hybrid Normalization)","which embeds geometric priors into normalization and pooling to capture fine-grained spatial details. Extensive experiments on ShapeNet55","ModelNet40","and ScanObjectNN demonstrate that PointESS achieves competitive or state-of-the-art results with substantially fewer parameters","highlighting its effectiveness in both generalization and efficiency.3D Object Classification"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Recent progress in point cloud modeling highlights the need for efficient pretraining strategies and task adaptability. We introduce PointESS, the first framework to leverage the Mamba2 architecture for point cloud representation learning. PointESS balances structural modeling with downstream flexibility and incorporates two key components to enhance local structural awareness and fine-tuning performance: Gate Adapter, which dynamically integrates pretrained and task-specific features for improved transferability, and KH-Norm (KNN-Hybrid Normalization), which embeds geometric priors into normalization and pooling to capture fine-grained spatial details.\nExtensive experiments on ShapeNet55, ModelNet40, and ScanObjectNN demonstrate that PointESS achieves competitive or state-of-the-art results with substantially fewer parameters, highlighting its effectiveness in both generalization and efficiency."},"_bibtex":{"value":"@misc{\ntsai2026pointess,\ntitle={{POINTESS}: {EFFICIENT} 3D {POINT} {CLOUD} {MODELING} {THROUGH} {ENHANCED} {STATE} {SPACE} {NETWORKS} {AND} {FINE}-{TUNING} {STRATEGIES}},\nauthor={Wen-Jiin Tsai},\nyear={2026},\nurl={https://openreview.net/forum?id=mgDoHyj4YL}\n}"},"title":{"value":"POINTESS: EFFICIENT 3D POINT CLOUD MODELING THROUGH ENHANCED STATE SPACE NETWORKS AND FINE-TUNING STRATEGIES"},"pdf":{"value":"/pdf/ce1a3e3103737c7dc0b5d8ed2353e4211c1dfb7b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"tsai|pointess_efficient_3d_point_cloud_modeling_through_enhanced_state_space_networks_and_finetuning_strategies"},"authorids":{"value":["~Wen-Jiin_Tsai1"]},"authors":{"value":["Wen-Jiin Tsai"]}},"version":2},{"content":{"summary":{"value":"The paper innovatively combines the Mamba architecture with Koopman operators, introducing the MamKO framework - the first application of a large model architecture to Koopman modeling and control. The authors effectively leverage Mamba's capabilities in time-varying modeling and improve the matrix generation mechanism by replacing the negative exponential function with CELU activation function, enabling the model to handle unstable systems. Furthermore, based on the generated time-varying Koopman operator, they designed an MPC controller that significantly enhances modeling and control performance while maintaining computational efficiency. In the experimental section, the authors demonstrate MamKO's practical performance across multiple application scenarios (such as CartPole and chemical processes) and compare it with other control methods (like SAC) to validate its superiority."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"To enhance the impact and persuasiveness of this work, the following questions merit consideration:\n\n1. A detailed analysis or proof regarding the theoretical stability and convergence properties of the MamKO framework would be valuable.\n\n2. Validation of selecting the Mamba structure over other SSM-based models (such as S4, H3) would strengthen the paper, along with elaboration on Mamba's mathematical advantages and rationale in control tasks.\n\n3. Could more evidence about the MamKO framework's generalization ability and robustness when handling systems with different dynamic characteristics be supported by experimental data?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":4},"strengths":{"value":"- Innovation: First-time integration of Mamba architecture with Koopman operators, introducing the MamKO framework, pioneering a new application of large models in control theory.\n\n- Performance: Demonstrated superior performance of MamKO-MPC across multiple practical application scenarios, with effective comparisons against existing control methods.\n\n- Model Enhancement: Improved the matrix generation mechanism in Mamba through CELU activation function, enabling the handling of unstable systems and enhancing model robustness.\n\n- Computational Efficiency: The designed MPC controller maintains high computational efficiency while improving modeling and control performance, reducing the computational burden associated with traditional online Koopman methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper has a clear overall structure, but lacks sufficient depth in key areas, particularly in justifying the choice of Mamba over other sequence model architectures. Furthermore, the limited explanation of theoretical background makes it challenging for readers to fully understand the design rationale and advantages of the MamKO framework. Specifically:\n\nTheoretical Analysis Needs Strengthening:\n- Lacks in-depth theoretical analysis of system stability and convergence. As a control method incorporating deep learning, it should provide more comprehensive theoretical guarantees for system stability\n\nInsufficient Model Selection Justification:\n- While the similarity between Mamba's SSM structure and Koopman operators is mentioned, a detailed mathematical comparison between the Mamba SSM structure and Koopman operators lacks and reasons for choosing Mamba over other sequence model architectures based on SSM (like S4, H3) inadequate\n\nLimited Control Task Types:\n- Control tasks primarily focus on setpoint tracking, missing evaluation of other control tasks (trajectory tracking, disturbance rejection, energy optimization)\n\nExperimental Evaluation Metrics Need Expansion:\n- Lacks systematic evaluation of control performance metrics (overshoot, steady-state error), recommend including more control performance indicators to comprehensively demonstrate MamKO-MPC's advantages"}},"nonreaders":[],"tmdate":1731428340564,"tcdate":1730534023739,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10895/Reviewer_PG54"],"signatures":["ICLR.cc/2025/Conference/Submission10895/Reviewer_PG54"],"forum":"hNjCVVm0EQ","number":1,"license":"CC BY 4.0","cdate":1730534023739,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10895/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428340564,"domain":"ICLR.cc/2025/Conference","replyto":"hNjCVVm0EQ","id":"yEvUJmiBkf","forumContent":{"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Koopman operator; model predictive control; nonlinear systems"]},"supplementary_material":{"value":"/attachment/0bf4c76c44568ae8539ede6166bb6a47f0d2be3a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The Koopman theory, which enables the transformation of nonlinear systems into linear representations, is a powerful and efficient tool to model and control nonlinear systems. However, the ability of the Koopman operator to model complex systems, particularly time-varying systems, is limited by the fixed linear state-space representation. To address the limitation, the large language model, Mamba, is considered a promising strategy for enhancing modeling capabilities while preserving the linear state-space structure.\nIn this paper, we propose a new framework, the Mamba-based Koopman operator (MamKO), which provides enhanced model prediction capability and adaptability, as compared to Koopman models with constant Koopman operators. Inspired by the Mamba structure, MamKO generates Koopman operators from online data; this enables the model to effectively capture the dynamic behaviors of the nonlinear system over time. A model predictive control system is then developed based on the proposed MamKO model. The modeling and control performance of the proposed method is evaluated through experiments on benchmark time-invariant and time-varying systems. The experimental results demonstrate the superiority of the proposed approach. Additionally, we perform ablation experiments to test the effectiveness of individual components of MamKO. This approach unlocks new possibilities for integrating large language models with control frameworks, and it achieves a good balance between advanced modeling capabilities and real-time control implementation efficiency."},"_bibtex":{"value":"@inproceedings{\nli2025mamko,\ntitle={Mam{KO}: Mamba-based Koopman operator for modeling and predictive control},\nauthor={ZHAOYANG LI and Minghao Han and Xunyuan Yin},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=hNjCVVm0EQ}\n}"},"title":{"value":"MamKO: Mamba-based Koopman operator for modeling and predictive control"},"pdf":{"value":"/pdf/49097a18837a65cc1415fac755f61c717559a104.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"li|mamko_mambabased_koopman_operator_for_modeling_and_predictive_control"},"authorids":{"value":["~ZHAOYANG_LI12","~Minghao_Han2","~Xunyuan_Yin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["ZHAOYANG LI","Minghao Han","Xunyuan Yin"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We present Causal Relation Networks (CausalRNs), the first all-MLP sequence modeling architecture with linear-time parallel training."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["All-MLP","Sequence Modeling","Multilayer Perceptron","Transformer"]},"supplementary_material":{"value":"/attachment/be16bead4ea44ac8e8dc4346fd0b4b80f8223c83.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We present Causal Relation Networks (CausalRNs), the first all-MLP sequence modeling architecture with linear-time parallel training.\nTo enable autoregressive modeling, we made Relation Networks (RNs) equivariant and causal through relaxation and masking.\nContrary to the earlier belief that RNs are quadratic-time, we show that when using exp(x) as the activation function, any RN is linear-time, fully parallelizable, and numerically stable.\nOur derivation spontaneously gave rise to familiar design choices adopted by state-of-the-art architectures, e.g. exponential gating and state expansion.\nSuch duality provided a new perspective, from which we not only validated popular design choices, but also discovered new design considerations.\nExperiments on autoregressive language modeling and image classification showed CausalRNs to be comparable to Linear Transformers.\nThe quadratic variant of CausalRNs achieved perfect retrieval on the copying task, which was previously only possible with Transformers."},"_bibtex":{"value":"@misc{\ncui2025lineartime,\ntitle={Linear-Time Sequence Modeling with {MLP}s},\nauthor={Chenwei Cui and Zehao Yan and Gedeon Muhawenayo and Hannah Kerner},\nyear={2025},\nurl={https://openreview.net/forum?id=dM1wO2OkbO}\n}"},"title":{"value":"Linear-Time Sequence Modeling with MLPs"},"pdf":{"value":"/pdf/605c012cef5e00685703c72c5fdb43358dd4590a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"cui|lineartime_sequence_modeling_with_mlps"},"authorids":{"value":["~Chenwei_Cui1","~Zehao_Yan3","~Gedeon_Muhawenayo1","~Hannah_Kerner1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chenwei Cui","Zehao Yan","Gedeon Muhawenayo","Hannah Kerner"]}},"tmdate":1738735880046,"tcdate":1727511231723,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13780/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13780/Authors"],"forum":"dM1wO2OkbO","license":"CC BY 4.0","number":13780,"cdate":1727511231723,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission13780/-/Full_Submission","ICLR.cc/2025/Conference/Submission13780/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735880046,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"dM1wO2OkbO","version":2},{"content":{"summary":{"value":"### This paper presents ​Mamba-Branching, a novel neural branching policy for Mixed-Integer Linear Programming (MILP) that leverages the Mamba architecture to model the sequential nature of Branch-and-Bound (B&B) trees. The approach incorporates contrastive learning, which is a novel embedding layer trained with a contrastive loss to improve discriminability between candidate variables, which often have similar features. Experiments on heterogeneous MILP benchmarks (MILP-S and MILP-L) show that Mamba-Branching outperforms previous neural policies."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. During inference, does the input sequence contain the model's own previous predictions or the expert's actions? If it's the latter, how do you justify this setup for a practical scenario?\n2. The results indicate that on standard (\"easy\") instances, all neural branching policies, including Mamba-Branching, still underperform the expert rule-based relpscost method. This raises two important questions: what explains this performance gap, and what would be needed for neural methods to become practically viable?\n3. Why choose the difficult instances shown in Table 17? Is there justification for that, given the numerous difficult instances in MIPLIB 2017?"},"rating":{"value":6},"details_of_ethics_concerns":{"value":"None"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"* This is the first work to explicitly model the sequential nature of B&B using a state-space model (Mamba). The idea of treating the search as a \"branching path\" is intuitive and well-motivated.\n* The method is well-designed, combining contrastive learning for representation learning with sequential modeling for decision-making. The use of Mamba is justified given the long-sequence nature of the problem.\n* The paper provides extensive experiments on real-world benchmarks, including easy and difficult instances. The results are interesting, demonstrating clear improvements over neural baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.  The description of the input sequence is confusing. It appears that at each step t, the model processes all candidate variable embeddings (e_t^1, ..., e_t^|C_t|) plus the expert's chosen variable embedding e_t^{a_t}. This leads to a sequence length of Σ(|C_t| + 1)over T steps. \n    * Concern 1:​​ Why include the expert action e_t^{a_t}in the input during imitation learning? During inference, the expert action is unavailable. This creates a train-test mismatch. The model might be learning to rely on seeing the expert's choice in the history.\n    * Concern 1:​​ The massive sequence length (Σ|C_t| + T) is acknowledged, but it's unclear how this is handled practically. How many tokens are typical for T=100? The paper mentions \"tens of thousands,\" which seems prohibitive even for Mamba. Please provide statistics on average sequence lengths in training/inference.\n2. The comparison to ​Transformer-Branching​ is unfair. Transformer-Branching is trained with only T = 9 steps due to hardware constraints, whereas Mamba-Branching uses T = 100. The performance gap is likely dominated by this difference in historical context rather than the architecture itself.\n3. The rationale for choosing relpscostover strong branching as the expert is crucial but relegated to the appendix. The argument that TreeGate features are more compatible with relpscost is interesting, but needs a more straightforward explanation in the main text \n4. The experimental setting on the difficult problem is limited. I would like to see more challenging instances being solved and compared to obtain a statistically persuasive experiment."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923003691,"tcdate":1761641906362,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12016/Reviewer_usDM"],"signatures":["ICLR.cc/2026/Conference/Submission12016/Reviewer_usDM"],"forum":"iQKzHeLDrN","number":1,"license":"CC BY 4.0","cdate":1761641906362,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12016/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923003691,"domain":"ICLR.cc/2026/Conference","replyto":"iQKzHeLDrN","id":"XQllddFcBO","forumContent":{"TLDR":{"value":"We propose a branching policy that utilizes the sequential branching path of the branch-and-bound tree to guide current decision."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Mixed-Integer Linear Programming","Branch-and-Bound","Branching Variable Selection Policy","Generalization","Mamba"]},"supplementary_material":{"value":"/attachment/92e991cc82f9ad41529abd7aef31e00b1f3c69c9.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"The branch-and-bound (B\\&B) method is a dominant exact algorithm for solving Mixed-Integer Linear Programming problems (MILPs). While recent deep learning approaches have shown promise in learning branching policies using instance-independent features, they often struggle to capture the sequential decision-making nature of B\\&B, particularly over long horizons with complex inter-step dependencies and intra-step variable interactions. To address these challenges, we propose Mamba-Branching, a novel learning-based branching policy that leverages the Mamba architecture for efficient long-sequence modeling, enabling effective capture of temporal dynamics across B\\&B steps. Additionally, we introduce a contrastive learning strategy to pre-train discriminative embeddings for candidate branching variables, significantly enhancing Mamba's performance. Experimental results demonstrate that Mamba-Branching outperforms all previous neural branching policies on real-world MILP instances and achieves superior computational efficiency compared to the advanced open-source solver SCIP. The source code can be accessed at https://github.com/doctor-watson626/Mamba-Branching/."},"_bibtex":{"value":"@inproceedings{\nzhang2026towards,\ntitle={Towards Better Branching Policies: Leveraging the Sequential Nature of Branch-and-Bound Tree},\nauthor={Ce Zhang and Bin Zhang and Guoliang Fan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=iQKzHeLDrN}\n}"},"title":{"value":"Towards Better Branching Policies: Leveraging the Sequential Nature of Branch-and-Bound Tree"},"pdf":{"value":"/pdf/252a128e120299bcb68a697f9200fb14e8927006.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|towards_better_branching_policies_leveraging_the_sequential_nature_of_branchandbound_tree"},"authorids":{"value":["~Ce_Zhang10","~Bin_Zhang12","~Guoliang_Fan3"]},"authors":{"value":["Ce Zhang","Bin Zhang","Guoliang Fan"]}},"version":2},{"content":{"summary":{"value":"This paper introduced BDGO a framework for protein function prediction. It combines Mamba architecture and a interaction module for ontology prediction with pre-training on both the sequence and spatial modules. The authors show marginal performance improvement over some existing sequence based and PPI(protein-protein interaction) network based methods for protein function prediction."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"1. What specific aspect of the proposed model contributed to the performance improvement? The use of ProtT5 against CFAGO is not a fair comparison because  CFAGO does not use protein language model as part of the featurization pipeline. \n2. How does a BOW encoding PPI matrix utilize the space state model proposed ? \n3. Why not include results for EC number prediction in your results? \n4. Why not also benchmark using a sequence similarity based split?\n\n[1]Altschul, Stephen F., et al. \"Basic local alignment search tool.\" *Journal of molecular biology* 215.3 (1990): 403-410.\n[2]Cho, Hyunghoon, Bonnie Berger, and Jian Peng. \"Compact integration of multi-network topology for functional analysis of genes.\" *Cell systems* 3.6 (2016): 540-548.\n[3]Barot, Meet, et al. \"NetQuilt: deep multispecies network-based protein function prediction using homology-informed network similarity.\" *Bioinformatics* 37.16 (2021): 2414-2422.\n[4]Elnaggar, Ahmed, et al. \"Prottrans: Toward understanding the language of life through self-supervised learning.\" *IEEE transactions on pattern analysis and machine intelligence* 44.10 (2021): 7112-7127.\n[5]Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" *arXiv preprint arXiv:2312.00752* (2023).\n[6] Kulmanov, Maxat, et al. \"Protein function prediction as approximate semantic entailment.\" *Nature Machine Intelligence* 6.2 (2024): 220-228.\n[7]Wu, Zhourun, et al. \"CFAGO: cross-fusion of network and attributes based on attention mechanism for protein function prediction.\" *Bioinformatics* 39.3 (2023): btad123."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"1. The authors demonstrated the Mamba architecture combined with protein language model can improve prediction performance on existing sequence based and PPI based methods such as BLAST[1] and interaction network based methods such as Mashup[2] and NetQuilt[3].\n2. Ablation studies showed the importance of the protein language models(ProtT5)[4] as a featurization tool."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Lack of motivation for the proposed architecture. The authors proposed the used of BiMamba as extension of the Mamba[5] architecture. However, no specific motivation was given on how the author hypothesize such architecture’s advantage on modeling protein sequences. very generic terms were used to justified such model selection. ‘’*BiMamba introduces a novel bidirectional selective scanning mechanism designed for protein information, which can take into account both the information at the beginning\nand end of spatial features. This design allows BiMamba to capture details and context information in the spatial features of proteins.*” How does BOW encoding of domain and localization benefits from your “Spatial” encoding? \n2. In the benchmarking effort, the authors only compared with methods that used naive sequence input and PPI network information. However, the proposed method uses protein language model to featurize the input sequence and it’s widely accepted in the community that the use of protein language model and significantly improve model performance[6]. The lack of acknowledgement in the benchmarking effort either shows the authors’ lack of knowledge in the field of protein function prediction or potentially intentional misleading benchmarking setup. \n3. The failure to extend the framework to enzyme function(EC) prediction. It has been the standard for protein function prediction framework to include enzyme function prediction as part of evaluation. The authors failed to include such results.\n4. The authors directly borrowed a dataset from a previous study CFAGO[7] which is a time stamp based split. For machine learning methods, it is very important to check for sequence similarity between the training and testing set to avoid leakage and inflated results. \n5. Overall, this study showed a arbitrary deep learning architecture applied to protein function prediction. The motivation and hypothesis is not well justified for use of such model and the benchmarking effort is lacking in both comprehensiveness and rigor."}},"nonreaders":[],"tmdate":1732493323086,"tcdate":1730233815345,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11120/Reviewer_2RUc"],"signatures":["ICLR.cc/2025/Conference/Submission11120/Reviewer_2RUc"],"forum":"nbia2X0urs","number":1,"license":"CC BY 4.0","cdate":1730233815345,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11120/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732493323086,"domain":"ICLR.cc/2025/Conference","replyto":"nbia2X0urs","id":"SlrmN40466","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Multimodal","protein function prediction","multi-label classification"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Protein function prediction is pivotal for uncovering the mechanisms of life processes. Protein function prediction is a multi-label classification task with numerous functional labels that exhibit hierarchical relationships. Relying solely on unimodal protein features is insufficient for computational models to capture complex protein functions adequately. Recently, several methods for protein function prediction have enhanced the performance by integrating multimodal protein features. However, since multimodal protein features describe protein functions from different perspectives, it is challenging to capture the intricate relationships among these multimodal features with different meanings and heterogeneity. Therefore, we propose a multimodal method for protein function prediction that can effectively utilize the intricate internal relationships between spatial structure features (i.e., protein-protein interaction network, subcellular location, and protein domains) and sequence features (i.e., amino acid sequence). In this work, we introduce the Bidirectional Interaction Module (BInM) to facilitate interactive learning between multimodal features by mapping spatial structure and sequence features of proteins to each other. Moreover, to deal with the difficulty of hierarchical multi-label classification in this task, a multi-branch Dynamic Selection Module (DSM) is designed to select the feature representation that is most favorable for current protein function prediction. Comprehensive experiments on human datasets demonstrate that our model outperforms state-of-the-art multimodal-based methods such as Graph2GO, DeepGraphGO, and CFAGO. Furthermore, we assess the efficacy of the features through Davies-Bouldin scores and t-SNE visualization experiments. The experimental results show that our method constructs more useful protein representations through bidirectional interaction and dynamic selection mechanisms, leading to improved accuracy in protein function prediction. The code in this work will be made public after its acceptance."},"_bibtex":{"value":"@misc{\nluo2025improving,\ntitle={Improving Multimodal Protein Function Prediction Using Bidirectional Interaction and Dynamic Selection Mechanisms},\nauthor={Xiaoling Luo and Peng Chen and Chengliang Liu and Xiaopeng Jin and Jie Wen},\nyear={2025},\nurl={https://openreview.net/forum?id=nbia2X0urs}\n}"},"title":{"value":"Improving Multimodal Protein Function Prediction Using Bidirectional Interaction and Dynamic Selection Mechanisms"},"pdf":{"value":"/pdf/9010bdebbe6341034c939400ca0aaf82322d8709.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"luo|improving_multimodal_protein_function_prediction_using_bidirectional_interaction_and_dynamic_selection_mechanisms"},"authorids":{"value":["~Xiaoling_Luo2","~Peng_Chen29","~Chengliang_Liu1","~Xiaopeng_Jin1","~Jie_Wen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xiaoling Luo","Peng Chen","Chengliang Liu","Xiaopeng Jin","Jie Wen"]}},"version":2},{"content":{"summary":{"value":"The paper introduces Gated Slot Attention (GSA), an enhancement of Gated Linear Attention (GLA), aimed at improving the efficiency of sequence modeling. GSA incorporates a selective gating mechanism to manage memory updates, leveraging a two-pass GLA structure. This approach allows GSA to be more context-aware and to retain a bounded memory footprint, making it suitable for long sequence tasks. \nThe key improvement is the incorporation of a softmax operation to retain sharp attention distributions, which enhances sequence modeling by reducing dilution."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"How does the inference performance of GSA compare to other SOTA models in terms of speed and memory usage?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- The GSA's gated update mechanism and two-pass structure offer a straightforward yet effective enhancement over GLA. This design ensures that memory usage remains bounded and manageable, which is critical for handling long sequences efficiently.\n- The experimental results provided by the authors demonstrate the advantages of the GSA mechanism. The results highlight GSA's ability to improve performance in sequence modeling tasks, validating the practical effectiveness of the proposed approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- While GSA presents improvements, it largely builds on existing GLA techniques. The enhancements, though valuable, might be seen as incremental rather than revolutionary, potentially limiting the perceived impact of the work.\n- Despite efforts to reduce computational overhead, the softmax operation on $QK^T$ still retains a quadratic complexity in training. This raises questions about the authors' claim of linear-time sequence modeling in the title, which could be misleading.\n- The authors do not specify explicitly whether GSA inherits the recurrent architecture of GLA. If this is the case, it would be necessary to compare GSA to other recurrent models like RWKV to provide a comprehensive evaluation of its performance and advantages in sequence modeling tasks."},"limitations":{"value":"Yes"}},"nonreaders":[],"tmdate":1730879745592,"tcdate":1719569187585,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission15000/Reviewer_eYuC"],"signatures":["NeurIPS.cc/2024/Conference/Submission15000/Reviewer_eYuC"],"forum":"jY4PhQibmg","number":1,"license":"CC BY 4.0","cdate":1719569187585,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission15000/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879745592,"domain":"NeurIPS.cc/2024/Conference","replyto":"jY4PhQibmg","id":"s4PaPZFRDb","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["sequence modeling","language modeling","hardware-efficient","linear attention"]},"primary_area":{"value":"natural_language_processing"},"abstract":{"value":"Linear attention Transformers and their gated variants, celebrated for enabling parallel training and efficient recurrent inference, still fall short in recall-intensive tasks compared to traditional Transformers and demand significant resources for training from scratch.\nThis paper introduces Gated Slot Attention (GSA), which enhances Attention with Bounded-memory-Control (ABC) by incorporating a gating mechanism inspired by Gated Linear Attention (GLA).\nEssentially, GSA comprises a two-layer GLA linked via $\\operatorname{softmax}$, utilizing context-aware memory reading and adaptive forgetting to improve memory capacity while maintaining compact recurrent state size.\nThis design greatly enhances both training and inference efficiency through GLA's hardware-efficient training algorithm and reduced state size.\nAdditionally, retaining the $\\operatorname{softmax}$ operation is particularly beneficial in ``finetuning pretrained Transformers to RNNs'' (T2R) settings, reducing the need for extensive training from scratch.\nExtensive experiments confirm GSA's superior performance in scenarios requiring in-context recall and in T2R settings."},"_bibtex":{"value":"@inproceedings{\nzhang2024gated,\ntitle={Gated Slot Attention for Efficient Linear-Time Sequence Modeling},\nauthor={Yu Zhang and Songlin Yang and Rui-Jie Zhu and Yue Zhang and Leyang Cui and Yiqiao Wang and Bolun Wang and Freda Shi and Bailin Wang and Wei Bi and Peng Zhou and Guohong Fu},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=jY4PhQibmg}\n}"},"title":{"value":"Gated Slot Attention for Efficient Linear-Time Sequence Modeling"},"pdf":{"value":"/pdf/1bda433926edbbc8370e798a20b99c748ba22694.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"zhang|gated_slot_attention_for_efficient_lineartime_sequence_modeling"},"authorids":{"value":["~Yu_Zhang36","~Songlin_Yang1","~Rui-Jie_Zhu2","~Yue_Zhang12","~Leyang_Cui1","~Yiqiao_Wang3","~Bolun_Wang1","~Freda_Shi1","~Bailin_Wang3","~Wei_Bi1","~Peng_Zhou10","~Guohong_Fu1"]},"authors":{"value":["Yu Zhang","Songlin Yang","Rui-Jie Zhu","Yue Zhang","Leyang Cui","Yiqiao Wang","Bolun Wang","Freda Shi","Bailin Wang","Wei Bi","Peng Zhou","Guohong Fu"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a novel method for feature matching between pairs of images. It builds upon the coarse-to-fine architectural design proposed in [1], but differs by incorporating a fusion of MAMBA state-space modeling and the attention mechanism to process features at the coarse level. In addition, the proposed model learns an input-dependent weighting factor to adaptively balance the contributions of the two fused components.\n\n[1] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. arXiv preprint arXiv:2104.00680, 2021."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- In Equation (5), g(⋅) denotes global statistics. Could the authors specify which global statistics were used?\n\n- In the experimental section, the authors compare their proposed fusion strategy with a baseline that performs a weighted sum between MAMBA and attention features. However, the distinction between the two approaches is not clearly explained. Does the baseline use a fixed weighting coefficient rather than a learned one? If so, was any hyperparameter search or tuning performed to determine the optimal static weight?\n\n- Since the proposed MAMBA component is causal along a chosen one-dimensional ordering of the two-dimensional feature map, the method may inherit a directional bias. A row-major sequence emphasizes horizontal context while underutilizing vertical dependencies, and vice versa for a column-major sequence. This may reduce rotation robustness and produce anisotropic features. It would useful to conduct an ablation study across different sequence orderings (row, column, snake, or space-filling curves such as Hilbert and Morton), as well as testing bidirectional and multi-axis SSM variants and evaluating rotation and flip robustness. Visualizing the fusion weights and feature spectra would also help clarify whether the attention branch compensates for this bias and whether order-invariant designs (e.g., multi-axis or bidirectional SSM, or order augmentation) are necessary.\n\n- Finally, could the authors justify how their approach achieves better computational efficiency than LoFTR, given that it employs both MAMBA and attention while LoFTR uses only attention in the coarse-level block?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper effectively combines MAMBA and attention to exploit complementary context modeling strategies: attention captures fine-grained spatial relationships, while MAMBA provides smooth, coherent state representations that stabilize long-range dependencies. The input-dependent fusion adaptively balances these components, improving the robustness of coarse-level descriptors and reducing false correspondences in low-texture or ambiguous regions. The learned weighting further enhances generalization by dynamically adjusting between global smoothness and local distinctiveness."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The authors claim that their method improves both the efficiency and accuracy of feature matching compared to approaches relying solely on attention. However, although MAMBA is theoretically more efficient than attention due to its linear-time state updates, the proposed fusion module computes both the attention and MAMBA branches in parallel and combines their outputs through an input-dependent weighting. As a result, the overall computational complexity remains dominated by the quadratic cost of attention, negating the intended efficiency advantage. In practice, the model may even experience higher inference latency and memory usage due to duplicated computations and additional fusion parameters.\n\nThe overall pipeline, including the coarse-to-fine refinement, correspondence formation, and positional encoding strategy, largely mirrors LoFTR. The primary modification is the replacement of the coarse-level transformer with a MAMBA–attention fusion block. While this hybridization is conceptually interesting, it may be viewed as an incremental architectural adjustment rather than a fundamentally new paradigm."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922113879,"tcdate":1761775746529,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10912/Reviewer_MkH5"],"signatures":["ICLR.cc/2026/Conference/Submission10912/Reviewer_MkH5"],"forum":"7lddcnHLCI","number":2,"license":"CC BY 4.0","cdate":1761775746529,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10912/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922113879,"domain":"ICLR.cc/2026/Conference","replyto":"7lddcnHLCI","id":"XEQmF2ckqt","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["SLAM","Feature Matching","State Space Models"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"SLAM (Simultaneous Localization and Mapping) systems depend on front-end components for feature detection and matching. Traditional methods use handcrafted or learning-based features, but both have limitations in robustness and efficiency. We propose a new SLAM front-end framework that fuses recent State Space Models (SSMs), specifically the Mamba architecture, with transformer-based attention. Our method exploits the linear efficiency of SSMs for visual feature processing and the global modeling of attention. By integrating these techniques, we achieve better feature matching on challenging datasets while keeping computation efficient. The fusion strategy adaptively balances local relationships from Mamba and global dependencies from attention. Experiments show our approach surpasses state-of-the-art methods in feature matching precision and recall, especially in scenes with repetitive patterns, lighting and viewpoint shifts."},"_bibtex":{"value":"@misc{\nli2025mambamatch,\ntitle={MambaMatch: {SLAM} Front-End Feature Matching with State Space Models},\nauthor={Shijun Li and Jingping qiao and Jiechao Gao and Yifan Liu and Hengyu Liu and Luyao Tang and Haochen You and Chenxin Li and Zheng Zhu},\nyear={2025},\nurl={https://openreview.net/forum?id=7lddcnHLCI}\n}"},"title":{"value":"MambaMatch: SLAM Front-End Feature Matching with State Space Models"},"pdf":{"value":"/pdf/6a3bf43c34b080e67dbbf216a0a26e52650ec583.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"li|mambamatch_slam_frontend_feature_matching_with_state_space_models"},"authorids":{"value":["~Shijun_Li4","~Jingping_qiao2","~Jiechao_Gao5","~Yifan_Liu8","~Hengyu_Liu2","~Luyao_Tang1","~Haochen_You1","~Chenxin_Li1","~Zheng_Zhu1"]},"authors":{"value":["Shijun Li","Jingping qiao","Jiechao Gao","Yifan Liu","Hengyu Liu","Luyao Tang","Haochen You","Chenxin Li","Zheng Zhu"]}},"version":2},{"content":{"summary":{"value":"This paper presents a novel approach to enhance long-term time-series forecasting (LTSF) through the use of Test-Time Training (TTT) modules. Traditional forecasting models, particularly those based on Transformers, face challenges in managing computational costs and effectively capturing long-range dependencies. To address these limitations, the authors propose integrating TTT modules into the TimeMachine model (a state-of-the-art architecture for LTSF) replacing the Mamba modules for greater adaptability and dependency capture during test time. The authors' primary contributions include the development of a new model architecture with quadruple TTT modules, specifically designed to handle extended sequence and prediction tasks. Through extensive benchmarking on datasets such as Electricity, Traffic, and Weather, the TTT-based model demonstrates notable improvements in forecasting accuracy compared to TimeMachine and other leading models. Additionally, the authors explore various convolutional configurations, such as Conv Stack 5, within the TTT framework, finding that even relatively simple configurations yield significant performance gains. This adaptability and scalability allow the TTT model to outperform competing models, especially when applied to larger prediction and sequence lengths. By establishing TTT as a powerful and scalable solution for LTSF, the paper sets a new benchmark and paves the way for further research into high-performance forecasting models."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"What is the theoretical basis for Test-Time Training’s (TTT) adaptability without catastrophic forgetting? Could the authors elaborate on the theoretical intuition behind why selective parameter updates at test time, as done in TTT, help capture long-term dependencies without leading to catastrophic forgetting? A more detailed theoretical framework or references to relevant works (e.g., continual learning or meta-learning) would be helpful in understanding the robustness of TTT’s adaptation.\n\nHow does TTT handle extreme distribution shifts? In cases where data distributions shift significantly, such as in time-series data with high volatility or sudden anomalies, how does TTT manage to adapt without compromising previous knowledge? Empirical or anecdotal insights on TTT’s performance under extreme shifts would clarify its robustness, as well as the practical limits of test-time adaptation in real-world applications. What is the computational overhead introduced by TTT during test time?\n\nHow does TTT affect computational resources, such as memory usage, latency, and runtime, particularly in high-frequency or large-scale data scenarios? If the authors could quantify the test-time computational costs of TTT relative to baseline models (e.g., Transformer, Mamba), this would provide clarity on the practicality and scalability of TTT for real-time applications. Why were specific convolutional architectures selected for the ablation study?\n\nCould the authors explain their choice of Conv Stack 5 and ModernTCN in the ablation study and provide any reasoning for omitting other architectures, such as dilated or multi-scale convolutions? An understanding of these choices would help readers assess the comprehensiveness of the convolutional architecture exploration and potential gains in performance. How generalizable is TTT to other sequence modeling tasks beyond time-series forecasting?\n\nThe paper demonstrates TTT’s efficacy on time-series forecasting, but to what extent might it be adaptable to other sequential tasks, such as language modeling or event prediction? If generalization is anticipated, the authors could discuss potential adaptations required for TTT in these tasks, or conversely, explain the unique suitability of TTT to time-series data. Does TTT benefit from any specific parameter initialization strategies?\n\nGiven TTT’s adaptation at test time, does it benefit from certain parameter initialization methods, particularly in the hidden states? Clarification on initialization methods, if any, that enhance TTT’s adaptability would give insight into optimal configurations and whether further fine-tuning might improve TTT’s performance.\n\nSuggestions\n\n1. Incorporate a failure case analysis for TTT in extreme settings.\n\nIncluding a discussion or experiment that explores the limitations of TTT in scenarios with sudden distribution shifts or heavy noise would provide a more balanced perspective. If the authors have preliminary findings on failure cases, this could help readers understand when TTT might be less effective, along with areas for potential improvement.\n\n2. Expand on the choice of hierarchical two-level context modeling.\n\nThe hierarchical design in TTT for high and low-resolution contexts is an innovative approach, but could the authors provide further intuition or references for why this structure specifically benefits time-series forecasting? This would clarify how hierarchical cues improve TTT’s ability to model long-term dependencies and adapt to multiscale patterns within time-series data.\nConsider additional benchmarks with noise robustness or temporal regularization methods.\n\nTo further validate TTT’s position relative to the state of the art, additional comparisons with models that handle noise or temporal regularization (e.g., DeepAR, TCN) could strengthen empirical findings. If direct comparisons are impractical, a discussion on how TTT theoretically or practically compares with these methods would be informative.\nAdd computational efficiency metrics in experiments.\n\nIncluding memory consumption, training time, and inference time as part of the experimental results would help assess TTT’s trade-offs in resource utilization. If TTT is found to be resource-intensive, the authors could discuss strategies for balancing adaptability and efficiency, especially in real-time or resource-constrained environments.\n\n3. Discuss potential applications of TTT in real-world systems.\n\nAn analysis of how TTT could be applied in real-world scenarios, such as financial prediction or adaptive traffic monitoring, could showcase its practical relevance and advantages. Simulated or hypothetical case studies would also help highlight TTT’s robustness and adaptability. Consider further discussion of generalization beyond time-series forecasting.\n\nIf TTT’s applicability is anticipated for other sequence modeling domains, discussing any potential adaptations (or limitations) of TTT would be valuable. If generalization is limited, clarifying why TTT is best suited to time-series forecasting would help contextualize its strengths."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Originality :\n\nThe paper demonstrates a high degree of originality by introducing Test-Time Training (TTT) modules specifically for time-series forecasting. This approach innovatively allows models to adapt during testing, a novel contribution in the field of forecasting where most models rely solely on training data and fixed parameters during inference. By integrating TTT modules with linear RNNs and exploring contextual hierarchies (high-resolution and low-resolution levels), the paper provides a fresh approach to capturing long-term dependencies—a significant challenge in time-series forecasting. The use of TTT for dynamic adaptation in response to evolving data patterns, especially without altering primary model parameters, marks a clear advancement over traditional models. It can be elaborated into :\n\nTest-Time Training (TTT) Module: This module enables weight updates during the testing phase, allowing the model to adapt to varying data patterns. This makes TTT effective in capturing complex patterns for long-term predictions.\n\nPerformance Improvement: Through experiments on benchmark datasets such as Electricity, Traffic, and Weather, the TTT-based model shows significant performance gains compared to other state-of-the-art models like Transformer-based models and Mamba. Performance indicators, including Mean Squared Error (MSE) and Mean Absolute Error (MAE), demonstrate that the TTT model is better suited for predictions with longer sequence lengths and prediction horizons.\n\nAblation Study on Convolutional Architecture: This paper also explores several convolutional architecture configurations within the TTT framework, such as Conv Stack 5 and ModernTCN. The results indicate that simpler convolutional configurations can deliver competitive results without adding unnecessary complexity.\n\n2. The paper’s quality is above average, particularly in its extensive and comprehensive empirical validation across multiple benchmark datasets, such as Electricity, Traffic, and Weather. The experimental design is robust, including comparisons with state-of-the-art (SOTA) models like Transformer and Mamba-based architectures. Additionally, the ablation study exploring different convolutional architectures (e.g., Conv Stack 5, ModernTCN) within the TTT framework shows careful attention to model component effectiveness and efficiency. Furthermore, performance metrics, including Mean Squared Error (MSE) and Mean Absolute Error (MAE), provide clear evidence of the model’s capabilities and improvements over existing models, adding rigor to the evaluation.\n\n3. The clarity of the paper is commendable, especially in presenting the layered architecture of TTT modules and explaining the two-level hierarchy for capturing contextual cues at high and low resolutions. This structuring aids in understanding the model’s mechanics and how it addresses long-range dependencies. The presentation of channel mixing and independence modes is also clear, helping readers comprehend the adaptation of the model to multivariate time-series data. However, there may still be room to expand theoretical explanations of TTT’s advantages, particularly regarding why selective adaptation during test time enhances stability and adaptability, which could further elevate clarity for a broader audience.\n\n4. The significance of this work is substantial due to its relevance in advancing time-series forecasting, particularly for long-term forecasting tasks that require handling complex temporal dependencies across large data streams. Given the growing use of time-series forecasting in fields like energy management, traffic monitoring, and financial prediction, a model that can adapt dynamically to new patterns has a wide range of practical applications. Moreover, by maintaining model efficiency through selective parameter updating during testing, TTT addresses a pressing need for models that can scale with real-time demands while remaining computationally feasible. This innovation not only pushes forward the boundaries in time-series forecasting but could also inspire similar adaptive techniques in other sequential modeling domains."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Theoretical Justification and Intuition\n\nWhile the empirical results demonstrate TTT’s effectiveness, the paper lacks an in-depth theoretical justification for why selective adaptation during testing works effectively for capturing long-term dependencies. The proposed model could benefit from a more comprehensive discussion of the underlying mechanisms of TTT and why it succeeds in enhancing the model's ability to adapt to long-range dependencies without suffering from catastrophic forgetting. Strengthening the theoretical foundation with a mathematical analysis of TTT’s adaptability, particularly how it avoids catastrophic forgetting during selective adaptation, could be a valuable addition. It would be beneficial to reference existing theoretical frameworks on continual learning (e.g., Kirkpatrick et al., 2017 on Overcoming catastrophic forgetting in neural networks) and contrast how TTT’s unique selective adaptation minimizes memory interference, particularly in long-term sequence contexts.\n\n2. Computational Complexity and Practicality\n\nWhile TTT shows strong empirical performance, the paper does not sufficiently discuss the computational overhead associated with adapting the model during testing. Test-time training requires frequent parameter updates, which could become computationally expensive, especially for real-time applications or in settings with limited resources. There is also no analysis comparing the memory and processing requirements of TTT versus the baseline models, making it difficult to assess the model’s practical feasibility. It would be better to include a detailed analysis of computational efficiency, such as memory consumption, latency, and runtime, particularly in high-frequency and high-volume time-series contexts. If TTT proves resource-intensive, consider exploring optimization strategies, like limiting parameter updates to specific intervals or only for data points that deviate significantly from training data, which could enhance efficiency without sacrificing adaptability.\n\n3. Insufficient Exploration of Convolutional Variants\n\nThe paper does conduct ablation studies on convolutional configurations (e.g., Conv Stack 5 and ModernTCN), but these variations remain relatively limited and do not fully explore convolutional architectures that could better capture multiscale temporal dependencies. As a result, it remains uncertain whether the TTT-based models are achieving optimal use of local temporal patterns within each channel and across channels in the multivariate context. We suggest broadening the convolutional architecture exploration to include dilated convolutions or multi-resolution features, which could potentially improve performance on long-term dependencies. Additionally, incorporating a study on whether certain convolutional structures offer better adaptability with TTT could provide deeper insights into architectural choices that optimize TTT’s efficacy.\n\n4. Overlooked Explanation of Failure Cases\n\nThe paper does not discuss scenarios where TTT might underperform or fail to deliver the expected benefits, which could be valuable for understanding limitations and areas for further improvement. For instance, if the model encounters unexpected shifts in data distributions or operates in domains with high-frequency noise, its adaptive capabilities may struggle, impacting overall performance. Adding a failure case analysis or discussion of situations where TTT might be less effective would be beneficial for readers. An evaluation of TTT’s robustness against significant distribution shifts or noise would provide a more balanced perspective. If these analyses indicate limitations, the authors could suggest future research directions, such as integrating TTT with methods for noise-robust learning or investigating methods to dynamically adjust the degree of test-time adaptation based on input characteristics.\n\n5. Benchmark Comparisons and Real-World Simulation\n\nWhile TTT is validated on benchmark datasets, these may not fully represent real-world complexities, such as fluctuating conditions and anomalies in time-series data. Also, TTT’s performance is primarily compared with Mamba-based and Transformer models, but comparisons with state-of-the-art time-series forecasting methods that employ different adaptation or regularization mechanisms would help place TTT’s effectiveness in context. Extending the experiments to include real-world or more challenging datasets that capture diverse time series behaviors, such as anomalies or irregular patterns, would effectively stress-test TTT’s adaptability. Including comparisons with models that use temporal regularization (e.g., DeepAR, TCN) or adversarial learning techniques would also provide additional benchmarks to validate TTT’s position as a state-of-the-art adaptive method."}},"nonreaders":[],"tmdate":1731428725699,"tcdate":1730706103848,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12519/Reviewer_neWB"],"signatures":["ICLR.cc/2025/Conference/Submission12519/Reviewer_neWB"],"forum":"WWymYrA48K","number":3,"license":"CC BY 4.0","cdate":1730706103848,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12519/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428725699,"domain":"ICLR.cc/2025/Conference","replyto":"WWymYrA48K","id":"amF0Xqer1r","forumContent":{"TLDR":{"value":"Test-Time Learning Applied To Long Time Series Forecasting"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forecasting","Test-Time Training","Mamba","Expressive Hidden States","Modern CNN"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We propose the use of Test-Time Training (TTT) modules in a cascade architecture to enhance performance in long-term time series forecasting. Through extensive experiments on standard benchmark datasets, we demonstrate that TTT modules consistently outperform state-of-the-art models, including Mamba-based TimeMachine, particularly in scenarios involving extended sequence and prediction lengths. Our results show significant improvements, especially on larger datasets such as Electricity, Traffic, and Weather, underscoring the effectiveness of TTT in capturing long-range dependencies. Additionally, we explore various convolutional architectures within the TTT framework, showing that convolutional blocks as hidden layer architectures can achieve competitive results."},"_bibtex":{"value":"@misc{\nchristou2025test,\ntitle={Test Time Learning for Time Series Forecasting},\nauthor={Panayiotis Christou and Shichu Chen and Xupeng Chen and Parijat Dube},\nyear={2025},\nurl={https://openreview.net/forum?id=WWymYrA48K}\n}"},"title":{"value":"Test Time Learning for Time Series Forecasting"},"pdf":{"value":"/pdf/810930cf247905bae4385ec4c728ed2cea6f2c24.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"christou|test_time_learning_for_time_series_forecasting"},"authorids":{"value":["~Panayiotis_Christou1","~Shichu_Chen1","~Xupeng_Chen1","~Parijat_Dube1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Panayiotis Christou","Shichu Chen","Xupeng Chen","Parijat Dube"]}},"version":2},{"content":{"venue":{"value":"ICLR 2026 Oral"},"TLDR":{"value":"Mamba-3, an inference-first SSM that pushes on core SSM principles: improved discretization for better quality, complex dynamics for new capabilities, and MIMO updates for efficient inference."},"keywords":{"value":["State Space Models","Mamba","LLMs","Subquadratic Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While current Transformer models deliver strong quality, their quadratic compute and linear memory requirements make inference expensive. This has spurred the development of sub-quadratic models with reduced compute and constant memory requirements.\nHowever, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice.\nGuided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule enabling richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation that improves model performance without increasing decode latency.\nTogether with architectural refinements, Mamba-3 achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with the MIMO variant further improving accuracy by an additional 1.2 points, for a total gain of 1.8 points.\nAcross state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half the state size. These results demonstrate that Mamba-3 advances the performance–efficiency frontier."},"_bibtex":{"value":"@inproceedings{\nlahoti2026mamba,\ntitle={Mamba-3: Improved Sequence Modeling using State Space Principles},\nauthor={Aakash Lahoti and Kevin Li and Berlin Chen and Caitlin Wang and Aviv Bick and J Zico Kolter and Tri Dao and Albert Gu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=HwCvaJOiCj}\n}"},"title":{"value":"Mamba-3: Improved Sequence Modeling using State Space Principles"},"pdf":{"value":"/pdf/09c67b76b7d1ad254ea30eeba839b8592de9bd78.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lahoti|mamba3_improved_sequence_modeling_using_state_space_principles"},"authorids":{"value":["~Aakash_Lahoti1","~Kevin_Li3","~Berlin_Chen1","~Caitlin_Wang1","~Aviv_Bick1","~J_Zico_Kolter1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Aakash Lahoti","Kevin Li","Berlin Chen","Caitlin Wang","Aviv Bick","J Zico Kolter","Tri Dao","Albert Gu"]}},"tmdate":1775877156905,"pdate":1769436159601,"tcdate":1758219116250,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13549/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13549/Authors"],"forum":"HwCvaJOiCj","license":"CC BY 4.0","number":13549,"cdate":1758219116250,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission13549/-/Full_Submission","ICLR.cc/2026/Conference/Submission13549/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission13549/-/Camera_Ready_Revision"],"mdate":1775877156905,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"HwCvaJOiCj","version":2},{"content":{"summary":{"value":"1.\tThis paper presents VF-Mamba, a novel semantic segmentation model for remote sensing imagery that introduces a saliency-driven scanning (SDS) mechanism into the Mamba state space model framework. \n2.\tThe core idea is to replace the fixed, linear scanning order of standard Mamba with a dynamic, content-aware order that prioritizes salient image patches, mimicking human visual perception. \n3.\tThe authors propose two variants of SDS—Full-SDS and Sparse-SDS—to balance precision and computational cost. The method is evaluated on two standard ISPRS datasets (Potsdam and Vaihingen), demonstrating competitive or superior performance compared to several state-of-the-art models, particularly on rare and challenging object categories."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1.  Could you provide a quantitative analysis (e.g., wall-clock time, FLOPs) of the computational overhead introduced by Full-SDS and Sparse-SDS compared to the base scanning mechanism?\n2.  The restoration of the original order after the S6 block is a key step. Could you elaborate on how the hidden states from the scrambled sequence are mapped back to the original 2D spatial layout?\n3.  Have you observed any failure cases or scenarios where the SDS mechanism does not lead to improvement, or even harms performance?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1.  The paper's central contribution—adapting Mamba's scanning order based on visual saliency—is innovative and well-grounded. The analogy to the human visual system is compelling and directly addresses a known limitation (fixed scanning order) of existing Mamba-based vision models. This is a meaningful step towards more spatially adaptive and efficient sequence modeling for images.\n2.  The proposed SDS mechanism is elegantly designed. Using the Wasserstein Distance (WD) to measure distributional distinctiveness is a sound choice for remote sensing data, as it is more robust to complex backgrounds and geometric structures than simpler metrics like Euclidean distance. The introduction of two strategies (Full-SDS for accuracy, Sparse-SDS for efficiency) shows thoughtful consideration of practical deployment scenarios.\n3. The paper provides comprehensive experimental results. The method achieves state-of-the-art or highly competitive results on both benchmark datasets, with a notable improvement on challenging categories. The consistent outperformance of other Mamba variants (VMamba, DefMamba, UNetMamba) strongly validates the effectiveness of the proposed adaptive scanning.\n4.The visual results effectively demonstrate the model's strength in preserving fine details, geometric regularity, and boundary integrity, especially for small objects like vehicles.\n5.The ablation study in Table 3 is excellent. It convincingly isolates the contribution of the SDS module by comparing against random, linear, and Euclidean-based scanning, and validates the design choice of using a learned Gaussian reference in Sparse-SDS."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.  While the paper correctly states that Full-SDS has higher complexity (O(L²)) than Sparse-SDS (O(L)), a more detailed analysis is needed. Reporting actual training/inference times, FLOPs, or memory usage for both variants compared to a baseline (e.g., VMamba) would provide a much clearer picture of the efficiency trade-off. This is crucial for readers to choose between Full-SDS and Sparse-SDS in practice.\n2.  The link between distributional distinctiveness and semantic saliency is asserted but could be more deeply explored. While the results support the claim, a brief analysis or visualization (e.g., showing the top salient patches selected by SDS) could provide more intuitive evidence that the mechanism is indeed prioritizing semantically important regions like rare objects, rather than just statistically anomalous background patches.\n3.  The description of how the saliency-sorted sequence is fed into the S6 block and how the original order is \"restored based on the indices\" is somewhat brief. A more detailed explanation or a reference to how this is compatible with the causal nature of the Mamba SSM would be helpful.\n4.  The discussion could be strengthened by explicitly stating the limitations of the proposed approach. For instance, how does the performance of Sparse-SDS degrade with highly non-Gaussian data? Is there a risk of the model becoming over-specialized on the specific statistical properties of the training data?\n5.While comparison to other remote sensing Mamba models is good, a brief comparison with a recent adaptive/computationally efficient Transformer (e.g., a specific linear attention variant) could better situate the contribution within the broader field beyond Mamba-family models."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918982842,"tcdate":1761787503608,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6685/Reviewer_jEnJ"],"signatures":["ICLR.cc/2026/Conference/Submission6685/Reviewer_jEnJ"],"forum":"5rbd5nvdPv","number":2,"license":"CC BY 4.0","cdate":1761787503608,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6685/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918982842,"domain":"ICLR.cc/2026/Conference","replyto":"5rbd5nvdPv","id":"BS0XZtj6MH","forumContent":{"TLDR":{"value":"we propose Visual Focus Mamba (VF-Mamba), which introduces a Saliency-Driven Scanning (SDS) mechanism to guide adaptive, content-aware processing."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Semantic segmentation","Mamba Model","Attention Mechanism","Remote Sensing"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"While Mamba models offer efficient global modeling, fixed scanning limits adaptation in complex scenes. This limitation is critical for remote sensing segmentation, where fine-grained understanding of rare objects is required. In this paper, we propose Visual Focus Mamba (VF-Mamba), a saliency-guided state space model that scans informative regions first. Our VF-Mamba is inspired by the phenomenon that the human visual system prioritizes salient regions for progressive scene interpretation. Based on this principle, we develop Saliency-Driven Scanning (SDS), which ranks patches according to their distributional distinctiveness. The underlying foundation of our approach is that samples exhibiting substantial distributional disparities are to be classified as either sparse categories or regions of interest, thus prioritizing scanning them can improve segmentation performance. For the purposes of enhancing robustness and efficiency, respectively, two strategies are hereby proposed: Full-SDS employs pairwise Wasserstein distances for precise measurement, while Sparse-SDS utilises a Gaussian reference for efficient approximation. The efficacy of VF-Mamba is evaluated using two benchmark datasets. The results demonstrate higher accuracy and better preservation of fine details in small classes compared to state-of-the-art methods, achieved a 3.28\\% IoU improvement on challenging categories."},"_bibtex":{"value":"@misc{\nma2025vfmamba,\ntitle={{VF}-Mamba: See What Matters First for Remote Sensing Semantic Segmentation},\nauthor={Jitao Ma and Qingchen Zeng and Weiying Xie and Ruofan Jia and Daixun Li and Yunsong Li and Leyuan Fang},\nyear={2025},\nurl={https://openreview.net/forum?id=5rbd5nvdPv}\n}"},"title":{"value":"VF-Mamba: See What Matters First for Remote Sensing Semantic Segmentation"},"pdf":{"value":"/pdf/f09bc95a73d09e7fe7942edf5bb1a7a6ae4a5e87.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"ma|vfmamba_see_what_matters_first_for_remote_sensing_semantic_segmentation"},"authorids":{"value":["~Jitao_Ma1","~Qingchen_Zeng1","~Weiying_Xie1","~Ruofan_Jia1","~Daixun_Li1","~Yunsong_Li1","~Leyuan_Fang1"]},"authors":{"value":["Jitao Ma","Qingchen Zeng","Weiying Xie","Ruofan Jia","Daixun Li","Yunsong Li","Leyuan Fang"]}},"version":2},{"content":{"summary":{"value":"The paper introduces autoregressive pretraining to the domain of vision mamba models. Drawing inspiration from \"Scalable Pre-training of Large Autoregressive Image Models\" (AIM), it shows how the method naturally adapts to Mamba models, which are inherently autoregressive, at least in their simplest form. This pretraining method shows positive scaling traits, improving downstream classification accuracy, and also allowing for larger variants to be trained than previously possible."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"4.4.4 Which normalization method are you referring to for pixel targets? I’m guessing you’re referring to MAE’s [3] method, but this needs to be explicitly cited.\n\nDo you have any experiments where you just vary the random seed to get a sense of the variance in model quality just due to randomness? For example, there are 6 examples of the word “significant”, but I assume this is not statistical significance. Taking table 4 as an example, there is a spread of 0.8% between a 1x1 cluster and a 4x4 cluster. Knowing the variance of your method on Top-1% would be really helpful for determining whether these difference are actually significant.\n\n[3] Masked Autoencoders Are Scalable Vision Learners (http://arxiv.org/abs/2111.06377)"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The paper finds an effective way to apply AIM to the space of vision-mamba variants, demonstrating that it improves the pareto frontier for mamba-based vision models. The pretraining unlocking the use of larger vision-mamba variants is a useful outcome."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"At a high level, the novelty in this paper is quite limited. It appears to mostly be “AIM [1], but with Mamba”. Similarly, I think table 2 needs to include AIM; I know this is difficult because they don’t provide finetuned accuracy numbers (only probes), but they do provide the model checkpoints, so even a casual attempt at finetuning their 0.6B model on imagenet would be helpful to contextualize on the huge models.\n\nGiven that ARM isn’t bringing anything new to the theory of SSSMs or autoregressive pretraining, it doesn’t feel like equations 1-6 are necessary, or at least could be greatly condensed [1, 2], . Particularly given the degree to which this paper builds on AIM, leveraging their section 4.1 would be better. Similarly, equations 7-9 are highly redundant. Distilling all of this down, all of the equations presented in this paper don’t seem to aid in clarity.\n\n“Prediction units”. This is defined in 3.2.2 as generally being patches, aside from iGPT which uses pixels. This seems to be reinforced by figure 2d and figure 3. However, section 4.4.1 confusingly switches to clusters being prediction units. Based on the figures, it seems like a patch is always a prediction unit. But a good amount of the paper is spent on processing order, with a distinction between tokens and clusters. Equation 9 seems to suggest that a full cluster is predicted simultaneously. Presumably ARM is patch based though, so it seems like clusters are mostly just a way to define a particular order of patches, and due to scan being an enforced sequential algorithm, a cluster isn’t actually predicted simultaneously. This would make figure 3 make sense.\n\nFigure 4: I’m struggling to find anywhere where the switch from 1 scan to 4 scans is justified. It would be good to at least demonstrate the difference in accuracy between 1 scan and 4 in your setting.\n\nTable 4 has clarity issues. It would be helpful to directly specify the cluster size instead of having it be $\\frac{cluster\\\\_size}{patch\\\\_size}$.\n\n[1] Scalable Pre-training of Large Autoregressive Image Models (http://arxiv.org/abs/2401.08541)\n\n[2] Mamba: Linear-Time Sequence Modeling with Selective State Spaces (https://arxiv.org/abs/2312.00752)"}},"nonreaders":[],"tmdate":1732566634015,"tcdate":1729724713240,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission7773/Reviewer_Xz8m"],"signatures":["ICLR.cc/2025/Conference/Submission7773/Reviewer_Xz8m"],"forum":"PQpvhUrA1C","number":1,"license":"CC BY 4.0","cdate":1729724713240,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission7773/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732566634015,"domain":"ICLR.cc/2025/Conference","replyto":"PQpvhUrA1C","id":"9G3MSNO2Rt","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Auto regressive Pretraining"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The vision community has started to build with the recently developed state space model, Mamba, as the new backbone for a range of tasks. This paper shows that Mamba's visual capability can be significantly enhanced through autoregressive pretraining, a direction not previously explored. Efficiency-wise, the autoregressive nature can well capitalize on the Mamba's unidirectional recurrent structure, enabling faster overall training speed compared to other training strategies like mask modeling. Performance-wise, autoregressive pretraining equips the Mamba architecture with markedly higher accuracy over its supervised-trained counterparts and, more importantly, successfully unlocks its scaling potential to large and even huge model sizes. For example, with autoregressive pretraining, a base-size Mamba attains 83.2\\% ImageNet accuracy, outperforming its supervised counterpart by 2.0\\%; our huge-size Mamba, the largest Vision Mamba to date, attains 85.0\\% ImageNet accuracy (85.5\\% when finetuned with $384\\times384$ inputs), notably surpassing all other Mamba variants in vision. The code is available at \\url{https://github.com/OliverRensu/ARM}."},"_bibtex":{"value":"@inproceedings{\nren2025autoregressive,\ntitle={Autoregressive Pretraining with Mamba in Vision},\nauthor={Sucheng Ren and Xianhang Li and Haoqin Tu and Feng Wang and Fangxun Shu and Lei Zhang and Jieru Mei and Linjie Yang and Peng Wang and Heng Wang and Alan Yuille and Cihang Xie},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=PQpvhUrA1C}\n}"},"title":{"value":"Autoregressive Pretraining with Mamba in Vision"},"pdf":{"value":"/pdf/49a1ba5ebe67bb002c241e051bd28e213cefe85a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"ren|autoregressive_pretraining_with_mamba_in_vision"},"authorids":{"value":["~Sucheng_Ren1","~Xianhang_Li1","~Haoqin_Tu1","~Feng_Wang14","~Fangxun_Shu1","~Lei_Zhang47","~Jieru_Mei2","~Linjie_Yang4","~Peng_Wang2","~Heng_Wang2","~Alan_Yuille1","~Cihang_Xie3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Sucheng Ren","Xianhang Li","Haoqin Tu","Feng Wang","Fangxun Shu","Lei Zhang","Jieru Mei","Linjie Yang","Peng Wang","Heng Wang","Alan Yuille","Cihang Xie"]}},"version":2},{"content":{"summary":{"value":"1. The paper presents *LightNet*, a new variant of State Space Models (SSMs) incorporating an additive decay/selectivity parameter. Unlike Mamba or Linear Attention (LA) with decay, LightNet avoids recursion-based parallelization, achieving faster parallelization using only matrix multiplications, similar to LA without decay.\n\n2. To address permutation invariance in non-causal settings (a characteristic it shares with LA), the authors introduce two multi-dimensional positional encodings MD-TPE and MD-LRPE.\n\n3. The model is evaluated across diverse tasks, including autoregressive and bidirectional language modeling, image classification and generation."},"soundness":{"value":4},"confidence":{"value":5},"questions":{"value":"1. In the paper you do claim that you have results on 1B language modeling as well. Could you please share them?\n2. Could you also compare against vanilla LA especially on larger scales. This helps prove/disprove my hypothesis.\n3. Could the authors please explain how is the method implemented through matrix multiplications in the causal setting? I am not able to figure out the vectorized formula for the same that computes in O(Sequence Length) time.\n4. Could the authors please clarify what they mean by \"parameter tiling strategy\" in Line 266, and if and why this is not implemented for non-causal settings?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. The authors identify that SSM-based methods like Mamba and LA with decay suffer from slow GPU performance due to recursion-based parallelization. In contrast vanilla LA is fully parallelizable via matrix multiplications but underperforms other SOTA SSMs.\n \n2. To introduce decay to improve LA's performance while retaining full parallelizability, they derive the mathematical formulation of the linear operation induced by a decayed recurrence and using this they motivate an “Additive” decay mechanism, that allows for a decay with matrix multiplication-based parallelization.\n\n3. They evaluate their model across a broad range of tasks and demonstrate near competitive performance against SOTA methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**1. Saturation of Additive Decay for Long Sequences**\n\nThe proposed additive decay mechanism, defined as $g(t) = \\sum_{s=1}^{t} \\delta(s)$, tends to saturate for long sequences. As the decay accumulates $\\delta$ values over the course of a long sequence, $g(t)/g(s)$ reaches a point where it can no longer distinguish between nearby tokens effectively. This may be problematic for language modeling as it has a strong local bias.\n\nThis saturation effect will likely be more pronounced with very long sequences or on large-scale models. **I am curious how LightNet might perform compared to Mamba in this setting**, which has the opposite issue: difficulty paying attention to very distant tokens despite selectivity. I suspect LightNet may struggle with paying attention locally at scale, potentially leading to underperformance on long sequences.\n\n**2. Comparing LA and LightNet on Long Sequences**\n\nIn line with the reasoning above, I feel that LA and LightNet may have similar performance at scale. This is due to LightNet's decay factor saturating to 1. To better understand this, an ablation study comparing LightNet directly with Linear Attention, keeping all other factors constant, would be useful. Specifically, I suggest replacing the additive decay with vanilla LA’s $\\phi(K)$ (substituting `softmax(K)` with $\\phi(K)$). Additionally, scaling up model sizes could reveal whether the performance gap between LightNet and Linear Attention narrows, providing insight into the effectiveness of additive decay.\n\n\n**3. It is unclear how much of the heavy lifting is being done by the position embeddings**. \n\nI believe there is a chance that the position embeddings may be doing a lot of the heavy lifting in getting good performance. For instance Toeplitz matrix in itself is a performant SSM [1] and may artificially boost the apparent efficacy of Additive Decay. Can authors maybe use 2D rope as the position embedding and see if they improve/compete with original LightNet, LightNet with no position embeddings and Linear Attention.\n\n[1] Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers. Sukjun Hwang, Aakash Lahoti, Tri Dao, Albert Gu"}},"nonreaders":[],"tmdate":1731428015164,"tcdate":1730720443773,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10364/Reviewer_tk6m"],"signatures":["ICLR.cc/2025/Conference/Submission10364/Reviewer_tk6m"],"forum":"qK3XElJUbq","number":4,"license":"CC BY 4.0","cdate":1730720443773,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10364/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428015164,"domain":"ICLR.cc/2025/Conference","replyto":"qK3XElJUbq","id":"FH3sVDa9UB","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LightNet","multi-dimensional sequential modeling"]},"supplementary_material":{"value":"/attachment/2a97e4fadc4a6ce9c6e7986164f3211e2a83ac63.pdf"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Linear attention mechanisms have gained prominence in causal language models due to their linear computational complexity and enhanced speed. However, the inherent decay mechanism in linear attention presents challenges when applied to multi-dimensional sequence modeling tasks, such as image processing and multi-modal learning. In these scenarios, the utilization of sequential scanning to establish a global receptive field necessitates multiple scans for multi-dimensional data, thereby leading to inefficiencies. This paper identifies the inefficiency caused by a \\enquote{multiplicative} linear recurrence and proposes an efficient alternative \\enquote{additive} linear recurrence to avoid the issue, as it can handle multi-dimensional data within a single scan. We further develop an efficient multi-dimensional sequential modeling framework called LightNet based on the new recurrence. Moreover, we present two new multi-dimensional linear relative positional encoding methods, MD-TPE and MD-LRPE to enhance the model's ability to discern positional information in multi-dimensional scenarios. Our empirical evaluations across various tasks, including image classification, image generation, bidirectional language modeling, and autoregressive language modeling, demonstrate the efficacy of LightNet, showcasing its potential as a versatile and efficient solution for multi-dimensional sequential modeling."},"_bibtex":{"value":"@misc{\nqin2025you,\ntitle={You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet},\nauthor={Zhen Qin and Yuxin Mao and Xuyang Shen and Dong Li and Jing Zhang and Yuchao Dai and Yiran Zhong},\nyear={2025},\nurl={https://openreview.net/forum?id=qK3XElJUbq}\n}"},"title":{"value":"You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet"},"pdf":{"value":"/pdf/d7cd71cad5f3ec30feba8bb187c5ea7e0360bfcd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"qin|you_only_scan_once_efficient_multidimension_sequential_modeling_with_lightnet"},"authorids":{"value":["~Zhen_Qin6","~Yuxin_Mao3","~Xuyang_Shen1","~Dong_Li11","~Jing_Zhang23","~Yuchao_Dai1","~Yiran_Zhong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zhen Qin","Yuxin Mao","Xuyang Shen","Dong Li","Jing Zhang","Yuchao Dai","Yiran Zhong"]}},"version":2},{"content":{"summary":{"value":"### Summary\n\n1. Extends Eigenvalue Analysis to LPV Models:\n\n   The paper builds on earlier work that analyzed eigenvalues in Linear Time-Invariant (LTI) State Space Models, where eigenvalues capture memory, stability, and selectivity. Eigenvalues near 1 mean good long-term memory, while those near 0 indicate forgetting or gating. \n\n   The authors extend this idea to Linear Parameter-Varying (LPV) systems using the Dynamical Systems Framework (DSF), which writes models like Attention and Mamba-2 as dynamical systems:\n   \n   $h_i = \\Lambda_i h_{i-1} + B_i u_i, \\quad y_i = C_i h_i + D_i u_i$\n\n   Here, the parameters change with input, allowing a similar eigenvalue analysis across model types.\n\n2. Analysis:\n\n   On memory, authors show the intuitive result that models that remember well keep eigenvalues close to 1. On language modeling (WikiText), the authors find that “gating” or eigenvalues near zero help model selectively store tokens.\n\n3. Architectural Modifications Based on the Analysis:\n\n   The paper tries to use these insights to improve models with changes including: Gating, Short Convs, Varying the number of layers etc"},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See weakness"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper’s central idea, that is extending eigenvalue analysis from LTI SSMs to LPV systems and attention, is natural and easy to follow.\n2. The connection between eigenvalues and memory retention/selectivity is intuitive and aligns with established understanding.\n3. The paper conducts experiments across a breadth of models (S4, Mamba-2, Self-Attention, Linear Attention etc) and tasks (LRA, MQAR, WikiText)"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"### Weaknesses\n\n1. **On memory tasks**\n\n   1. The analysis of memory is intuitive and largely reiterates known understanding—that eigenvalues near one preserve information and those near zero lead to forgetting.\n   2. The attention results, though potentially interesting, are underexplained. The paper attributes attention’s poor performance on LRA to having both very low and very high eigenvalues, but I suspect this explanation may be correlational rather than causal. Attention is known to perform strongly on other memory-intensive retrieval tasks (e.g., NIAH [1]). It would have been useful if the paper discussed this discrepancy and explored why attention struggles on LRA despite its well-documented ability to retain long-term information in other settings.\n\n2. **On gating/selectivity tasks**\n\n   1. The interpretation that “gating” corresponds to zeroing out eigenvalues in WikiText models like Mamba-2 is reasonable but not novel as it directly follows from the Mamba-2's design of selection mechanism.\n   2. The follow-up “add gate” experiment on attention has a conceptual mismatch---The gating mechanism used in the experiments differs from Mamba-2’s within-sequence gating, and is instead applied per-token AFTER sequence mixing, making the analogy weak.\n   3. The task choice and conclusions seem inconsistent: gating is shown to help on IMDb, a memory-heavy task where (if i understand correctly) it should theoretically hurt (or at-least not help). Furthermore, ListOps, which is also memory intensive, unexpectedly develops gating, which remains unexplained. \n\n       I expected to see a gating-improved task, like WikiText, show improvement when gating is added to attention.\n\n3. **On convolution and varying number of layers**\n\n   1. These experiments are not motivated by eigenvalue analysis and are instead motivated from Mamba-2's strong performance which feels disconnected from the main argument.\n   2. The claim that short convolutions “take over long-range memory” seems to be incorrect—short convolutions are \"short\" (of size 4) and cannot provide such capacity. The paper employs this argument to justify why a one layer attention model with convolution could solve MQAR. I believe this justification is incorrect. Prior work [1] has shown that MQAR performance requires \"induction-head formation\": first sequence mixer mixes keys and values and the second sequence mixer retrieves the correct value. Convolution helps because it performs the first task, not because it replaces long-range retrieval.\n\n4. **On evaluation scope**\n\n   1. For the proposed architectural changes, the experiments are limited to small-scale settings and on synthetic tasks.\n   2. It would be more convincing to test these modifications on language modeling and downstream evals at across multiple scales (e.g., 125M, 350M, 750M, 1.3B) to assess their importance.\n\n5. **On scope and completeness**\n\n   1. The paper briefly mentions that eigenvalues evolve during training, suggesting potential task-dependent initialization, but this idea is never explored.\n   2. The architectural additions should be tested on language modeling for validation.\n   3. Overall, the paper reads as an incremental extension of prior DSF analyses, with several unexplained experimental results.\n\n-------\n[1]: Mechanistic evaluation of Transformers and state space models. Aryaman Arora, Neil Rathi, Nikil Roashan Selvam, Róbert Csordás, Dan Jurafsky, Christopher Potts"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762940622778,"tcdate":1762122283750,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21047/Reviewer_N9x6"],"signatures":["ICLR.cc/2026/Conference/Submission21047/Reviewer_N9x6"],"forum":"ALf0xHCCaP","number":2,"license":"CC BY 4.0","cdate":1762122283750,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission21047/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762940622778,"domain":"ICLR.cc/2026/Conference","replyto":"ALf0xHCCaP","id":"QsCkLROH9C","forumContent":{"TLDR":{"value":"This paper shows that eigenvalues offer a metric for memory retention and selective forgetting in sequence models, guiding model design choices to meet task requirements."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Sequence models","dynamical systems","memory dynamics"]},"supplementary_material":{"value":"/attachment/a34ab7106844c8b1974675fdf300085c394ec367.pdf"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Although softmax attention drives state-of-the-art performance for sequence models, its quadratic complexity limits scalability, motivating linear alternatives such as state space models (SSMs). While these alternatives improve efficiency, their fundamental differences in information processing remain poorly understood. In this work, we leverage the recently proposed dynamical systems framework to represent softmax, norm and linear attention as dynamical systems, enabling a structured comparison with SSMs by analyzing their respective eigenvalue spectra. Since eigenvalues capture essential aspects of dynamical system behavior, we conduct an extensive empirical analysis across diverse sequence models and benchmarks. We first show that eigenvalues influence essential aspects of memory and long-range dependency modeling, revealing spectral signatures that align with task requirements. Building on these insights, we then investigate how architectural modifications in sequence models impact both eigenvalue spectra and task performance.  This correspondence further strengthens the position of eigenvalue analysis as a principled metric for interpreting, understanding, and ultimately improving the capabilities of sequence models."},"_bibtex":{"value":"@misc{\nrickenbach2026tasklevel,\ntitle={Task-Level Insights from Eigenvalues across Sequence Models},\nauthor={Rahel Rickenbach and Jelena Trisovic and Alexandre Didier and Jerome Sieber and Melanie Zeilinger},\nyear={2026},\nurl={https://openreview.net/forum?id=ALf0xHCCaP}\n}"},"title":{"value":"Task-Level Insights from Eigenvalues across Sequence Models"},"pdf":{"value":"/pdf/ef15be1e4ca44ec34adb76dbbbb5e30448f4997c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"rickenbach|tasklevel_insights_from_eigenvalues_across_sequence_models"},"authorids":{"value":["~Rahel_Rickenbach1","~Jelena_Trisovic1","~Alexandre_Didier1","~Jerome_Sieber1","~Melanie_Zeilinger1"]},"authors":{"value":["Rahel Rickenbach","Jelena Trisovic","Alexandre Didier","Jerome Sieber","Melanie Zeilinger"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2402.05892v5"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"li|mamband_selective_state_space_modeling_for_multidimensional_data"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Shufan_Li:","https://dblp.org/search/pid/api?q=author:Harkanwar_Singh:","~Aditya_Grover1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2402.05892"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2402-05892,\n  publtype={informal},\n  author={Shufan Li and Harkanwar Singh and Aditya Grover},\n  title={Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2402.05892},\n  url={https://doi.org/10.48550/arXiv.2402.05892}\n}\n"},"abstract":{"value":"In recent years, Transformers have become the de-facto architecture for sequence modeling on text and a variety of multi-dimensional data, such as images and video. However, the use of self-attention layers in a Transformer incurs prohibitive compute and memory complexity that scales quadratically w.r.t. the sequence length. A recent architecture, Mamba, based on state space models has been shown to achieve comparable performance for modeling text sequences, while scaling linearly with the sequence length. In this work, we present Mamba-ND, a generalized design extending the Mamba architecture to arbitrary multi-dimensional data. Our design alternatively unravels the input data across different dimensions following row-major orderings. We provide a systematic comparison of Mamba-ND with several other alternatives, based on prior multi-dimensional extensions such as Bi-directional LSTMs and S4ND. Empirically, we show that Mamba-ND demonstrates performance competitive with the state-of-the-art on a variety of multi-dimensional benchmarks, including ImageNet-1K classification, HMDB-51 action recognition, and ERA5 weather forecasting."},"title":{"value":"Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data"},"authors":{"value":["Shufan Li","Harkanwar Singh","Aditya Grover"]}},"tmdate":1727720318188,"pdate":1704067200000,"tcdate":1727720315105,"writers":["~"],"signatures":["~Aditya_Grover1"],"forum":"9fYouXg4Q0","license":"CC BY-SA 4.0","number":122669,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1727720318188,"domain":"DBLP.org","id":"9fYouXg4Q0","version":2},{"content":{"summary":{"value":"This work presents Mamba-HMIL, a hierarchical MIL method leveraging the state space modeling for weakly-supervised tasks in computational pathology. MAMBA-HMIL includes several components: (1) state-space modeling (Mamba), (2) Mixture of Experts (MoE) blocks, and (3) sequence fusion / adaptive fusion blocks. Mamba-HMIL is evaluated on cancer subtyping and survival prediction tasks, and compared with relevant baselines in the literature (ABMIL, CLAM, DSMIL, HIPT, and Mamba extensions to MIL). Ablation experiments for each component is performed."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"See above."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- Overall, the experimental design is comprehensive and thorough. Mamba-HMIL is compared against many-to-all relevant baselines in the literature, from simple permutation-invariant pooling baselines (ABMIL, CLAM-MB), to Transformer MIL architectures that learn token dependencies (TransMIL, HIPT), to also direct extensions of Mamba applied to MIL architectures (Mamba+ABMIL, Mamba+DSMIL, Mamba+CLAM-MB) as well as Mamba-MIL. Evaluation on survival prediction is also appreciated and validates the strength of Mamba-HMIL in learning context-aware features for understanding the tumor microenvironment. Good attention-to-detail to the survival prediction baselines in evaluating recent SOTA early multimodal fusion architectures like MCAT and PIBD. This study also presents good depth of experiments, in not only ablating the components of Mamba-HMIL (MoE, SF/AF), but also validating other components in MIL including different pretrained encoders (ResNet-50 vs UNI) and hierarchical feature extraction (10X, 20X, 10X+20X).\n- While direct extensions of Mamba are not a technical novelty, Mamba-HMIL has good performance gains and consistently achieves the best performance across all tasks. These performance gains are on top of comparisons to MIL architectures with Mamba extensions, which suggests that the architecture modifications are not ad hoc.\n- Interestingly, unimodal Mamba-HMIL outperforms many multimodal fusion comparisons, including Mamba+CLAM-SB, PORPOISE, and MCAT. This is a good finding that should be highlighted more."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Though method seems strong, many of the components of Mamba-HMIL itself are either not novel or not studied enough to demonstrate why we see strong improvement in performance. I would like to understand how these components \"stabilize SSM training\". It is not clear how SSM training is unstable when direct extension of Mamba works quite well for all MIL architectures across all tasks.\n- I cannot review the code for this submission. As many of the contributions are empirical, it would be valuable to validate the contributions of this work empirically before acceptance.\n- Is it possible to visualize token-to-token interactions by Mamba-HMIL, besides attention weights from global attention pooling? How do token interactions change across different Mamba layers?\n- There are several outstanding typos in this work. There is missing-or-extra spacing after Mamba-HMIL on many lines. There are many misspellings like \"Propoise\". In the data description of survival prediction on Line 327, TCGA-LUSC instead of TCGA-BLCA is written. Many citations are missing and marked as ?."}},"nonreaders":[],"tmdate":1731427421221,"tcdate":1731052795689,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1404/Reviewer_UNQv"],"signatures":["ICLR.cc/2025/Conference/Submission1404/Reviewer_UNQv"],"forum":"0yVP49SDg0","number":4,"license":"CC BY 4.0","cdate":1731052795689,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1404/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427421221,"domain":"ICLR.cc/2025/Conference","replyto":"0yVP49SDg0","id":"vI6nTpKpeR","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Whole Slide Images","Hierarchical Multiple Instance Learning","State Space Model."]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Multiple instance learning (MIL) has been widely employed for gigapixel whole slide image (WSI) classification. Existing MIL methods, however, are found wanting to align with the clinical practice of pathologists, who typically scrutinize WSIs at varied scales and compare the local regions in a global perspective. Given that WSIs usually boast immense dimensions peppered with large regions not pertinent to diagnosis, we propose a novel hierarchical multiple instance learning method based on the state space model, called Mamba-HMIL, for WSI classification. Mamba-HMIL consists of three primary modules to enhance the performance of MIL. First, the hierarchical feature extractor harvests features across diverse scales. Second, for capturing the correlation among patches, the state space model demonstrates robust modeling capabilities. A Mixture of Experts (MoE) module is for stable SSM training. Third, the adaptive selection model strives to reduce redundancies by focusing on disease-positive regions. We evaluate Mamba-HMIL on two WSI subtype datasets (TCGA-NSCLC and TCGA-RCC) and two WSI survival datasets (TCGA-BRCA and TCGA-BLCA). Our results suggest that Mamba-HMIL outperforms existing MIL methods on both WSI tasks. Our code will be made publicly available."},"_bibtex":{"value":"@misc{\nlu2024mambahmil,\ntitle={Mamba-{HMIL}: Hierarchical Multiple Instance Learning via State Space Model for Whole Slide Image Diagnosis},\nauthor={Mengkang Lu and Tianyi Wang and Qiwei Fu and Qingjie Zeng and ZhaoyangLIU and shu minglei and Yong Xia},\nyear={2024},\nurl={https://openreview.net/forum?id=0yVP49SDg0}\n}"},"title":{"value":"Mamba-HMIL: Hierarchical Multiple Instance Learning via State Space Model for Whole Slide Image Diagnosis"},"pdf":{"value":"/pdf/b9c0ba4bff85871877abf382a53613dc757358f5.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"lu|mambahmil_hierarchical_multiple_instance_learning_via_state_space_model_for_whole_slide_image_diagnosis"},"authorids":{"value":["~Mengkang_Lu1","~Tianyi_Wang4","~Qiwei_Fu1","~Qingjie_Zeng1","~ZhaoyangLIU1","~shu_minglei1","~Yong_Xia1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Mengkang Lu","Tianyi Wang","Qiwei Fu","Qingjie Zeng","ZhaoyangLIU","shu minglei","Yong Xia"]}},"version":2},{"content":{"summary":{"value":"This paper proposes COMBA, a novel framework designed to apply State Space Models (SSMs), such as Mamba, to the task of learning on large-scale graphs. The core motivation is to leverage the linear-time complexity of SSMs while adapting them to the non-sequential, neighborhood-centric structure of graph data."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1.  Why were no efficiency baselines (training time, memory usage) included in the complexity analysis? A direct comparison against ClusterGCN, which did not run OOM on Ogbn-product, is essential to validate the scalability claims.\n2.  Can the authors provide a hyperparameter sensitivity analysis for the hop length `k` and the context window size `w`?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1.  **Novel Motivation and Problem Formulation:** The paper does an excellent job of identifying a key challenge in applying SSMs to graphs: the fundamental mismatch between the 1D sequential dependency of SSMs and the multi-hop neighborhood dependency of graphs.\n2.  **Interesting Core Mechanisms:** Both of the paper's core contributions are well-motivated. \n    * \"Cross Batch Aggregation\" is a practical approach to scaling GNNs, and the theoretical justification (Theorem 1) for its error reduction is a good addition.\n    * \"Graph Context Gating\" is an intelligent modification of the Mamba architecture, which elegantly repurposes the sequence-modeling capability of SSMs to handle multi-hop graph contexts."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.  **Lacking Efficiency Baselines:** The paper's main motivation is to \"tackle large graph learning\" and scale SSMs. However, the complexity analysis (Section 5.3, Figure 4) is limited *only* to the proposed COMBA model. There is **zero** comparison against any baseline in terms of wall-clock training time, inference time, or memory usage.\n    * The authors state that baselines like Nagphormer and Graph Mamba-1/2 ran \"OOM\" (Out of Memory) on Ogbn-product, which supports their memory-efficiency claim.\n    * However, **ClusterGCN (a key scalability baseline) did *not* OOM**, yet it is *excluded* from all efficiency/complexity comparisons. This is a major omission. The scalability claim is therefore entirely unsubstantiated.\n\n2.  **Superficial Ablation Study:** The ablation study (Table 2) is too simple. It only demonstrates that removing the two core components degrades performance. This is a weak form of ablation. A more rigorous study would have included:\n    * A comparison of \"Graph Context Gating\" against simpler gating alternatives (e.g., a standard fixed-projection Mamba, or a gate that only depends on the current hop $Z'_k$).\n    * An analysis of the GNN backbone (the paper uses Gated-GCN) to show if the COMBA framework is robust to different GNN choices (e.g., GCN, GAT).\n\n3.  **No Hyperparameter Sensitivity Analysis:** This is a crucial flaw. The model introduces at least two key new hyperparameters: the hop length `k` (how many hops to unroll into a sequence) and the context window size `w` (for the gating mechanism). The paper provides **no analysis** of how model performance changes with different values of `k` and `w`. It is impossible to know if the model is robust or if its performance is an artifact of extensive, un-falsifiable tuning."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942227939,"tcdate":1761808335659,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22461/Reviewer_t7Fe"],"signatures":["ICLR.cc/2026/Conference/Submission22461/Reviewer_t7Fe"],"forum":"E4Pm2M5m09","number":2,"license":"CC BY 4.0","cdate":1761808335659,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22461/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942227939,"domain":"ICLR.cc/2026/Conference","replyto":"E4Pm2M5m09","id":"TV0TKHo38j","forumContent":{"TLDR":{"value":"In this paper, we propose COMBA to tackle large graph learning using state space models, with graph context gating and cross batch aggregation."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Graph neural networks","State space models","Batch learning","Homogeneous graphs","Classification"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"State space models (SSMs) have recently emerged for modeling long-range dependency in sequence data, with much simplified computational costs than modern alternatives, such as transformers. Advancing SMMs to graph structured data, especially for large graphs, is a significant challenge because SSMs are sequence models and the shear graph volumes make it very expensive to convert graphs as sequences for effective learning. In this paper, we propose COMBA to tackle large graph learning using state space models, with two key innovations: graph context gating and cross batch aggregation. Graph context refers to different hops of neighborhood for each node, and graph context gating allows COMBA to use such context to learn best control of neighbor aggregation. For each graph context, COMBA samples nodes as batches, and train a graph neural network (GNN), with information being aggregated cross batches, allowing COMBA to scale to large graphs. Our theoretical study asserts that cross-batch aggregation guarantees lower error than training GNN without aggregation. Experiments on benchmark networks demonstrate significant performance gains compared to baseline approaches. Code and benchmark datasets will be released for public access."},"_bibtex":{"value":"@misc{\nshen2025comba,\ntitle={{COMBA}: Cross Batch Aggregation for Learning Large Graphs with Context Gating State Space Models},\nauthor={Jiajun Shen and Yufei Jin and Yi He and Xingquan Zhu},\nyear={2025},\nurl={https://openreview.net/forum?id=E4Pm2M5m09}\n}"},"title":{"value":"COMBA: Cross Batch Aggregation for Learning Large Graphs with Context Gating State Space Models"},"pdf":{"value":"/pdf/58113009b83859a859d68373caa03caa11a889f6.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"shen|comba_cross_batch_aggregation_for_learning_large_graphs_with_context_gating_state_space_models"},"authorids":{"value":["~Jiajun_Shen6","~Yufei_Jin1","~Yi_He4","~Xingquan_Zhu1"]},"authors":{"value":["Jiajun Shen","Yufei Jin","Yi He","Xingquan Zhu"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.02178v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"yadav|audio_mamba_selective_state_spaces_for_selfsupervised_audio_representations"},"authorids":{"value":["~Sarthak_Yadav1","https://dblp.org/search/pid/api?q=author:Zheng-Hua_Tan:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.02178"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-02178,\n  publtype={informal},\n  author={Sarthak Yadav and Zheng-Hua Tan},\n  title={Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.02178},\n  url={https://doi.org/10.48550/arXiv.2406.02178}\n}\n"},"abstract":{"value":"Despite its widespread adoption as the prominent neural architecture, the Transformer has spurred several independent lines of work to address its limitations. One such approach is selective state space models, which have demonstrated promising results for language modelling. However, their feasibility for learning self-supervised, general-purpose audio representations is yet to be investigated. This work proposes Audio Mamba, a selective state space model for learning general-purpose audio representations from randomly masked spectrogram patches through self-supervision. Empirical results on ten diverse audio recognition downstream tasks show that the proposed models, pretrained on the AudioSet dataset, consistently outperform comparable self-supervised audio spectrogram transformer (SSAST) baselines by a considerable margin and demonstrate better performance in dataset size, sequence length and model size comparisons."},"title":{"value":"Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations"},"authors":{"value":["Sarthak Yadav","Zheng-Hua Tan"]}},"tmdate":1727698939619,"pdate":1704067200000,"tcdate":1727698936090,"writers":["~"],"signatures":["~Sarthak_Yadav1"],"forum":"p9hNIgJ23G","license":"CC BY-SA 4.0","number":110270,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1727698939619,"domain":"DBLP.org","id":"p9hNIgJ23G","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2505.17932v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"casti|selection_mechanisms_for_sequence_modeling_using_linear_state_space_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Umberto_Casti:","https://dblp.org/search/pid/api?q=author:Sandro_Zampieri:","~Fabio_Pasqualetti1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2505.17932"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2505-17932,\n  publtype={informal},\n  author={Umberto Casti and Sandro Zampieri and Fabio Pasqualetti},\n  title={Selection Mechanisms for Sequence Modeling using Linear State Space Models},\n  year={2025},\n  month={May},\n  cdate={1746057600000},\n  journal={CoRR},\n  volume={abs/2505.17932},\n  url={https://doi.org/10.48550/arXiv.2505.17932}\n}\n"},"abstract":{"value":"Recent advancements in language modeling tasks have been driven by architectures such as Transformers and, more recently, by Selective State Space Models (SSMs). In this paper, we introduce an alternative selection mechanism inspired by control theory methodologies. Specifically, we propose a novel residual generator for selection, drawing an analogy to fault detection strategies in Linear Time-Invariant (LTI) systems. Unlike Mamba, which utilizes Linear Time-Varying (LTV) systems, our approach combines multiple LTI systems, preserving their beneficial properties during training while achieving comparable selectivity. To evaluate the effectiveness of the proposed architecture, we test its performance on synthetic tasks. While these tasks are not inherently critical, they serve as benchmarks to test the selectivity properties of different cores architecture. This work highlights the potential of integrating theoretical insights with experimental advancements, offering a complementary perspective to deep learning innovations at the intersection of control theory and machine learning."},"title":{"value":"Selection Mechanisms for Sequence Modeling using Linear State Space Models"},"authors":{"value":["Umberto Casti","Sandro Zampieri","Fabio Pasqualetti"]}},"tmdate":1769467686040,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2505-17932"],"tcdate":1769467635541,"writers":["~"],"signatures":["~Fabio_Pasqualetti1"],"forum":"goqou8sYID","license":"CC BY-SA 4.0","number":804014,"cdate":1746057600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769467686040,"domain":"DBLP.org","id":"goqou8sYID","version":2},{"content":{"venue":{"value":"Crossref"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-031-72083-3_28.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"yang|mambamil_enhancing_long_sequence_modeling_with_sequence_reordering_in_computational_pathology"},"html":{"value":"https://doi.org/10.1007/978-3-031-72083-3_28"},"abstract":{"value":"Multiple Instance Learning (MIL) has emerged as a dominant paradigm to extract discriminative feature representations within Whole Slide Images (WSIs) in computational pathology. Despite driving notable progress, existing MIL approaches suffer from limitations in facilitating comprehensive and efficient interactions among instances, as well as challenges related to time-consuming computations and overfitting. In this paper, we incorporate the Selective Scan Space State Sequential Model (Mamba) in Multiple Instance Learning (MIL) for long sequence modeling with linear complexity, termed as MambaMIL. By inheriting the capability of vanilla Mamba, MambaMIL demonstrates the ability to comprehensively understand and perceive long sequences of instances. Furthermore, we propose the Sequence Reordering Mamba (SR-Mamba) aware of the order and distribution of instances, which exploits the inherent valuable information embedded within the long sequences. With the SR-Mamba as the core component, MambaMIL can effectively capture more discriminative features and mitigate the challenges associated with overfitting and high computational overhead. Extensive experiments on two public challenging tasks across nine diverse datasets demonstrate that our proposed framework performs favorably against state-of-the-art MIL methods. The code is released at https://github.com/isyangshu/MambaMIL."},"title":{"value":"MambaMIL: Enhancing Long Sequence Modeling with Sequence Reordering in Computational Pathology"},"authors":{"value":[{"fullname":"Shu Yang","username":"~Shu_Yang5"},{"fullname":"Yihui Wang","username":"~Yihui_Wang1"},{"fullname":"Hao Chen"}]}},"tmdate":1789093063024,"pdate":1704067200000,"externalIds":["doi:10.1007/978-3-031-72083-3_28"],"tcdate":1762667914685,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Shu_Yang5"],"forum":"4NiUHys2CQ","license":"CC BY-SA 4.0","number":11592,"cdate":1728845238028,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Authorship_Claim","OpenReview.net/-/Edit"],"mdate":1789093063024,"domain":"OpenReview.net/Public_Article","id":"4NiUHys2CQ","version":2},{"content":{"venue":{"value":"MICCAI (4) 2024"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-031-72083-3_28.pdf"},"venueid":{"value":"dblp.org/conf/MICCAI/2024"},"paperhash":{"value":"yang|mambamil_enhancing_long_sequence_modeling_with_sequence_reordering_in_computational_pathology"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Shu_Yang_0004:","~Yihui_Wang1","https://dblp.org/search/pid/api?q=author:Hao_Chen_0011:"]},"html":{"value":"https://doi.org/10.1007/978-3-031-72083-3_28"},"_bibtex":{"value":"@inproceedings{DBLP:conf/miccai/YangWC24,\n  author={Shu Yang and Yihui Wang and Hao Chen},\n  title={MambaMIL: Enhancing Long Sequence Modeling with Sequence Reordering in Computational Pathology},\n  year={2024},\n  cdate={1704067200000},\n  pages={296-306},\n  url={https://doi.org/10.1007/978-3-031-72083-3_28},\n  booktitle={MICCAI (4)},\n  crossref={conf/miccai/2024-4}\n}\n"},"abstract":{"value":"Multiple Instance Learning (MIL) has emerged as a dominant paradigm to extract discriminative feature representations within Whole Slide Images (WSIs) in computational pathology. Despite driving notable progress, existing MIL approaches suffer from limitations in facilitating comprehensive and efficient interactions among instances, as well as challenges related to time-consuming computations and overfitting. In this paper, we incorporate the Selective Scan Space State Sequential Model (Mamba) in Multiple Instance Learning (MIL) for long sequence modeling with linear complexity, termed as MambaMIL. By inheriting the capability of vanilla Mamba, MambaMIL demonstrates the ability to comprehensively understand and perceive long sequences of instances. Furthermore, we propose the Sequence Reordering Mamba (SR-Mamba) aware of the order and distribution of instances, which exploits the inherent valuable information embedded within the long sequences. With the SR-Mamba as the core component, MambaMIL can effectively capture more discriminative features and mitigate the challenges associated with overfitting and high computational overhead. Extensive experiments on two public challenging tasks across nine diverse datasets demonstrate that our proposed framework performs favorably against state-of-the-art MIL methods. The code is released at https://github.com/isyangshu/MambaMIL."},"title":{"value":"MambaMIL: Enhancing Long Sequence Modeling with Sequence Reordering in Computational Pathology"},"authors":{"value":["Shu Yang","Yihui Wang","Hao Chen"]}},"tmdate":1737376034431,"pdate":1704067200000,"tcdate":1737376031375,"writers":["~"],"signatures":["~Yihui_Wang1"],"forum":"WUYkJmADLP","license":"CC BY-SA 4.0","number":268848,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1737376034431,"domain":"DBLP.org","id":"WUYkJmADLP","version":2},{"content":{"comment":{"value":"Dear Reviewer,\n\nThank you very much for your thoughtful and constructive feedback, as well as the recognition of the contributions in our paper. Your insights are invaluable, and we are sincerely grateful for the time and effort you have invested in reviewing our work.\n\nWe must first apologize for the delay in responding to your comments. We truly appreciate your patience and understanding as we strive to address your points with the rigor they deserve.\n\nWe completely agree with your observation regarding the importance of comparing our proposed iLLaMA model with recent advancements like Mamba-based [6] and xLSTM-based [7] architectures. Both families of models have shown significant progress in handling complex tasks in the vision and language domains, making them critical baselines to consider.\n\nTo address this, we have conducted additional experiments to systematically compare iLLaMA with Mamba-based models (e.g., Vim [1], Mamba-2D [2], PlainMamba [3], VMamba [4]) and xLSTM-based models (e.g., ViL [5]). Note that we conducted a **system-level comparison** (the results of iLLaMA is achieved with ImageNet-21K pre-training, as the dataset is publicly available and can enhance iLLaMA's performance through pre-training). \n\nAs shown in Table 1 below, we provide a comprehensive comparison in terms of model parameters, computational cost, and accuracy on the ImageNet-1K [8] dataset. The results highlight that iLLaMA achieves superior performance while maintaining competitive efficiency in both parameter size and computational demands.\n\nWe hope this additional analysis clarifies iLLaMA's relative advantages and contributes to a more comprehensive understanding of its strengths. Thank you again for pointing out this important aspect, which has helped us further strengthen our evaluation.\n\n\n\nTable 1: System-level comparison on ImageNet-1K dataset. \n\n| Model             | Params (M) | MACs (G)      | ImageNet-1K Accuracy |\n| ----------------- | ---------- | ------------- | -------------------- |\n| Vim-B [1]         | 98         | -             | 83.2                 |\n| Mamba-2D-B [2]    | 92         | -             | 83.0                 |\n| PlainMamba-L3 [3] | 50         | 14.4          | 82.3                 |\n| VMamba-B [4]      | 89         | 15.4          | 83.9                 |\n| ViL-B [5]         | 89         | 18.6          | 82.4                 |\n| iLLaMA-B          | 86.3       | 17.6 (224px)  | 83.6                 |\n| iLLaMA-B          | 86.3       | 55.5 (384px)  | 85.0                 |\n| iLLaMA-L          | 310.2      | 62.8 (224px)  | 84.8                 |\n| iLLaMA-L          | 310.2      | 194.7 (384px) | 86.0                 |\n\n\n\n\n\nWe will update the main body of our paper with some results to provide readers with a clear and convenient comparison. We believe this update will enhance the accessibility of our findings and allow readers to better understand iLLaMA’s performance relative to these important baselines.\n\n\n\n\n\n[1] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model\n\n[2] Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data\n\n[3] PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition\n\n[4] VMamba: Visual State Space Model\n\n[5] Vision-LSTM: xLSTM as Generic Vision Backbone\n\n[6] Mamba: Linear-time sequence modeling with selective state spaces\n\n[7] xlstm: Extended long short-term memory\n\n[8] Imagenet: A large-scale hierarchical image database"},"title":{"value":"Response Part 1: System-Level Comparison with Mamba and xLSTM Based Models on ImageNet-1K"}},"tmdate":1732646265404,"tcdate":1732631822621,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5995/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5995/Authors"],"forum":"GraOHDxFjl","number":14,"license":"CC BY 4.0","cdate":1732631822621,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5995/-/Official_Comment"],"mdate":1732646265404,"domain":"ICLR.cc/2025/Conference","replyto":"Vnscugxs97","id":"4wpF2jUrto","forumContent":{"TLDR":{"value":"A successful application of the decoder-only architecture adapted to the visual Transformer."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Decoder-only; Vision Transformer; LLaMA"]},"supplementary_material":{"value":"/attachment/78197926ac2be36d5095510da4673e9230ea4da2.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Using the same architecture for text and image is important for AI standardization. Recent multimodal models use a decoder-only Transformer to generate text and an encoder-only Transformer to extract image features. Can images use exactly the same language architecture? To answer this question, we aim at a LLaMa decoder as vision Transformer (ViT) classifier in this paper. Specifically, we start\nour trajectory by “LLaMAfy” a standard ViT step-by-step, i.e., feed-forward net, normalization layer, causal self-attention and positional embedding, and point out a key issue—attention collapse—that result in the failure to the network training. Motivated by this observation, we propose post-sequence class token, enabling causal self-attention to efficiently capture the entire image’s information. To improve model optimization behavior and enhance performance, we then introduce a soft mask strategy to gradually transform the attention from bi-directional to causal mode. The tailored model, dubbed as image LLaMA (iLLaMA), maintains high consistency with LLaMA architecture, while matching up well against ViT, achieving 75.1% ImageNet top-1 accuracy with only 5.7M parameters. Scaling the model to ∼310M and pre-training on ImageNet-21K further enhances the accuracy to 86.0%. Its causal self-attention boosts computational efficiency and learns complex representation by elevating attention map ranks. Extensive experiments demonstrate iLLaMA’s reliable properties: shape-texture bias, calibration, quantization compatibility, ADE20K segmentation and CIFAR transfer learning. We hope our study can kindle fresh views to visual architectures in the era of LLMs and contributes to standardized AI models."},"_bibtex":{"value":"@misc{\nwang2025llama,\ntitle={{LL}a{MA} Decoder As Vision Transformer},\nauthor={Jiahao Wang and Wenqi Shao and Mengzhao Chen and Chengyue Wu and Yong Liu and Taiqiang Wu and Kaipeng Zhang and Songyang Zhang and Kai Chen and Ping Luo},\nyear={2025},\nurl={https://openreview.net/forum?id=GraOHDxFjl}\n}"},"title":{"value":"LLaMA Decoder As Vision Transformer"},"pdf":{"value":"/pdf/e480cfc7e55c005251065560b4b34abe6cfd1d6d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"wang|llama_decoder_as_vision_transformer"},"authorids":{"value":["~Jiahao_Wang1","~Wenqi_Shao2","~Mengzhao_Chen1","~Chengyue_Wu1","~Yong_Liu16","~Taiqiang_Wu1","~Kaipeng_Zhang1","~Songyang_Zhang1","~Kai_Chen4","~Ping_Luo2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jiahao Wang","Wenqi Shao","Mengzhao Chen","Chengyue Wu","Yong Liu","Taiqiang Wu","Kaipeng Zhang","Songyang Zhang","Kai Chen","Ping Luo"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Baleen, a new family of SSMs that integrates stochastic selective memory with an information bottleneck objective to improve interpretability and robustness. Extending Mamba, Baleen models state-selection gates as random variables with a variational KL regularizer, balancing context retention and compression while preserving linear-time recurrence. Two variants based on Bernoulli gating and exponential time sampling, Baleen-B and Baleen-E, yield self-interpretable token-level importance scores derived from expected gate values. The formulation provides an information-theoretic refinement of Mamba, explicitly controlling how much input history is retained, and promoting minimal sufficient representations that enhance robustness and explanation fidelity."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. How sensitive are Baleen’s results to the choice of the IB regularization strength (the weight β in KL term in Eq. 4)?\n\n2. Can Baleen effectively handle tasks requiring many pieces of information to be retained over long sequences?\n\n3. Since the gates are stochastic, have the authors examined whether the expected gate values are stable across different runs, and whether the top-selected tokens align with known important words or ground-truth rationales?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper presents a principled integration of information bottleneck theory into SSMs, yielding stochastic gate regularization that improves both interpretability and robustness.\n\n2. The proposed Bernoulli and exponential gating variants are theoretically well-motivated and retain Mamba’s linear computational complexity.\n\n3. The method demonstrates strong robustness under input perturbation and adversarial settings."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. While formulating a variational information bottleneck on Mamba’s gating is interesting, the conceptual novelty is incremental. The method essentially adds a KL-based sparsity regularizer to encourage selective gating, similar to variational dropout or selective masking. The core SSM architecture remains unchanged, and the stochastic gates are parameterized similarly to Mamba’s deterministic ones. Thus, Baleen may be viewed as “Mamba2 + KL regularizer,” raising the question of whether similar gains could be achieved with simpler methods (e.g., L0​/L1​ penalties or tuned dropout). Nonetheless, the formalization through the IB framework provides a theoretically grounded lens on sparsity and interpretability, which may justify the contribution despite its incremental nature.\n\n2. Baleen inherits the fixed hidden-state constraint of SSMs, which limits the amount of information it can store and retrieve. This becomes problematic for tasks requiring the distributed recall of multiple facts across long contexts. The IB objective improves information quality but may over-compress, discarding useful signals needed later. Since the number of independent bits is bounded by the state dimension, Baleen cannot fully overcome memory saturation when the sequence length greatly exceeds the state size. Without adaptive or external memory, it may still struggle with tasks requiring long-term or multi-fact reasoning.\n\n3. Although Baleen consistently outperforms prior SSMs, its empirical gains are modest. Improvements vary across tasks, with some regressions, and certain baselines (e.g., Gated-DeltaNet) outperform it on individual benchmarks. Moreover, comparisons are limited to SSM and linear RNN families; Transformer baselines, or hybrid models of comparable scale, are missing. Without such results, it remains unclear whether Baleen approaches or still lags Transformer-level performance."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925363884,"tcdate":1762154214028,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15041/Reviewer_pniM"],"signatures":["ICLR.cc/2026/Conference/Submission15041/Reviewer_pniM"],"forum":"976tPua6Hd","number":4,"license":"CC BY 4.0","cdate":1762154214028,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15041/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925363884,"domain":"ICLR.cc/2026/Conference","replyto":"976tPua6Hd","id":"ETwSyzyTvg","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Interpretation","Robustness","Large Language Models","State-space models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"We introduce \\textit{\\textbf{Baleen}}, a family of state space models that unifies \\textbf{stochastic selection} with \\textbf{information bottleneck} to build interpretable and robust long‑context learners. Unlike Mamba/Mamba2’s deterministic gates, Baleen treats selection as a random variable and regularizes it with a closed‑form KL to a sparsity prior: (i) \\textit{\\textbf{\\bib}} samples Bernoulli state‑transition gates; (ii) \\textit{\\textbf{\\gib}} samples Exponential  time‑intervals. This yields an explicit trade‑off between retention and compression and exposes token‑level selection heatmaps at inference for self‑interpretation. On language benchmarks, \\ib  improves average accuracy over Mamba2 by +0.95 at 370M pretraining and +1.38 at 7B finetuning. Baleen delivers stronger robustness to localized perturbations and adversarial attacks: under CIFAR‑10 sequence perturbation, prefix damage falls to 0.6\\% vs 26.5\\% for Mamba2 (average under attacks 0.542 vs 0.385). Finally, Baleen’s self‑interpretations outperform IG/Grad‑CAM on average fidelity across four text classification tasks. We will release our Baleen‑7B models on Hugging Face with code, checkpoints, and an interactive selection‑heatmap demo."},"_bibtex":{"value":"@misc{\nzhu2026baleen,\ntitle={Baleen: Self\\nobreakdash-Interpretable, Robust {SSM}s with Stochastic Selective Memory},\nauthor={Jiajun Zhu and Yuehao Wang and Peihao Wang and Ruisi Cai and Siqi Miao and Liliang Ren and Pan Li and Jason D. Lee and Zhangyang Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=976tPua6Hd}\n}"},"title":{"value":"Baleen: Self‑Interpretable, Robust SSMs with Stochastic Selective Memory"},"pdf":{"value":"/pdf/651f6e0f7dafa731cf8adeb1471c685a32749a87.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhu|baleen_selfinterpretable_robust_ssms_with_stochastic_selective_memory"},"authorids":{"value":["~Jiajun_Zhu1","~Yuehao_Wang1","~Peihao_Wang1","~Ruisi_Cai1","~Siqi_Miao1","~Liliang_Ren1","~Pan_Li2","~Jason_D._Lee1","~Zhangyang_Wang1"]},"authors":{"value":["Jiajun Zhu","Yuehao Wang","Peihao Wang","Ruisi Cai","Siqi Miao","Liliang Ren","Pan Li","Jason D. Lee","Zhangyang Wang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2412.14518v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"wang|efficient_selfsupervised_video_hashing_with_selective_state_spaces"},"authorids":{"value":["~Jinpeng_Wang3","~Niu_Lian1","https://dblp.org/search/pid/api?q=author:Jun_Li:","https://dblp.org/search/pid/api?q=author:Yuting_Wang:","https://dblp.org/search/pid/api?q=author:Yan_Feng:","https://dblp.org/search/pid/api?q=author:Bin_Chen_0011:","https://dblp.org/search/pid/api?q=author:Yongbing_Zhang_0002:","https://dblp.org/search/pid/api?q=author:Shu-Tao_Xia:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2412.14518"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-14518,\n  publtype={informal},\n  author={Jinpeng Wang and Niu Lian and Jun Li and Yuting Wang and Yan Feng and Bin Chen and Yongbing Zhang and Shu-Tao Xia},\n  title={Efficient Self-Supervised Video Hashing with Selective State Spaces},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2412.14518},\n  url={https://doi.org/10.48550/arXiv.2412.14518}\n}\n"},"abstract":{"value":"Self-supervised video hashing (SSVH) is a practical task in video indexing and retrieval. Although Transformers are predominant in SSVH for their impressive temporal modeling capabilities, they often suffer from computational and memory inefficiencies. Drawing inspiration from Mamba, an advanced state-space model, we explore its potential in SSVH to achieve a better balance between efficacy and efficiency. We introduce S5VH, a Mamba-based video hashing model with an improved self-supervised learning paradigm. Specifically, we design bidirectional Mamba layers for both the encoder and decoder, which are effective and efficient in capturing temporal relationships thanks to the data-dependent selective scanning mechanism with linear complexity. In our learning strategy, we transform global semantics in the feature space into semantically consistent and discriminative hash centers, followed by a center alignment loss as a global learning signal. Our self-local-global (SLG) paradigm significantly improves learning efficiency, leading to faster and better convergence. Extensive experiments demonstrate S5VH's improvements over state-of-the-art methods, superior transferability, and scalable advantages in inference efficiency. Code is available at https://github.com/gimpong/AAAI25-S5VH."},"title":{"value":"Efficient Self-Supervised Video Hashing with Selective State Spaces"},"authors":{"value":["Jinpeng Wang","Niu Lian","Jun Li","Yuting Wang","Yan Feng","Bin Chen","Yongbing Zhang","Shu-Tao Xia"]}},"tmdate":1784015768470,"pdate":1704067200000,"tcdate":1747598985894,"writers":["~"],"signatures":["~Jinpeng_Wang3"],"forum":"UOKzqSPFSg","license":"CC BY-SA 4.0","number":522635,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784015768470,"domain":"DBLP.org","id":"UOKzqSPFSg","version":2},{"content":{"venue":{"value":"AAAI 2025"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/32835/34990"},"venueid":{"value":"dblp.org/conf/AAAI/2025"},"paperhash":{"value":"wang|efficient_selfsupervised_video_hashing_with_selective_state_spaces"},"authorids":{"value":["~Jinpeng_Wang3","~Niu_Lian1","~Jun_Li47","https://dblp.org/search/pid/api?q=author:Yuting_Wang:","https://dblp.org/search/pid/api?q=author:Yan_Feng:","https://dblp.org/search/pid/api?q=author:Bin_Chen_0011:","https://dblp.org/search/pid/api?q=author:Yongbing_Zhang_0002:","https://dblp.org/search/pid/api?q=author:Shu-Tao_Xia:"]},"html":{"value":"https://doi.org/10.1609/aaai.v39i7.32835"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/0002LLWFCZX25,\n  author={Jinpeng Wang and Niu Lian and Jun Li and Yuting Wang and Yan Feng and Bin Chen and Yongbing Zhang and Shu-Tao Xia},\n  title={Efficient Self-Supervised Video Hashing with Selective State Spaces},\n  year={2025},\n  cdate={1735689600000},\n  pages={7753-7761},\n  url={https://doi.org/10.1609/aaai.v39i7.32835},\n  booktitle={AAAI},\n  crossref={conf/aaai/2025}\n}\n"},"abstract":{"value":"Self-supervised video hashing (SSVH) is a practical task in video indexing and retrieval. Although Transformers are predominant in SSVH for their impressive temporal modeling capabilities, they often suffer from computational and memory inefficiencies. Drawing inspiration from Mamba, an advanced state-space model, we explore its potential in SSVH to achieve a better balance between efficacy and efficiency. We introduce S5VH, a Mamba-based video hashing model with an improved self-supervised learning paradigm. Specifically, we design bidirectional Mamba layers for both the encoder and decoder, which are effective and efficient in capturing temporal relationships thanks to the data-dependent selective scanning mechanism with linear complexity. In our learning strategy, we transform global semantics in the feature space into semantically consistent and discriminative hash centers, followed by a center alignment loss as a global learning signal. Our self-local-global (SLG) paradigm significantly improves learning efficiency, leading to faster and better convergence. Extensive experiments demonstrate S5VH's improvements over state-of-the-art methods, superior transferability, and scalable advantages in inference efficiency."},"title":{"value":"Efficient Self-Supervised Video Hashing with Selective State Spaces"},"authors":{"value":["Jinpeng Wang","Niu Lian","Jun Li","Yuting Wang","Yan Feng","Bin Chen","Yongbing Zhang","Shu-Tao Xia"]}},"tmdate":1784015768396,"pdate":1735689600000,"tcdate":1747598985732,"writers":["~"],"signatures":["~Jinpeng_Wang3"],"forum":"C8v60aVx81","license":"CC BY-SA 4.0","number":522633,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784015768396,"domain":"DBLP.org","id":"C8v60aVx81","version":2},{"content":{"summary":{"value":"This paper proposes an SSM (state space model) architecture call S7, to provide an input-dependent mechanism for an existing work (S5), and showing this architecture can efficiently and accurately deal with sequence modeling tasks. The experiments show it performs better than Mamba(S6) in LRA tasks, while worse than other SSM-like models, and show it achieves good performance in neuromorphic datasets and dynamic prediction tasks."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1.\tWhat does S7 mean? (is the ”7” with some particular meaning?)\nOther questions please see the above weaknesses."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"The paper proposes an input-independent mechanism for SSM, with stable reparameterization techniques, and it provides the stability of this reparameterization through comprehensive theoretical derivations."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tThe novelty is not very clear to me, as mentioned in the paper, it is claimed that this paper provides efficient input-dependent state transition dynamics based on S5, but conceptually, it is the same as what Mamba (S6) did for S4, that introduced learnable matrices A, B, C\n2.\tThe paper claims this S7 is more efficient than Mamba, but I did not find any experiments data on the efficiency comparison with Mamba/Mamba2. Theoretically, without using parallel technologies like selective scan or others, how could one run the S7 in an efficient way when at each time step one needs to update the dynamics of A, B, C, D? \n3.\tThe experiments do not well support the claims: the performance of S7 in LRA is substantially worse than many other methods, and there’s no results showing it’s better than mamba in general language modeling tasks (which is an important selling point of mamba-like models). This leads to a question: what is the use case of S7?"}},"nonreaders":[],"tmdate":1731427551309,"tcdate":1730695552875,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2167/Reviewer_jZ6a"],"signatures":["ICLR.cc/2025/Conference/Submission2167/Reviewer_jZ6a"],"forum":"4wtcXV0kbi","number":3,"license":"CC BY 4.0","cdate":1730695552875,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2167/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427551309,"domain":"ICLR.cc/2025/Conference","replyto":"4wtcXV0kbi","id":"NlbWaK0OxK","forumContent":{"TLDR":{"value":"A new sequence model for long sequence modeling employed on various tasks."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["state space models","neural network architectures","deep learning architectures","sequence modeling","event-based vision","event cameras","neural odes"]},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"A central challenge in sequence modeling is efficiently handling tasks with extended contexts. While recent state-space models (SSMs) have made significant progress in this area, they often lack input-dependent filtering or require substantial increases in model complexity to handle input variability. We address this gap by introducing S7, a simplified yet powerful SSM that can handle input dependence while incorporating stable reparameterization and specific design choices to dynamically adjust state transitions based on input content, maintaining efficiency and performance. We prove that this reparameterization ensures stability in long-sequence modeling by keeping state transitions well-behaved over time. Additionally, it controls the gradient norm, enabling efficient training and preventing issues like exploding or vanishing gradients. S7 significantly outperforms baselines across various sequence modeling tasks, including neuromorphic event-based datasets, Long Range Arena benchmarks, and various physical and biological time series. Overall, S7 offers a more straightforward approach to sequence modeling without relying on complex, domain-specific inductive biases, achieving significant improvements across key benchmarks."},"_bibtex":{"value":"@misc{\nsoydan2024s,\ntitle={S7: Selective and Simplified State Space Layers for Sequence Modeling},\nauthor={Taylan Soydan and Nikola Zubic and Nico Messikommer and Siddhartha Mishra and Davide Scaramuzza},\nyear={2024},\nurl={https://openreview.net/forum?id=4wtcXV0kbi}\n}"},"title":{"value":"S7: Selective and Simplified State Space Layers for Sequence Modeling"},"pdf":{"value":"/pdf/1fffab6aa0f02c9b5326306d49f2faa5435961a9.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"soydan|s7_selective_and_simplified_state_space_layers_for_sequence_modeling"},"authorids":{"value":["~Taylan_Soydan1","~Nikola_Zubic1","~Nico_Messikommer1","~Siddhartha_Mishra1","~Davide_Scaramuzza1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Taylan Soydan","Nikola Zubic","Nico Messikommer","Siddhartha Mishra","Davide Scaramuzza"]}},"version":2},{"content":{"summary":{"value":"This paper proposes leveraging Mamba blocks for 3D point cloud modeling in the form of 3DET-Mamba, an application of SSM for 3D object detection. Similar to the prior 3DETR model, this approach partitions the point cloud into \"patch\" using Mamba blocks to capture local information, complemented by global modeling through dual Mamba blocks (organized by farthest and nearest point order). Additionally, query-aware Mamba blocks, akin to transformer blocks, are designed to decode objects in a DETR-like manner. The novelty of the approach lies in its ability to enhance 3DETR by integrating SSMs innovatively for 3D indoor scene understanding, utilizing ScanNet and SUN RGB-D datasets."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Besides weakness part, there are some other questions:\n1. Some terms could potentially cause confusion. For instance, FPS is an acronym for farthest point sampling, but in the context of the dual Mamba, it refers to farthest point order, while NPS refers to nearest point order.\n2. Are there any existing studies that use Mamba in a query-based method? It would be beneficial to reference these works to better understand the uniqueness of this module."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The proposed architecture adopts a Mamba-style approach and introduces 3D-specific customizations, such as partitioning the point clouds using an inner Mamba network, employing FPS/NPS ordering of points, and implementing a query-aware Mamba block.\n2. The experiments detailed in Tables 2, 3, and 4 conduct ablation studies that demonstrate the effectiveness of the Mamba block compared to baseline designs using transformers, although these comparisons do not include assessments of speed."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Mamba is praised for handling linear complexity and long-range sequences better than transformers, which have higher computational demands. However, the paper lacks theoretical analysis or experimental evidence to compare time and space complexities between Mamba and traditional architectures. The effects of changing point resolutions or patch sizes on performance are also not explored.\n\n2. The clarity of some ablation studies is lacking. For example, the role of ranked queries, a key contribution, is not evaluated. The relevance of NPS and FPS ordering in modeling patch sequences remains ambiguous. Additionally, the rationale for query-aware Mamba blocks is unconvincing; they seem to be simply added to cross-attention blocks with extra links. Table 4 does not clearly show that improvements are due to the Mamba block rather than just increased computational power.\n\n3. The paper does not provide solid proof of Mamba’s effectiveness in terms of performance or speed. It omits a comparison with the once leading FCAF3D detector, which scored 71.5 on ScanNet, without explanation. Comparisons of inference speeds across architectures are also absent."},"limitations":{"value":"Yes."}},"nonreaders":[],"tmdate":1730878625338,"tcdate":1720887282382,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission112/Reviewer_m4H4"],"signatures":["NeurIPS.cc/2024/Conference/Submission112/Reviewer_m4H4"],"forum":"iOleSlC80F","number":3,"license":"CC BY 4.0","cdate":1720887282382,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission112/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878625338,"domain":"NeurIPS.cc/2024/Conference","replyto":"iOleSlC80F","id":"sEF9Zirmwx","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["3D Object Detection","Mamba"]},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Transformer-based architectures have been proven successful in detecting 3D objects from point clouds. However, the quadratic complexity of the attention mechanism struggles to encode rich information as point cloud resolution increases. Recently, state space models (SSM) such as Mamba have gained great attention due to their linear complexity and long sequence modeling ability for language understanding. To exploit the potential of Mamba on 3D scene-level perception, for the first time, we propose 3DET-Mamba, which is a novel SSM-based model designed for indoor 3d object detection. Specifically, we divide the point cloud into different patches and use a lightweight yet effective Inner Mamba to capture local geometric information. To observe the scene from a global perspective, we introduce a novel Dual Mamba module that models the point cloud in terms of spatial distribution and continuity. Additionally, we design a Query-aware Mamba module that decodes context features into object sets under the guidance of learnable queries. Extensive experiments demonstrate that 3DET-Mamba surpasses previous 3DETR on indoor 3D detection benchmarks such as ScanNet, improving AP25/AP50 from 65.0\\%/47.0\\% to 70.4\\%/54.4\\%, respectively."},"_bibtex":{"value":"@inproceedings{\nli2024detmamba,\ntitle={3{DET}-Mamba: Causal Sequence Modelling for End-to-End 3D Object Detection},\nauthor={Mingsheng Li and Jiakang Yuan and Sijin Chen and Lin Zhang and Anyu Zhu and Xin Chen and Tao Chen},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=iOleSlC80F}\n}"},"title":{"value":"3DET-Mamba: Causal Sequence Modelling for End-to-End 3D Object Detection"},"pdf":{"value":"/pdf/e562d505611831135151fdb8395d9283e0291da6.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"li|3detmamba_causal_sequence_modelling_for_endtoend_3d_object_detection"},"authorids":{"value":["~Mingsheng_Li2","~Jiakang_Yuan1","~Sijin_Chen1","~Lin_Zhang15","~Anyu_Zhu1","~Xin_Chen16","~Tao_Chen6"]},"authors":{"value":["Mingsheng Li","Jiakang Yuan","Sijin Chen","Lin Zhang","Anyu Zhu","Xin Chen","Tao Chen"]}},"version":2},{"content":{"summary":{"value":"This paper focuses on heterogeneous graph representation learning, aiming to simultaneously capture long-range semantic dependencies and localized structural patterns, which are often overlooked by existing methods. The authors propose a novel framework named DVCL, which integrates a Graph Schema View Module (GSVM) for structural modeling via relational graph neural networks and a Semantic Meta-Path Mamba Module (SMPMM) for high-order semantic modeling using a Mamba-based sequence encoder. A bidirectional contrastive learning mechanism aligns these views to produce robust representations. Experimental results on four datasets show that DVCL outperforms state-of-the-art methods in node classification and clustering tasks."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"See weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":4},"strengths":{"value":"1.\tThe use of Mamba architecture for meta-path sequences in heterogeneous graphs is innovative, as it addresses long-range dependency modeling with linear complexity.\n2.\tComprehensive evaluations on four datasets provide strong empirical support, and the inclusion of ablation studies and the hyperparameter analysis validates component contributions.\n3.\tThe paper is well-structured, with clear module descriptions and algorithm outlines. Figure 1 gives a very detailed overview of the proposed model."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tThe hyperparameter analysis only examines embedding dimension $h$, temperature $\\tau$, and loss weight $\\lambda$, ignoring critical parameters like the number of Mamba layers.\n2.\tNo theoretical or experimental analysis of convergence, complexity bounds, or robustness guarantees is included.\n3.\tThe authors provide complexity analysis in Section 2.5, but the paper does not discuss practical scalability on very large graphs or compare with baselines empirically.\n4. Although meta-path mamba module is novel to me, the motivation of using it should be clarified in introduction."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919711144,"tcdate":1761477506835,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7632/Reviewer_A832"],"signatures":["ICLR.cc/2026/Conference/Submission7632/Reviewer_A832"],"forum":"4xjkRxcYHV","number":1,"license":"CC BY 4.0","cdate":1761477506835,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7632/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919711144,"domain":"ICLR.cc/2026/Conference","replyto":"4xjkRxcYHV","id":"XUegeIHYfL","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Multi-instance/Multi-view Learning","Representation Learning"]},"supplementary_material":{"value":"/attachment/f71c2821ac88806a7a65d3c1babe3248a60c0c84.zip"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Heterogeneous graph representation learning leverage the rich semantics and complex structural relationships within heterogeneous graphs. However, existing methods often fail to capture long-range semantic dependencies and localized structural patterns simultaneously. Therefore, we propose a novel Dual-View Contrastive Learning framework (DVCL) for heterogeneous graph representation learning. Specifically, the Graph Schema View Module (GSVM) is conducted to model the structural dependencies by leveraging a relational graph neural network with type-aware message passing and adaptive residual connections. Then, the Semantic Meta-Path Mamba Module (SMPMM) is designed to capture high-order semantic dependencies through a globally enhanced Mamba backbone, equipped with multi-resolution fusion and directional positional encodings. Moreover, a dynamic bidirectional contrastive learning is constructed to integrate the semantic view and structural view, treating each view as a learnable augmentation of the other to ensure robust and complementary representations. Extensive experiments on four datasets demonstrate that the proposed method consistently outperforms state-of-the-art methods, in terms of classification and clustering tasks."},"_bibtex":{"value":"@misc{\nyang2026a,\ntitle={A Dual-View Contrastive Learning Framework for Heterogeneous Graph Representation Learning},\nauthor={Binyun Yang and Zhi Chen and Qiaoqin Li and Rongyao Hu and Yongguo Liu},\nyear={2026},\nurl={https://openreview.net/forum?id=4xjkRxcYHV}\n}"},"title":{"value":"A Dual-View Contrastive Learning Framework for Heterogeneous Graph Representation Learning"},"pdf":{"value":"/pdf/d3fd3e481eaac57079e6957d7ae785c6e6ec268d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yang|a_dualview_contrastive_learning_framework_for_heterogeneous_graph_representation_learning"},"authorids":{"value":["~Binyun_Yang2","~Zhi_Chen13","~Qiaoqin_Li1","~Rongyao_Hu1","~Yongguo_Liu1"]},"authors":{"value":["Binyun Yang","Zhi Chen","Qiaoqin Li","Rongyao Hu","Yongguo Liu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Bi-Mamba Bayesian-MAGAC, combining bidirectional selective state-space models (Bi-Mamba) with Bayesian multi-head adaptive graph attention convolution (MAGAC) for financial time-series forecasting. Bi-Mamba processes sequences bidirectionally with linear complexity, while MAGAC constructs adaptive graph topologies and performs spectral filtering. Experiments on NASDAQ, NYSE, and DJIA show improvements over baselines."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"See the Weakness above."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The integration of linear-time Mamba for temporal dynamics with graph convolution for cross-asset relationships effectively addresses both long-range dependencies and inter-asset correlations in financial forecasting.\n\n2. The Bayesian treatment captures both epistemic and graph structural uncertainty through MC-Dropout and DropEdge, with closed-form variance propagation maintaining computational efficiency."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Evaluation uses only three datasets with implausibly high IC values (~0.94) compared to baselines (~0.2-0.3), raising concerns about potential data leakage or experimental setup issues.\n\n2. The code is not available for reproducibility."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918902284,"tcdate":1762056679948,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6564/Reviewer_7kbC"],"signatures":["ICLR.cc/2026/Conference/Submission6564/Reviewer_7kbC"],"forum":"KqCU5rfcMm","number":4,"license":"CC BY 4.0","cdate":1762056679948,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6564/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918902284,"domain":"ICLR.cc/2026/Conference","replyto":"KqCU5rfcMm","id":"tzfZGYei73","forumContent":{"TLDR":{"value":"Bi-Mamba & Bayesian MAGAC combines bidirectional selective state-space models with Bayesian graph attention for scalable, uncertainty-aware financial forecasting in linear time."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["mamba","transformer","bayesian networks"]},"primary_area":{"value":"probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)"},"abstract":{"value":"This paper presents \\textbf{BIMAMBA \\& Bayesian-MAGAC}, a unified framework that integrates bidirectional Selective State-Space Models with Bayesian Multi-head Adaptive Graph Attention Convolution for uncertainty-aware financial forecasting. The framework addresses two fundamental challenges: capturing long-range temporal dependencies across volatile market regimes while maintaining linear complexity, and learning adaptive cross-sectional structure with calibrated predictive uncertainty. BIMAMBA processes sequences bidirectionally via reversible state-space filters, extracting complementary temporal features while preserving strict causality. MAGAC constructs dynamic adjacencies through Gaussian kernel and attention blending, followed by Chebyshev spectral filtering for multi-scale aggregation. The Bayesian extension treats adjacencies and spectral filters as stochastic variables via Monte Carlo Dropout and DropEdge, yielding posterior predictive distributions with closed-form variance propagation at $\\mathcal{O}(N)$ complexity. Comprehensive evaluations on U.S. equity indices demonstrate that the architecture achieves substantial improvements in both point prediction accuracy and uncertainty calibration compared to established baselines, with statistically significant correlation between predicted uncertainty and prediction difficulty, suggesting practical utility for risk-aware decision making in financial applications.\n\\textit{The code is available on GitHub but has been hidden to preserve anonymity during the review process.}"},"_bibtex":{"value":"@misc{\ndang2026integrating,\ntitle={Integrating Selective State-Space Models and Bayesian Graph Attention for Uncertainty-aware Time-Series Analysis},\nauthor={Quang-Vinh Dang and SonAn Nguyen Ngoc},\nyear={2026},\nurl={https://openreview.net/forum?id=KqCU5rfcMm}\n}"},"title":{"value":"Integrating Selective State-Space Models and Bayesian Graph Attention for Uncertainty-aware Time-Series Analysis"},"pdf":{"value":"/pdf/2df5bc4bd83d8fe264929f03c5c341ba93bd102f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"dang|integrating_selective_statespace_models_and_bayesian_graph_attention_for_uncertaintyaware_timeseries_analysis"},"authorids":{"value":["~Quang-Vinh_Dang1","~SonAn_Nguyen_Ngoc1"]},"authors":{"value":["Quang-Vinh Dang","SonAn Nguyen Ngoc"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2501.11729v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"rando|serpent_selective_resampling_for_expressive_state_space_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Stefano_Rando:","https://dblp.org/search/pid/api?q=author:Luca_Romani:","~Matteo_Migliarini1","https://dblp.org/search/pid/api?q=author:Luca_Franco:","https://dblp.org/search/pid/api?q=author:Denis_A._Gudovskiy:","https://dblp.org/search/pid/api?q=author:Fabio_Galasso:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2501.11729"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2501-11729,\n  publtype={informal},\n  author={Stefano Rando and Luca Romani and Matteo Migliarini and Luca Franco and Denis A. Gudovskiy and Fabio Galasso},\n  title={SeRpEnt: Selective Resampling for Expressive State Space Models},\n  year={2025},\n  month={January},\n  cdate={1735689600000},\n  journal={CoRR},\n  volume={abs/2501.11729},\n  url={https://doi.org/10.48550/arXiv.2501.11729}\n}\n"},"abstract":{"value":"State Space Models (SSMs) have recently enjoyed a rise to prominence in the field of deep learning for sequence modeling, especially as an alternative to Transformers. Their success stems from avoiding two well-known drawbacks of attention-based models: quadratic complexity with respect to the sequence length and inability to model long-range dependencies. The SSM variant Mamba has demonstrated performance comparable to Transformers without any form of attention, thanks to the use of a selective mechanism for the state parameters. Selectivity, however, is only evaluated empirically and the reasons of its effectiveness remain unclear. In this work, we show how selectivity is related to the sequence processing. Our analysis shows that selective time intervals in Mamba act as linear approximators of information. Then, we propose our SeRpEnt architecture, a SSM that further exploits selectivity to compress sequences in an information-aware fashion. It employs a resampling mechanism that aggregates elements based on their information content. Our empirical results in the Long Range Arena benchmark and other language modeling tasks show benefits of the SeRpEnt's resampling mechanism."},"title":{"value":"SeRpEnt: Selective Resampling for Expressive State Space Models"},"authors":{"value":["Stefano Rando","Luca Romani","Matteo Migliarini","Luca Franco","Denis A. Gudovskiy","Fabio Galasso"]}},"tmdate":1771350919915,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2501-11729"],"tcdate":1771350914793,"writers":["~"],"signatures":["~Matteo_Migliarini1"],"forum":"fxcplRimcn","license":"CC BY-SA 4.0","number":825159,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1771350919915,"domain":"DBLP.org","id":"fxcplRimcn","version":2},{"content":{"summary":{"value":"This manuscript proposes a novel linear-complexity separable self-attention mechanism called Vision Mamba Inspired Separable self-Attention (VMI-SA), which draws inspiration from the SSM/Mamba while avoiding integrating any SSM blocks and featuring an attention computation process distinct from existing mechanisms. It first distill four design principles by analyzing the strengths and weaknesses of separable self-attention, then design a recurrent formulation of VMI-SA and a matrix formulation to enhance token dependency modeling and computational efficiency. Based on VMI-SA, they construct two proof-of-concept networks, VMINet and VMIFormer, and conduct fair comparisons with state-of-the-art Transformers, CNNs, and ViMs by controlling parameters, FLOPs, and encoder counts. Experimental results show that VMINet and VMIFormer achieve competitive performance in ImageNet-1K image classification, MSCOCO object detection, and ADE20K semantic segmentation, demonstrating VMI-SA’s effectiveness in balancing performance and efficiency."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. How does each detailed structure affect the final performance?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This paper introduces an interesting model, which incorporates separate self-attention modules into the Mamba marco design.\n2. The experiment are conducted on competitive benchmarks, e.g., ImageNet, COCO and ADE20K.\n3. The final model have linear complexity, which is a very promising research topic to explore."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The novelty is limited. This paper incorporated the minor design in separable self-attention into the Mamba marco design, titled Mamba Inspired Separable self-Attention. It is very similar to MLLA (Mamba-Inspired Linear Attention)[1] , which incroporate the Mamba minor design into the vision transformer marco design.\n2. The paper also lacks the method comparsion and performance comparsion with MLLA[1].\n3. Although the authors claim this is a linear model, the performance when the token length varies is missing.\n4. There is nearly no ablation in the submission. Only ablation in mask type in Tab 5. However, how each minor design affects the final result is unclear.\n\n[1] Han et al, Demystify Mamba in Vision: A Linear Attention Perspective, in NeurIPS 2024."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918912702,"tcdate":1762007926998,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6579/Reviewer_t7qr"],"signatures":["ICLR.cc/2026/Conference/Submission6579/Reviewer_t7qr"],"forum":"1KUMxnrhnH","number":3,"license":"CC BY 4.0","cdate":1762007926998,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6579/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918912702,"domain":"ICLR.cc/2026/Conference","replyto":"1KUMxnrhnH","id":"b7shFsvRST","forumContent":{"TLDR":{"value":"Inspired by the state space model at the core of Vision Mamba, we propose a novel separable self-attention mechanism termed Vision Mamba-Inspired Separable Self-Attention."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Separable Self-attention","State Space Models","Vision Mamba"]},"supplementary_material":{"value":"/attachment/0cd46f2349f939d5a4b0d712c9497c03b36d205a.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Separable self-attention is an early attention mechanism with linear complexity. When parameters and FLOPs are comparable, lightweight networks built upon separable self-attention and its variants underperform the recent Vision Mamba (ViM). By analyzing the strengths and weaknesses of separable self-attention, we distill four design principles and, inspired by the State Space Model (SSM) serving as the core of ViM, propose a novel separable self-attention termed Vision Mamba Inspired Separable self-Attention (VMI-SA). Notably, VMI-SA does not incorporate any SSM blocks, and its attention computation process differs from all existing attention mechanisms to the best of our knowledge. We introduce proof-of-concept networks, VMINet and VMIFormer, enabling fair comparisons with ViMs through deliberate control of parameters, FLOPs, and encoder numbers. Compared to state-of-the-art Transformers, CNNs, and ViMs, VMINet and VMIFormer achieve competitive results in image classification and high-resolution dense prediction tasks."},"_bibtex":{"value":"@misc{\nzhang2026a,\ntitle={A Separable Self-attention Inspired by the State Space Model for Computer Vision},\nauthor={Juntao Zhang and Jun Zhou and Kun Bian and zhou you and Jianning Liu},\nyear={2026},\nurl={https://openreview.net/forum?id=1KUMxnrhnH}\n}"},"title":{"value":"A Separable Self-attention Inspired by the State Space Model for Computer Vision"},"pdf":{"value":"/pdf/763025fad779b7f900ba99ff6984267142f9e588.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|a_separable_selfattention_inspired_by_the_state_space_model_for_computer_vision"},"authorids":{"value":["~Juntao_Zhang1","~Jun_Zhou16","~Kun_Bian1","~zhou_you1","~Jianning_Liu1"]},"authors":{"value":["Juntao Zhang","Jun Zhou","Kun Bian","zhou you","Jianning Liu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes WARP, a novel class of recurrent neural networks (RNNs) that perform sequence modeling directly in weight-space, blending linear recurrence with non-linear decoding. Unlike standard RNNs, which maintain a hidden state that is a result of propagating the sequence through the network, the proposed model’s hidden state is instead equal to the parameter vector of a so called auxiliary (“root”) MLP and is updated over time via a linear recurrence. Each update is driven by consecutive input differences, and the auxiliary MLP provides nonlinear decoding using an input that encodes the canonical ordering of the sequence.  This formulation enables gradient-free adaptation and in-context learning during inference, as well as the injection of physics priors through explicit parameter constraints.\nExtensive experiments are conducted across time-series analysis, dynamical system reconstruction, and multivariate time-series classification. WARP shows consistent or superior performance to state-of-the-art baselines on most tasks. A physics-informed variant, WARP-Phys, achieves significant improvements on physical dynamics reconstruction tasks."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Have you evaluated WARP’s performance on noisy measurement scenarios for dynamical system reconstruction?\n2. Could you discuss related sequence modeling approaches in weight space and explain why these were not included as baselines in any of the tasks?\n3. For the Energy Prediction experiment: what are the current state-of-the-art results? In general, for all experiments, how were baselines tuned, and how much effort was spent on hyperparameter optimization? Also, performance of S4 is reported only on MNIST, and ConvCNP only on CelebA. Full performance tables in the appendix would improve transparency.\n4. Have you considered benchmarking WARP on long-range dependency benchmarks, such as Long Range Arena (LRA)? This could contextualize WARP’s capabilities relative to other established long-sequence models that were evaluated on these tasks."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. Novel conceptual framing:
 The idea of treating the recurrent hidden state of a linear state-space model as the weights of another neural network is both elegant and novel. It bridges ideas from fast weights, meta-learning, hypernetworks and structured state-space models while maintaining linear recurrence efficiency. Additionally, it offers a built-in support for gradient-free adaptation, in-context learning, and physics-informed modeling in a single architecture.\n2. Computational Efficiency: Once the model has learned from the context, the final root network can be extracted and reused to process subsequent queries without reevaluating the entire sequence, yielding significant computational savings compared to other in-context learning models. Furthermore, the proposed architecture leverages a dual training mode that combines linear recurrence with a parallel scan operator – a well-established technique in the State Space Model (SSM) literature – to accelerate state propagation. Together, these design choices lead to notable computational efficiency improvements.\n3. Strong empirical results: Competitive or superior performance on time-series analysis, especially Traffic Flow Forecasting (despite ignoring graph priors) and Image completion, as well as dynamical system reconstruction. The inclusion of a physics-informed variant further demonstrates the framework’s adaptability and potential for interpretability.\n4. Clarity and completeness:
 The paper is well-written, includes high-quality figures, ablations, detailed appendices, and clear pseudocode."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Scalability constraints: 
The main bottleneck is the large transition matrix which scales quadratically with the number of root-network parameters. Experiments are thus limited to moderate model sizes, raising questions about feasibility for large-scale models.\n2. Limited theoretical grounding:
 While the empirical evidence is compelling, the theoretical analysis of representational capacity and stability (e.g., under linear recurrence updates) still remains to be established. \n3. Computational cost reporting:
 Although the recurrence is linear, updating and decoding weight vectors remains costly. Memory and compute scaling with model size are not fully quantified and are only provided in the appendix, but entirely missing from the main body of the paper.\n4. Limited setting for dynamical system reconstruction:
 While the possibility of making the network physics-informed is compelling, the shown examples illustrate this for relatively simple systems with a small number of parameters. While the proposed method clearly allows for in-context learning, and hence does not need to retrain a network for each new dynamical system (from the same category), the setups are done for what appears to be noiseless input-output data, and a low number of parameters. One could, instead of learning the entire mapping of the system, learn only its phase or exponential mapping for any other sequence model that allows for ICL, in the same way as demonstrated in this work. This alternative formulation would serve as a fairer baseline for comparison.\n5. Novelty relative to prior work:
 There is conceptual overlap with other concepts briefly outlined in the related work (e.g., fast weight RNNs), though the authors’ framing is distinctive. A more explicit comparison to those baselines would help to better position WARP’s contribution."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917574881,"tcdate":1761920579154,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4784/Reviewer_3Pnu"],"signatures":["ICLR.cc/2026/Conference/Submission4784/Reviewer_3Pnu"],"forum":"zHdKaF3ZM7","number":2,"license":"CC BY 4.0","cdate":1761920579154,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4784/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917574881,"domain":"ICLR.cc/2026/Conference","replyto":"zHdKaF3ZM7","id":"K7gWau6xml","forumContent":{"TLDR":{"value":"We introduce a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["physics-informed machine learning","weight-space learning","meta-learning","deep sequence modeling","linear recurrence","test-time training"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"We introduce WARP (**W**eight-space **A**daptive **R**ecurrent **P**rediction), a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into fixed-dimensional hidden states, WARP explicitly parametrizes its hidden state as the weights and biases of a distinct auxiliary neural network, and uses input differences to drive its recurrence. This brain-inspired formulation enables efficient gradient-free adaptation of the auxiliary network at test-time, in-context learning abilities, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state-of-the-art baselines on diverse classification tasks, featuring in the top three in 4 out of 6 real-world challenging datasets. Furthermore, extensive experiments across sequential image completion, multivariate time series forecasting, and dynamical system reconstruction demonstrate its expressiveness and generalization capabilities. Remarkably, a physics-informed variant of our model outperforms the next best model by more than 10x. Ablation studies confirm the architectural necessity of key components, solidifying weight-space linear RNNs as a transformative paradigm for adaptive machine intelligence."},"_bibtex":{"value":"@inproceedings{\nnzoyem2026weightspace,\ntitle={Weight-Space Linear Recurrent Neural Networks},\nauthor={Roussel Desmond Nzoyem and Nawid Keshtmand and Enrique Crespo-Fernandez and Idriss TSAYEM and Raul Santos-Rodriguez and David A.W. Barton and Tom Deakin},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zHdKaF3ZM7}\n}"},"title":{"value":"Weight-Space Linear Recurrent Neural Networks"},"pdf":{"value":"/pdf/c01f4c1dbdaf1c6e821cde510623f871a07c3404.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"nzoyem|weightspace_linear_recurrent_neural_networks"},"authorids":{"value":["~Roussel_Desmond_Nzoyem1","~Nawid_Keshtmand1","~Enrique_Crespo-Fernandez1","~Idriss_TSAYEM1","~Raul_Santos-Rodriguez1","~David_A.W._Barton1","~Tom_Deakin1"]},"authors":{"value":["Roussel Desmond Nzoyem","Nawid Keshtmand","Enrique Crespo-Fernandez","Idriss TSAYEM","Raul Santos-Rodriguez","David A.W. Barton","Tom Deakin"]}},"version":2},{"content":{"summary":{"value":"This paper implements a variant of state-space models inspired by the information bottleneck principle. Thy argue that this information bottleneck principle can result in an inherently robust and interpretable architecture, whereas standard MAMBA does not have any incentives to select only useful parts of the context and can \"overfit\" in various ways with longer contexts. They propose a variant of the infromation bottleneck for the sequence modelling problem and provide a variational upper-bound which can be realized via stochastic transition dynamics in Mamba. They provide two formulations:  Baleen-B samples transitions from a bernoulli distributions and Bernoulli E samples the time-interval parameter randomly. They show that Baleen improves language modelling performance relative to MAMBA, can yield inherent interpretability, and appears to induce superior robustness to adversarial attacks/perturbations."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"(1) Could the authors please clarify whether their problem statement is related to that shown in the Stuffed Mamba paper? \n(2) Could the authors clarify the hyper-parameter tuning details for the prior parameters?\n(3) Could the authors mention how is stochasticity of the parameters utilized at inference time.\n(4) Could the authors mention how they expect the benefits of BALEEN to scale at different sequence lengths/state sizes?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"I think that this paper attacks an important problem facing state-space models: their seeming difficulty in rejecting irrelevant historical information in Mamba models. This has, for example, been documented in [1] where they attribute that a inability to learn to forget information contributes to performance degradations when Mamba models are tested beyond their training context size. Additionally, I find that the connection to information bottleneck principle is interesting and they clearly explain the derivations along the way to their variational upper bound and training algorithm. I believe that it is interesting that they derive two possible instantiations of their approach based off Bernoulli sampling of the transition matrix and . They also perform evaluations on a range of model properties, including interpretability, robustness, and general performance -- which I find are interesting.\n[1] Chen, Yingfa, et al. \"Stuffed Mamba: Oversized States Lead to the Inability to Forget.\" arXiv preprint arXiv:2410.07145 (2024)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Motivations for the Method Could Be Presented Better**: The authors introduce their problem, in my opinion, far too abstractly. They comment on there being \"no incentive\" for the MAMBA model to compress irrelevant contextual information out. They provide some vague intuitions that this may be connected to worse performance/adversarial vulnerability etc. However, these claims are largely intuitive and do not seem entirely convincing.  I believe that the authors could better motivate their methods by discussing more direct evidence that MAMBA models detrimentally fail to remove knowledge. For example, connecting with the observations in [1] and/or providing more detailed results tying together the inability to forget with adversarial vulnerability/overfitting could be very beneficial for this paper.\n\n**Better Clarity in the Implementations** Although the authors do a good job of presenting the mathematical derivations behind their method, the implementation details could be more clear. In fact, I would argue that too much time is spent on the mathematical derivation, relative to implementation details. After reading the paper, for example, I was a bit confused on several implementation details. For example, it was not clear to me how is inference performed in Baleen? The authors mention that the expectation of the stochastic matrices is used for interpretation, but it remained unclear if this is for the general forward pass or just the interpretations. Similarly, how are the hyperparameters for the prior distributions chosen? This seems to be important for the reproducability of the work but is highly unclear from the paper.\n\n**Show Results Across Architecture and Sequence Length** The authors performed some experiments at a 20B token scale and superior performance along some baselines. However, I would be interested to know more about whether these benefits only arrive in a small-scale training setting. For example, in [1] they note that the relationship between the state parameters and the sequence length control how much forgetting is learned. Thus, it would be interesting to learn more about whether the benefits of BALEEN are amplified or mitigated depending on the relationship between Architecture v.s. Sequence Length. It would be good to haver more clarity about in what regimes the information bottleneck is particularly helpful, as this would improve our understanding of the method and its benefits.\n\n\n**Provide Qualitative Examples of the Difference Between Standard Mamba and BALEEN Interpretations**  Currently, the paper provides some examples. of the BALEEN interpretability an argues that they are superior to Mamba. There are some quantitative results on this as well. However, I feel that the it would help my understanding of the paper if there were qualitative examples of where a comparable setup with Mamba fails.\n\n\n[1] Chen, Yingfa, et al. \"Stuffed Mamba: Oversized States Lead to the Inability to Forget.\" arXiv preprint arXiv:2410.07145 (2024)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925365569,"tcdate":1761875904552,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15041/Reviewer_vu9z"],"signatures":["ICLR.cc/2026/Conference/Submission15041/Reviewer_vu9z"],"forum":"976tPua6Hd","number":2,"license":"CC BY 4.0","cdate":1761875904552,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15041/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925365569,"domain":"ICLR.cc/2026/Conference","replyto":"976tPua6Hd","id":"8bucd9xppo","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Interpretation","Robustness","Large Language Models","State-space models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"We introduce \\textit{\\textbf{Baleen}}, a family of state space models that unifies \\textbf{stochastic selection} with \\textbf{information bottleneck} to build interpretable and robust long‑context learners. Unlike Mamba/Mamba2’s deterministic gates, Baleen treats selection as a random variable and regularizes it with a closed‑form KL to a sparsity prior: (i) \\textit{\\textbf{\\bib}} samples Bernoulli state‑transition gates; (ii) \\textit{\\textbf{\\gib}} samples Exponential  time‑intervals. This yields an explicit trade‑off between retention and compression and exposes token‑level selection heatmaps at inference for self‑interpretation. On language benchmarks, \\ib  improves average accuracy over Mamba2 by +0.95 at 370M pretraining and +1.38 at 7B finetuning. Baleen delivers stronger robustness to localized perturbations and adversarial attacks: under CIFAR‑10 sequence perturbation, prefix damage falls to 0.6\\% vs 26.5\\% for Mamba2 (average under attacks 0.542 vs 0.385). Finally, Baleen’s self‑interpretations outperform IG/Grad‑CAM on average fidelity across four text classification tasks. We will release our Baleen‑7B models on Hugging Face with code, checkpoints, and an interactive selection‑heatmap demo."},"_bibtex":{"value":"@misc{\nzhu2026baleen,\ntitle={Baleen: Self\\nobreakdash-Interpretable, Robust {SSM}s with Stochastic Selective Memory},\nauthor={Jiajun Zhu and Yuehao Wang and Peihao Wang and Ruisi Cai and Siqi Miao and Liliang Ren and Pan Li and Jason D. Lee and Zhangyang Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=976tPua6Hd}\n}"},"title":{"value":"Baleen: Self‑Interpretable, Robust SSMs with Stochastic Selective Memory"},"pdf":{"value":"/pdf/651f6e0f7dafa731cf8adeb1471c685a32749a87.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhu|baleen_selfinterpretable_robust_ssms_with_stochastic_selective_memory"},"authorids":{"value":["~Jiajun_Zhu1","~Yuehao_Wang1","~Peihao_Wang1","~Ruisi_Cai1","~Siqi_Miao1","~Liliang_Ren1","~Pan_Li2","~Jason_D._Lee1","~Zhangyang_Wang1"]},"authors":{"value":["Jiajun Zhu","Yuehao Wang","Peihao Wang","Ruisi Cai","Siqi Miao","Liliang Ren","Pan Li","Jason D. Lee","Zhangyang Wang"]}},"version":2},{"content":{"summary":{"value":"This paper addresses high-dimensional time series prediction with correlated units (e.g., stocks, loans, sensors). It needs to capture cross-sectional dependencies among M units and temporal dynamics over T steps, but direct joint modeling is computationally infeasible. Existing methods rely on hand-crafted cross-sectional features, fail to adapt to dynamic unit counts, and miss latent correlations—though the problem itself is important.\n\nThe paper proposes the Set-Sequence model (Set module for linear-complexity permutation-invariant cross-sectional summaries; Sequence module for unit temporal dynamics). Evaluated on a synthetic loan default task, 36,600 U.S. equities, and 5M mortgage records, it outperforms baselines (Transformer, S4, domain models): 4.4–10.2× lower KL divergence, 22%–42% higher equity Sharpe ratio, and 0.683 mortgage AUC."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Sensitivity to look-back parameter 𝐿: The paper fixes 𝐿=3 without justification. How sensitive is the model to this choice? Does increasing \n𝐿 enhance long-term dependency modeling or simply increase computation? A short sensitivity analysis would clarify its role.\n\nRobustness of linear aggregation: The Set module employs mean pooling for cross-sectional summarization. Given potential heterogeneity among units, is this pooling stable or biased toward large or noisy samples? Have the authors explored normalization or adaptive weighting schemes?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"Strength 1: Set-Sequence models cross-sectional dependencies as a permutation-invariant set summary. Linear pooling plus a shallow readout is proven to uniformly approximate any continuous permutation-invariant target. This result supplies an expressive lower bound under exchangeability and directly informs the choice of pooling order and hyper-parameters.\n\nStrength 2: The module plugs into any temporal backbone with Θ(M) complexity. A single scan and per-unit temporal update reduce FLOPs by an order of magnitude relative to self-attention when M≥2500. Units can be added or dropped at inference without retraining, and memory grows linearly, so one A6000 GPU handles 5 M samples.\n\nStrength 3: Consistent, measurable gains are observed on synthetic contagion, equity-portfolio, and mortgage-default tasks. Compared to strong baselines, it achieves 10× KL reduction, +42 % Sharpe, and +4 average AUC, with improvements that grow monotonically with pool size. The learned summary correlates 0.67 with the realized foreclosure rate, delivering an auditable macro signal for risk management."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Limited methodological novelty: The proposed Set-Sequence model lacks substantial innovation. It simply combines an existing permutation-invariant set encoder with a standard sequential backbone, following the design principles of Deep Sets and Set Transformer with only minor modifications. Theoretical results are mostly restatements of known approximation properties rather than new insights. Overall, the contribution feels more like an engineering integration than a novel modeling paradigm.\n\nOutdated and insufficient experimental comparisons: The experiments rely on relatively old baselines and omit recent state-of-the-art time-series architectures such as iTransformer, TimeMixer, PatchTST, and Mamba. The reported improvements over older models therefore do not convincingly demonstrate the superiority of the proposed method. Moreover, the empirical evaluation lacks comprehensive analyses such as cross-task generalization, robustness, or hyperparameter sensitivity.\n\nShallow theoretical and architectural analysis: The claimed linear scalability is only valid under a simple mean-pooling assumption, which is not evaluated in heterogeneous or strongly correlated scenarios. The paper does not analyze how this simplification affects representational capacity or generalization performance, nor does it study how the look-back parameter interacts with temporal dependencies. Consequently, the theoretical analysis remains shallow and fails to substantiate the claimed advantages of the architecture."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922781542,"tcdate":1761806836671,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11754/Reviewer_eEp8"],"signatures":["ICLR.cc/2026/Conference/Submission11754/Reviewer_eEp8"],"forum":"c1A3DCni3K","number":2,"license":"CC BY 4.0","cdate":1761806836671,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11754/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922781542,"domain":"ICLR.cc/2026/Conference","replyto":"c1A3DCni3K","id":"LCngThQPbj","forumContent":{"TLDR":{"value":"When numerous time series are exchangeable, the Set-Sequence model efficiently captures cross-sectional dependencies"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Set modeling","Sequence modeling","High-dimensional time series","Cross-sectional dependencies","Interpretable AI","Synthetic data","Mortgage risk prediction","Exchangeability","Time series prediction","Financial AI","Deep learning"]},"supplementary_material":{"value":"/attachment/325e76334150d1a8a8e96c235d088931acd8c9ae.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Many prediction problems across science and engineering, especially in finance and economics, involve large cross-sections of individual time series, where each unit (e.g., a loan, stock, or customer) is driven by unit-level features and latent cross-sectional dynamics. While sequence models have advanced per-unit temporal prediction, capturing cross-sectional effects often still relies on hand-crafted summary features. We propose Set-Sequence, a model that learns cross-sectional structure directly, enhancing expressivity and eliminating manual feature engineering. At each time step, a permutation-invariant Set module summarizes the unit set; a Sequence module then models each unit’s dynamics conditioned on both its features and the learned summary. The architecture accommodates unaligned series, supports varying numbers of units at inference, integrates with standard sequence backbones (e.g., Transformers), and scales linearly in cross-sectional size. Across a synthetic contagion task and two large-scale real-world applications—equity portfolio optimization and loan risk prediction—Set-Sequence significantly outperforms strong baselines, delivering higher Sharpe ratios, improved AUCs, and interpretable cross-sectional summaries."},"_bibtex":{"value":"@misc{\nepstein2026a,\ntitle={A Set-Sequence Model for Time Series},\nauthor={Elliot L Epstein and Apaar Sadhwani and Kay Giesecke},\nyear={2026},\nurl={https://openreview.net/forum?id=c1A3DCni3K}\n}"},"title":{"value":"A Set-Sequence Model for Time Series"},"pdf":{"value":"/pdf/af39a1344d262612bd1bf64c15a33dbcc3a7f6ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"epstein|a_setsequence_model_for_time_series"},"authorids":{"value":["~Elliot_L_Epstein1","~Apaar_Sadhwani1","~Kay_Giesecke1"]},"authors":{"value":["Elliot L Epstein","Apaar Sadhwani","Kay Giesecke"]}},"version":2},{"content":{"summary":{"value":"This paper introduces two Linear RNN gating parameterizations that enhance expressivity while retaining efficiency. The first one, Higher-order Linear Recurrent Units (H-LRU), generalizes first-order recurrence to m-th order, enabling temporal mixing across multiple past hidden states. The second one, Block-Diagonal Linear Recurrent Units (BD-LRU), replaces purely diagonal state-transition matrices with a block-diagonal structure, permitting richer intra-block channel mixing. Both architectures incorporate selective gating and L1-normalization of the transition blocks (per-channel for H-LRU, per-row for BD-LRU) to ensure stability. The authors also provide a theoretical guarantee (Proposition 1) that under this L1-normalized gating, the hidden-state norm remains bounded. \n\nEmpirical results show that both H-LRU and BD-LRU achieve strong performance on the MAD benchmark tasks (compression, associative recall and compression). Moreover, BD-LRU matches or exceeds recent linear baselines (Mamba, DeltaNet, DeltaProduct) and even LSTM on the permutation tasks that require state tracking. Finally, the authors provide a parallel-scan implementation that is shown to yield throughput comparable to diagonal LRNNs for moderate block sizes."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. To what extent can the BD-LRU architecture support eigenvalues outside the positive range (i.e., negative or complex eigenvalues) in the state transition matrix, as discussed by Grazzi et al. 2025? How does the L1 norm, block size m or order in H-LRU affect the eigenspectrum?\n\n2. Outside the permutation tasks, have you also evaluated on regular language tasks such as parity or modular counting, where such eigenvalues were shown to be necessary for state-tracking?\n\n3. Can you also provide latency/throughput for varying block sizes (e.g., m=2, 4, 8, 16) and higher sequence lengths to better assess the runtime trade-off for the parallel-scan implementation?\n\n\n-- References --\n\n*Gu, A., Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv 2023*\n\n*Orvieto, A., Smith, S.L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., De, S.. Resurrecting recurrent neural networks for long sequences, in: International Conference on Machine Learning 2023*\n\n*Gu, A., Goel, K., Ré, C.. Efficiently modeling long sequences with structured state spaces. arXiv 2021*\n\n*J. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, R. Grazzi. DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products. In NeurIPS 2025.*\n\n*R. Grazzi, J. Siems, A. Zela, J.K.Franke, F. Hutter, M. Pontil. Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. In ICLR 2025*\n\n*M. Poli, A. W. Thomas, E. Nguyen, et al. Mechanistic Design and Scaling of Hybrid Architectures. arXiv 2024*"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. The proposed architectural parameterizations (H-LRU, BD-LRU) are conceptually elegant and clearly motivated. The authors show that higher-order recurrence and block structure are natural ways to increase mixing without resorting to full dense transitions. Both architectures incorporate input-dependent selective gates with L1-normalization (per-channel or per-row), ensuring forward-pass stability and bounded dynamics.\n\n2. The authors provide a theoretical justification (Proposition 1) that this normalization guarantees dynamical stability and normalized hidden-state evolution, showcasing that these architectures are well-behaved and stable by construction.\n\n3. Overall, the paper is well-organized. The progression from model definition (Sec 2) to normalization (Sec 3) and experiments (Sec 4, 5) is nicely structured. Figures and tables presenting results are clear too.\n\n4. The results are strong on the considered synthetic tasks. BD-LRU, in particular, shows significant gains over existing linear RNNs (Mamba, DeltaNet and DeltaProduct) and LSTMs also on state-tracking tasks like permutation tasks (Table 2).\n\n5. The proposed parallel-scan algorithm demonstrates that block-diagonal recurrences can achieve throughput competitive with diagonal ones (Figure 4), addressing a practical concern for deployment."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Limited evaluations**: The tasks considered in the empirical evaluations are only synthetic ones. While the use of the MAD benchmark is very useful, demonstrating competitive performance on large-scale language or long-sequence datasets (e.g., LRA, language modeling). This makes it hard to assess if the structural benefits of the proposed architectures transfer to more practical usage. The authors acknowledge this as future work though.\n\n2. The core contributions are about the structure of state mixing. Yet, the authors do not provide empirical analysis on the actual learned matrices and their properties, e.g., their eigenvalue spectra. This would provide crucial insight into what kind of mixing the model is learning (e.g., is BD-LRU learning negative eigenvalues?) and why it succeeds on state-tracking (permutation task).\n\n3. The discussion mentions FLOPs and throughput but lacks detailed runtime comparisons across block sizes, hidden dims and sequence lengths. It’s unclear how block‐size m scales in practice (for m large).\n\n4. The paper provides key code snippets in Appendix G, however, the full code is not yet released."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927459972,"tcdate":1762198143447,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17598/Reviewer_A46F"],"signatures":["ICLR.cc/2026/Conference/Submission17598/Reviewer_A46F"],"forum":"fF72TYOKjZ","number":4,"license":"CC BY 4.0","cdate":1762198143447,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17598/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927459972,"domain":"ICLR.cc/2026/Conference","replyto":"fF72TYOKjZ","id":"R1CfbGgXoV","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["linear RNNs","higher-order recurrence","block diagonal recurrence","normalization","synthetic tasks"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Linear recurrent networks (LRNNs) and linear state space models (SSMs) promise computational and memory efficiency on long-sequence modeling tasks, yet their diagonal state transitions limit expressivity. Dense and/or nonlinear architectures (e.g., LSTMs) on the other hand are provably more expressive, but computationally costly. \nHere, we explore how expressivity in LRNNs can be increased via richer state mixing across time and channels while maintaining competitive efficiency. Specifically, we introduce two structured LRNN architectures: (i) Higher-order Linear Recurrent Units (H-LRU), which generalize first-order recurrence to $m$-th order, mixing multiple past states, and (ii) Block-Diagonal LRUs (BD-LRU), which enable dense intra-block channel mixing. Per-channel (H-LRU) / per-row (BD-LRU) L1-normalization of selective gates stabilizes training and allows for scaling window/block sizes. In synthetic sequence-modeling benchmarks (compression, selective copying, associative recall), H-LRU is found to be the most parameter-efficient in compression, while the performance of BD-LRU matches or exceeds those of linear SSMs (Mamba), low-rank LRNNs (DeltaNet) and LSTM baselines. In permutation composition tasks ($S_3$-$S_5$), BD-LRU is found to efficiently solve these tasks at moderate block sizes, outperforming both linear and non-linear baselines. A parallel-scan implementation of the proposed architectures keeps the throughput competitive with diagonal LRNNs for moderate orders (H-LRU) and block sizes (BD-LRU), while preserving the efficiency that motivated LRNNs. \nThese results indicate that the structure of state mixing rather than width alone shapes expressivity of LRNNs, offering a practical route to closing the efficiency–expressivity gap in linear sequence models."},"_bibtex":{"value":"@misc{\ndubinin2026improved,\ntitle={Improved state mixing in higher-order and block diagonal linear recurrent networks},\nauthor={Igor Dubinin and Antonio Orvieto and Felix Effenberger},\nyear={2026},\nurl={https://openreview.net/forum?id=fF72TYOKjZ}\n}"},"title":{"value":"Improved state mixing in higher-order and block diagonal linear recurrent networks"},"pdf":{"value":"/pdf/7a4d7bd95df82a2db1bd82c38287ef7ad0d06fc5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"dubinin|improved_state_mixing_in_higherorder_and_block_diagonal_linear_recurrent_networks"},"authorids":{"value":["~Igor_Dubinin1","~Antonio_Orvieto3","~Felix_Effenberger1"]},"authors":{"value":["Igor Dubinin","Antonio Orvieto","Felix Effenberger"]}},"version":2},{"content":{"summary":{"value":"This paper proposes TFMAudio, a text-to-audio generation framework that integrates Time-Frequency Mamba (TFM) and Energy-Aware Guidance (EAG). The model achieves linear-time long-form generation and maintains temporal and spectral consistency, producing up to 30-minute high-fidelity 44.1 kHz audio with strong semantic alignment."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"**1. On the Effectiveness of Energy-Aware Guidance (EAG)**\n\nThe ablation results suggest that EAG provides only marginal gains across objective metrics.\nCould the authors elaborate on specific scenarios or qualitative aspects where EAG meaningfully contributes to stability or audio fidelity?\n\n\n**2. On Length Controllability During Generation**\n\nCan TFMAudio support arbitrary-length generation (e.g., 13s or 27s) beyond fixed training configurations?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"**1. Effective Dual-Axis Modeling via Time-Frequency Mamba**\n\nThe proposed Time-Frequency Mamba performs 1D scans along both temporal and frequency axes, allowing the model to jointly capture temporal causality and spectral correlation — a capability that conventional Transformers struggle to achieve.\n\n**2. Linear-Time Complexity for Long Sequences**\n\nThe Mamba-based recurrent formulation enables linear computational complexity O(L) with respect to sequence length, offering a far more efficient alternative to the quadratic O(L²) cost of Transformers while maintaining long-range dependencies.\n\n**3. Stable and Scalable Audio Generation with Energy-Aware Guidance**\n\nThe introduced Energy-Aware Guidance (EAG) mitigates state drift by decomposing the flow-matching velocity field and adaptively damping unstable components, enabling reliable ultra-long (30-minute) 44.1 kHz audio generation with temporal consistency."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**1. Marginal Impact of Energy-Aware Guidance (EAG)**\n\nAccording to the ablation results, the performance improvement from EAG is minimal, with only slight differences in objective metrics.\n\n**2. Lack of Flexible Length Control in Generation**\n\nWhile the paper demonstrates 10s and 30s generations, it is unclear whether the model allows arbitrary-length synthesis (e.g., 13s or 27s) or only supports pre-defined durations tied to training configurations."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918253226,"tcdate":1761552129837,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5772/Reviewer_xDov"],"signatures":["ICLR.cc/2026/Conference/Submission5772/Reviewer_xDov"],"forum":"WFgtMF6vk6","number":2,"license":"CC BY 4.0","cdate":1761552129837,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5772/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918253226,"domain":"ICLR.cc/2026/Conference","replyto":"WFgtMF6vk6","id":"v2ko9AABB5","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Text-to-Audio","Flow-Matching","Mamba","SSM","Diffusion"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Recent advancements in audio generation have been dominated by transformer-based diffusion models, which face challenges in extrapolating positional encodings and exhibit quadratic complexity in self-attention, limiting their consistency and efficiency for long-form generation.\nTo address these limitations, we propose TFMAudio, a novel latent audio generation model that integrates the strengths of Flow Matching and a custom-designed TFMamba backbone.\nTFMamba employs a dual-scan mechanism: TimeMamba captures long-range causal dependencies with linear complexity, while FrequencyMamba models spectral correlations such as harmonic structures. To enhance stability, we further introduce Energy-Aware Guidance (EAG), which mitigates state drift by adaptively regularizing classifier-free guidance. Experiments demonstrate that TFMAudio achieves state-of-the-art performance on text-to-audio benchmarks and exhibits robust extrapolation to ultra-long sequences. Remarkably, our model generates 30-minute high-fidelity audio while preserving temporal consistency and semantic alignment, significantly advancing the scalability and usability of text-to-audio models.\n  Demo:https://huggingface.co/spaces/tfmaudio/TFMAudio"},"_bibtex":{"value":"@misc{\ndai2026tfmaudio,\ntitle={{TFMA}udio: High-Fidelity Long-Form Text-to-Audio via Mamba-based Flow Matching},\nauthor={Hao Dai and Jagmohan Chauhan},\nyear={2026},\nurl={https://openreview.net/forum?id=WFgtMF6vk6}\n}"},"title":{"value":"TFMAudio: High-Fidelity Long-Form Text-to-Audio via Mamba-based Flow Matching"},"pdf":{"value":"/pdf/5833cc8c9f598257ed7359368750dd201f830de2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"dai|tfmaudio_highfidelity_longform_texttoaudio_via_mambabased_flow_matching"},"authorids":{"value":["~Hao_Dai1","~Jagmohan_Chauhan1"]},"authors":{"value":["Hao Dai","Jagmohan Chauhan"]}},"version":2},{"content":{"summary":{"value":"The paper introduces Vision Mamba Inspired Separable Self-Attention (VMI-SA), a new separable self-attention mechanism drawing design principles from State Space Models, particularly Mamba. The authors propose VMINet—a prototype vision backbone built purely from stacking VMI-SA blocks and downsampling layers. Through extensive experimentation across image classification, detection, and segmentation tasks, VMINet is shown to outperform state-space-based Vim models and be competitive with strong baselines in lightweight settings."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. The format of the paper is a little messy with large blanks and unaligned equations.\n\n2. The \"Related Works\" section is missing, making it confusing to position this paper in some lines of research and show its unique advantages."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"This paper analyzes different designs principles of self-attention, vision mamba, separable self-attention, and conclude the results into four rules to guide the design of the vision models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The involvement of causal mask does not make sense for most of the vision tasks since there are no causal hypotheses in the spatial dimension of the images and videos. That is why Mamba models [1,2,3] in vision need to define one or several complicated scanning sequence to ensure the visual signals are correctly modeled. In VMI-SA, the authors use two set of learnable gating parameters  $\\alpha$s and  $\\beta$s to control the proportion between the causal contexts and the direct contexts. It is of vital importance to carefully analyze how and go for different inputs and in different layers, which can provide meaningful insights on how these two types of context affect the model on vision tasks. Another important work [4] points out that the causal modeling in vision mamba models could be regarded as a forced local modeling pattern, which is also helpful. However, these analyses are missing in current submission, which fade the technical depth of the paper.\n\n2. For the discussion part of Effectiveness of VMI-SA, the authors replace the VMI-SA block with an FC layer. However, this design choice does not resemble ConvNeXt block since the normalization layers are not the same. Current drop of the accuracy cannot support the assertion. The authors could adopt the MetaFormer [5] archictecture equipped with VMI-SA, Pooling, and Self-attention, respectively to verify the impact of the spatial modeling module.\n\n3. The experiment results do not report the model variants in larger sizes, e.g, GFLOPs for inputs on the ImageNet-1K datasets, and models with longer sequence inputs, e.g., input resolutions. It is hard to distinguish the proposed VMINet out of the baselines such as ViM [1].\n\n[1] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. \"Vision mamba: Efficient visual representation learning with bidirectional state space model.\", ICML 2024\n\n[2] Liu, Yue, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. \"Vmamba: Visual state space model.\", NeurIPS 2024\n\n[3] Yang, Chenhongyi, Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot J. Crowley. \"Plainmamba: Improving non-hierarchical mamba in visual recognition.\", BMVC 2024\n\n[4] Han, Dongchen, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. \"Demystify mamba in vision: A linear attention perspective.\", NeurIPS 2024\n\n[5] Yu, Weihao, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. \"Metaformer baselines for vision.\", IEEE TPAMI"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918912246,"tcdate":1762159868467,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6579/Reviewer_RwYd"],"signatures":["ICLR.cc/2026/Conference/Submission6579/Reviewer_RwYd"],"forum":"1KUMxnrhnH","number":4,"license":"CC BY 4.0","cdate":1762159868467,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6579/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918912246,"domain":"ICLR.cc/2026/Conference","replyto":"1KUMxnrhnH","id":"dKY3anoCX2","forumContent":{"TLDR":{"value":"Inspired by the state space model at the core of Vision Mamba, we propose a novel separable self-attention mechanism termed Vision Mamba-Inspired Separable Self-Attention."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Separable Self-attention","State Space Models","Vision Mamba"]},"supplementary_material":{"value":"/attachment/0cd46f2349f939d5a4b0d712c9497c03b36d205a.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Separable self-attention is an early attention mechanism with linear complexity. When parameters and FLOPs are comparable, lightweight networks built upon separable self-attention and its variants underperform the recent Vision Mamba (ViM). By analyzing the strengths and weaknesses of separable self-attention, we distill four design principles and, inspired by the State Space Model (SSM) serving as the core of ViM, propose a novel separable self-attention termed Vision Mamba Inspired Separable self-Attention (VMI-SA). Notably, VMI-SA does not incorporate any SSM blocks, and its attention computation process differs from all existing attention mechanisms to the best of our knowledge. We introduce proof-of-concept networks, VMINet and VMIFormer, enabling fair comparisons with ViMs through deliberate control of parameters, FLOPs, and encoder numbers. Compared to state-of-the-art Transformers, CNNs, and ViMs, VMINet and VMIFormer achieve competitive results in image classification and high-resolution dense prediction tasks."},"_bibtex":{"value":"@misc{\nzhang2026a,\ntitle={A Separable Self-attention Inspired by the State Space Model for Computer Vision},\nauthor={Juntao Zhang and Jun Zhou and Kun Bian and zhou you and Jianning Liu},\nyear={2026},\nurl={https://openreview.net/forum?id=1KUMxnrhnH}\n}"},"title":{"value":"A Separable Self-attention Inspired by the State Space Model for Computer Vision"},"pdf":{"value":"/pdf/763025fad779b7f900ba99ff6984267142f9e588.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|a_separable_selfattention_inspired_by_the_state_space_model_for_computer_vision"},"authorids":{"value":["~Juntao_Zhang1","~Jun_Zhou16","~Kun_Bian1","~zhou_you1","~Jianning_Liu1"]},"authors":{"value":["Juntao Zhang","Jun Zhou","Kun Bian","zhou you","Jianning Liu"]}},"version":2},{"content":{"comment":{"value":"**[Reply 2/2]**\n\n**[W3]. Finally, I remain suspicious of the empirical results given that no learning rate tuning was reported. Without such tuning I find the results hard to trust.**\n\n**Answer:** We would like to clarify that our hyperparameter settings follow those established in previous works, specifically Mamba [Gu2023] for language modeling task on WikiText103 and MambaVision [Hatamizadeh2024] for image classification on ImageNet. Additionally, we have provided our code in the supplementary materials, ensuring that our results can be fully reproduced. \n\n**References**\n[Gu2023] Gu, A., & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752.\n[Hatamizadeh2024] Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083, 2024.\n\n-----\nWe hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience."}},"tmdate":1732161676491,"tcdate":1732161676491,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5608/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5608/Authors"],"forum":"qtTIP5Gjc5","number":10,"license":"CC BY 4.0","cdate":1732161676491,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5608/-/Official_Comment"],"mdate":1732161676491,"domain":"ICLR.cc/2025/Conference","replyto":"1U1sOyC6t6","id":"6K5j85g8Ug","forumContent":{"TLDR":{"value":"We describe the dynamical properties of tokens in a deep selective state-space model, and based on that, we suggest improvements to the model by excluding negatively impactful scenarios and reordering tokens using importance scores."},"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Selective state-space model","continuous-time limit","dynamical system","asymptotic behavior","token reordering"]},"supplementary_material":{"value":"/attachment/f01b2b77a31c531086f0fefd1143d89ad8415a39.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity.  We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance.  For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training.  Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance.  Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications."},"_bibtex":{"value":"@inproceedings{\nvo2025demystifying,\ntitle={Demystifying the Token Dynamics of Deep Selective State Space Models},\nauthor={Thieu Vo and Duy-Tung Pham and Xin T. Tong and Tan Minh Nguyen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=qtTIP5Gjc5}\n}"},"title":{"value":"Demystifying the Token Dynamics of Deep Selective State Space Models"},"pdf":{"value":"/pdf/abc1efe1f910ee48b743f7ba26000b1d5e622f85.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"vo|demystifying_the_token_dynamics_of_deep_selective_state_space_models"},"authorids":{"value":["~Thieu_Vo1","~Duy-Tung_Pham1","~Xin_T._Tong2","~Tan_Minh_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Thieu Vo","Duy-Tung Pham","Xin T. Tong","Tan Minh Nguyen"]}},"version":2},{"content":{"summary":{"value":"The paper investigates the memory decay problem in state-space models with a focus on the Mamba architecture and presents MemMamba, a memory-augmented variant designed to improve long-range information retention while preserving linear complexity. The authors conduct both mathematical and information-theoretic analyses to explain Mamba’s exponential memory decay and introduce two metrics, Expected Token Memory Fidelity (ETMF) and Expected Cross-Layer Memory Fidelity (ECLMF), which quantify information loss across tokens and layers. MemMamba incorporates a Note Block for dynamic state summarization together with sparse cross-token and cross-layer attention to restore salient information. Experiments on long-sequence benchmarks such as PG19 and Passkey Retrieval demonstrate improved stability and efficiency compared with Mamba and Transformer baselines."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"See the weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The topic is timely and relevant given the growing interest in SSM-based long-sequence modeling.\n- The paper presents a clear analysis of memory decay and provides intuitive metrics (ETMF / ECLMF) to visualize horizontal and vertical information loss.\n- The proposed architecture is well-motivated and empirically improves robustness on long-context benchmarks such as PG19 and Passkey Retrieval.\n- Experimental presentation and ablations are thorough, and the writing is generally clear."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**1. Methodological novelty is minimal within the hybrid SSM + attention family**\n\nMemMamba combines a selective state-space model with sparse cross-token and cross-layer attention, but this pattern has already been explored in Compressive Transformer, RetNet, LongMamba, and other recent variants. The Note Block functions similarly to prior compression or summarization modules, offering only minor procedural differences rather than a genuinely new architecture.\n\n**2. Theoretical analysis lacks rigor and depth**\n\nThe paper’s mathematical treatment mostly restates well-known properties of linear recurrent systems, such as exponential decay under $|A| < 1$. The proposed ETMF and ECLMF metrics are intuitive but not derived from principled information-theoretic foundations, and their empirical correlation with downstream performance remains unclear.\n\n**3. Experimental gains are not sufficiently validated against strong baselines**\n\nWhile results on PG19 and Passkey Retrieval are promising, comparisons exclude important contemporaries such as Mamba-2, RetNet, and RWKV. Parameter counts and training setups also differ, leaving uncertainty over whether improvements stem from the proposed mechanisms or from implementation choices.\n\n**4. Claimed efficiency improvements are weakly supported**\n\nThe reported 48% inference speedup is measured on a single GPU under a limited setup. No analysis is provided for scaling behavior or memory usage under multi-GPU or distributed inference conditions, making the efficiency claim difficult to generalize.\n\n**5. Writing occasionally overstates the contribution**\n\nPhrases like “breakthrough” and “new paradigm” exaggerate the paper’s significance given its incremental contribution. A more balanced presentation would strengthen credibility and highlight the genuine empirical strengths."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919697270,"tcdate":1761890121624,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7614/Reviewer_sU2y"],"signatures":["ICLR.cc/2026/Conference/Submission7614/Reviewer_sU2y"],"forum":"jPYJuvzof2","number":2,"license":"CC BY 4.0","cdate":1761890121624,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7614/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919697270,"domain":"ICLR.cc/2026/Conference","replyto":"jPYJuvzof2","id":"8m7il0VQ0X","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Long-sequence modeling","Mamba","Memory mechanisms"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the explosive growth of data, long-sequence modeling has become increasingly important in tasks such as natural language processing and bioinformatics. However, existing methods face inherent trade-offs between efficiency and memory. Recurrent neural networks suffer from gradient vanishing and explosion, making them hard to scale. Transformers can model global dependencies but are constrained by quadratic complexity. Recently, selective state-space models such as Mamba have demonstrated high efficiency with $O(n)$ time and $O(1)$ recurrent inference, yet their long-range memory decays exponentially. In this work, we conduct mathematical derivations and information-theoretic analysis to systematically uncover the memory decay mechanism of Mamba, answering a fundamental question: what is the nature of Mamba’s long-range memory and how does it retain information? To quantify key information loss, we further introduce horizontal–vertical memory fidelity metrics that capture degradation both within and across layers. Inspired by how humans distill and retain salient information when reading long documents, we propose MemMamba, a novel architectural framework that integrates state summarization mechanism together with cross-layer and cross-token attention, which alleviates long-range forgetting while preserving linear complexity. MemMamba achieves significant improvements over existing Mamba variants and Transformers on long-sequence benchmarks such as PG19-PPL and Passkey Retrieval, while delivering a 48\\% speedup in inference efficiency. Both theoretical analysis and empirical results demonstrate that MemMamba achieves a breakthrough in the complexity–memory trade-off, offering a new paradigm for ultra-long sequence modeling. The code and the pre-trained models will be released upon acceptance."},"_bibtex":{"value":"@misc{\nwang2026memmamba,\ntitle={MemMamba: Rethinking Memory Patterns in State Space Model},\nauthor={Youjin Wang and Yangjingyi Chen and Jiahao yan and Jiaxuan Lu and Xiao Sun},\nyear={2026},\nurl={https://openreview.net/forum?id=jPYJuvzof2}\n}"},"title":{"value":"MemMamba: Rethinking Memory Patterns in State Space Model"},"pdf":{"value":"/pdf/26376d032f685fb35659e4ab241730a5eb60a8ff.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wang|memmamba_rethinking_memory_patterns_in_state_space_model"},"authorids":{"value":["~Youjin_Wang1","~Yangjingyi_Chen1","~Jiahao_yan1","~Jiaxuan_Lu1","~Xiao_Sun8"]},"authors":{"value":["Youjin Wang","Yangjingyi Chen","Jiahao yan","Jiaxuan Lu","Xiao Sun"]}},"version":2},{"content":{"TLDR":{"value":"We provide the first systematic study of textual backdoors in Mamba language models, uncover their state-write and readout mechanisms, and propose TASWB to strengthen trigger activation."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Textual Backdoor Attacks; Mamba; Selective State Space Models; Backdoor Mechanism Analysis; Trigger Activation"]},"supplementary_material":{"value":"/attachment/22124c17387e8b4cd59021279f7d289f324afa4b.zip"},"primary_area":{"value":"alignment, fairness, safety, privacy, and societal considerations"},"abstract":{"value":"Textual backdoor attacks have been extensively studied for attention-based natural language processing (NLP) backbones, but their behavior in selective state-space language models remains unclear. We investigate how backdoors are activated in Mamba, where trigger information must be written into a recurrent state, preserved through subsequent updates, and exposed at the classifier readout. Across four datasets, we compare Mamba and Pythia at two model scales under BadNL, SynBkd, StyBkd, and BGMAttack. The results show that architecture and scale jointly shape backdoor vulnerability: Mamba achieves higher clean accuracy at small scale, while larger models make non-lexical triggers substantially more reliable. Mechanism analysis further reveals that Mamba backdoors are readout-aligned and partially concentrated at the classifier interface.The input-dependent time step \n is a strong diagnostic of triggered computation and attack success, whereas component replacement identifies the mixer, gate, scan-output, and state-write pathways as the causal drivers. Guided by this insight, we propose Trigger-Aware State-Write Boosting (TASWB), which strengthens trigger-aware state writes and target-class margins. Together, our findings provide the first systematic study of Mamba backdoor activation and expose a state-write-mediated attack surface in selective state-space language models."},"_bibtex":{"value":"@inproceedings{\nanonymous2026when,\ntitle={When Backdoors Meet Selective State Spaces: Mechanisms and Attacks in Mamba Language Models},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=t2dxTYa2z6},\nnote={under review}\n}"},"title":{"value":"When Backdoors Meet Selective State Spaces: Mechanisms and Attacks in Mamba Language Models"},"pdf":{"value":"/pdf/583e73b18cf1d87cddd78222bc85b2ff99b40a79.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791227282074,"tcdate":1789276418912,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission15652/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission15652/Authors"],"forum":"t2dxTYa2z6","license":"CC BY 4.0","number":15652,"cdate":1789276418912,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission15652/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791227282074,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"t2dxTYa2z6","version":2},{"content":{"summary":{"value":"This paper introduces CAB (Cross-architecture distillation via Attention Bridge), a framework for distilling knowledge from Transformer-based teacher models to state-space model (SSM) students, specifically Mamba. The core innovation is a lightweight, MLP-based \"Attention Bridge\" that aligns the implicit attention carriers in Mamba (the token-dependent projections $B$ and $C$) with the explicit key ($K$) and query ($Q$) representations from the Transformer.\n\nAfter the distillation process is complete, the MLP weights are discarded. The final product is a standalone Mamba model that has internalized the attention-based inductive biases from the Transformer teacher, with no inference-time overhead from the bridge components. The distillation process does not freeze the Mamba weights; instead, the gradients from the attention alignment loss and the standard output distillation loss are backpropagated through the MLP bridge and into all weights of the student model.\n\n\n\nThis method enables fine-grained, token-level supervision without the quadratic memory cost of aligning full attention matrices. It also employs a flexible, proportional layer-wise alignment strategy to handle architectural depth mismatches. The authors demonstrate CAB's effectiveness and data efficiency through extensive experiments on both vision (ImageNet) and language (OpenWebText, C4, WikiText) tasks, showing consistent improvements over standard and recent cross-architecture distillation baselines, particularly in low-data regimes."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"* Why was a 2-layer MLP chosen for the bridge instead of a simpler linear projection? Was this choice ablated?\n\n* Does the CAB method require the teacher and student to share the same tokenizer?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"* The formulation of the \"Attention Bridge\" is a creative and novel solution to the cross-architecture distillation problem. The insight to treat Mamba's $B$ and $C$ projections as analogous to the Transformer's $K$ and $Q$ under a linear attention approximation is theoretically grounded and is a natural extension of State-Space Duality. \n\n* The experimental evaluation is thorough, spanning multiple domains, model scales, and data regimes. The consistent and often substantial improvements over strong baselines is very convincing. The ablation studies are well-designed and provide crucial insights into the contribution of each component.\n\n* The paper is well-written. The problem is motivated, the method is explained with sufficient detail, and the results are presented logically and convincingly. \n\n* This work has high practical significance. By enabling data-efficient knowledge transfer from Transformers to Mamba, it lowers the barrier to adopting high-performance SSMs in real-world scenarios where data is limited and computational resources are constrained."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The authors use a 2-layer MLP with SiLU activations for the bridge modules ($\\phi_B$) and ($\\phi_C$). While effective, the paper does not ablate this choice against a simpler linear projection. A brief justification or experimental result showing the superiority of a non-linear transformation over a linear one would strengthen this design decision, especially given the focus on efficiency.\n\n* The proportional layer mapping $g(l)$ is a simple and effective heuristic. However, the paper does not explore or ablate against other possible strategies (e.g., learned alignment, manual assignment based on layer depth). A brief discussion on the choice and potential alternatives would be useful.\n\n* A key, unexplored trade-off is whether distilling a Transformer's inductive biases into a Mamba model also transfers the Transformer's weakness in length generalization. State-space models like Mamba are prized for their ability to generalize to sequences longer than those seen in training, a known challenge for Transformers. The paper lacks any analysis or discussion of whether the distilled models retain their robustness to longer sequence lengths."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922764659,"tcdate":1761664905278,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11726/Reviewer_bjGZ"],"signatures":["ICLR.cc/2026/Conference/Submission11726/Reviewer_bjGZ"],"forum":"WyyawATBhs","number":2,"license":"CC BY 4.0","cdate":1761664905278,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11726/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922764659,"domain":"ICLR.cc/2026/Conference","replyto":"WyyawATBhs","id":"w4C3LWmic6","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Knowledge distillation","SSM"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"State-space models (SSMs) have emerged as efficient alternatives to Transformers for sequence modeling, offering superior scalability through recurrent structures. \nHowever, their training remains costly and the ecosystem around them is far less mature than that of Transformers. \nMoreover, the structural heterogeneity between SSMs and Transformers makes it challenging to efficiently distill knowledge from pretrained attention models. \nIn this work, we propose **C**ross-architecture distillation via **A**ttention **B**ridge (**CAB**), a novel data-efficient distillation framework that efficiently transfers attention knowledge from Transformer teachers to state-space student models. \nUnlike conventional knowledge distillation that transfers knowledge only at the output level, CAB enables token-level supervision via a lightweight bridge and flexible layer-wise alignment, improving both efficiency and transferability.\nWe further introduce flexible layer-wise alignment strategies to accommodate architectural discrepancies between teacher and student. \nExtensive experiments across vision and language domains demonstrate that our method consistently improves the performance of state-space models, even under limited training data, outperforming both standard and cross-architecture distillation methods.\nOur findings suggest that attention-based knowledge can be efficiently transferred to recurrent models, enabling rapid utilization of Transformer expertise for building a stronger SSM community."},"_bibtex":{"value":"@misc{\nwang2026data,\ntitle={Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge},\nauthor={Penghao Wang and Yuhao Zhou and Mengxuan Wu and Panpan Zhang and Zhangyang Wang and Kai Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=WyyawATBhs}\n}"},"title":{"value":"Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge"},"pdf":{"value":"/pdf/1107d4419b0a58a0ec493949579e38815bca3071.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"wang|data_efficient_any_transformertomamba_distillation_via_attention_bridge"},"authorids":{"value":["~Penghao_Wang3","~Yuhao_Zhou4","~Mengxuan_Wu1","~Panpan_Zhang1","~Zhangyang_Wang1","~Kai_Wang8"]},"authors":{"value":["Penghao Wang","Yuhao Zhou","Mengxuan Wu","Panpan Zhang","Zhangyang Wang","Kai Wang"]}},"version":2},{"content":{"summary":{"value":"This paper presents \\modelname, a training-free, one-shot pruning technique for Mamba-based state-space language models. The method adapts the Optimal Brain Surgeon framework to accommodate the time-shared and discretized nature of the state-transition matrix in selective state-space modules, and incorporates a sensitivity-based approach for pruning feed-forward network components. The authors report that their algorithm can prune up to 50% of SSM weights without fine-tuning and with no observed drop in zero-shot accuracy, outperforming several baseline pruning strategies. The work offers a modest contribution to efficient compression of state-space models as an extension to an existing method."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please address the weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"Originality: The paper introduces an extension to the OBS pruning framework tailored to Mamba-based state-space models, addressing architectural challenges—like time-shared and discretized parameters—not considered in prior work. \n\nQuality: The approach has rigorous theoretical derivations and an empirical evaluation spanning different model sizes, pruning granularities, and both language modeling and reasoning benchmarks. The inclusion of ablation studies and sensitivity analyses validates the method’s effectiveness and robustness.\n\nClarity: The presentation is clear and organized, offering step-by-step explanations and visual aids that facilitate understanding of both the technical challenges and solutions. Key concepts and distinctions from prior work are well-articulated.\n\nSignificance: Efficient pruning of SSMs is practically important as these models see wider use. Achieving high sparsity without accuracy loss makes deployment more feasible and compression methods are important in this emerging area."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Limited Baseline Comparison: The experimental evaluation primarily contrasts the proposed method with magnitude pruning, SparseGPT, and Mamba-Shedder. However, recent pruning approaches such as Wanda are not included. Further evaluation would clarify whether the paper’s architectural adaptations are essential and show real empirical advantages.\n\n2. Experimental Scope and Generalization: Although multiple scales of Mamba models are evaluated, the experiments are limited to Mamba architecture alone. The claim that OBS-style pruning generalizes to state-space models would be much stronger if demonstrated on alternative SSMs (e.g., S4 or S5) and hybrid models (e.g., JAMBA, Zamba). Additionally, assessing the robustness of the pruning method under longer context lengths or distributional shifts—key strengths of SSMs—would provide more convincing evidence of practical utility.\n\n3. Evaluation Metrics and Practical Impact: The focus is primarily on perplexity and zero-shot accuracy. The benefits of pruning for deployment (such as latency, memory savings, or throughput improvements) are asserted but not quantified. Structured pruning is said to improve efficiency, but actual hardware speedup, resource consumption, or inference time should be reported across diverse sparsity settings, ideally with comparisons to pruned transformer models.\n\n4. Theoretical Clarification: The adaptation of Hessian-based saliency scores to time-shared and discretized SSM parameters is reasonable, but the discussion could more thoroughly explain why generic second-order pruning (such as SparseGPT) fails or underperforms in this context, both mathematically and experimentally. Demonstrating fundamental limitations or failure cases of alternative methods would strengthen the motivation for the approach."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915755779,"tcdate":1761887844391,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1381/Reviewer_H9zR"],"signatures":["ICLR.cc/2026/Conference/Submission1381/Reviewer_H9zR"],"forum":"bzb9OJVfCP","number":3,"license":"CC BY 4.0","cdate":1761887844391,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1381/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915755779,"domain":"ICLR.cc/2026/Conference","replyto":"bzb9OJVfCP","id":"KxLesfazCq","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We introduce an OBS–based pruning methodology tailored to SSM-based architectures such as Mamba, effectively reducing redundancy while preserving model performance."},"keywords":{"value":["Neural network pruning","Mamba","Hessian-based pruning"]},"supplementary_material":{"value":"/attachment/6b5f630754dd0367304e126456137bc23a85f5bf.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"State-space language models such as Mamba match Transformer quality while permitting linear complexity inference, yet still comprise billions of parameters that hinder deployment. Existing one-shot pruning methods are tailored to attention blocks and fail to account for the time-shared and discretized state-transition matrix at the heart of the selective state-space module (SSM). In this paper, we introduce SparseSSM, the first training-free pruning framework that extends the classic optimal brain surgeon (OBS) framework to state space architectures. Our layer-wise algorithm (i) derives an approximate second-order saliency score that aggregates Hessian-trace information across time steps, (ii) incorporates a component sensitivity analysis to guide feed-forward network (FFN) pruning, which also sheds light on where redundancy resides in mamba architecture, (iii) can be easily extended to semi-structured and structured sparsity. Empirically, we prune 50% of SSM weights without fine-tuning and observe no zero-shot accuracy loss, achieving the current state-of-the-art pruning algorithm for Mamba-based LLMs."},"_bibtex":{"value":"@misc{\ntuo2026sparsessm,\ntitle={Sparse{SSM}: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot},\nauthor={Kaiwen Tuo and Huan Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=bzb9OJVfCP}\n}"},"title":{"value":"SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot"},"pdf":{"value":"/pdf/8e219971d001aa872a5a0e48ae02be58f8abb56b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"tuo|sparsessm_efficient_selective_structured_state_space_models_can_be_pruned_in_oneshot"},"authorids":{"value":["~Kaiwen_Tuo1","~Huan_Wang3"]},"authors":{"value":["Kaiwen Tuo","Huan Wang"]}},"version":2},{"content":{"summary":{"value":"This paper proposes SOR-Mamba to solve the sequential order bias introduced by mamba capturing channel dependencies. Model SOR-Mamba incorporates a regularization strategy to minimize the distance between embeddings from data and reversed data and eliminates the 1D-convolution in the original Mamba block. And a pretraining task channel correlation modeling is introduced to preserve the channel correlation from the data space to the latent space. The effectiveness of the proposed method is demonstrated by the extensive experiment results."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"* When randomly shuffling the channel order, how is it done? Is a fixed shuffling method used for all sequences in a dataset, or is it shuffled randomly each time?\n* In the last row of Table 1, is the calculation formula written incorrectly?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"* This paper uses a single unidirectional Mamba with regularization and non-1D convolutions to capture channel correlations. By modifying the traditional Mamba module, it can better address the issue of sequential order bias. The research problem is clearly defined, and the solution is reasonable. And the pretraining task channel correlation modeling also enhances the model's generalization and performance.\n\n* The details of the model are mostly clear, with each module explained and supported by equations. The experiments  include time series forecasting, transfer learning and many ablation studies to validate the effectiveness of each module."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The proposed method is mostly constructed on existing models -- reverse modeling has been widely used in literature and removing the 1d-conv is trival.\n* The experimental results show that the model's performance improvement across various datasets is not significant, mostly in the thousandth.\n* The removal of 1D-conv negatively impacts PEMS dataset in tale 7 and figure 5. The necessity to remove 1D-conv is uncertain.\n* The model uses the Mamba backbone, but the baseline only includes a single Mamba model: S-Mamba. Providing more models from the Mamba framework for comparison would be better."}},"nonreaders":[],"tmdate":1731427192982,"tcdate":1730687571763,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission69/Reviewer_PJuw"],"signatures":["ICLR.cc/2025/Conference/Submission69/Reviewer_PJuw"],"forum":"fyl82vAale","number":6,"license":"CC BY 4.0","cdate":1730687571763,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission69/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427192982,"domain":"ICLR.cc/2025/Conference","replyto":"fyl82vAale","id":"zOscO3IUW3","forumContent":{"TLDR":{"value":"We propose SOR-Mamba, a time series forecasting method that uses Mamba with regularization to address the sequential order bias in capturing channel dependencies."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","State-space Model","Time Series"]},"supplementary_material":{"value":"/attachment/a3e6bf94e3ea1b0642f98af3f08c4590c0c5d03e.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data.\nHowever, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.\nExtensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios."},"_bibtex":{"value":"@misc{\nlee2025sequential,\ntitle={Sequential Order-Robust Mamba for Time Series Forecasting},\nauthor={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\nyear={2025},\nurl={https://openreview.net/forum?id=fyl82vAale}\n}"},"title":{"value":"Sequential Order-Robust Mamba for Time Series Forecasting"},"pdf":{"value":"/pdf/2172851f730b05caa6144b837024da19c84263c0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"lee|sequential_orderrobust_mamba_for_time_series_forecasting"},"authorids":{"value":["~Seunghan_Lee1","~Juri_Hong1","~Kibok_Lee1","~Taeyoung_Park1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seunghan Lee","Juri Hong","Kibok Lee","Taeyoung Park"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2404.16112v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"patro|mamba360_survey_of_state_space_models_as_transformer_alternative_for_long_sequence_modelling_methods_applications_and_challenges"},"authorids":{"value":["~Badri_Narayana_Patro1","https://dblp.org/search/pid/api?q=author:Vijay_Srinivas_Agneeswaran:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2404.16112"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2404-16112,\n  publtype={informal},\n  author={Badri Narayana Patro and Vijay Srinivas Agneeswaran},\n  title={Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2404.16112},\n  url={https://doi.org/10.48550/arXiv.2404.16112}\n}\n"},"abstract":{"value":"Sequence modeling is a crucial area across various domains, including Natural Language Processing (NLP), speech recognition, time series forecasting, music generation, and bioinformatics. Recurrent Neural Networks (RNNs) and Long Short Term Memory Networks (LSTMs) have historically dominated sequence modeling tasks like Machine Translation, Named Entity Recognition (NER), etc. However, the advancement of transformers has led to a shift in this paradigm, given their superior performance. Yet, transformers suffer from $O(N^2)$ attention complexity and challenges in handling inductive bias. Several variations have been proposed to address these issues which use spectral networks or convolutions and have performed well on a range of tasks. However, they still have difficulty in dealing with long sequences. State Space Models(SSMs) have emerged as promising alternatives for sequence modeling paradigms in this context, especially with the advent of S4 and its variants, such as S4nd, Hippo, Hyena, Diagnol State Spaces (DSS), Gated State Spaces (GSS), Linear Recurrent Unit (LRU), Liquid-S4, Mamba, etc. In this survey, we categorize the foundational SSMs based on three paradigms namely, Gating architectures, Structural architectures, and Recurrent architectures. This survey also highlights diverse applications of SSMs across domains such as vision, video, audio, speech, language (especially long sequence modeling), medical (including genomics), chemical (like drug design), recommendation systems, and time series analysis, including tabular data. Moreover, we consolidate the performance of SSMs on benchmark datasets like Long Range Arena (LRA), WikiText, Glue, Pile, ImageNet, Kinetics-400, sstv2, as well as video datasets such as Breakfast, COIN, LVU, and various time series datasets. The project page for Mamba-360 work is available on this webpage.\\url{https://github.com/badripatro/mamba360}."},"title":{"value":"Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges"},"authors":{"value":["Badri Narayana Patro","Vijay Srinivas Agneeswaran"]}},"tmdate":1731564362178,"pdate":1704067200000,"tcdate":1731564358883,"writers":["~"],"signatures":["~Badri_Narayana_Patro1"],"forum":"1HRyR3j7Gz","license":"CC BY-SA 4.0","number":238409,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1731564362178,"domain":"DBLP.org","id":"1HRyR3j7Gz","version":2},{"content":{"summary":{"value":"This paper develops a methodology for linearizing nonlinear neural networks by applying linear space theory. First, it introduces an algebra (vector operations and a metric) that endows curved spaces with the structure of inner product spaces using invertible neural transformations. Using this, it treats neural networks as linear mappings on curved vector spaces transformed by these invertible networks. As applications, this paper proposes one-step inference via flow-matching, interpolation for style transfer, and the construction of idempotent networks."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"See the Weaknesses section, particularly the third point."},"rating":{"value":4},"details_of_ethics_concerns":{"value":"N.A."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"Several studies have explored deep learning models that perform linear operations in latent spaces, such as Mixup and state-space models. However, few have provided a theoretical framework for such operations. This paper offers a theoretical framework that reframes neural networks as linear mappings in curved spaces. To my knowledge, no other paper has framed neural networks in this manner."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. As the authors point out, the proposed linearizable neural networks have limitations in their expressive power. Therefore, I question whether they are theoretically useful architectures.\n2. The experiments are limited to generation tasks on relatively small image datasets, such as MNIST and CelebA. Therefore, their significance is limited from the perspective of practical usefulness.\n3. I would like more clarification on the benefit of reframing deep learning models as linear mappings in curved spaces. If all neural networks could be interpreted this way, it would be convincing. However, as this paper itself points out, this is not the case. Therefore, it seems more reasonable to view this paper as proposing a specific type of architecture that combines invertible networks with linear mappings."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917302393,"tcdate":1761889327080,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4332/Reviewer_TT5Z"],"signatures":["ICLR.cc/2026/Conference/Submission4332/Reviewer_TT5Z"],"forum":"RcxM3e59ga","number":3,"license":"CC BY 4.0","cdate":1761889327080,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4332/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917302393,"domain":"ICLR.cc/2026/Conference","replyto":"RcxM3e59ga","id":"kTTTg3k7Xz","forumContent":{"TLDR":{"value":"We define vector spaces where networks are linear, then apply linear algebra to tasks like one-step diffusion."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Linear","flow","idempotent","diffeomorphism","vector-space"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, $f$$:$$\\mathcal{X}$$\\to$$\\mathcal{Y}$. Is it possible to identify a pair of non-standard vector spaces for which a conventionally nonlinear function is, in fact, linear? This paper introduces a method that makes such vector spaces explicit by construction. We find that if we sandwich a linear operator $A$ between two invertible neural networks, $f(x)=g_y^{-1}(A g_x(x))$, then the corresponding vector spaces $\\mathcal{X}$ and $\\mathcal{Y}$ are induced by newly defined addition and scaling actions derived from $g_x$ and $g_y$. We term this kind of architecture a Linearizer. This framework makes the entire arsenal of linear algebra, including SVD, pseudo-inverse, orthogonal projection and more, applicable to nonlinear mappings. Furthermore, we show that the composition of two Linearizers that share a neural network is also a Linearizer. We leverage this property and demonstrate that training diffusion models using our architecture makes the hundreds of sampling steps collapse into a single step. We further utilize our framework to enforce \nidempotency (i.e. $f(f(x))=f(x)$) on networks leading to a globally projective generative model and to demonstrate modular style transfer."},"_bibtex":{"value":"@misc{\nberman2026who,\ntitle={Who Said Neural Networks Aren't Linear?},\nauthor={Nimrod Berman and Assaf Hallak and Assaf Shocher},\nyear={2026},\nurl={https://openreview.net/forum?id=RcxM3e59ga}\n}"},"title":{"value":"Who Said Neural Networks Aren't Linear?"},"pdf":{"value":"/pdf/cddd05a450a90366f9938889e633ffe6621ae3ee.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"berman|who_said_neural_networks_arent_linear"},"authorids":{"value":["~Nimrod_Berman1","~Assaf_Hallak1","~Assaf_Shocher1"]},"authors":{"value":["Nimrod Berman","Assaf Hallak","Assaf Shocher"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Social-Mamba, a novel and efficient architecture for human trajectory forecasting that replaces quadratic-cost attention mechanisms with a Mamba-based state-space model to achieve linear-time complexity while maintaining state-of-the-art accuracy. Its core contributions include the Cycle Mamba block, which enables continuous bidirectional information flow for richer context modeling, and a structured social triplet factorization that decomposes interactions into temporal, egocentric, and goal-centric scans processed via semantically ordered sequences on an ego-centric grid. Extensive experiments on benchmarks demonstrate that Social-Mamba outperforms existing methods in accuracy while significantly improving parameter and computational efficiency."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"See  weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper is logically structured and visually coherent, featuring well-designed figures, concise tables, and fluent writing. The clarity of presentation allows readers to grasp the methodology and results with ease.\n\n2. The experimental analysis is thorough and well-executed."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Novelty**: Mamba has been available for more than two years, and a large number of studies have already explored its application in various domains. Even within the trajectory forecasting field for both vehicles and humans, several works [1,2,3] have investigated the use of Mamba. Therefore, simply replacing the attention mechanism with Mamba for efficiency improvement is not sufficiently convincing or novel.\n\n2. **efficiency**: Reporting only the number of parameters and GFLOPs is not enough to demonstrate the efficiency of the proposed method. A more comprehensive evaluation—including inference time and memory usage compared with other approaches—is necessary to substantiate the efficiency claims.\n\n**References**:\n\n[1] Trajectory Mamba: Efficient Attention-Mamba Forecasting Model Based on Selective SSM, CVPR 2025.\n\n[2] DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States, NeurIPS 2024.\n\n[3] MambaPTP: Exploring the Potential of Mamba for Pedestrian Trajectory Prediction, IEEE Transactions on Circuits and Systems for Video Technology."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360171255,"tcdate":1761304981210,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission3412/Reviewer_4div"],"signatures":["ICLR.cc/2026/Conference/Submission3412/Reviewer_4div"],"forum":"9XCnaA2CQm","number":1,"license":"CC BY 4.0","cdate":1761304981210,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission3412/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360171255,"domain":"ICLR.cc/2026/Conference","replyto":"9XCnaA2CQm","id":"CIBQxhitWw","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Human trajectory prediction","Selective state-space models"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Human trajectory forecasting is crucial for safe navigation in crowded environments, requiring models that balance accuracy with computational efficiency. Efficiently modeling social interactions is key to performance in dense crowds. Yet, most recent methods rely on attention mechanisms, which are effective at capturing complex dependencies, but incur quadratic computational costs that scale poorly with the growing number of neighbors. Recently, Selective State-Space Models have provided a linear-time alternative; however, their inherently sequential design is misaligned with the unstructured and dynamic nature of social interactions. To address this challenge, we propose Social-Mamba, a forecasting architecture that reformulates social interactions as structured sequential processes. At its core is the Cycle Mamba block, a novel module that enables continuous bidirectional information flow. Social-Mamba organizes agents on a semantically ordered egocentric grid and introduces social triplet factorization, which decomposes interactions into temporal, egocentric, and goal-centric scans. These are dynamically integrated through a learnable social gate and global scan to generate accurate and efficient trajectory predictions. Extensive experiments on five trajectory forecasting benchmarks show that Social-Mamba achieves state-of-the-art accuracy while offering superior parameter efficiency and computational scalability. Furthermore, embedding Social-Mamba into a flow-matching framework further enhances both accuracy and efficiency, establishing it as a flexible and robust foundation for future trajectory forecasting research."},"_bibtex":{"value":"@misc{\nluan2026socialmamba,\ntitle={Social-Mamba: Efficient Human Trajectory Forecasting with State-Space Models},\nauthor={Po-Chien Luan and Wuyang Li and Yang Gao and Alexandre Alahi},\nyear={2026},\nurl={https://openreview.net/forum?id=9XCnaA2CQm}\n}"},"title":{"value":"Social-Mamba: Efficient Human Trajectory Forecasting with State-Space Models"},"pdf":{"value":"/pdf/e8274978dcc35c7a13afdaa601d4b91dd99b4728.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"luan|socialmamba_efficient_human_trajectory_forecasting_with_statespace_models"},"authorids":{"value":["~Po-Chien_Luan1","~Wuyang_Li1","~Yang_Gao15","~Alexandre_Alahi3"]},"authors":{"value":["Po-Chien Luan","Wuyang Li","Yang Gao","Alexandre Alahi"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2405.16440v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"cai|mambats_improved_selective_state_space_models_for_longterm_time_series_forecasting"},"authorids":{"value":["~Xiuding_Cai1","https://dblp.org/search/pid/api?q=author:Yaoyao_Zhu:","https://dblp.org/search/pid/api?q=author:Xueyao_Wang:","https://dblp.org/search/pid/api?q=author:Yu_Yao:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2405.16440"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2405-16440,\n  publtype={informal},\n  author={Xiuding Cai and Yaoyao Zhu and Xueyao Wang and Yu Yao},\n  title={MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2405.16440},\n  url={https://doi.org/10.48550/arXiv.2405.16440}\n}\n"},"abstract":{"value":"In recent years, Transformers have become the de-facto architecture for long-term sequence forecasting (LTSF), but faces challenges such as quadratic complexity and permutation invariant bias. A recent model, Mamba, based on selective state space models (SSMs), has emerged as a competitive alternative to Transformer, offering comparable performance with higher throughput and linear complexity related to sequence length. In this study, we analyze the limitations of current Mamba in LTSF and propose four targeted improvements, leading to MambaTS. We first introduce variable scan along time to arrange the historical information of all the variables together. We suggest that causal convolution in Mamba is not necessary for LTSF and propose the Temporal Mamba Block (TMB). We further incorporate a dropout mechanism for selective parameters of TMB to mitigate model overfitting. Moreover, we tackle the issue of variable scan order sensitivity by introducing variable permutation training. We further propose variable-aware scan along time to dynamically discover variable relationships during training and decode the optimal variable scan order by solving the shortest path visiting all nodes problem during inference. Extensive experiments conducted on eight public datasets demonstrate that MambaTS achieves new state-of-the-art performance."},"title":{"value":"MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting"},"authors":{"value":["Xiuding Cai","Yaoyao Zhu","Xueyao Wang","Yu Yao"]}},"tmdate":1727875818656,"pdate":1704067200000,"tcdate":1727875815553,"writers":["~"],"signatures":["~Xiuding_Cai1"],"forum":"84Nvd1lBPi","license":"CC BY-SA 4.0","number":136798,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1727875818656,"domain":"DBLP.org","id":"84Nvd1lBPi","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2403.18257v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"jiang|dualpath_mamba_short_and_longterm_bidirectional_selective_structured_state_space_models_for_speech_separation"},"authorids":{"value":["~Xilin_Jiang1","~Cong_Han1","https://dblp.org/search/pid/api?q=author:Nima_Mesgarani:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.18257"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-18257,\n  publtype={informal},\n  author={Xilin Jiang and Cong Han and Nima Mesgarani},\n  title={Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.18257},\n  url={https://doi.org/10.48550/arXiv.2403.18257}\n}\n"},"abstract":{"value":"Transformers have been the most successful architecture for various speech modeling tasks, including speech separation. However, the self-attention mechanism in transformers with quadratic complexity is inefficient in computation and memory. Recent models incorporate new layers and modules along with transformers for better performance but also introduce extra model complexity. In this work, we replace transformers with Mamba, a selective state space model, for speech separation. We propose dual-path Mamba, which models short-term and long-term forward and backward dependency of speech signals using selective state spaces. Our experimental results on the WSJ0-2mix data show that our dual-path Mamba models of comparably smaller sizes outperform state-of-the-art RNN model DPRNN, CNN model WaveSplit, and transformer model Sepformer. Code: https://github.com/xi-j/Mamba-TasNet"},"title":{"value":"Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation"},"authors":{"value":["Xilin Jiang","Cong Han","Nima Mesgarani"]}},"tmdate":1767726920557,"pdate":1704067200000,"externalIds":["dblp:journals/corr/abs-2403-18257"],"tcdate":1767672409753,"writers":["~"],"signatures":["~Xilin_Jiang1"],"forum":"KRgbcz3Krv","license":"CC BY-SA 4.0","number":717762,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1767726920557,"domain":"DBLP.org","id":"KRgbcz3Krv","version":2},{"content":{"summary":{"value":"This paper proposes Masked Mamba, an efficient model for pathological image classification. To adapt Mamba for pathological images, it introduce a patch ghosting module to capture multi-scale features and employ masked autoencoders to extract robust feature representations. Experimental results demonstrate that Masked Mamba achieves state-of-the-art performance."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"No question"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1.\tThe proposed patch ghosting module effectively enhances locality with multi-scale information, making it highly practical.\n2.\tExtending Mamba from 1D sequence modeling to 2D image modeling is a valuable and well-motivated approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tThe combination of Vision-Mamba and masked autoencoders shows limited novelty, as neither component is original to the authors.\n2.\tThe patch ghosting module should be compared to the commonly used Local Perception Unit described in [1].\n3.\tAlthough the paper claims that Masked Mamba is efficient, it lacks a table comparing parameters or FLOPs to substantiate this claim.\n4.\tThis paper resizes all pathological images to 224×224, which fails to convincingly demonstrate that Mamba is applicable to high-resolution pathological images.\n5.\tThe writing and expression could be improved, as the contributions in the abstract and introduction are inconsistent. \n[1] Guo J, Han K, Wu H, et al. Cmt: Convolutional neural networks meet vision transformers[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 12175-12185."}},"nonreaders":[],"tmdate":1731428774578,"tcdate":1730385659868,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6954/Reviewer_F4TG"],"signatures":["ICLR.cc/2025/Conference/Submission6954/Reviewer_F4TG"],"forum":"V9UsZBbTvZ","number":1,"license":"CC BY 4.0","cdate":1730385659868,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6954/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428774578,"domain":"ICLR.cc/2025/Conference","replyto":"V9UsZBbTvZ","id":"udF2j3Bz4L","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Pathological image classification","Mamba model","Self-supervised learning"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Extracting visual representations is a crucial challenge in the domain of computational histopathology. Considering the powerful strength of deep learning algorithms and the dearth of annotated samples, self-supervised learning presents itself as a compelling strategy to extract effective visual representations from unlabeled histopathology images. Although some self-supervised learning methods have been specifically proposed for histopathology image classification, most of them have certain drawbacks that may affect the functionality or representation capacity. In this work, we propose Masked Mamba, a novel self-supervised visual representation learning method tailored for histopathology images that can adequately extract local-global features. The proposed method consists of two stages: local perception positional encoding (LPPE) and directional Mamba vision backbone (DM). In addition, we use masked autoencoder (MAE) pretraining to unleashing directional Mamba vision backbone's potential. Masked Mamba makes good use of domain-specific knowledge and requires no side information, which means good rationality and versatility. Experimental results demonstrate the effectiveness and robustness of masked Mamba on common histopathology classification tasks. Furthermore, ablation studies prove that the local perception positional encoding and directional Mamba vision backbone in masked Mamba can complement and enhance each other."},"_bibtex":{"value":"@misc{\nchai2025masked,\ntitle={Masked Mamba: An Efficient Self-Supervised Framework for Pathological Image Classification},\nauthor={Enhui Chai and Xingyu Li and Zheng Lu and Tianxiang Cui},\nyear={2025},\nurl={https://openreview.net/forum?id=V9UsZBbTvZ}\n}"},"title":{"value":"Masked Mamba: An Efficient Self-Supervised Framework for Pathological Image Classification"},"pdf":{"value":"/pdf/3b4dccaf614da783c9697fc3a87a7a734a9f67d5.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chai|masked_mamba_an_efficient_selfsupervised_framework_for_pathological_image_classification"},"authorids":{"value":["~Enhui_Chai1","~Xingyu_Li8","~Zheng_Lu4","~Tianxiang_Cui1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Enhui Chai","Xingyu Li","Zheng Lu","Tianxiang Cui"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a novel method for processing 3D point clouds using State Space Models (SSMs), specifically the Mamba model. The key innovation is a strategy to convert 3D point clouds into 1D sequences that preserves the spatial structure without requiring data replication, thus enabling efficient sequential processing by Mamba in a permutation-invariant manner. The authors claim that their method surpasses Transformer-based models in accuracy and efficiency and does not require positional embeddings. The paper reports state-of-the-art results on ModelNet40 and ScanObjectNN datasets and demonstrates improved robustness against data transformations such as rotations and jittering."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"Refer to weakness part."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":2},"strengths":{"value":"1、The linear complexity of SSMs like Mamba, as opposed to the quadratic complexity of Transformer models, makes NIMBA highly scalable for high-resolution 3D data.\n\n2、The paper shows that NIMBA is more robust to data transformations such as rotations and jittering, which is crucial for real-world applications where data can be subject to various distortions.\n\n3、The elimination of positional embeddings and sequence replication makes the model more principled and less reliant on artificial constructs for sequence ordering."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1、One of the contributions proposed in this paper is the reordering strategy. However, this serialization strategy should be compared and discussed with the Point Cloud Mamba[1]. In the Point Cloud Mamba, many methods are discussed and compared, but they are all missing in this paper. There is even no specific discussion and comparison in the ablation studies. Furthermore, the performance of NIMBA's reordering strategy is dependent on the choice of the threshold parameter, which may require careful tuning for different datasets.\n\n2、The paper notes that NIMBA shows limited improvement when scaled, suggesting that there may be optimization challenges that need to be addressed. There is a noted decline in performance when integrating NIMBA with Mamba2 or in hybrid architectures, indicating potential issues with model integration.\n\n3、Comparative Analysis: The paper primarily compares NIMBA with Mamba-based models; a more comprehensive comparison with other state-of-the-art methods, especially those using different SSMs [1], [2], could provide a fuller picture of NIMBA's performance. Furthermore, this paper claims to surpass the transformer-based method, but it lacks many comparisons with such methods, such as PointBert[3], PointM2AE[4].\n\n4、 How was the threshold parameter determined, and how sensitive is the model's performance to changes in this parameter? How does the removal of positional embeddings affect the model's interpretability, and can the learned representations be easily understood?\n\n5、While the paper claims efficiency improvements, are there specific computational cost analyses, especially for large-scale datasets?\n\n[1] Zhang T, Li X, Yuan H, et al. Point could mamba: Point cloud learning via state space model[J]. arXiv preprint arXiv:2403.00762, 2024.\n\n[2] Han X, Tang Y, Wang Z, et al. Mamba3d: Enhancing local features for 3d point cloud analysis via state space model[J]. arXiv preprint arXiv:2404.14966, 2024.\n\n[3] Yu X, Tang L, Rao Y, et al. Point-bert: Pre-training 3d point cloud transformers with masked point modeling[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 19313-19322.\n\n[4] Zhang R, Guo Z, Gao P, et al. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training[J]. Advances in neural information processing systems, 2022, 35: 27061-27074."}},"nonreaders":[],"tmdate":1731428593285,"tcdate":1730376760675,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10086/Reviewer_9zEE"],"signatures":["ICLR.cc/2025/Conference/Submission10086/Reviewer_9zEE"],"forum":"aHSoKImhc2","number":2,"license":"CC BY 4.0","cdate":1730376760675,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10086/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428593285,"domain":"ICLR.cc/2025/Conference","replyto":"aHSoKImhc2","id":"w4d1NSos1u","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["point clouds","SSMs","state-space models","Mamba","3D","transformers","pointcloud"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Transformers have become dominant in large-scale deep learning tasks across various domains, including text, 2D and 3D vision. However, the quadratic complexity of their attention mechanism limits their efficiency as the sequence length increases, particularly in high-resolution 3D data such as point clouds. Recently, state space models (SSMs) like Mamba have emerged as promising alternatives, offering linear complexity, scalability, and high performance in long-sequence tasks. The key challenge in the application of SSMs in this domain lies in reconciling the non-sequential structure of point clouds with the inherently directional (or bi-directional) order-dependent processing of recurrent models like Mamba. To achieve this, previous research proposed reorganizing point clouds along multiple directions or predetermined paths in 3D space, concatenating the results to produce a single 1D sequence capturing different views. In our work we introduce a method to convert point clouds into 1D sequences that maintains 3D spatial structure with no need for data replication, allowing Mamba’s sequential processing to be applied effectively in an almost permutation-invariant manner. In contrast to other works, we found that our method does not require positional embeddings, and allows for shorter sequence lengths while still achieving state-of-the-art results in ModelNet40 and ScanObjectNN datasets and surpassing Transformer-based models in both accuracy and efficiency."},"_bibtex":{"value":"@misc{\nkoprucu2025nimba,\ntitle={{NIMBA} : Towards Robust and Principled Processing of Point Clouds With {SSM}s},\nauthor={Nursena Koprucu and Destiny Okpekpe and Antonio Orvieto},\nyear={2025},\nurl={https://openreview.net/forum?id=aHSoKImhc2}\n}"},"title":{"value":"NIMBA : Towards Robust and Principled Processing of Point Clouds With SSMs"},"pdf":{"value":"/pdf/0e67e526c59087fbadbaa49be2898a7f6c87b665.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"koprucu|nimba_towards_robust_and_principled_processing_of_point_clouds_with_ssms"},"authorids":{"value":["~Nursena_Koprucu1","~Destiny_Okpekpe1","~Antonio_Orvieto3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Nursena Koprucu","Destiny Okpekpe","Antonio Orvieto"]}},"version":2},{"content":{"summary":{"value":"This paper introduce a new Mamba-based model for long term time series forecasting named SOR-Mamba which utilizes modified Mamba structure to capture channel dependencies. Furthermore, the authors also introduces channel correlation modeling as a pretraining task aimed at enhancing the ability to capture channel dependencies."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See Weaknesses above.\n3. In line 369-370, I’m curious about how to calculate the global correlation."},"rating":{"value":5},"details_of_ethics_concerns":{"value":"I think these if no ethics concerns."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The proposed SOR-Mamba uses a modified Mamba structure (also with a regularization strategy) to eliminate the sequential order bias of Mamba which is more useful in capture channel dependencies.\n2. The authors also introduce a pretraining task named channel correlation modeling (CCM) to preserve correlations between channels from the data space to the latent space in order to enhance the ability to capture CD."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Since Mamba is a sequential-based model, and the channels lake sequential order, why we need Mamba to extract channel dependencies? It is important to clarify the importance of capturing channel dependencies with Mamba.\n2. The experimental comparisons appear unfair, as models like PatchTST exhibit better performance with longer lookback window. I strongly suggest the authors conduct additional experiments, such as setting $L=336$."}},"nonreaders":[],"tmdate":1731427193448,"tcdate":1730364640668,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission69/Reviewer_vPYA"],"signatures":["ICLR.cc/2025/Conference/Submission69/Reviewer_vPYA"],"forum":"fyl82vAale","number":2,"license":"CC BY 4.0","cdate":1730364640668,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission69/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427193448,"domain":"ICLR.cc/2025/Conference","replyto":"fyl82vAale","id":"sZWs0V7cvG","forumContent":{"TLDR":{"value":"We propose SOR-Mamba, a time series forecasting method that uses Mamba with regularization to address the sequential order bias in capturing channel dependencies."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","State-space Model","Time Series"]},"supplementary_material":{"value":"/attachment/a3e6bf94e3ea1b0642f98af3f08c4590c0c5d03e.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data.\nHowever, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.\nExtensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios."},"_bibtex":{"value":"@misc{\nlee2025sequential,\ntitle={Sequential Order-Robust Mamba for Time Series Forecasting},\nauthor={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\nyear={2025},\nurl={https://openreview.net/forum?id=fyl82vAale}\n}"},"title":{"value":"Sequential Order-Robust Mamba for Time Series Forecasting"},"pdf":{"value":"/pdf/2172851f730b05caa6144b837024da19c84263c0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"lee|sequential_orderrobust_mamba_for_time_series_forecasting"},"authorids":{"value":["~Seunghan_Lee1","~Juri_Hong1","~Kibok_Lee1","~Taeyoung_Park1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seunghan Lee","Juri Hong","Kibok Lee","Taeyoung Park"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2509.17711v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"kang|damamba_dialogueaware_selective_statespace_model_for_multimodal_engagement_estimation"},"html":{"value":"https://doi.org/10.48550/arXiv.2509.17711"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2509-17711,\n  publtype={informal},\n  author={Shenwei Kang and Xin Zhang and Wen Liu and Bin Li and Yujie Liu and Bo Gao},\n  title={DA-Mamba: Dialogue-aware selective state-space model for multimodal engagement estimation},\n  year={2025},\n  month={September},\n  cdate={1756684800000},\n  journal={CoRR},\n  volume={abs/2509.17711},\n  url={https://doi.org/10.48550/arXiv.2509.17711}\n}\n"},"abstract":{"value":"Human engagement estimation in conversational scenarios is essential for applications such as adaptive tutoring, remote healthcare assessment, and socially aware human--computer interaction. Engagement is a dynamic, multimodal signal conveyed by facial expressions, speech, gestures, and behavioral cues over time. In this work we introduce DA-Mamba, a dialogue-aware multimodal architecture that replaces attention-heavy dialogue encoders with Mamba-based selective state-space processing to achieve linear time and memory complexity while retaining expressive cross-modal reasoning. We design a Mamba dialogue-aware selective state-space model composed of three core modules: a Dialogue-Aware Encoder, and two Mamba-based fusion mechanisms: Modality-Group Fusion and Partner-Group Fusion, these modules achieve expressive dialogue understanding. Extensive experiments on three standard benchmarks (NoXi, NoXi-Add, and MPIIGI) show that DA-Mamba surpasses prior state-of-the-art (SOTA) methods in concordance correlation coefficient (CCC), while reducing training time and peak memory; these gains enable processing much longer sequences and facilitate real-time deployment in resource-constrained, multi-party conversational settings. The source code will be available at: https://github.com/kksssssss-ssda/MMEA."},"title":{"value":"DA-Mamba: Dialogue-aware selective state-space model for multimodal engagement estimation"},"authors":{"value":[{"fullname":"Shenwei Kang","username":""},{"fullname":"Xin Zhang","username":""},{"fullname":"Wen Liu","username":""},{"fullname":"Bin Li","username":"~Bin_Li14"},{"fullname":"Yujie Liu","username":""},{"fullname":"Bo Gao","username":""}]}},"tmdate":1785633778596,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2509-17711"],"tcdate":1785633761996,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Bin_Li14"],"forum":"Mn4yjOqeW7","license":"CC BY-SA 4.0","number":113300,"cdate":1756684800000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1785633778596,"domain":"OpenReview.net/Public_Article","id":"Mn4yjOqeW7","version":2},{"content":{"summary":{"value":"The paper addresses the challenge of multi-modal fusion in deep learning. Current fusion methods struggle to capture complex intra- and inter-modality correlations. Recent state space models like Mamba show promise but are limited in fusing multiple modalities efficiently. The paper propose Coupled Mamba, key aspects include: A coupled state transition scheme that allows information exchange between modalities while maintaining individual propagation. A state summation and transition mechanism to enable parallel computing with multiple inputs. Derivation of a global convolution kernel for efficient parallelization.\nThe model architecture consists of multiple Coupled Mamba blocks, each processing different modalities and aggregating states before transitioning to the next time step.\n\nExperiments were conducted on three multi-modal sentiment analysis datasets: CMU-MOSEI, CH-SIMS, and CH-SIMSV2. \nResults show improved performance over existing methods as well as faster inference and GPU memory savings compared to transformer-based approaches. Effective performance on both aligned and unaligned multi-modal data."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. In table 9, what does Mamba Fusion mean?\n\n2. Can the authors provide more interpretability behind combining to mamba structure and explain how the modalities interact between the two mamba?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The method demonstrates consistent performance and good efficiency compared with existing works.\n2. The paper provides a detailed mathematical derivation of the coupled state transition process,\n3. The idea of coupling mamba is novel."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The authors seem to have combined two Mamba models without providing enough insights into how this combination actually improves multi-modal fusion. \n\nThere's no clear analysis or interpretability of what the model is learning or how it's fusing information across modalities. This makes it difficult to understand why their approach is fundamentally better than existing methods, including transformers. \n\nThe speed and linear complexity benefits are inherited from the original mamba model, so they are apparent advantages for all state space-related works. \n\nThe experiments are limited to sentiment analysis tasks. The generalizability of the method to other multi-modal tasks or domains is not demonstrated or discussed."},"limitations":{"value":"The authors have addressed the limitations"}},"nonreaders":[],"tmdate":1730878771015,"tcdate":1720666065640,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2377/Reviewer_FSJZ"],"signatures":["NeurIPS.cc/2024/Conference/Submission2377/Reviewer_FSJZ"],"forum":"UXEo3uNNIX","number":3,"license":"CC BY 4.0","cdate":1720666065640,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2377/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878771015,"domain":"NeurIPS.cc/2024/Conference","replyto":"UXEo3uNNIX","id":"S7eQjpB13v","forumContent":{"TLDR":{"value":"Enhanced Multimodal Fusion with Coupled State Space Model"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","multi-modal fusion","multi-modal sentiment analysis"]},"supplementary_material":{"value":"/attachment/1af5552642a4f1fdcea9c586b54a5450f6c0c432.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"The essence of multi-modal fusion lies in exploiting the complementary information inherent in diverse modalities.However, most prevalent fusion methods rely on traditional neural architectures and are inadequately equipped to capture the dynamics of interactions across modalities, particularly in presence of complex intra- and inter-modality correlations.Recent advancements in State Space Models (SSMs), notably exemplified by the Mamba model, have emerged as promising contenders. Particularly, its state evolving process implies stronger modality fusion paradigm, making multi-modal fusion on SSMs an appealing direction. However, fusing multiple modalities is challenging for SSMs due to its hardware-aware parallelism designs. To this end, this paper proposes the Coupled SSM model, for coupling state chains of multiple modalities while maintaining independence of intra-modality state processes. Specifically, in our coupled scheme, we devise an inter-modal hidden states transition scheme, in which the current state is dependent on the states of its own chain and that of the neighbouring chains at the previous time-step. To fully comply with the hardware-aware parallelism, we obtain the global convolution kernel by deriving the state equation while introducing the historical state.Extensive experiments on CMU-MOSEI, CH-SIMS, CH-SIMSV2 through multi-domain input verify the effectiveness of our model compared to current state-of-the-art methods, improved F1-Score by 0.4%, 0.9%, and 2.3% on the three datasets respectively, 49% faster inference and 83.7% GPU memory save. The results demonstrate that Coupled Mamba model is capable of enhanced multi-modal fusion."},"_bibtex":{"value":"@inproceedings{\nli2024coupled,\ntitle={Coupled Mamba: Enhanced Multimodal Fusion with Coupled State Space Model},\nauthor={Wenbing Li and Hang Zhou and Junqing Yu and Zikai Song and Wei Yang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=UXEo3uNNIX}\n}"},"title":{"value":"Coupled Mamba: Enhanced Multimodal Fusion with Coupled State Space Model"},"pdf":{"value":"/pdf/c0967468999c3d38c161a8dd93682b7d1707ad5e.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"li|coupled_mamba_enhanced_multimodal_fusion_with_coupled_state_space_model"},"authorids":{"value":["~Wenbing_Li1","~Hang_Zhou12","~Junqing_Yu1","~Zikai_Song1","~Wei_Yang3"]},"authors":{"value":["Wenbing Li","Hang Zhou","Junqing Yu","Zikai Song","Wei Yang"]}},"version":2},{"content":{"venue":{"value":"ICASSP 2025"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/10887540/10887541/10888514.pdf"},"venueid":{"value":"dblp.org/conf/ICASSP/2025"},"paperhash":{"value":"jiang|dualpath_mamba_short_and_longterm_bidirectional_selective_structured_state_space_models_for_speech_separation"},"authorids":{"value":["~Xilin_Jiang1","~Cong_Han1","https://dblp.org/search/pid/api?q=author:Nima_Mesgarani:"]},"html":{"value":"https://doi.org/10.1109/ICASSP49660.2025.10888514"},"_bibtex":{"value":"@inproceedings{DBLP:conf/icassp/JiangHM25,\n  author={Xilin Jiang and Cong Han and Nima Mesgarani},\n  title={Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation},\n  year={2025},\n  cdate={1735689600000},\n  pages={1-5},\n  url={https://doi.org/10.1109/ICASSP49660.2025.10888514},\n  booktitle={ICASSP},\n  crossref={conf/icassp/2025}\n}\n"},"abstract":{"value":"Transformers have been the most successful architecture for various speech modeling tasks, including speech separation. However, the self-attention mechanism in transformers with quadratic complexity is inefficient in computation and memory. Recent models incorporate new layers and modules along with transformers for better performance but also introduce extra model complexity. In this work, we replace transformers with Mamba, a selective state space model, for speech separation. We propose dual-path Mamba, which models short-term and long-term forward and backward dependency of speech signals using selective state spaces. Our experimental results on the WSJ0-2mix data show that our dual-path Mamba models of comparably smaller sizes outperform state-of-the-art RNN model DPRNN, CNN model WaveSplit, and transformer model Sepformer. Our largest model reaches an SI-SNRi of 23.4 dB. Code available: https://github.com/xi-j/Mamba-TasNet"},"title":{"value":"Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation"},"authors":{"value":["Xilin Jiang","Cong Han","Nima Mesgarani"]}},"tmdate":1767726920625,"pdate":1735689600000,"externalIds":["dblp:conf/icassp/JiangHM25"],"tcdate":1767672409352,"writers":["~"],"signatures":["~Xilin_Jiang1"],"forum":"ddfuZ3pj4t","license":"CC BY-SA 4.0","number":717753,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1767726920625,"domain":"DBLP.org","id":"ddfuZ3pj4t","version":2},{"content":{"summary":{"value":"This paper proposes, for the first time, training deep learning models on encoded video bytes as opposed to raw RGB frames. They claim that encoded video bytes are similar to RGB frames in the sense that they can be decoded to into RGB frames. Thus, the semantic mapping learned by deep models trained on RGB frames should also work with encoded bytes. Due to the increased sequence length of video bytes as opposed to RGB videos, they propose using state space models over transformers due to their linear complexity. They perform training on a large pre-training RGB video dataset with their \"Bytes\" model and show experiments on various downstream action recognition and long video understanding datasets."},"soundness":{"value":1},"confidence":{"value":5},"questions":{"value":"See weaknesses section. Overall, I think the research question is interesting, but the scope, narrative, experiments, and writing of the paper requires a lot of improvement."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"1. The idea of training deep learning models with bytes vs. conventional RGB frames is extremely interesting.\n2. Many experiments are provided to validate that training a deep learning model with video bytes is feasible."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Extremely limited novelty** Besides the research problem of training deep learning models with encoded video bytes, there is no additional novelty introduced in the paper. The proposed Bytes model is basically a standard Mamba model, with stacked Mamba layers and some dimensional pooling in-between layers. The idea of computing sequences without a hidden state and updating after the fact was already proposed in the original Mamba paper [1]. The rest of the methodology section simply discuss trivial engineering details such as embedding size and pooling structure.\n\n2. **Scattered/Incomplete Experiments** Many of the experiments and tables in the paper are incomplete or do not have a coherent message. In Table 1, why are multi-modal models used as a comparison? The main idea should be to compare simple, uni-modal models that were ideally pre-trained on the same dataset as the Bytes model for a fairer comparison. Instead, a random assortment of multi-modal models, pre-trained on all different datasets, are used as comparison. Furthermore, I question the validity of Table 3 as the only VideoMamba variant that has 28.4 TFLOPS was VideoMamba trained on K400 in a self-supervised manner, whereas the authors claim that the table denotes classification results.  Moreover, the fact that simply doubling the size of model leads to a 25% increase in performance is surprising, and yet there is no discussion whatsoever on that point. Further, Tables 4 and 5 show results on a relatively unknown dataset, with no meaningful baselines. These are just a few examples of the inconsistency/unclear narrative around the experiments (see below).\n\n3. **Unsupported Claims** There are many claims made throughout the paper which require further support to confirm - especially given the new domain of training on encoded bytes. For example:\n\n- \"Bytes that are far apart have no correlation, so attention is not needed to understand video bytes.\" (Lines 127-128)\n- \"Recent SSM models [...] still don't fully address some challenges such as [...]. We present a different approach which preserves better the sequence signal.\" (Lines 160-165). The authors simply use stacked Mamba layers, so how does their method preserve the sequence better? Does this mean the comparison with baseline Mamba in Table 9 is a single Mamba layer?\n\n- \"We observed that the model very easily overfits when trained on encoded videos\" (Line 265). Not only are none of these experiments shown, but does this also imply that without augmentations, the Bytes model would get 4% on any test data due to overfitting? Yet simply by applying augmentations to the RGB video, which in turn changes the byte string, the authors are then able to attain nearly 85% on Kinetics? This also further raises questions about if nearly \"half the string\" is changed when applying small augmentations in the RGB image, how are encoded bytes a reliable enough modality for deep learning models to perform video understanding? The fact that even image-level understanding is not tackled first before investigating video/long-video understanding leads me to believe there is a lot more experimentation required to validate the large claims made in this paper.\n\n[1] Gu, A., & Dao, T. (2024, May). Mamba: Linear-time sequence modeling with selective state spaces. In First conference on language modeling."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916043699,"tcdate":1761777192209,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2135/Reviewer_ZNXf"],"signatures":["ICLR.cc/2026/Conference/Submission2135/Reviewer_ZNXf"],"forum":"psLcyKiuCp","number":2,"license":"CC BY 4.0","cdate":1761777192209,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2135/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916043699,"domain":"ICLR.cc/2026/Conference","replyto":"psLcyKiuCp","id":"1giVTThlFc","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["video understanding"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"We present an approach to understand video from encoded bytes, e.g., mp4s. These compressed videos are 99\\% smaller than the RGB pixel representations which are currently commonly used for video understanding. Encoded videos are able to compress the pixels by taking advantage of the redundant information across the frames using special encoding, such as key frames and motion residuals to handle this. However, standard video understanding models do not take advantage of this significant compression already available for each video, and instead either heavily subsample the frames or only work on short segments of the video. Here, we present an approach to understanding video from encoded bytes directly. We note that simply applying existing models, e.g., Transformers or State-Space models, to video byte sequences does not work, both due to difficulty in handling very long video byte sequences and easy overfitting. To address these challenges, we design a State-Space model with sequence parallelism to handle very long byte sequences, reaching 15 Million tokens in training, and essentially unlimited tokens in inference. We also propose a multilevel SSM activation fusion that reduces sequence length, which we find also benefits video understanding. We evaluate on common video understanding and natural extension to video + audio understanding tasks and demonstrate competitive performance, illustrating, for the first time, the feasibility of learning from compressed video byte representations."},"_bibtex":{"value":"@misc{\npiergiovanni2025learning,\ntitle={Learning from Encoded Video Bytes},\nauthor={AJ Piergiovanni and Ganesh Satish Mallya and Dahun Kim and Anelia Angelova},\nyear={2025},\nurl={https://openreview.net/forum?id=psLcyKiuCp}\n}"},"title":{"value":"Learning from Encoded Video Bytes"},"pdf":{"value":"/pdf/2ff5bbde8303355e1416ad1c6c4e027210586f69.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"piergiovanni|learning_from_encoded_video_bytes"},"authorids":{"value":["~AJ_Piergiovanni1","~Ganesh_Satish_Mallya1","~Dahun_Kim1","~Anelia_Angelova1"]},"authors":{"value":["AJ Piergiovanni","Ganesh Satish Mallya","Dahun Kim","Anelia Angelova"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a modification to the recent state space model (SSM) Mamba. Instead of the diagonal transition matrices of previous structured SSMs such as S4 and Mamba, this paper borrows ideas from classical control theory, in particular that of controllability and observability, to propose a different class of structure transition matrices $A$: more precisely, where it has the form of a companion matrix or transposed companion matrix."},"soundness":{"value":1},"confidence":{"value":5},"questions":{"value":"The entire method description is unfortunately too vague that I cannot evaluate the technical content of the submission."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"- Borrowing classical ideas from prior statistical or control work on state-based models is always a great direction to continue understanding and improving them.\n- The particular form of structured matrices used and the ideas of controllability and observability make sense."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Method**\n\nThe method details are very not clear. For example, one crucial detail in Mamba is that the $B$, $C$, and $A$ matrices all depend on the input. In this method, I cannot figure out if a still depends on the input (i.e. if the vector $(a_0, \\dots, a_n)$ of coefficients can vary per timestep of the input sequence). Additionally, no algorithm is provided to compute the model. The exact distinctions between Mamba(-2) should be made more clear.\n\n\n**Writing**\n- Background reads more like a survey; there are 4 full pages of background out of an 8 page paper\n- Numerous typos and formatting issues throughout\n- Main figure (Figure 1) is very hard to understand and there is little description\n\nAdditionally, SpaceTime [1] should be mentioned as they were the first to use companion matrices in structured SSMs.\n\n[1] Zhang et al. \"Effectively Modeling Time Series with Simple Discrete State Spaces\""}},"nonreaders":[],"tmdate":1731427831083,"tcdate":1729983111449,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12587/Reviewer_CwE7"],"signatures":["ICLR.cc/2025/Conference/Submission12587/Reviewer_CwE7"],"forum":"vOfDGYGVyj","number":2,"license":"CC BY 4.0","cdate":1729983111449,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12587/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427831083,"domain":"ICLR.cc/2025/Conference","replyto":"vOfDGYGVyj","id":"ZoCV9ksPhr","forumContent":{"TLDR":{"value":"Sparse Mamba"},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","state space models","natural language processing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"In this work, we introduce the concept of controllability and observability to the Mamba SSM's architecture in our Sparse-Mamba (S-Mamba) for natural language processing (NLP) applications. The structured state space model (SSM) development in recent studies, such as Mamba and Mamba2, outperformed and solved the computational inefficiency of transformers and large language models at small to medium scale. The Mamba SSMs architecture drops the need for attention layers or multilayer perception blocks in transformers. However, current Mamba models lack reinforcement of controllability in state-space equations for computing the $A$, $B$, $C$, and $D$ matrices at each time step, leading to increased complexity and computational costs. In this paper, we demonstrate a reduction of parameters in comparison to the first published Mamba and Mamba2. We showcase an improvement in perplexity by 5\\% and a decrease in training time by 3\\% after reinforcing controllability and observability on the original Mamba architecture in our proposed S-Mamba. The controllable $n \\times n$ state  matrix $A$ is sparse and it has only $n$ free parameters. Our novel approach will ensure a controllable system which will be the gate key for Mamba3."},"_bibtex":{"value":"@misc{\nhamdan2024sparse,\ntitle={Sparse Mamba: Reinforcing Controllability In Structural State Space Models},\nauthor={Emadeldeen Hamdan and Hongyi Pan and Ahmet Cetin},\nyear={2024},\nurl={https://openreview.net/forum?id=vOfDGYGVyj}\n}"},"title":{"value":"Sparse Mamba: Reinforcing Controllability In Structural State Space Models"},"pdf":{"value":"/pdf/56eae743dc6d789a0b97e323821a349c3bf415ff.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"hamdan|sparse_mamba_reinforcing_controllability_in_structural_state_space_models"},"authorids":{"value":["~Emadeldeen_Hamdan1","~Hongyi_Pan1","~Ahmet_Cetin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Emadeldeen Hamdan","Hongyi Pan","Ahmet Cetin"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2601.00920v2"},"venueid":{"value":"dblp.org/journals/CORR/2026"},"paperhash":{"value":"chen|mode_efficient_time_series_prediction_with_mamba_enhanced_by_lowrank_neural_odes"},"authorids":{"value":["~Xingsheng_Chen3","","","","","","~Kwok-Yan_Lam1",""]},"html":{"value":"https://doi.org/10.48550/arXiv.2601.00920"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2601-00920,\n  publtype={informal},\n  author={Xingsheng Chen and Regina Zhang and Bo Gao and Xingwei He and Xiaofeng Liu and Pietro Lio and Kwok-Yan Lam and Siu-Ming Yiu},\n  title={MODE: Efficient Time Series Prediction with Mamba Enhanced by Low-Rank Neural ODEs},\n  year={2026},\n  month={January},\n  cdate={1767225600000},\n  journal={CoRR},\n  volume={abs/2601.00920},\n  url={https://doi.org/10.48550/arXiv.2601.00920}\n}\n"},"abstract":{"value":"Time series prediction plays a pivotal role across diverse domains such as finance, healthcare, energy systems, and environmental modeling. However, existing approaches often struggle to balance efficiency, scalability, and accuracy, particularly when handling long-range dependencies and irregularly sampled data. To address these challenges, we propose MODE, a unified framework that integrates Low-Rank Neural Ordinary Differential Equations (Neural ODEs) with an Enhanced Mamba architecture. As illustrated in our framework, the input sequence is first transformed by a Linear Tokenization Layer and then processed through multiple Mamba Encoder blocks, each equipped with an Enhanced Mamba Layer that employs Causal Convolution, SiLU activation, and a Low-Rank Neural ODE enhancement to efficiently capture temporal dynamics. This low-rank formulation reduces computational overhead while maintaining expressive power. Furthermore, a segmented selective scanning mechanism, inspired by pseudo-ODE dynamics, adaptively focuses on salient subsequences to improve scalability and long-range sequence modeling. Extensive experiments on benchmark datasets demonstrate that MODE surpasses existing baselines in both predictive accuracy and computational efficiency. Overall, our contributions include: (1) a unified and efficient architecture for long-term time series modeling, (2) integration of Mamba's selective scanning with low-rank Neural ODEs for enhanced temporal representation, and (3) substantial improvements in efficiency and scalability enabled by low-rank approximation and dynamic selective scanning."},"title":{"value":"MODE: Efficient Time Series Prediction with Mamba Enhanced by Low-Rank Neural ODEs"},"authors":{"value":["Xingsheng Chen","Regina Zhang","Bo Gao","Xingwei He","Xiaofeng Liu","Pietro Lio","Kwok-Yan Lam","Siu-Ming Yiu"]}},"tmdate":1784586576720,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2601-00920"],"tcdate":1773305069671,"writers":["~"],"signatures":["~Kwok-Yan_Lam1"],"forum":"lreMXppP7H","license":"CC BY-SA 4.0","number":850966,"cdate":1767225600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784586576720,"domain":"DBLP.org","id":"lreMXppP7H","version":2},{"content":{"summary":{"value":"This paper introduces the Log-Normal State-Space Model (LNSSM), a novel architecture designed to bridge the performance gap between efficient State-Space Models (SSMs) and expressive self-attention mechanisms. The authors observe that the strong performance of modern SSMs (like Mamba and RWKV) relies not only on their time-mixing (recurrent) layers but also on intricately designed channel-mixing layers, which adds complexity and reduces interpretability. To address this, LNSSM leverages the theoretical insight that softmax attention weights approximately follow a log-normal distribution. By reformulating causal Linear Log-Normal (LLN) attention into a recurrent state-update form with an exponential feature map and a state decay mechanism inspired by RWKV, LNSSM aims to achieve the expressiveness of self-attention while maintaining the linear-time complexity and constant memory footprint of SSMs. Experimental results on language modeling (perplexity), question answering (BABILong), and long-form generation (ASQA, ELI5, WikiLarge) demonstrate that LNSSM outperforms several SSM baselines and remains robust with increasing sequence lengths, where standard transformers degrade."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"* To directly validate the claim of reduced reliance on specialized channel-mixing, could you provide a performance comparison where LNSSM (with GPT-2 MLP) is directly compared against Mamba-2 with its original channel-mixing layer under identical settings and parameter budgets?\n\n* Table 4 shows a dramatic speedup with a custom CUDA kernel. Since custom kernels are standard for performance I wanted to ask if there's anything specific to LNSSM that allows for such speedups?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"* The paper is strengthened by a clear, critical observation: modern SSMs rely heavily on complex, specialized channel-mixing layers. The authors use this to motivate a highly controlled experimental setup, comparing all time-mixing methods using an identical, simple GPT-2 MLP. This elegantly isolates the variable of interest and proves that LNSSM's performance gains stem directly from its novel time-mixing core, not auxiliary architectural components. I quite like the methodology. \n\n* By creating a time-mixing layer powerful enough to work with a standard MLP, LNSSM offers a path toward more efficient, interpretable, and expressive sequence models, addressing a central challenge in the field often overlooked by others."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The comparison, while including strong baselines, could be more definitive. A comparison against the standard Mamba (or Mamba-2) block *with its original channel-mixing layer* is crucial to fully substantiate the claim that LNSSM achieves comparable performance with a simpler (GPT-2) MLP. The paper should state this crucial methodological choice explicitly and unambiguously in the main text of Section 5, not just bury it in a figure caption. This is by far my biggest concern. \n\n* All experiments are conducted at a relatively small scale (~72M parameters). The conclusions would be significantly stronger if supported by results at a larger scale (e.g., 100M+ parameters) to ensure the findings hold as model size increases.\n\n* The contribution of individual components in the overall LNSSM design (QK normalization, KV-shift, output gate, and most importantly, the state decay matrix A) is not systematically ablated. It is unclear how much each component contributes to the final performance.\n\n* The extensive presentation of hand-derived gradient formulas and custom CUDA kernel algorithms in the appendix, while impressive from an engineering standpoint, is atypical for a core conference paper and distracts from the primary scientific contribution. A high-level summary of the implementation challenge and its solution would suffice. Note that I have *not* lowered my score because of this."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915863222,"tcdate":1761614093238,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1709/Reviewer_knEo"],"signatures":["ICLR.cc/2026/Conference/Submission1709/Reviewer_knEo"],"forum":"XSsI8KXOed","number":2,"license":"CC BY 4.0","cdate":1761614093238,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1709/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915863222,"domain":"ICLR.cc/2026/Conference","replyto":"XSsI8KXOed","id":"62m5kqpMig","forumContent":{"TLDR":{"value":"An efficient state space model achieving high performance without extra MLPs."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Linear attention","State-space model","Log-normal distribution","Long- range dependency","Sequence modeling"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"State space model (SSM) have emerged as a strong alternative to transformer owing to its linear-time complexity and state retention mechanism where the computation efficiency and memory capability are enhanced especially in long-sequence tasks. However, the features derived from state updates in SSM still exhibit weaker representation than those generated by self-attention in transformer. In this work, we propose a new architecture that preserves the linear-time efficiency of SSMs while enabling state-update features to approach the expressiveness of self-attention, thereby achieving both computation efficiency and memory enhancement. Our code is available at https://anonymous.4open.science/r/Log-Normal-State-Space-Model-8301/.gitignore"},"_bibtex":{"value":"@misc{\nlin2025lognormal,\ntitle={Log-Normal State-Space Model},\nauthor={Yin-Wen Lin and Ming-Yen Tsai and Feng-Ting Liao and Davide Buffelli and Da-shan Shiu and Jen-Tzung Chien},\nyear={2025},\nurl={https://openreview.net/forum?id=XSsI8KXOed}\n}"},"title":{"value":"Log-Normal State-Space Model"},"pdf":{"value":"/pdf/5e7376ab341675184c8f7d092a39c8d7ee4ed7d4.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"lin|lognormal_statespace_model"},"authorids":{"value":["~Yin-Wen_Lin1","~Ming-Yen_Tsai1","~Feng-Ting_Liao1","~Davide_Buffelli1","~Da-shan_Shiu1","~Jen-Tzung_Chien1"]},"authors":{"value":["Yin-Wen Lin","Ming-Yen Tsai","Feng-Ting Liao","Davide Buffelli","Da-shan Shiu","Jen-Tzung Chien"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2502.16249v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"afzal|linear_attention_for_efficient_bidirectional_sequence_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Arshia_Afzal:","https://dblp.org/search/pid/api?q=author:Elías_Abad-Rocamora:","~Leyla_Naz_Candogan1","~Pol_Puigdemont1","https://dblp.org/search/pid/api?q=author:Francesco_Tonin:","https://dblp.org/search/pid/api?q=author:Yongtao_Wu:","~Mahsa_Shoaran1","https://dblp.org/search/pid/api?q=author:Volkan_Cevher:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2502.16249"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2502-16249,\n  publtype={informal},\n  author={Arshia Afzal and Elías Abad-Rocamora and Leyla Naz Candogan and Pol Puigdemont and Francesco Tonin and Yongtao Wu and Mahsa Shoaran and Volkan Cevher},\n  title={Linear Attention for Efficient Bidirectional Sequence Modeling},\n  year={2025},\n  month={February},\n  cdate={1738368000000},\n  journal={CoRR},\n  volume={abs/2502.16249},\n  url={https://doi.org/10.48550/arXiv.2502.16249}\n}\n"},"abstract":{"value":"Transformers with linear attention enable fast and parallel training. Moreover, they can be formulated as Recurrent Neural Networks (RNNs), for efficient linear-time inference. While extensively evaluated in causal sequence modeling, they have yet to be extended to the bidirectional setting. This work introduces the LION framework, establishing new theoretical foundations for linear transformers in bidirectional sequence modeling. LION constructs a bidirectional RNN equivalent to full Linear Attention. This extends the benefits of linear transformers: parallel training, and efficient inference, into the bidirectional setting. Using LION, we cast three linear transformers to their bidirectional form: LION-LIT, the bidirectional variant corresponding to (Katharopoulos et al., 2020); LION-D, extending RetNet (Sun et al., 2023); and LION-S, a linear transformer with a stable selective mask inspired by selectivity of SSMs (Dao & Gu, 2024). Replacing the attention block with LION (-LIT, -D, -S) achieves performance on bidirectional tasks that approaches that of Transformers and State-Space Models (SSMs), while delivering significant improvements in training speed. Our implementation is available in http://github.com/LIONS-EPFL/LION."},"title":{"value":"Linear Attention for Efficient Bidirectional Sequence Modeling"},"authors":{"value":["Arshia Afzal","Elías Abad-Rocamora","Leyla Naz Candogan","Pol Puigdemont","Francesco Tonin","Yongtao Wu","Mahsa Shoaran","Volkan Cevher"]}},"tmdate":1747300875749,"pdate":1735689600000,"tcdate":1747112551868,"writers":["~"],"signatures":["~Mahsa_Shoaran1"],"forum":"4JUPAGcoFx","license":"CC BY-SA 4.0","number":436386,"cdate":1738368000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747300875749,"domain":"DBLP.org","id":"4JUPAGcoFx","version":2},{"content":{"summary":{"value":"This paper investigates the dynamical properties of tokens in a pre-trained Mamba model, a type of selective state space model (SSM). Despite the empirical success of Mamba, a comprehensive theoretical understanding of its internal mechanisms is lacking in various areas. The authors address this gap in one area by deriving the dynamical system governing the continuous-time limit of the Mamba model and analyzing its asymptotic behavior. In the one-dimensional case, they prove that the system exhibits one of two behaviors: either all tokens converge to zero or diverge to infinity. They provide criteria based on model parameters to determine which scenario occurs. Empirically, they verify that the convergent scenario negatively impacts model performance, while in the divergent scenario, tokens diverge at different rates, contributing unequally during training.\nBased on these findings, the authors propose two refinements to improve model performance:\n* Excluding the Convergent Scenario: By ensuring certain conditions on model parameters, they prevent the tokens from converging to zero.\n* Reordering Tokens: They reorder tokens based on their importance scores to account for the unequal contribution during training.\n\nExperimental results on language modeling and image classification tasks validate these refinements, showing improved accuracy and convergence speed."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Could you extend the theoretical analysis to higher-dimensional cases?\n2. Can you provide more details on how the token importance scores are computed and how the reordering is implemented? What is the computational overhead of this process?\n3. Do your findings and proposed refinements generalize to other selective state space models beyond Mamba?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- Originality: The paper addresses a gap in the theoretical understanding of selective state space models, specifically Mamba, by analyzing the continuous-time dynamics of tokens.\n- Theoretical Rigor: Provides rigorous mathematical proofs for the asymptotic behavior of tokens, offering clear criteria based on model parameters.\n- Practical Implications: The findings lead to actionable refinements that improve model performance, validated through experiments on language modeling and image classification tasks.\n- Clarity: The paper is well-structured, with logical progression and clear explanations that make complex mathematical concepts accessible."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Simplifying Assumptions: Using time-independent parameters and excluding other layers for mathematical convenience may oversimplify Mamba's actual behavior in practical settings.\n- Experimental Scope: Though supportive, experimental validation could be more extensive. Including additional datasets and comparisons with other models would strengthen the claims.\n- Limited Discussion on Limitations: The paper could benefit from a more thorough discussion of the limitations of the proposed refinements and potential challenges in practical implementation."}},"nonreaders":[],"tmdate":1732539602007,"tcdate":1730719267762,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5608/Reviewer_FpJk"],"signatures":["ICLR.cc/2025/Conference/Submission5608/Reviewer_FpJk"],"forum":"qtTIP5Gjc5","number":4,"license":"CC BY 4.0","cdate":1730719267762,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5608/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732539602007,"domain":"ICLR.cc/2025/Conference","replyto":"qtTIP5Gjc5","id":"I6vBIt5mfc","forumContent":{"TLDR":{"value":"We describe the dynamical properties of tokens in a deep selective state-space model, and based on that, we suggest improvements to the model by excluding negatively impactful scenarios and reordering tokens using importance scores."},"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Selective state-space model","continuous-time limit","dynamical system","asymptotic behavior","token reordering"]},"supplementary_material":{"value":"/attachment/f01b2b77a31c531086f0fefd1143d89ad8415a39.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity.  We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance.  For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training.  Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance.  Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications."},"_bibtex":{"value":"@inproceedings{\nvo2025demystifying,\ntitle={Demystifying the Token Dynamics of Deep Selective State Space Models},\nauthor={Thieu Vo and Duy-Tung Pham and Xin T. Tong and Tan Minh Nguyen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=qtTIP5Gjc5}\n}"},"title":{"value":"Demystifying the Token Dynamics of Deep Selective State Space Models"},"pdf":{"value":"/pdf/abc1efe1f910ee48b743f7ba26000b1d5e622f85.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"vo|demystifying_the_token_dynamics_of_deep_selective_state_space_models"},"authorids":{"value":["~Thieu_Vo1","~Duy-Tung_Pham1","~Xin_T._Tong2","~Tan_Minh_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Thieu Vo","Duy-Tung Pham","Xin T. Tong","Tan Minh Nguyen"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2411.11843v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"tang|bimamba_towards_accurate_1bit_state_space_models"},"authorids":{"value":["~Shengkun_Tang1","https://dblp.org/search/pid/api?q=author:Liqun_Ma:","https://dblp.org/search/pid/api?q=author:Haonan_Li:","https://dblp.org/search/pid/api?q=author:Mingjie_Sun:","~Zhiqiang_Shen1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2411.11843"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2411-11843,\n  publtype={informal},\n  author={Shengkun Tang and Liqun Ma and Haonan Li and Mingjie Sun and Zhiqiang Shen},\n  title={Bi-Mamba: Towards Accurate 1-Bit State Space Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2411.11843},\n  url={https://doi.org/10.48550/arXiv.2411.11843}\n}\n"},"abstract":{"value":"The typical selective state-space model (SSM) of Mamba addresses several limitations of Transformers, such as quadratic computational complexity with sequence length and significant inference-time memory requirements due to the key-value cache. However, the growing size of Mamba models continues to pose training and deployment challenges and raises environmental concerns due to considerable energy consumption. In this work, we introduce Bi-Mamba, a scalable and powerful 1-bit Mamba architecture designed for more efficient large language models with multiple sizes across 780M, 1.3B, and 2.7B. Bi-Mamba models are trained from scratch on data volume as regular LLM pertaining using an autoregressive distillation loss. Extensive experimental results on language modeling demonstrate that Bi-Mamba achieves performance comparable to its full-precision counterparts (e.g., FP16 or BF16) and much better accuracy than post-training-binarization (PTB) Mamba baselines, while significantly reducing memory footprint and energy consumption compared to the original Mamba model. Our study pioneers a new linear computational complexity LLM framework under low-bit representation and facilitates the future design of specialized hardware tailored for efficient 1-bit Mamba-based LLMs."},"title":{"value":"Bi-Mamba: Towards Accurate 1-Bit State Space Models"},"authors":{"value":["Shengkun Tang","Liqun Ma","Haonan Li","Mingjie Sun","Zhiqiang Shen"]}},"tmdate":1747135111665,"pdate":1704067200000,"tcdate":1738818103655,"writers":["~"],"signatures":["~Zhiqiang_Shen1"],"forum":"urfaI4AqbY","license":"CC BY-SA 4.0","number":301054,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747135111665,"domain":"DBLP.org","id":"urfaI4AqbY","version":2},{"content":{"summary":{"value":"The author applies a Mamba-based backbone to the one-stream visual tracking framework. The author explores various scan patterns among the template and search frames for this Mamba based tracker. Other widely-applied tricks, such as dynamic templates and motion tokens, are also applied to the tracker. The experiments show that TrackMamba outperforms Transformers at the same resolution while reducing memory usage with larger resolutions."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please see the weakness. I am glad to see the new Mamba model applied in tracking, and I do not emphasize SOTA performance too much. The reason I did not give a positive review at the beginning is that the explorations on Mamba were not adequate. However, I would raise the rating if my concerns are addressed."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. This paper has an interesting topic, which applies the Mamba-based structure to the visual tracking framework. The motivation also makes sense as the Mamba can have linear computational complexity to the transformer-based architecture. \n\n2. The overall framework is neat, though some widely applied tricks, such as motion tokens to strengthen the temporal modeling, are applied. The Mamba-based tracking framework has the potential to contribute to developing next-generation trackers. \n\n3. The experiments are also extensive. The benchmark results show some SOTA performance under specific settings. The author also strengthens the computation flops when the input resolution is larger, which validates the strength of Mamba-based models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. I am most concerned that the application of the Mamba-based backbone does not show enough strengths. Firstly, many widely adopted tricks from transformer-based one-stream trackers, such as dynamic templates and motion tokens, are applied. The author should develop the specific modules or functions under the long-sequence modeling framework (mamba)， such as more templates. The larger input resolution setting is good, but the speed of the method is around 30FPS, which is far lag behind the transformer trackers (typically larger than 80FPS). Thus, though the computation flops are reduced, the inference speed is not competitive. Thus, it makes the proposed tracker not as competitive as the 2024 year sota trackers in various benchmarks. \n\n2. The second point is that the overall backbone network is borrowed from the MambaVision paper, which was developed for general vision tasks as a backbone network. The difference only lies in the scanning pattern of the template and search images.  The author's exploration of the scanning pattern is straightforward and only involves the order of the template and search frames. It lacks enough insights, and I think the casual modeling of the Mamba should be explored for this video task.\n\n3. As addressed by point 1, the performance of the method is not competitive, removing the additional tricks. Also the speed \n(30fps) which is important for tracking, is really not satisfying as the pipeline is generally neat and simple."}},"nonreaders":[],"tmdate":1731428180225,"tcdate":1729245184751,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4593/Reviewer_PjTd"],"signatures":["ICLR.cc/2025/Conference/Submission4593/Reviewer_PjTd"],"forum":"V7QRVEZ0le","number":1,"license":"CC BY 4.0","cdate":1729245184751,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4593/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428180225,"domain":"ICLR.cc/2025/Conference","replyto":"V7QRVEZ0le","id":"4ShRUwf6xb","forumContent":{"TLDR":{"value":"A mamba-transformer tracking framework is proposed for resolution-scalable visual tracking."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Single Object Tracking"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Current one-stream Transformer-based trackers are quality but unfriendly to memory consumption of large resolution and long sequence, both of which are crucial keys to tracking tasks. Recently structured state space model (SSM) demonstrates promising performance and efficiency in sequence modeling but struggles to retrieve due to the limited hidden state number. To solve the computation challenge and explore the potential of Mamba, we propose TrackMamba, a Mamba-Transformer tracker containing TrackMamba Blocks and Attention Blocks. In order to better harness the scanning in TrackMamba Blocks for inter- and intra-frame modeling, we introduce various scan patterns for rearrangement and flipping. Furthermore, we propose Target Enhancement, including Temporal Token for target aggregation and search enhancement, and Temporal Mamba for target information cross-frame propagation. Extensive experiments show TrackMamba performs better than the first-generation one-stream Transformer-based tracker at same resolution and mitigates consumption growth when enlarging resolution, exhibiting the potential of Mamba-based model for large-resolution tracking."},"_bibtex":{"value":"@misc{\nzhang2024trackmamba,\ntitle={TrackMamba: Mamba-Transformer Tracking},\nauthor={Jiaming Zhang and Cheng Liang and Yutao Cui and Xiangbo Shu and Gangshan Wu and Limin Wang},\nyear={2024},\nurl={https://openreview.net/forum?id=V7QRVEZ0le}\n}"},"title":{"value":"TrackMamba: Mamba-Transformer Tracking"},"pdf":{"value":"/pdf/b6fd4058e862b558436284a5d3bc1c3711911485.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhang|trackmamba_mambatransformer_tracking"},"authorids":{"value":["~Jiaming_Zhang6","~Cheng_Liang1","~Yutao_Cui1","~Xiangbo_Shu1","~Gangshan_Wu1","~Limin_Wang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jiaming Zhang","Cheng Liang","Yutao Cui","Xiangbo Shu","Gangshan Wu","Limin Wang"]}},"version":2},{"content":{"comment":{"value":"**References**:\n\n[1] He, Zhenyu, et al. \"Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation.\" Forty-first International Conference on Machine Learning. 2024.\n\n[2] Shaham, Uri, et al. \"SCROLLS: Standardized CompaRison Over Long Language Sequences.\" Conference on Empirical Methods in Natural Language Processing. 2022.\n\n[3] Li, Shanda, et al. \"Functional Interpolation for Relative Positions improves Long Context Transformers.\" The Twelfth International Conference on Learning Representations. 2023.\n\n[4] Hoffmann, Jordan, et al. \"Training compute-optimal large language models.\" arXiv preprint arXiv:2203.15556 (2022).\n\n[5] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" First Conference on Language Modeling. 2024.\n\n[6] Chen, Yukang, et al. \"LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models.\" The Twelfth International Conference on Learning Representations. 2023."},"title":{"value":"Further Response to Reviewer 8dnP (3/3)"}},"tmdate":1732431852025,"tcdate":1732431852025,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1199/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission1199/Authors"],"forum":"Us1RXG1Ji2","number":14,"license":"CC BY 4.0","cdate":1732431852025,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1199/-/Official_Comment"],"mdate":1732431852025,"domain":"ICLR.cc/2025/Conference","replyto":"CgTDH0IEow","id":"W3Cl9NO1zW","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Positional Encoding","Equivariance","Addressing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Transformers rely on both content-based and position-based addressing mechanisms to make predictions, but existing positional encoding techniques often diminish the effectiveness of position-based addressing. Many current methods enforce rigid patterns in attention maps, limiting the ability to model long-range dependencies and adapt to diverse tasks. Additionally, most positional encodings are learned as general biases, lacking the specialization required for different instances within a dataset.\nTo address this, we propose TAPE: con**T**extualized equivari**A**nt **P**osition **E**mbedding, \na novel framework that enhances positional embeddings by incorporating sequence content across layers. TAPE introduces dynamic, context-aware positional encodings, overcoming the constraints of traditional fixed patterns. By enforcing permutation and orthogonal equivariance, TAPE ensures the stability of positional encodings during updates, improving robustness and adaptability. Our method can be easily integrated into pre-trained transformers, offering parameter-efficient fine-tuning with minimal overhead.\nExtensive experiments shows that TAPE achieves superior performance in language modeling, arithmetic reasoning, and long-context retrieval tasks compared to existing positional embedding techniques."},"_bibtex":{"value":"@misc{\nzhu2025rethinking,\ntitle={Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding},\nauthor={Jiajun Zhu and Peihao Wang and Ruisi Cai and Jason D. Lee and Pan Li and Zhangyang Wang},\nyear={2025},\nurl={https://openreview.net/forum?id=Us1RXG1Ji2}\n}"},"title":{"value":"Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding"},"pdf":{"value":"/pdf/fe2770a73fc402eb6d9af0f59272641e74b1995a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhu|rethinking_addressing_in_language_models_via_contextualized_equivariant_positional_encoding"},"authorids":{"value":["~Jiajun_Zhu1","~Peihao_Wang1","~Ruisi_Cai1","~Jason_D._Lee1","~Pan_Li2","~Zhangyang_Wang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jiajun Zhu","Peihao Wang","Ruisi Cai","Jason D. Lee","Pan Li","Zhangyang Wang"]}},"version":2},{"content":{"comment":{"value":"Thank you for your reply. Your rebuttal has solved most of my concerns, but some are still not addressed. \nThe remaining concerns almost overlap with Reviewer C5pc's new concerns.\n\nFor example, \n\n(1) why your mamba-transformer is faster than DiT? The rebuttal shows that DIMSUM is faster than DiT on ImageNet 256 (256 tokens considering the downsampler of the pre-trained autoencoder), however, a single Mamba module is faster than a transformer module only when the number of tokens is greater than 1000 (please refer to Fig. 8 in [1]);\n\n(2) the performance of DiMSUM after longer training on ImageNet 256 is 2.11 which is worse than SiT (FiD=2.06) [2], though both of these methods use flow matching. Thus, the performance gain of Mamba remains ambiguous (at least marginal).\n\n[1] Gu A, Dao T. Mamba: Linear-time sequence modeling with selective state spaces[J]. arXiv preprint arXiv:2312.00752, 2023.\n\n[2] Ma N, Goldstein M, Albergo M S, et al. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers[J]. arXiv preprint arXiv:2401.08740, 2024."}},"tmdate":1730890686916,"tcdate":1723528321379,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission13587/Reviewer_RLU7"],"signatures":["NeurIPS.cc/2024/Conference/Submission13587/Reviewer_RLU7"],"forum":"KqbLzSIXkm","number":9,"license":"CC BY 4.0","cdate":1723528321379,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission13587/-/Official_Comment","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730890686916,"domain":"NeurIPS.cc/2024/Conference","replyto":"YORjhv05fH","id":"V5nJ84FVdW","forumContent":{"TLDR":{"value":"A new Mamba architecture for image diffusion models"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Diffusion models","Mamba","Wavelet transformation"]},"supplementary_material":{"value":"/attachment/1281fd9115978629e9ed30829686033a665680cb.zip"},"primary_area":{"value":"generative_models"},"abstract":{"value":"We introduce a novel state-space architecture for diffusion models, effectively harnessing spatial and frequency information to enhance the inductive bias towards local features in input images for image generation tasks. While state-space networks, including Mamba, a revolutionary advancement in recurrent neural networks, typically scan input sequences from left to right, they face difficulties in designing effective scanning strategies, especially in the processing of image data. Our method demonstrates that integrating wavelet transformation into Mamba enhances the local structure awareness of visual inputs and better captures long-range relations of frequencies by disentangling them into wavelet subbands, representing both low- and high-frequency components. These wavelet-based outputs are then processed and seamlessly fused with the original Mamba outputs through a cross-attention fusion layer, combining both spatial and frequency information to optimize the order awareness of state-space models which is essential for the details and overall quality of image generation. Besides, we introduce a globally-shared transformer to supercharge the performance of Mamba, harnessing its exceptional power to capture global relationships. Through extensive experiments on standard benchmarks, our method demonstrates superior results compared to DiT and DIFFUSSM, achieving faster training convergence and delivering high-quality outputs. The codes and pretrained models are released at https://github.com/VinAIResearch/DiMSUM.git."},"_bibtex":{"value":"@inproceedings{\nphung2024dimsum,\ntitle={Di{MSUM}: Diffusion Mamba - A Scalable and Unified Spatial-Frequency Method for Image Generation},\nauthor={Hao Phung and Quan Dao and Trung Tuan Dao and Hoang Phan and Dimitris N. Metaxas and Anh Tuan Tran},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=KqbLzSIXkm}\n}"},"title":{"value":"DiMSUM: Diffusion Mamba - A Scalable and Unified Spatial-Frequency Method for Image Generation"},"pdf":{"value":"/pdf/0ae8bfdeeec0ac6c1b9be00728313d0eee7040d2.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"phung|dimsum_diffusion_mamba_a_scalable_and_unified_spatialfrequency_method_for_image_generation"},"authorids":{"value":["~Hao_Phung1","~Quan_Dao1","~Trung_Tuan_Dao1","~Hoang_Phan1","~Dimitris_N._Metaxas1","~Anh_Tuan_Tran2"]},"authors":{"value":["Hao Phung","Quan Dao","Trung Tuan Dao","Hoang Phan","Dimitris N. Metaxas","Anh Tuan Tran"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a Mamba-based backbone for 3D object detection focused on more effective foreground voxel encoding. The method selects top-scoring foreground voxels, serializes them using a rotation-augmented Hilbert curve, and processes them with RGSW for propagating local and global context, and SASFMamba to enhance context representation via state space modeling. Experiments are performed on nuScenes, KITTI, and Waymo benchmarks, along with ablation studies to justify each component."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. How robust is the method to imperfect, noisy, or ambiguous foreground scoring? Do detection performance or ablations degrade gracefully if the scoring network fails to highlight the correct voxels?\n2. How do the authors envision handling the omission of background information in domains where background context could be a valuable cue (e.g., occlusions, instance support)?\n3. Please elaborate on any observed failure modes (missed detection, false positives/negatives), especially with reference to dataset bias, rare classes, or environmental conditions."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper clearly identifies a major inefficiency in prior Mamba-based 3D object detectors: unnecessary computation over background voxels, and the resulting context loss when naïvely restricting to the foreground.\n2. Foreground selection is performed using a trainable scoring mechanism with effective sampling and serialization, alleviating regional truncation problems. It is well-motivated both by experiments and design illustrations.\n3. The RGSW and the SASFMamba fusion modules are described in substantial detail, with concrete mathematical rationale, and the implementation roadmap is well clarified.\n4. The method is compared against a rich spectrum of modern baselines on three benchmarks. Relative improvements are numerically strong."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Limited discussion of causal sequence modeling approaches: Although the method addresses linear sequence encoding with foreground focus, it omits explicit comparison to other causal sequence modeling alternatives or other advanced state-space models.\n2. Modeling limitations: The method largely treats background voxels as non-informative, but in autonomous driving, background structure or clutter can contain subtle contextual cues (for object contextual priors, occlusion estimation, etc.). The effect of completely de-emphasizing background is not discussed in terms of potential blindspots for categories with ambiguous boundaries."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762921613311,"tcdate":1761920930554,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10258/Reviewer_cXEa"],"signatures":["ICLR.cc/2026/Conference/Submission10258/Reviewer_cXEa"],"forum":"e4t1775UJ1","number":4,"license":"CC BY 4.0","cdate":1761920930554,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10258/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762921613311,"domain":"ICLR.cc/2026/Conference","replyto":"e4t1775UJ1","id":"VcvZ9cIFvD","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["3D Object Detection","Mamba","Foreground Representation"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Linear modeling methods like Mamba have been merged as the effective backbone for the 3D object detection task. However, previous Mamba-based methods utilize the bidirectional encoding for the whole non-empty voxel sequence, which contains abundant useless background information in the scenes. Though directly encoding foreground voxels appears to be a plausible solution, it tends to degrade detection performance. We attribute this to the response attenuation and restricted context representation in the linear modeling for fore-only sequences. To address this problem, we propose a novel backbone, termed Fore-Mamba3D, to focus on the foreground enhancement by modifying Mamba-based encoder. The foreground voxels are first sampled according to the predicted scores. Considering the response attenuation existing in the interaction of foreground voxels across different instances, we design a regional-to-global slide window (RGSW) to propagate the information from regional split to the entire sequence. Furthermore, a semantic-assisted and state spatial fusion module (SASFMamba) is proposed to enrich contextual representation by enhancing semantic and geometric awareness within the Mamba model. Our method emphasizes foreground-only encoding and alleviates the distance-based and causal dependencies in the linear autoregression model. The superior performance across various benchmarks demonstrates the effectiveness of Fore-Mamba3D in the 3D object detection task."},"_bibtex":{"value":"@inproceedings{\nning2026foremambad,\ntitle={Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection},\nauthor={Zhiwei Ning and Xuanang Gao and Jiaxi Cao and Runze Yang and Huiying Xu and Xinzhong Zhu and JIE YANG and Wei Liu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=e4t1775UJ1}\n}"},"title":{"value":"Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection"},"pdf":{"value":"/pdf/c88d1a4b5a3d89c3710599dac8b4ca8c7c9c1d6e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"ning|foremamba3d_mambabased_foregroundenhanced_encoding_for_3d_object_detection"},"authorids":{"value":["~Zhiwei_Ning2","~Xuanang_Gao1","~Jiaxi_Cao1","~Runze_Yang2","~Huiying_Xu1","~Xinzhong_Zhu1","~JIE_YANG18","~Wei_Liu38"]},"authors":{"value":["Zhiwei Ning","Xuanang Gao","Jiaxi Cao","Runze Yang","Huiying Xu","Xinzhong Zhu","JIE YANG","Wei Liu"]}},"version":2},{"content":{"summary":{"value":"The paper proposed a Mamba-based protein language model, ProtMamba, using concatenated sequences from protein families, with a FIM training objective. This approach allows for faster training and inference speeds. Experiments demonstrate ProtMamba’s versatility across protein fitness prediction and context-conditioned generation."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- I’m curious about the impact of the number of context sequences on ProtMamba’s performance. For example, how does performance change with 0, 5, or more sequences, and is there a threshold beyond which additional context sequences no longer contribute to model performance? In your experiments on scaling FIM perplexity with the number of context sequences, it seems perplexity stabilizes with around 30 context sequences. Could you elaborate on this?\n\n- Inference Efficiency: Could you report on ProtMamba’s efficiency during inference? Additionally, how does ProtMamba’s performance, memory usage, and computational efficiency scale with increasing context length compared to transformer-based models, like PoET?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- **Novelty**: This is one of the first works to incorporate state-space models (SSMs) in protein language modeling, utilizing the Mamba architecture for efficient long-context handling. \n\n- **Innovative Input Design**: The input consists of a concatenation of unaligned homologous sequences separated by CLS tokens, with a carefully designed masking strategy. This design effectively leverages long homology contexts, maximizing the model’s ability to capture evolutionary information. Training with a Fill-In-The-Middle (FIM) objective enables flexible application to tasks like mutational effect prediction.\n\n- **Comprehensive Model Implementation and Training**: The authors have put substantial effort into implementing and training ProtMamba, incorporating techniques inspired by DNA modeling, such as callback mechanisms and sequence length warmup."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Performance**: ProtMamba does not show significant performance improvements over strong baselines such as ESM-2 and Tranception, which may limit its competitive edge.\n\n- **Additional Comparisons**: Including comparisons with other baseline models, such as PoET-205M[1], SaProt[2], ProtHyena[3], or PTM-Mamba[4] would provide a fair evaluation and offer a more comprehensive view of ProtMamba’s strengths and weaknesses.\n\n- One advantage of Mamba is its faster generation capability compared to transformer-based models. The authors could extend ProtMamba’s use cases by addressing protein sequence generative tasks, such as unconditional generation. A more detailed discussion in Section 3.4 comparing ProtMamba to other generative models would strengthen the paper. You could follow the setting and metrics in PROTEINBENCH[5] paper.\n\n\n[1] Truong Jr, T., & Bepler, T. (2023). Poet: A generative model of protein families as sequences-of-sequences. Advances in Neural Information Processing Systems, 36, 77379-77415.\n\n[2] Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., & Yuan, F. (2023). Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, 2023-10.\n\n[3] Zhang, Y. (2024). Prothyena: A fast and efficient foundation protein language model at single amino acid resolution. bioRxiv, 2024-01.\n\n[4] Peng, Z., Schussheim, B., & Chatterjee, P. (2024). PTM-Mamba: A PTM-aware protein language model with bidirectional gated Mamba blocks. bioRxiv.\n\n[5] Ye, F., Zheng, Z., Xue, D., Shen, Y., Wang, L., Ma, Y., ... & Gu, Q. (2024). ProteinBench: A Holistic Evaluation of Protein Foundation Models. arXiv preprint arXiv:2409.06744."}},"nonreaders":[],"tmdate":1733663346733,"tcdate":1730554703595,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6891/Reviewer_SxMF"],"signatures":["ICLR.cc/2025/Conference/Submission6891/Reviewer_SxMF"],"forum":"BMfHO2lXGe","number":3,"license":"CC BY 4.0","cdate":1730554703595,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6891/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733663346733,"domain":"ICLR.cc/2025/Conference","replyto":"BMfHO2lXGe","id":"u9BqeUaoPk","forumContent":{"TLDR":{"value":"ProtMamba is a protein state space model that can handle very long context and enables autoregressive generation and fitness prediction conditioned on a set of homologous sequences."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["proteins","protein sequence","protein language model","computational biology","generative model","protein engineering","protein fitness prediction","protein design"]},"supplementary_material":{"value":"/attachment/dae9053c91a055be470f2724557660f77a9bb545.zip"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Protein design has important implications for drug discovery, personalized medicine, and biotechnology. Models based on multiple sequence alignments efficiently capture the evolutionary information in homologous protein sequences, but multiple sequence alignment construction is imperfect. We present ProtMamba, a homology-aware but alignment-free protein language model based on the Mamba architecture. In contrast with attention-based models, ProtMamba efficiently handles very long context, comprising hundreds of protein sequences. We train ProtMamba on a large dataset of concatenated homologous sequences, using two GPUs. We combine autoregressive modeling and masked language modeling through a fill-in-the-middle training objective. This makes the model adapted to various protein design applications. We demonstrate ProtMamba's usefulness for the generation of novel sequences and for fitness prediction. ProtMamba reaches competitive performance with other protein language models despite its smaller size, which sheds light on the importance of long-context conditioning."},"_bibtex":{"value":"@misc{\nsgarbossa2025protmamba,\ntitle={ProtMamba: a homology-aware but alignment-free protein state space model},\nauthor={Damiano Sgarbossa and Cyril Malbranke and Anne-Florence Bitbol},\nyear={2025},\nurl={https://openreview.net/forum?id=BMfHO2lXGe}\n}"},"title":{"value":"ProtMamba: a homology-aware but alignment-free protein state space model"},"pdf":{"value":"/pdf/b0e6d7eb3e3ff7a0d5496fa74b52528e6ef7a954.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sgarbossa|protmamba_a_homologyaware_but_alignmentfree_protein_state_space_model"},"authorids":{"value":["~Damiano_Sgarbossa1","~Cyril_Malbranke1","~Anne-Florence_Bitbol1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Damiano Sgarbossa","Cyril Malbranke","Anne-Florence Bitbol"]}},"version":2},{"content":{"summary":{"value":"The paper addresses the challenge of distilling a pretrained Transformer (Pythia) into Mamba to achieve faster inference and lower memory usage while retaining performance. Specifically, it proposes a two-stage distillation recipe: 1. Distilling the Transformer’s softmax Attention into a Linear Attention based on the Hedgehog approach. 2. Using this linearized Attention to initialize and fine-tune a Mamba-based architecture (HedgeMamba). The method is evaluated on the OpenWebText dataset, and the proposed HedgeMamba achieves superior performance compared to the simple direct distillation into Mamba and Hedgehog distillation into Linear Attention."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please see major weakness. As to the third point, if it is difficult to evaluate with large models, it can be interesting if you can show some evaluations that the distillation improves the weak points of the vanilla Mamba described in some papers such as [3, 4]. In addition, if HedgeMamba is efficient, I want to raise the rating.\n\n[3] Park, J., Park, J., Xiong, Z., Lee, N., Cho, J., Oymak, S., ... & Papailiopoulos, D. (2024). Can mamba learn how to learn? a comparative study on in-context learning tasks. arXiv preprint arXiv:2402.04248. \n[4] You, W., Tang, Z., Li, J., Yao, L., & Zhang, M. (2024). Revealing and Mitigating the Local Pattern Shortcuts of Mamba. arXiv preprint arXiv:2410.15678."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":2},"strengths":{"value":"- The two-stage distillation process is novel and reasonable. It works much better than the simple direct distillation from Transformer to (Hedge)Mamba. Also, thanks to the superior expressiveness of Mamba over that of Linear Attention, the distilled HedgeMamba works better than Linear Attention models.  \n- The ablation studies effectively show the importance of the two-stage recipe.  \n- The paper is well written and provides enough details about their implementations, including the limitations."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"#### Major Weaknesses  \n- The important information about the efficiency of HedgeMamba architecture is missing. Because the motivation of the distillation in this paper is to create an efficient model, the author should compare the inference speed or FLOPs. I doubt the efficiency of the HedgeMamba because the hidden stage dimension is huge (2048) compared to Mamba (16 to 64), in addition to the newly introduced layers.  \n- Why is the large SSM hidden state used? Although it is related to the previous weakness point, the inference speed should be much slower compared to the original Mamba (I guess 2~5 times slower) due to the large hidden state size. Did you use Mamba2 instead of Mamba? (Mamba2 should be relatively faster even if the hidden state size is large.)  \n- A gap exists between the motivation and the evaluation. Although the motivation is to borrow the strong performance of large Transformers, the experiments are conducted with the 1B size model. The Pythia-1B is inferior to Mamba-790M, and, to this end, HedgeMamba is inferior to Mamba-790M. In addition, the distillation cost (12 days with 8xA100 GPUs) seems not so small compared with the scratch training cost of Mamba.  \n- The lack of comparison against other methods proposing the Transformer to Mamba distillation [1, 2]. Although the authors mentioned that the experimental setups are different, comparisons can be possible by trying previous methods with this paper’s setup. Different architectures can be compared with accuracy-efficiency trade-off.  \n\n#### Minor Weakness\n- Typo  \n    - L037) tokens representations -> token representations  \n\n[1] Wang, J., Paliotta, D., May, A., Rush, A., & Dao, T. (2024). The mamba in the llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems, 37, 62432-62457.  \n[2] Bick, Aviv, et al. \"Transformers to ssms: Distilling quadratic knowledge to subquadratic models.\" Advances in Neural Information Processing Systems 37 (2024): 31788-31812."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931329004,"tcdate":1761657014871,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19414/Reviewer_g5YJ"],"signatures":["ICLR.cc/2026/Conference/Submission19414/Reviewer_g5YJ"],"forum":"EXtPGwYdWe","number":2,"license":"CC BY 4.0","cdate":1761657014871,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19414/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931329004,"domain":"ICLR.cc/2026/Conference","replyto":"EXtPGwYdWe","id":"ZPX6co2XU2","forumContent":{"TLDR":{"value":"We propose a novel method for cross-architecture distillation from a Transformer model to a State-Space Model"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Distillation","Mamba","Transformers","Linear Attention","SSM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"State Space Models (SSMs) such as Mamba have become a popular alternative to Transformer models, due to their reduced memory consumption and higher throughput at generation compared to their Attention-based counterparts. On the other hand, the community has built up a considerable body of knowledge on how to train Transformers, and many pretrained Transformer models are readily available. \nTo facilitate the adoption of SSMs while leveraging existing pretrained Transformers, we aim to identify an effective recipe to distill an Attention-based model into a Mamba-like architecture. In prior work on cross-architecture distillation, however, it has been shown that a naive distillation procedure from Transformers to Mamba fails to preserve the original teacher performance, a limitation often overcome with hybrid solutions combining Attention and SSM blocks.\nThe key argument from our work is that, by equipping Mamba with a principled initialization, we can recover an overall better recipe for cross-architectural distillation. To this end, we propose a principled two-stage approach: first, we distill knowledge from a traditional Transformer into a linearized version of Attention, using an adaptation of the _kernel trick_. Then, we distill the linearized version into an adapted Mamba model that does not use any Attention block.\nOverall, the distilled Mamba model is able to preserve the original Pythia-1B Transformer performance in downstream tasks, maintaining a perplexity of 14.11 close to the teacher's 13.86. To show the efficacy of our recipe, we conduct thorough ablations at 1B scale with 10B tokens varying sequence mixer architecture, scaling analysis on model sizes and total distillation tokens, and a sensitivity analysis on tokens allocation between stages."},"_bibtex":{"value":"@misc{\nmoudgil2026attention,\ntitle={Attention to Mamba: A Recipe for Cross-Architecture Distillation},\nauthor={Abhinav Moudgil and Ningyuan Huang and Eeshan Gunesh Dhekane and Pau Rodriguez and Aswathy Balagopalan and Luca Zappella and Federico Danieli},\nyear={2026},\nurl={https://openreview.net/forum?id=EXtPGwYdWe}\n}"},"title":{"value":"Attention to Mamba: A Recipe for Cross-Architecture Distillation"},"pdf":{"value":"/pdf/ebc83b2b76691471b32e825bc66daf09e6371d95.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"moudgil|attention_to_mamba_a_recipe_for_crossarchitecture_distillation"},"authorids":{"value":["~Abhinav_Moudgil1","~Ningyuan_Huang1","~Eeshan_Gunesh_Dhekane1","~Pau_Rodriguez2","~Aswathy_Balagopalan1","~Luca_Zappella1","~Federico_Danieli1"]},"authors":{"value":["Abhinav Moudgil","Ningyuan Huang","Eeshan Gunesh Dhekane","Pau Rodriguez","Aswathy Balagopalan","Luca Zappella","Federico Danieli"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.12418v4"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"li|stgmamba_spatialtemporal_graph_learning_via_selective_state_space_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Lincan_Li:","~Hanchen_Wang2","~Wenjie_Zhang3","https://dblp.org/search/pid/api?q=author:Adelle_Coster:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.12418"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-12418,\n  publtype={informal},\n  author={Lincan Li and Hanchen Wang and Wenjie Zhang and Adelle Coster},\n  title={STG-Mamba: Spatial-Temporal Graph Learning via Selective State Space Model},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.12418},\n  url={https://doi.org/10.48550/arXiv.2403.12418}\n}\n"},"abstract":{"value":"Spatial-Temporal Graph (STG) data is characterized as dynamic, heterogenous, and non-stationary, leading to the continuous challenge of spatial-temporal graph learning. In the past few years, various GNN-based methods have been proposed to solely focus on mimicking the relationships among node individuals of the STG network, ignoring the significance of modeling the intrinsic features that exist in STG system over time. In contrast, modern Selective State Space Models (SSSMs) present a new approach which treat STG Network as a system, and meticulously explore the STG system's dynamic state evolution across temporal dimension. In this work, we introduce Spatial-Temporal Graph Mamba (STG-Mamba) as the first exploration of leveraging the powerful selective state space models for STG learning by treating STG Network as a system, and employing the Spatial-Temporal Selective State Space Module (ST-S3M) to precisely focus on the selected STG latent features. Furthermore, to strengthen GNN's ability of modeling STG data under the setting of selective state space models, we propose Kalman Filtering Graph Neural Networks (KFGN) for dynamically integrate and upgrade the STG embeddings from different temporal granularities through a learnable Kalman Filtering statistical theory-based approach. Extensive empirical studies are conducted on three benchmark STG forecasting datasets, demonstrating the performance superiority and computational efficiency of STG-Mamba. It not only surpasses existing state-of-the-art methods in terms of STG forecasting performance, but also effectively alleviate the computational bottleneck of large-scale graph networks in reducing the computational cost of FLOPs and test inference time. The implementation code is available at: \\url{https://github.com/LincanLi98/STG-Mamba}."},"title":{"value":"STG-Mamba: Spatial-Temporal Graph Learning via Selective State Space Model"},"authors":{"value":["Lincan Li","Hanchen Wang","Wenjie Zhang","Adelle Coster"]}},"tmdate":1747108635685,"pdate":1704067200000,"tcdate":1738822972875,"writers":["~"],"signatures":["~Hanchen_Wang2"],"forum":"1DHqCimm0R","license":"CC BY-SA 4.0","number":302378,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747108635685,"domain":"DBLP.org","id":"1DHqCimm0R","version":2},{"content":{"pdf":{"value":"/pdf/e1b901eb74cada10f9b09c456fa014222623c15c.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"li|stgmamba_spatialtemporal_graph_learning_via_selective_state_space_model"},"authorids":{"value":["~Lincan_Li1","~Hanchen_Wang2","~Wenjie_Zhang3"]},"html":{"value":"https://arxiv.org/pdf/2403.12418"},"abstract":{"value":"Spatial-Temporal Graph (STG) data is characterized as dynamic, heterogenous, and non-stationary, leading to the continuous challenge of spatial-temporal graph learning. In the past few years, various GNN-based methods have been proposed to solely focus on mimicking the relationships among node individuals of the STG network, ignoring the significance of modeling the intrinsic features that exist in STG system over time. In contrast, modern Selective State Space Models (SSSMs) present a new approach which treat STG Network as a system, and meticulously explore the STG system's dynamic state evolution across temporal dimension. In this work, we introduce Spatial-Temporal Graph Mamba (STG-Mamba) as the first exploration of leveraging the powerful selective state space models for STG learning by treating STG Network as a system, and employing the Spatial-Temporal Selective State Space Module (ST-S3M) to precisely focus on the selected STG latent features. Furthermore, to strengthen GNN's ability of modeling STG data under the setting of selective state space models, we propose Kalman Filtering Graph Neural Networks (KFGN) for dynamically integrate and upgrade the STG embeddings from different temporal granularities through a learnable Kalman Filtering statistical theory-based approach. Extensive empirical studies are conducted on three benchmark STG forecasting datasets, demonstrating the performance superiority and computational efficiency of STG-Mamba. It not only surpasses existing state-of-the-art methods in terms of STG forecasting performance, but also effectively alleviate the computational bottleneck of large-scale graph networks in reducing the computational cost of FLOPs and test inference time. The implementation code is available at: https://github.com/LincanLi98/STG-Mamba."},"title":{"value":"STG-Mamba: Spatial-temporal graph learning via selective state space model"},"authors":{"value":["Lincan Li","Hanchen Wang","Wenjie Zhang"]}},"tmdate":1736623709103,"pdate":1716008400000,"tcdate":1736623709103,"writers":["~Lincan_Li1","~Hanchen_Wang2","~Wenjie_Zhang3"],"signatures":["~Lincan_Li1"],"forum":"FZGF2FQYZ6","license":"CC BY-NC-ND 4.0","number":31987,"cdate":1736623709103,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1736623709103,"domain":"OpenReview.net/Archive","id":"FZGF2FQYZ6","version":2},{"content":{"summary":{"value":"This study introduces a low-latency streaming zero-shot TTS system with several key advancements: 1) the adoption of Mamba architecture for linear-time decoding, 2) the incorporation of rotary positional embeddings within the cross-attention module to manage infinite text streams, and 3) the implementation of semantic guidance to enhance system robustness. The proposed system achieves comparable performance in both offline and online inference settings."},"soundness":{"value":1},"confidence":{"value":5},"questions":{"value":"- The paper states, \"The embeddings can remain unchanged or be updated any time during streaming.\" on L191. Could you elaborate on the mechanisms for updating the enrollment speech embeddings? Furthermore, were these embeddings actually updated in the experiments conducted?\n\n- Could you provide statistical information for the reference enrollment speech? From experience, the duration of the reference enrollment speech greatly influences the speaker similarity."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- This work presents a novel scenario for zero-shot TTS systems and highlights its non-trivial challenges.\n  \n- This work validates the feasibility of Mamba, a linear-time decoding sequence modeling architecture, for zero-shot TTS tasks, providing insights that may inspire future research.\n  \n- The proposed streaming zero-shot TTS model achieves comparable performance in a simluated online inference setting."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Some key motivations in this paper, such as those mentioned in Table 1, haven't been thoroughly validated. Specifically:\n  \n  - The authors should consider adding experiments that test the model's robustness with sufficiently long text streams to support its claim of \"infinitely long speech streaming\" capabilities. Additionally, it would be beneficial for the authors to design experiments that demonstrate the advantages of modeling entire text streams compared to synthesizing them sentence by sentence.\n    \n  - The authors should compare the performance gap between complete texts and incomplete streaming texts to verify the model's ability to make \"seamless transitions between short speech chunks\".\n    \n  - Furthermore, regarding the \"text-audio stream synchronization\" capability, there seems to be a lack of clear explanation and corresponding experiments.\n    \n  - In addition, the paper positions itself as a low-latency streaming zero-shot TTS system, yet it falls short of providing a latency comparison.\n    \n- Some experimental details are missing, which could affect the reproducibility of the proposed system. Please refer to the Questions."}},"nonreaders":[],"tmdate":1731427414224,"tcdate":1729958858674,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1359/Reviewer_aFLh"],"signatures":["ICLR.cc/2025/Conference/Submission1359/Reviewer_aFLh"],"forum":"RK3Gj9J5my","number":1,"license":"CC BY 4.0","cdate":1729958858674,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1359/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427414224,"domain":"ICLR.cc/2025/Conference","replyto":"RK3Gj9J5my","id":"Ko7MfIU0Pr","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We introduce a zero-shot text-to-speech model that is capable of generating speech from a continuous text stream, with support for on-device streaming."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["zero-shot text-to-speech","streaming","state-space models"]},"supplementary_material":{"value":"/attachment/eb3ee0f6f9fc423a83e128f51cb138fde2d018cc.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Existing zero-shot text-to-speech (TTS) systems are typically designed to process complete sentences and are constrained by the maximum duration for which they have been trained. However, in many streaming applications, texts arrive continuously in short chunks, necessitating instant responses from the system. We identify the essential capabilities required for chunk-level streaming and introduce L3Speech, a stream-aware model that supports non-stop speech generation, text-audio stream synchronization, and seamless transitions between short speech chunks. To achieve these, we propose (1) adopting Mamba, a class of sequence modeling distinguished by linear-time decoding, which is augmented by cross-attention mechanisms for conditioning, (2) utilizing rotary positional embeddings in the computation of cross-attention, enabling the model to process a non-stop text stream by sliding a window, and (3) decoding with semantic guidance, a technique that aligns speech with the transcript during inference with minimal overhead. Experimental results demonstrate that our models are competitive with state-of-the-art language model-based zero-shot TTS models, while also providing flexibility to support a wide range of streaming scenarios."},"_bibtex":{"value":"@misc{\ndang2025zeroshot,\ntitle={Zero-Shot Text-to-Speech from Continuous Text Streams},\nauthor={Trung Dang and David Aponte and Dung Tran and Tianyi Chen and Kazuhito Koishida},\nyear={2025},\nurl={https://openreview.net/forum?id=RK3Gj9J5my}\n}"},"title":{"value":"Zero-Shot Text-to-Speech from Continuous Text Streams"},"pdf":{"value":"/pdf/ae6d24286703bb7fbfcd629543b4f9e8d221ff52.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"dang|zeroshot_texttospeech_from_continuous_text_streams"},"authorids":{"value":["~Trung_Dang1","~David_Aponte1","~Dung_Tran2","~Tianyi_Chen3","~Kazuhito_Koishida1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Trung Dang","David Aponte","Dung Tran","Tianyi Chen","Kazuhito Koishida"]}},"version":2},{"content":{"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["State Space Model; Mixture of Experts; Mamba"]},"primary_area":{"value":"deep_learning"},"abstract":{"value":"State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant per-step inference-time computation and memory complexity. Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations, positioning them as strong alternatives to Transformers for long sequence modeling. However, efficiently scaling the expressive power of SSMs, particularly with Mixture of Experts (MoE), remains challenging, as naive integration attempts often falter or degrade performance. In this work, we introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts. By sharing routing decisions between projection layers and lightweight sub-modules within Mamba across experts, RoM leverages synergies among linear projection experts for effective and efficient sparse scaling of Mamba layers. At a scale of 1.3B active parameters (10B total) and 16K training sequence length, RoM achieves language modeling performance equivalent to a dense Mamba model requiring over 2.3$\\times$ more active parameters, and demonstrates consistent perplexity across context lengths. Experimental results further show RoM effectively scales hybrid language models, yielding a 23% FLOPS saving compared to dense Mamba scaling for similar performance. We release our training codebase at https://github.com/zhanzheng8585/Routing-Mamba."},"_bibtex":{"value":"@inproceedings{\nzhan2025routing,\ntitle={Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection},\nauthor={Zheng Zhan and Liliang Ren and Shuohang Wang and Liyuan Liu and Yang Liu and Yeyun Gong and Yanzhi Wang and yelong shen},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=lqywifxoo1}\n}"},"title":{"value":"Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection"},"pdf":{"value":"/pdf/189421ae12f7679f6560b0b200b7f59631788726.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"zhan|routing_mamba_scaling_state_space_models_with_mixtureofexperts_projection"},"authorids":{"value":["~Zheng_Zhan3","~Liliang_Ren1","~Shuohang_Wang1","~Liyuan_Liu3","~Yang_Liu50","~Yeyun_Gong2","~Yanzhi_Wang3","~yelong_shen1"]},"authors":{"value":["Zheng Zhan","Liliang Ren","Shuohang Wang","Liyuan Liu","Yang Liu","Yeyun Gong","Yanzhi Wang","yelong shen"]}},"tmdate":1783627429586,"pdate":1758216682571,"tcdate":1746291340084,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission6467/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission6467/Authors"],"forum":"lqywifxoo1","license":"CC BY 4.0","number":6467,"cdate":1746291340084,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission6467/-/Full_Submission","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission6467/-/Camera_Ready_Revision"],"mdate":1783627429586,"odate":1761704792026,"domain":"NeurIPS.cc/2025/Conference","id":"lqywifxoo1","version":2},{"content":{"summary":{"value":"The paper proposes a wavelet transform-based positional representation for transformer models. It begins with highlighting the properties of existing position representation techniques such as relative position bias, RoPE and ALiBi. The authors then show how RoPE can be interpreted as a single scale wavelet transform with a Haar-like wavelet. Then the authors present a position embedding based on wavelet transform. This multi-scale embedding can be viewed as a generalization of RoPE and possess attractive \"multi-window\" properties of ALiBi. Experiment results have been reported on short and longer-context scenarios showing improved perplexity over existing methods and better extrapolation properties."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See above."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- The paper is well-written barring some typographical errors. It motivates the problem well, starts with a well-defined goal, describes existing methods clearly, and presents the proposed method in a manner which is easy to appreciate.\n- The paper tackles a critical problem of context length extrapolation which often arises in practical settings. The method holds significance, not just for the language modeling community, but also other domains such as time series forecasting where context length extrapolation may improve the performance of models on high-frequency data (see [1]). \n- The proposed method is novel to the best of my knowledge. It creatively relates RoPE to time-frequency analysis and proposes a promising method based on wavelet transforms. \n- The method outperforms alternatives and scales gracefully with extrapolated context lengths. \n\n[1] Ansari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" arXiv preprint arXiv:2403.07815 (2024)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- While the discussion of related methods is generally well done, discussion of RoPE scaling techniques (linear, NTK-aware) is missing. A discussion and comparison with these techniques would significantly improve the positioning of this work.\n- The results reported in sections 6 and 7 are excellent proofs of concept but they lack comprehensiveness. Particularly, in section 7, evaluations beyond the CodeParrot dataset would be needed to thoroughly appreciate the proposed method. Furthermore, the paper only evaluates the model in term of the perplexity. For a stronger evaluation, this needs to be augmented with task-based evaluations. Apart from common language tasks, you might also want to consider associative recall tasks [2] and the long range arena [3] since the focus of this work is context length extrapolation. To further demonstrate the robustness of the proposed method, domains beyond natural language may also be considered, for example, DNA modeling [4], audio generation [4] and time series forecasting [1]. \n\nI am willing to raise my score if the evaluation is strengthened. \n\n[1] Ansari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" arXiv preprint arXiv:2403.07815 (2024).    \n[2] Arora, Simran, et al. \"Zoology: Measuring and improving recall in efficient language models.\" arXiv preprint arXiv:2312.04927 (2023).    \n[3] Tay, Yi, et al. \"Long range arena: A benchmark for efficient transformers.\" arXiv preprint arXiv:2011.04006 (2020).    \n[4] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023)."}},"nonreaders":[],"tmdate":1731428996941,"tcdate":1730593922793,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13404/Reviewer_DFcs"],"signatures":["ICLR.cc/2025/Conference/Submission13404/Reviewer_DFcs"],"forum":"OhauMUNW8T","number":2,"license":"CC BY 4.0","cdate":1730593922793,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13404/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428996941,"domain":"ICLR.cc/2025/Conference","replyto":"OhauMUNW8T","id":"Rmom89YIBT","forumContent":{"TLDR":{"value":"We found that RoPE can be interpreted as a restricted wavelet transform. And we propose a new position representation method that captures window sizes by leveraging wavelet transforms without limiting the model's attention field."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Positional Encoding","Extrapolation","Wavelet Transform","Transformers","RoPE","ALiBi","NLP"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"In the realm of large-scale language models, a significant challenge arises when extrapolating sequences beyond the maximum allowable length. \nThis is because the model's position embedding mechanisms are limited to positions encountered during training, thus preventing effective representation of positions in longer sequences.\nWe analyzed conventional position encoding methods for long contexts and found the following characteristics.\n(1) When the representation dimension is regarded as the time axis, Rotary Position Embedding (RoPE) can be interpreted as a restricted wavelet transform using Haar-like wavelets. \nHowever, because it uses only a fixed scale parameter, it does not fully exploit the advantages of wavelet transforms, which capture the fine movements of non-stationary signals using multiple scales (window sizes). \nThis limitation could explain why RoPE performs poorly in extrapolation.\n(2)\nPrevious research as well as our own analysis indicates that Attention with Linear Biases (ALiBi) functions similarly to windowed attention, using windows of varying sizes.\nHowever, it has limitations in capturing deep dependencies because it restricts the receptive field of the model.\nFrom these insights, we propose a new position representation method that captures multiple scales (i.e., window sizes) by leveraging wavelet transforms without limiting the model's attention field.\nExperimental results show that this new method improves the performance of the model in both short and long contexts. \nIn particular, our method allows extrapolation of position information without limiting the model's attention field."},"_bibtex":{"value":"@inproceedings{\noka2025waveletbased,\ntitle={Wavelet-based Positional Representation for Long Context},\nauthor={Yui Oka and Taku Hasegawa and Kyosuke Nishida and Kuniko Saito},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=OhauMUNW8T}\n}"},"title":{"value":"Wavelet-based Positional Representation for Long Context"},"pdf":{"value":"/pdf/d5ed2fc8f4eb31801279781b07970b112229814c.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"oka|waveletbased_positional_representation_for_long_context"},"authorids":{"value":["~Yui_Oka1","~Taku_Hasegawa1","~Kyosuke_Nishida2","~Kuniko_Saito1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yui Oka","Taku Hasegawa","Kyosuke Nishida","Kuniko Saito"]}},"version":2},{"content":{"summary":{"value":"The paper introduces DPMFormer, a dual-path Mamba–Transformer hybrid designed for all-in-one image restoration. It combines a Dual-Path Mamba Block (DPMB)—one branch using Mamba for long-range dependency modeling and the other using a lightweight Transformer for local feature aggregation. A Path Interaction Unit (PIU) fuses global and local cues, while a Degradation-Aware Guidance Module (DGM) provides task conditioning via learned degradation priors. Experiments on multiple degradation benchmarks (rain, haze, low-light, noise) show improvements over several transformer and Mamba-based baselines"},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Could the authors provide FLOPs and throughput comparisons with Restormer and VMamba to justify efficiency claims?\n2. What are the qualitative differences between features extracted by the Mamba path and Transformer path? (e.g., visualization or layer attention maps)\n3. How does DPMFormer perform on real-capture datasets such as LOL-V2, RainDS, or SOTS-real?\n4. Have the authors compared their design to Swin-Mamba or other existing Mamba–Transformer hybrids?\n5. Does the DGM generalize to unseen degradation mixtures, or is it trained with supervision on specific degradation types?\n6. How sensitive is performance to the relative weighting or depth of the two paths? Could a single-path Mamba or Transformer with the same parameter budget achieve comparable results?\n7. The PSNR gains are small—can the authors include statistical significance or variance over multiple runs?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This paper proposes a hybrid Mamba–Transformer architecture, integrating sequence modeling and spatial self-attention. The dual-path structure is intuitively appealing for balancing long-range reasoning and local fidelity. Degradation-aware conditioning provides some adaptivity for mixed degradations.\n2. Experiments cover common restoration tasks and compare with both transformer (Restormer, Uformer) and Mamba-based baselines. \n3. Ablation studies isolate the effect of each module (Mamba path, Transformer path, DGM).\n4. Extends Mamba-based modeling into restoration, which remains relatively new."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Combining Mamba and Transformer paths is a logical but incremental step; there is little theoretical or architectural innovation beyond simple concatenation and gating.\n2. The PIU fusion resembles standard cross-attention or gating mechanisms used in hybrid CNN–Transformer or Swin–MLP models.\n3. The paper lacks rigorous analysis on why or when the Mamba path improves over pure Transformer designs. No detailed exploration of information flow or path synergy (e.g., attention entropy, frequency response, or token dependency visualization).\n4. Reported gains are modest (≈0.2–0.4 dB PSNR) and often within noise margins. On several datasets, DPMFormer lags behind recent AIR systems (PromptIR, UniRestorer) in unseen or composite degradations.\n5. All experiments are conducted on synthetic benchmarks; no evaluation on real-world degradation datasets or perceptual metrics (LPIPS, NIQE). Efficiency and scalability (especially GPU memory and throughput vs. Restormer or VMamba) are not reported."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762932907145,"tcdate":1761901967072,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20005/Reviewer_N7bx"],"signatures":["ICLR.cc/2026/Conference/Submission20005/Reviewer_N7bx"],"forum":"yk95hwXbUZ","number":3,"license":"CC BY 4.0","cdate":1761901967072,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20005/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762932907145,"domain":"ICLR.cc/2026/Conference","replyto":"yk95hwXbUZ","id":"z6GV1vROnV","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Transformer","Mamba","Dual-Path Network","Lightweight Network"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Vision Transformers have achieved outstanding performance in image super-resolution (SR), but existing lightweight models rely on window-based attention, limiting their ability to model global dependencies essential for high-quality reconstruction. To address these challenges, we present DPMFormer, a Dual-Path Mamba–Transformer architecture for lightweight image super-resolution.  Rather than a simple combination of a state-space model and a Transformer, the design couples two streams throughout the network. On the Transformer side, an Enhanced Transformer Layer (ETL) replaces self-attention with Spatial–Channel Correlation (SCC) and a Depthwise-SwiGLU Feed-Forward (DW-SwiFFN). On the Mamba side, Lightweight Bi-directional Mamba Layers (LBi-ML) implement single-pass bidirectionality via channel split and sequence reversal with additive cross coupling. The streams interact at two levels: within each block, a Cross-Attention Layer (CAL) performs fixed, non-overlapping cross fusion,\nand across blocks, Inter-branch Exchange Bridges (IEB) use resolution-preserving 1 × 1 adapters around tokenization to align channel spaces in both directions. Besides, we employ RMSNorm to reduce normalization overhead and, under our setup, observe modest, configuration-dependent gains. Extensive experiments show that DPMFormer reduces FLOPs by 47.3G (21\\%) and parameters by 21K under 2 × upsampling compared to HiT-SR, while almost achieving state-of-the-art performance across five benchmarks. Measured on an RTX 4090, our method reaches 668 ms latency, yielding 1.59 × and 2.33 × speedups over MambaIR and CATANet, respectively. The code will be publicly released."},"_bibtex":{"value":"@misc{\nyin2026dpmformer,\ntitle={{DPMF}ormer: Dual-Path Mamba-Transformer for Efficient Image Super\\nobreakdash-Resolution},\nauthor={Zhengnan Yin},\nyear={2026},\nurl={https://openreview.net/forum?id=yk95hwXbUZ}\n}"},"title":{"value":"DPMFormer: Dual-Path Mamba-Transformer for Efficient Image Super‑Resolution"},"pdf":{"value":"/pdf/933d7f41da3c9fd9c83cfe8eb7c75a778305cf0a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yin|dpmformer_dualpath_mambatransformer_for_efficient_image_superresolution"},"authorids":{"value":["~Zhengnan_Yin2"]},"authors":{"value":["Zhengnan Yin"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.13600v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"qiao|vlmamba_exploring_state_space_models_for_multimodal_learning"},"authorids":{"value":["~Yanyuan_Qiao1","~Zheng_Yu2","https://dblp.org/search/pid/api?q=author:Longteng_Guo:","https://dblp.org/search/pid/api?q=author:Sihan_Chen:","https://dblp.org/search/pid/api?q=author:Zijia_Zhao:","~Mingzhen_Sun1","https://dblp.org/search/pid/api?q=author:Qi_Wu_0001:","https://dblp.org/search/pid/api?q=author:Jing_Liu_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.13600"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-13600,\n  publtype={informal},\n  author={Yanyuan Qiao and Zheng Yu and Longteng Guo and Sihan Chen and Zijia Zhao and Mingzhen Sun and Qi Wu and Jing Liu},\n  title={VL-Mamba: Exploring State Space Models for Multimodal Learning},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.13600},\n  url={https://doi.org/10.48550/arXiv.2403.13600}\n}\n"},"abstract":{"value":"Multimodal large language models (MLLMs) have attracted widespread interest and have rich applications. However, the inherent attention mechanism in its Transformer structure requires quadratic complexity and results in expensive computational overhead. Therefore, in this work, we propose VL-Mamba, a multimodal large language model based on state space models, which have been shown to have great potential for long-sequence modeling with fast inference and linear scaling in sequence length. Specifically, we first replace the transformer-based backbone language model such as LLama or Vicuna with the pre-trained Mamba language model. Then, we empirically explore how to effectively apply the 2D vision selective scan mechanism for multimodal learning and the combinations of different vision encoders and variants of pretrained Mamba language models. The extensive experiments on diverse multimodal benchmarks with competitive performance show the effectiveness of our proposed VL-Mamba and demonstrate the great potential of applying state space models for multimodal learning tasks."},"title":{"value":"VL-Mamba: Exploring State Space Models for Multimodal Learning"},"authors":{"value":["Yanyuan Qiao","Zheng Yu","Longteng Guo","Sihan Chen","Zijia Zhao","Mingzhen Sun","Qi Wu","Jing Liu"]}},"tmdate":1744711947225,"pdate":1704067200000,"tcdate":1731461115630,"writers":["~"],"signatures":["~Zheng_Yu2"],"forum":"EiwQ1euACb","license":"CC BY-SA 4.0","number":184240,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1744711947225,"domain":"DBLP.org","id":"EiwQ1euACb","version":2},{"content":{"summary":{"value":"This paper proposed a framework for long-term time series forecasting. The model enhances the Mamba state space model by integrating three components. Experimental results on eight benchmark datasets show that Flames achieves good performance, outperforming Transformer- and MLP-based baselines with better noise robustness, scalability, and computational efficiency."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"- How to simplify or modularize the integration of multi-scale processing, Adaptive Fourier Filters, and the Inverse Laplace Transform to make the model easier to implement and fine-tune?\n- What experimental approaches or visual analyses could be added to empirically validate how the model captures transient dynamics and multi-scale temporal features?\n- How can the model adaptively determine or learn the optimal number of scales for different datasets or forecasting horizons wthout manual tuning?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- Combines Fourier and Laplace transforms with state space modeling for a unique approach to capturing multi-scale and transient temporal patterns.\n- Demonstrates consistent improvement over leading baselines across multiple datasets and noise levels, with extensive ablation and robustness studies.\n- Achieves linear computational complexity, lower parameter counts, and faster inference while maintaining high accuracy."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The combination of multi-scale processing, Mamba, Adaptive Fourier Filters, and the Inverse Laplace Transform makes the overall model architecture intricate and potentially complex to implement and fine-tune compared to simpler linear or pure Mamba models The formulation, particularly for the Laplace transform integration, is dense.\n- While claims about interpretability are made, there is little empirical validation of how the model captures transient dynamics or multi-scale features in practice. \n- The paper notes that the optimal number of scales varies with the prediction length, suggesting that a fixed choice (like M=3 for efficiency) may not be universally optimal and would require a trade-off analysis for new datasets or forecasting horizons."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917360082,"tcdate":1761943463595,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4431/Reviewer_Q1SQ"],"signatures":["ICLR.cc/2026/Conference/Submission4431/Reviewer_Q1SQ"],"forum":"eEmTxqxLtU","number":2,"license":"CC BY 4.0","cdate":1761943463595,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4431/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917360082,"domain":"ICLR.cc/2026/Conference","replyto":"eEmTxqxLtU","id":"mZq0KEOSGz","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Time Series Forecasting","Adaptive Fourier Filter","Mamba","Multi- Scale","Laplace Transform"]},"supplementary_material":{"value":"/attachment/c0b5728e9c85ba05bbc40d733bfdf8d5f1b474c9.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Time series data usually exhibit intricate characteristics such as non-stationarity, noise, multi-scale periodicity, and transient dynamics, posing significant challenges to long-term time series forecasting (LTSF). While transformer-based models effectively capture long-range dependencies, their practical applications are hindered by high computational cost with quadratic complexity, noise sensitivity, and overfitting on small datasets. Moreover, time series present distinct patterns at different temporal resolutions, containing both fine-grained (micro) and coarse-grained (macro) information. To address these issues, we propose a novel framework, Flames (multi-scale Fourier Filter Mamba with Laplace), designed for efficient and robust LTSF. Specifically: (i) We introduce an adaptive Fourier filter with a selection module embedded into Mamba. At each scale, the neural operator uses Fourier analysis to refine feature representations, applies learnable thresholds for noise reduction, and captures inter-frequency interactions via global-local semantic filters through element multiplication. (ii) We incorporate the Laplace transform to capture transient dynamics. Extensive experiments on multiple benchmarks demonstrate that Flames consistently outperforms SOTA methods, achieving superior accuracy–efficiency trade-offs. Results highlight its strong robustness and scalability, particularly in noisy or transient settings."},"_bibtex":{"value":"@misc{\nwu2026flames,\ntitle={Flames: Multi-Scale Mamba with Adaptive Fourier Filters and Laplace Transform for Time Series Forecasting},\nauthor={Yuhan Wu and Xiyu Meng and Peiheng Li and Xin Zhang and Peiliang Gong and Canran Xiao and Yabo Dong and Dongming Lu},\nyear={2026},\nurl={https://openreview.net/forum?id=eEmTxqxLtU}\n}"},"title":{"value":"Flames: Multi-Scale Mamba with Adaptive Fourier Filters and Laplace Transform for Time Series Forecasting"},"pdf":{"value":"/pdf/307b599472799e0765bbdf39d7638180867a1b17.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wu|flames_multiscale_mamba_with_adaptive_fourier_filters_and_laplace_transform_for_time_series_forecasting"},"authorids":{"value":["~Yuhan_Wu6","~Xiyu_Meng1","~Peiheng_Li2","~Xin_Zhang33","~Peiliang_Gong1","~Canran_Xiao1","~Yabo_Dong1","~Dongming_Lu1"]},"authors":{"value":["Yuhan Wu","Xiyu Meng","Peiheng Li","Xin Zhang","Peiliang Gong","Canran Xiao","Yabo Dong","Dongming Lu"]}},"version":2},{"content":{"venue":{"value":"AAAI 2025"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/33336/35491"},"venueid":{"value":"dblp.org/conf/AAAI/2025"},"paperhash":{"value":"liu|sigma_selective_gated_mamba_for_sequential_recommendation"},"authorids":{"value":["~Ziwei_Liu9","~Qidong_Liu2","~Yejing_Wang1","~Wanyu_Wang1","~Pengyue_Jia1","~Maolin_Wang1","~Zitao_Liu1","~Yi_Chang4","~Xiangyu_Zhao1"]},"html":{"value":"https://doi.org/10.1609/aaai.v39i12.33336"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/Liu0WWJWLCZ25,\n  author={Ziwei Liu and Qidong Liu and Yejing Wang and Wanyu Wang and Pengyue Jia and Maolin Wang and Zitao Liu and Yi Chang and Xiangyu Zhao},\n  title={SIGMA: Selective Gated Mamba for Sequential Recommendation},\n  year={2025},\n  cdate={1735689600000},\n  pages={12264-12272},\n  url={https://doi.org/10.1609/aaai.v39i12.33336},\n  booktitle={AAAI},\n  crossref={conf/aaai/2025}\n}\n"},"abstract":{"value":"Sequential Recommender Systems (SRS) has stood out as a highly promising technique in numerous domains due to its impressive capability of capturing complex user preferences. Current SRS have employed transformer-based models to give the next-item prediction. Nevertheless, its quadratic computational complexity has often resulted in notable inefficiencies, posing a significant obstacle to real-time recommendation processes. Recently, Mamba has demonstrated its exceptional effectiveness in time series prediction, delivering substantial improvements in both efficiency and effectiveness. However, directly applying Mamba to SRS poses certain challenges. Its unidirectional structure may impede the ability to capture contextual information in user-item interactions, while its instability in state estimation may hinder the ability to capture short-term patterns in interaction sequences. To address these issues, we propose a novel framework called Selective Gated Mamba for Sequential Recommendation (SIGMA). By introducing the Partially Flipped Mamba (PF-Mamba), we construct a special bi-directional structure to address the context modeling challenge. Then, to consolidate PF-Mamba's performance, we employed an input-dependent Dense Selective Gate (DS Gate) to allocate the weights of the two directions and further filter the sequential information. Moreover, for short sequence modeling, we devise a Feature Extract GRU (FE-GRU) to capture the short-term dependencies. Experimental results demonstrate that SIGMA significantly outperforms existing baselines across five real-world datasets. Our implementation code is available in Supplementary Material to ease reproducibility."},"title":{"value":"SIGMA: Selective Gated Mamba for Sequential Recommendation"},"authors":{"value":["Ziwei Liu","Qidong Liu","Yejing Wang","Wanyu Wang","Pengyue Jia","Maolin Wang","Zitao Liu","Yi Chang","Xiangyu Zhao"]}},"tmdate":1769056872199,"pdate":1735689600000,"externalIds":["dblp:conf/aaai/Liu0WWJWLCZ25"],"tcdate":1753977899755,"writers":["~"],"signatures":["~Qidong_Liu2"],"forum":"LpjnJpwnfh","license":"CC BY-SA 4.0","number":605065,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769056872199,"domain":"DBLP.org","id":"LpjnJpwnfh","version":2},{"content":{"pdf":{"value":"/pdf/8b61fa855bfb2521738e99295994a6a04dba06c9.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"liu|sigma_selective_gated_mamba_for_sequential_recommendation"},"authorids":{"value":["~Ziwei_Liu9","~Qidong_Liu2","~Yejing_Wang1","~Wanyu_Wang1","~Pengyue_Jia1","~Maolin_Wang1","~Zitao_Liu1","~Yi_Chang4","~Xiangyu_Zhao1"]},"html":{"value":"https://ojs.aaai.org/index.php/AAAI/article/view/33336"},"abstract":{"value":"Sequential Recommender Systems (SRS) have emerged as\na promising technique across various domains, excelling at\ncapturing complex user preferences. Current SRS have employed transformer-based models to give the next-item prediction. However, their quadratic computational complexity\noften lead to notable inefficiencies, posing a significant obstacle to real-time recommendation processes. Recently, Mamba\nhas demonstrated its exceptional effectiveness in time series\nprediction, delivering substantial improvements in both efficiency and effectiveness. However, directly applying Mamba\nto SRS poses certain challenges. Its unidirectional structure\nmay impede the ability to capture contextual information in\nuser-item interactions, while its instability in state estimation may hinder the ability to capture short-term patterns in\ninteraction sequences. To address these issues, we propose\na novel framework called SelectIve Gated MAmba for Sequential Recommendation (SIGMA). By introducing the Partially Flipped Mamba (PF-Mamba), we construct a special\nbi-directional structure to address the context modeling challenge. Then, to consolidate PF-Mamba’s performance, we\nemploy an input-dependent Dense Selective Gate (DS Gate)\nto allocate the weights of the two directions and further filter the sequential information. Moreover, for short sequence\nmodeling, we devise a Feature Extract GRU (FE-GRU) to\ncapture the short-term dependencies. Experimental results\ndemonstrate that SIGMA significantly outperforms existing\nbaselines across five real-world datasets. Our implementation code is available at https://github.com/Applied-MachineLearning-Lab/SIMGA"},"title":{"value":"SIGMA: Selective Gated Mamba for Sequential Recommendation"},"authors":{"value":["Ziwei Liu","Qidong Liu","Yejing Wang","Wanyu Wang","Pengyue Jia","Maolin Wang","Zitao Liu","Yi Chang","Xiangyu Zhao"]}},"tmdate":1753970886039,"pdate":1744300800000,"tcdate":1753970886039,"writers":["~Ziwei_Liu9","~Qidong_Liu2","~Yejing_Wang1","~Wanyu_Wang1","~Pengyue_Jia1","~Maolin_Wang1","~Zitao_Liu1","~Yi_Chang4","~Xiangyu_Zhao1"],"signatures":["~Ziwei_Liu9"],"forum":"8IGTtK0TrR","license":"CC BY 4.0","number":38282,"cdate":1753970886039,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1753970886039,"domain":"OpenReview.net/Archive","id":"8IGTtK0TrR","version":2},{"content":{"summary":{"value":"Addressing the core challenge in Zero-Shot Learning (ZSL)—the difficulty of jointly modeling local features and global correlations—this paper proposes UMZero, a hybrid framework that represents the first ZSL solution fusing a pre-trained CNN with Mamba (a State Space Model, SSM). The framework resolves limitations of existing methods through three core modules: 1) the High-order Global Aggregator (HGA), which enhances feature expressiveness by capturing high-order statistical dependencies across channels; 2) the Mamba Interaction Module (MIM), which combines the fine-grained local extraction capability of CNNs with Mamba’s long-range dependency modeling to enable cross-modal bidirectional information flow and alignment; and 3) the prototype learning module, which introduces a prototype contrastive loss to construct a semantically structured embedding space, improving intra-class compactness and inter-class discriminability.\n\nThe authors validated UMZero’s performance on three mainstream ZSL datasets (CUB, SUN, and AWA2)."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. The authors state that UMZero is the \"first CNN-Mamba fused ZSL framework,\" but Vision Mamba (Zhu et al., 2024) has already attempted to apply Mamba to visual tasks, and some works (e.g., ZeroMamba) have introduced Mamba into ZSL. We ask: 1) What are the essential differences between UMZero’s CNN-Mamba fusion strategy (e.g., feature splitting, channel shuffle) and Vision Mamba’s \"Mamba adapted for vision\"? 2) Why was \"CNN + Mamba\" chosen instead of \"ViT + Mamba\" fusion? Is there experimental or theoretical evidence to prove that the advantage of CNNs in local feature extraction provides a more significant gain for ZSL than ViTs?\n2. The \"Baseline\" in the ablation study does not clearly state whether it includes a basic semantic alignment module. If the Baseline is only a pure CNN classification model, it may underestimate the baseline performance of existing methods.\n3. How was the \"channel splitting ratio\" of the MIM module determined? Are there experiments to verify the impact of different splitting ratios (e.g., 1:1, 1:2, 2:1) on performance?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. UMZero is the first to deeply fuse the local feature extraction advantages of CNNs with the efficient global modeling capabilities of Mamba.\n2. The bidirectional cross-modal interaction mechanism (BAAM component) in the MIM module fills the gap in existing ZSL models, which \"only rely on semantics to guide visual updates and lack bidirectional visual-semantic feedback.\" This allows semantic prototypes to be dynamically optimized based on visual information, effectively bridging the representational gap between visual and semantic spaces."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Insufficient depth of innovation and ambiguous boundary of innovation in CNN-Mamba fusion. Although the authors claim UMZero is the \"first CNN-Mamba fused ZSL framework,\" they do not fully compare it with related work on \"CNN-SSM fusion in the visual domain\" (e.g., variants of Vision Mamba (Zhu et al., 2024)). The essential differences between UMZero’s fusion strategy (e.g., feature splitting, channel shuffle, bidirectional interaction) and these works are not clearly explained, which may weaken the uniqueness of its innovation.\n2. Omission of key baselines in related work. In the current ZSL field, methods based on \"bidirectional cross-modal interaction\" (e.g., CoAR-ZSL (Du et al., 2023) in 2023 and GNDAN (Chen et al., 2024a) in 2024) have also attempted to address visual-semantic alignment. However, the authors do not deeply analyze the differences in interaction mechanisms between UMZero’s BAAM component and these methods in both related work and experimental comparisons, resulting in insufficient horizontal comparison support for the innovation of \"bidirectional interaction.\"\n3. Vague explanation of core mechanisms in the MIM module. The authors mention that the CMamba branch of MIM splits features into two parts (for input to convolution and Mamba respectively) through \"channel splitting,\" but the basis for selecting the \"splitting ratio\" (e.g., 1:1 channel split) is not explained, nor is the impact of the splitting ratio on performance analyzed. Additionally, the BAAM component only provides the final update equations (Equations (7)–(10)) for \"bidirectional cross-modal attention calculation,\" without detailed derivation of the attention weight calculation process, reducing the reproducibility of the key module.\n4. Inadequate theoretical justification for the prototype contrastive loss. The authors define the prototype contrastive loss as a combination of \"prototype alignment loss + attribute contrastive loss,\" but do not theoretically explain why this combination can alleviate \"domain shift.\" For example, there is no comparison with other loss functions in the ZSL field (e.g., contrastive loss (Chen et al., 2022b), triplet loss (Wang et al., 2021)), nor is the basis for selecting the optimal ratio of λ₁ and λ₂ analyzed, leaving the design of the loss function without theoretical support.\n5. Insufficient depth of experimental design and robustness validation. When ablating \"CMamba,\" only the performance with and without CMamba is compared, and the individual contributions of the \"convolution branch\" and \"Mamba branch\" are not decomposed. This makes it impossible to clarify their specific roles in local-global modeling, weakening the persuasiveness of the ablation conclusions. The selection of hyperparameters is based on empirical choices from the results, and the authors did not explain the reasons for choosing each parameter in this manner.\n6. Many images and chapters in the text do not correspond accurately, and the writing expression and formula explanations are also insufficiently precise. For example: “In UMZero (Fig. 6), we integrate a hybrid CNN–Mamba architecture for ZSL.” “The imlementation deatils are provided in Appendix A.2.”"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942598388,"tcdate":1761548947631,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23305/Reviewer_H5bk"],"signatures":["ICLR.cc/2026/Conference/Submission23305/Reviewer_H5bk"],"forum":"UZJPcLtQkx","number":2,"license":"CC BY 4.0","cdate":1761548947631,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23305/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942598388,"domain":"ICLR.cc/2026/Conference","replyto":"UZJPcLtQkx","id":"zMxfoFDg1K","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Zero-shot learning (ZSL)；State space models；Cross-modal interactions; Prototype Contrastive"]},"supplementary_material":{"value":"/attachment/3db11224bd373e868c09ba3e0ed90d1d03585fb8.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"In zero-shot learning (ZSL), accurately recognizing unseen classes relies heavily on deeply understanding both local details and global correlations between visual and semantic modalities. While prior methods have leveraged CNNs for local feature extraction or utilized Transformers and Mamba models to capture global contextual information, these approaches often lack an integrated mechanism to jointly model both aspects. This limitation hampers their ability to fully exploit complex cross-modal interactions. To overcome this challenge, we propose UMZero, a novel hybrid ZSL framework that synergistically combines a pretrained CNN with a state space model (SSM), effectively uniting fine-grained local feature extraction with long-range dependency modeling. UMZero is composed of three core modules: the High-order Global Aggregator (HGA), the Mamba Interaction Module (MIM), and a prototype learning unit. Specifically, the HGA enhances feature expressiveness by capturing high-order statistical dependencies across channels; the MIM enables deep cross-modal fusion and alignment by jointly modeling local-global interactions and supporting bidirectional information flow between modalities; and the prototype learning module constructs a semantically structured embedding space, promoting compact intra-class and discriminative inter-class representations. Extensive evaluations on several well-established ZSL datasets confirm that UMZero surpasses current state-of-the-art methods, delivering superior performance and demonstrating robust generalization capabilities. The source code will be publicly released upon paper acceptance."},"_bibtex":{"value":"@misc{\nlei2026umzero,\ntitle={{UMZ}ero: A Unified {CNN}-Mamba Framework for Zero-Shot learning},\nauthor={Yu Lei and Junping Wang and Quanxue Gao and ShunxinCheng and Qin Li},\nyear={2026},\nurl={https://openreview.net/forum?id=UZJPcLtQkx}\n}"},"title":{"value":"UMZero: A Unified CNN-Mamba Framework for Zero-Shot learning"},"pdf":{"value":"/pdf/5a00bafead3301124fd587813fe4aba52bc85553.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"lei|umzero_a_unified_cnnmamba_framework_for_zeroshot_learning"},"authorids":{"value":["~Yu_Lei12","~Junping_Wang3","~Quanxue_Gao1","~ShunxinCheng1","~Qin_Li6"]},"authors":{"value":["Yu Lei","Junping Wang","Quanxue Gao","ShunxinCheng","Qin Li"]}},"version":2},{"content":{"summary":{"value":"The paper proposes to use invertible neural networks as non linear coordinate transformations of vector spaces R^n and to consider the linear structure within one such coordinate representation, leading to a non classical linear structure in the other and to a non-linear structure w.r.t. the standard operations on R^n. The authors thereafter consider the linear operations with respect to the deformed addition and scalar multiiplication. When seeing the results from the angle of non linear coordinates of a vector space, the results of the given Lemma are pretty straight forward, but valuable for less mathematically oriented readers. \nInterestingly, the authors can then employ their linearization approach in learning image synthesis style transfer. The FM generation paths that usually require the solution of a non linear ODE can, in the linearized representation be represented by a learned linear map that is obtained as the time integral of a linear vector field A_t v which is represented by a neural network. When the solution operator to A_t is computed once, this can be used for a essentially lossless one step generation.\nThe authors demonstrate the viability of their approach for MNIST and Celebrity Faces image generation tasks and neural style transfer. The results are decent and show the potential of the author's point of view."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- Can you give a detailed and reproducible account on how the coordinate maps g are obtained in the respective application context?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":4},"strengths":{"value":"* the paper provides a really original idea to consider invertible neural networks in vector spaces and then build machine learning tasks on linear methods in the constructed latent space\n* The authors show that this approach can be combined with generative learning tasks in the image domain, in particular that the time integration of the FM vector field can be effectively represented by a linear solution operator obtained as the time integration of (time dependent) linear vector fields.\n* The theoretical documentation is rather complete and easy to follow"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The core idea of the paper - looking at vector spaces in non linear coordinates - could be presented somewhat clearer. Many results given in the lemma would then just be clear and in parts superfluous (like Lemma 4). Also all the linear algebra checked in detail would follow from one structural argument. \n- Better visualizations would be really helpful for a better representation of the main idea. \n- For me the missing part is a detailed investigation of the interaction of the chosen 'coordinate maps' g with the learning task. Obviously not any g would do (e.g. not the identity or linear g's).\n- Related to this, I would appreciate a much more extensive analysis of the training details, especially with regard to the previous point.\n- Also the interaction between the learning task and the expressivity of the linearizers would be of interest. Is the reproduction of the FM time integration with a linear flow compatible with other approaches that do not have as 'stiff' vector fields as FM - take e.g. a likelihood based trained neuralODE. Similar questions arise for the style transfer task.\n- The title, due to its generality, somehow promises more than is kept by the rather specific application domain.\n- The authors address flow matching as diffusion. As an ODE based method it is not a diffusion process, unlike SDE based DDPM. In a mathematically based paper, such inaccuracies should be avoided."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917302048,"tcdate":1761992607816,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4332/Reviewer_ojRH"],"signatures":["ICLR.cc/2026/Conference/Submission4332/Reviewer_ojRH"],"forum":"RcxM3e59ga","number":4,"license":"CC BY 4.0","cdate":1761992607816,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4332/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917302048,"domain":"ICLR.cc/2026/Conference","replyto":"RcxM3e59ga","id":"3t2IRdDLF7","forumContent":{"TLDR":{"value":"We define vector spaces where networks are linear, then apply linear algebra to tasks like one-step diffusion."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Linear","flow","idempotent","diffeomorphism","vector-space"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, $f$$:$$\\mathcal{X}$$\\to$$\\mathcal{Y}$. Is it possible to identify a pair of non-standard vector spaces for which a conventionally nonlinear function is, in fact, linear? This paper introduces a method that makes such vector spaces explicit by construction. We find that if we sandwich a linear operator $A$ between two invertible neural networks, $f(x)=g_y^{-1}(A g_x(x))$, then the corresponding vector spaces $\\mathcal{X}$ and $\\mathcal{Y}$ are induced by newly defined addition and scaling actions derived from $g_x$ and $g_y$. We term this kind of architecture a Linearizer. This framework makes the entire arsenal of linear algebra, including SVD, pseudo-inverse, orthogonal projection and more, applicable to nonlinear mappings. Furthermore, we show that the composition of two Linearizers that share a neural network is also a Linearizer. We leverage this property and demonstrate that training diffusion models using our architecture makes the hundreds of sampling steps collapse into a single step. We further utilize our framework to enforce \nidempotency (i.e. $f(f(x))=f(x)$) on networks leading to a globally projective generative model and to demonstrate modular style transfer."},"_bibtex":{"value":"@misc{\nberman2026who,\ntitle={Who Said Neural Networks Aren't Linear?},\nauthor={Nimrod Berman and Assaf Hallak and Assaf Shocher},\nyear={2026},\nurl={https://openreview.net/forum?id=RcxM3e59ga}\n}"},"title":{"value":"Who Said Neural Networks Aren't Linear?"},"pdf":{"value":"/pdf/cddd05a450a90366f9938889e633ffe6621ae3ee.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"berman|who_said_neural_networks_arent_linear"},"authorids":{"value":["~Nimrod_Berman1","~Assaf_Hallak1","~Assaf_Shocher1"]},"authors":{"value":["Nimrod Berman","Assaf Hallak","Assaf Shocher"]}},"version":2},{"content":{"summary":{"value":"The paper proposes Time Filter (TiF), combining a new Fourier Neural Filter (FNF) backbone with a Dual Branch Decoupler (DBD) architecture. FNF extends FNO with an input-dependent kernel (selective activation), complex transform, and adaptive truncation to modulate time–frequency information; DBD uses parallel temporal/spatial branches to improve gradient flow and capacity from an information-bottleneck perspective. Experiments on 12 benchmarks (eight LTSF datasets plus four PeMS short-term sets) report strong results, along with ablations and an efficiency plot."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See in weakness."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. FNF formalizes an input-dependent spectral operator with selective activation and adaptive truncation; the math exposition (Definitions. 1–7, Remarks 1–6) is explicit and links capabilities to Transformer functions and complexity.\n\n2. DBD’s motivation via the information bottleneck and gradient-path analysis is well argued.\n\n3. Results span LTSF and PeMS with comparisons to Transformer/CNN/MLP/Fourier baselines; Table 2 claims lookback grid search (96–720) for all methods, which ensures a fair comparison.\n\n4. Efficiency analysis (Traffic) and component ablations (AT/SA, LS/GS) give some insight."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. FNF’s contributions (input-dependent kernel, selective activation, adaptive truncation) are close in spirit to prior spectral/fractional operators (e.g., FNO/AFNO/FITS/FreMLP). The paper does not clearly establish what capability FNF enables that prior spectral blocks cannot, beyond architectural composition; DBD overlaps with known parallel/dual-path decouplers (e.g., Leddam [1], xPatch [2], MTGNN [3], TimeMixer++ [4], just to name a few). A sharper comparison or operator-replacement study is needed.\n\n2. There is no working anonymous repo or pseudo code for review, which limits the reproducibility of the proposed method.\n\n3. For PeMS, the lookback is fixed to 96 for all baselines, which can bias results; fair practice tunes input length per method (as you already did for Table 2). Please align protocols across tasks.\n\n4. Table 2 lacks more recent strong baselines (e.g., TimeMixer++ [4] (ICLR25), PatchMLP [5] (AAAI25), TQNet [6], and TimeBridge [7] (ICML25)), which undermines the SOTA claim; please add with identical splits/tuning.\n\n5. The proposed FNF looks conceptually overlapped with selective state-space models such as Mamba: both realize an input-conditioned long filter with a gated residual/skip path. In FNF, the frequency-domain parameterization (F → complex transform → adaptive truncation → F⁻¹) effectively implements a learnable long convolution; Mamba parameterizes a similar operation via SSM kernels and a selective gate. The manuscript should explicitly position FNF against Mamba/S4-family—clarifying what FNF does that a selective SSM cannot—and include quantitative comparisons.\n\n*[1] Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling.*\n\n*[2] xPatch: Dual-Stream Time Series Forecasting with Exponential Seasonal-Trend Decomposition.*\n\n*[3] Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks.*\n\n*[4] TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis.*\n\n*[5] Unlocking the Power of Patch: Patch-Based MLP for Long-Term Time Series Forecasting.*\n\n*[6] Temporal Query Network for Efficient Multivariate Time Series Forecasting.*\n\n*[7] TimeBridge: Non-Stationarity Matters for Long-term Time Series Forecasting.*"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918424655,"tcdate":1760600456471,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6038/Reviewer_oAZB"],"signatures":["ICLR.cc/2026/Conference/Submission6038/Reviewer_oAZB"],"forum":"LhFnafBlv5","number":1,"license":"CC BY 4.0","cdate":1760600456471,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6038/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918424655,"domain":"ICLR.cc/2026/Conference","replyto":"LhFnafBlv5","id":"XVP3u1iOZ2","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["time series forecasting"]},"supplementary_material":{"value":"/attachment/b48a29d6aeeb7c8745bb870c88c9570b750e56e9.zip"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Multivariate time series forecasting has been suffering from the challenge of capturing both temporal dependencies within variables and spatial correlations across variables simultaneously. Current approaches predominantly repurpose backbones from Natural Language Processing (NLP) or Computer Vision (CV) (e.g., Transformers), which fail to adequately address the unique properties of time series (e.g., periodicity and fluctuation). The research community lacks dedicated backbones incorporating temporal-specific inductive biases, depending on domain-agnostic backbones supplemented with auxiliary techniques (e.g., signal decomposition). We introduce Fourier Neural Filter (FNF) as the backbone and Dual Branch Decoupler (DBD) as the architecture to provide exceptional learning capabilities and optimal learning pathways for spatiotemporal modeling, respectively. Our theoretical analysis proves that FNF integrates time-domain and frequency-domain analysis while enabling adaptive truncation of noise components within a unified backbone that extends naturally to spatial modeling. Through the lens of information bottleneck theory, we reveal that DBD delivers superior gradient flow and representation capacity, enabling it to effectively capture local and global, temporal and spatial information comprehensively. Our empirical evaluation on 12 public benchmark datasets, encompassing both multivariate long-term and short-term forecasting tasks, demonstrates state-of-the-art performance compared to existing advanced baseline models. Notably, our approach achieves these results without any auxiliary techniques, suggesting that properly designed neural architectures can capture the inherent properties of time series, potentially transforming time series modeling in scientific and industrial applications."},"_bibtex":{"value":"@misc{\nxu2026multivariate,\ntitle={Multivariate Time Series Forecasting with Fourier Neural Filter},\nauthor={Chenheng Xu and Dan Wu and Yixin Zhu and Ying Nian Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=LhFnafBlv5}\n}"},"title":{"value":"Multivariate Time Series Forecasting with Fourier Neural Filter"},"pdf":{"value":"/pdf/92a1c6f3d97f0a96813a9301610928f024b214ee.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xu|multivariate_time_series_forecasting_with_fourier_neural_filter"},"authorids":{"value":["~Chenheng_Xu1","~Dan_Wu7","~Yixin_Zhu1","~Ying_Nian_Wu1"]},"authors":{"value":["Chenheng Xu","Dan Wu","Yixin Zhu","Ying Nian Wu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes the Error-Correcting Code Mamba-Transformer (ECCM), a hybrid neural decoder architecture for linear block codes. The design alternates between Mamba (State-Space Model) layers and standard Transformer attention layers, aiming to leverage the linear-time complexity of Mamba for sequential modeling while retaining the global contextual modeling capabilities of the Transformer.\n\nThe authors claim three primary contributions: 1) The **hybrid Mamba-Transformer architecture** itself, 2) a new **layer-wise mask for Mamba layers,** **$f(H)$,** derived from parity check matrix, 3) a **progressive layer-wise loss** **function** combined with a syndrome-based early-exit mechanism to stabilize training and reduce inference latency.\n\nExperiments are conducted on BCH, Polar, and LDPC codes. The results show that the proposed ECCM model achieves performance that is on par with, or in some cases (notably BCH) superior to, existing Transformer-only decoders, while offering improved computational complexity."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. **Runtime evidence:** Were empirical measurements (e.g., samples/sec, latency, GPU memory) taken to support the “*low and improving inference speed*” claim? If so, please include them or provide a runtime benchmark appendix.\n2. **Clarification on the SCL Baseline for Polar Code Comparison:** In Appendix 12.2 and Table 4, the paper compares ECCM and CrossMPT against a classical \"SCL (L=32)\" baseline. Based on this, the authors conclude that ECCM \"meaningfully closes the distance to the strong SCL baseline\". However, the baseline is not specified as being CRC-Aided (CA-SCL), which is the standard high-performance SCL decoder configuration in modern literature. Could the authors clarify whether the \"SCL (L=32)\" baseline used in Table 4 includes a CRC? If it does not, how can the claim of closing the gap against a \"strong SCL baseline\" be justified, given that a ‘plain’ SCL is a significantly weaker baseline than the standard CA-SCL?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":3},"strengths":{"value":"1. **Novel and Timely Architecture:** The integration of Mamba, a recent and efficient SSM, into the ECC decoding problem is a new architecture for model-free channel decoders. The hybrid approach achieves strong decoding performance. \n2. **Strong Empirical Performance:** The model achieves highly competitive, and in the case of several BCH codes, state-of-the-art performance compared to strong Transformer-based baselines like AECCT and CrossMPT. This validates the practical efficacy of the paper’s approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Inconsistencies, Typos, and Presentation Issues:** The paper suffers from many typographical errors and presentation issues that reduce clarity and undermine its professional quality.\n    - **Formulation and Notation Errors**\n        - Eq (30) & (31): The calculation of error vector $z$ and the estimated codeword $\\hat{c}^{i}$ should reference the original received vector $y[]$, not the model input $y_{in}[]$ since $y_{in}=[|y|, s]$.\n        - Eq (28): The output dimension $N$ of $o^{i}$ should be $n$, given its relationship to the error vector $z$.\n        - Eq (4): The notation uses a semicolon ($;$), which conventionally denotes vertical concatenation, whereas the operation required by the stated dimensions is horizontal concatenation.\n        - Eq (31): The iterator should use a different variable than $n$ (e.g., $m$), and the upper value should be $n$ (code length).\n        - Eq (28) and (29): The paper used the following rule for hard-decision vector: positive y → 0 (binary) and negative y → 1 (on page 3). Because of this rule, it seems that (28) should be $s^i = H \\cdot (o^i < 0.5)$ instead of $s^i = H \\cdot (o^i > 0.5)$.\n    - **Figure-Text Inconsistency**\n        \n        In Figure 1(b), the Mamba block diagram is misleading. It correctly shows a SiLU activation for the $z$ path, but omits the SiLU activation for the $u$ path after the Conv block. This contradicts the text, as Eq (7) explicitly states $u_{conv}^{i}=SiLU(Conv^{i}(u^{i}))$.\n        \n    - **Poor Presentation**\n    In Figure 1(b), the final output label ($y^i$) is poorly rendered and overlaps with the output arrow, demonstrating a lack of attention to detail and reducing the diagram's readability.\nDefinition discrepancy in Mamba mask $f(H)$: The mathematical formulation in Eq (13) and (14) is misleading. The paper’s Mamba mask $f(H)$ is defined with dimensions $(n-k) \\times (2n-k)$, yet the paper applies it using indices $d $ and $s $ (model/state dimensions), as seen in $f(H)[l, d]$ and $f(H)[l, s]$. This indexing is dimensionally inconsistent with the mask's definition.\n3. **Lack of empirical runtime metrics:** The conclusion states that ECCM achieves *“low and improving inference speed”*. However, the main text provides only the big-O complexity (Section 7.2) without any empirical latency, throughput, or FLOPs."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915979564,"tcdate":1761959371756,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1978/Reviewer_v6DD"],"signatures":["ICLR.cc/2026/Conference/Submission1978/Reviewer_v6DD"],"forum":"BlKB91pkCd","number":3,"license":"CC BY 4.0","cdate":1761959371756,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1978/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915979564,"domain":"ICLR.cc/2026/Conference","replyto":"BlKB91pkCd","id":"6xVxua7IPM","forumContent":{"TLDR":{"value":"A Mamba–Transformer hybrid error correcting codes decoder"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Error-Correcting Codes","Mamba"]},"supplementary_material":{"value":"/attachment/79782159142db6be93456ee89de2c7c8ab69d7fd.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"We introduce a novel deep learning method for decoding error correction codes based on the Mamba architecture, enhanced with Transformer layers. Our approach proposes a hybrid decoder that leverages Mamba’s efficient sequential modeling while maintaining the global context capabilities of Transformers. To further improve performance, we design a novel layer-wise masking strategy applied to each Mamba layer, allowing selective attention to relevant code features at different depths. Additionally, we introduce a progressive layer-wise loss, supervising the network at intermediate stages and promoting robust feature extraction throughout the decoding process. Comprehensive experiments across a range of linear codes demonstrate that our method significantly outperforms Transformer-only decoders and standard Mamba models."},"_bibtex":{"value":"@misc{\ncohen2026hybrid,\ntitle={Hybrid Mamba{\\textendash}Transformer Decoder for Error-Correcting Codes},\nauthor={Shy-El Cohen and Yoni Choukroun and Eliya Nachmani},\nyear={2026},\nurl={https://openreview.net/forum?id=BlKB91pkCd}\n}"},"title":{"value":"Hybrid Mamba–Transformer Decoder for Error-Correcting Codes"},"pdf":{"value":"/pdf/e4d1c6cde1b5afdf339765037e957bdd3c51da3f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cohen|hybrid_mambatransformer_decoder_for_errorcorrecting_codes"},"authorids":{"value":["~Shy-El_Cohen1","~Yoni_Choukroun1","~Eliya_Nachmani1"]},"authors":{"value":["Shy-El Cohen","Yoni Choukroun","Eliya Nachmani"]}},"version":2},{"content":{"summary":{"value":"The paper identifies a failure mode of many kernelized/linear-attention variants namely “global context collapse,” where compressing all keys/values into a single global summary limits rank and drives attention toward near-uniform distributions as sequence length grow . \n\nAutores proposes Multi-Head Linear Attention (MHLA), which partitions tokens into non-overlapping blocks and learns a block-wise mixing of local KV summaries via a nonnegative, normalized coefficient matrix, restoring query-conditioned diversity while keeping linear-time scaling . The authors derive rank bounds showing MHLA can achieve substantially higher attainable rank than global linear attention and empirically observe lower entropy (sharper sparsity) of attention maps ."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"How are the mixing coefficients parameterized to ensure nonnegativity and row normalization in practice (softmax per row, or other), and how sensitive are results to this choice and to the locality-biased initialization? What values of M were used per model and task, and how does wall-clock time and peak memory scale with N and M on H100 across DeiT/DiT/NLP? Please include profiles showing the point where O(M²d²) becomes non-negligible .\n\n\nAlso, can you provide multi-seed runs (≥3) with mean±std for ImageNet accuracy and DiT FID/IS/sFID, including sample counts and evaluation details?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- Clear diagnosis of “global context collapse” with complementary theoretical indicators, e.g. rank upper bound ≤d for global linear attention; MHLA’s additive blockwise rank potential;  and empirical entropy analyses .\n\n\n- Simple, hardware-friendly construction: blockwise summaries + learned nonnegative mixing; retains linear-time leading term and is compatible with chunkwise/streaming execution .\n\n\n- Strong cross-domain results: (i) ImageNet improvements over linear attention baselines and competitive with transformer/mamba-style models, (ii) sizable FID gains across DiT scales (including XL) with near-linear-attention throughput, and (iii) modest yet positive NLP benchmarks without positional embeddings .\n\n\n- Useful ablations: initialization vs learnable mixing and head-number sensitivity give practical guidance (e.g., M=16 works well) ."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Results are mostly single-number comparisons; there is no reporting of multiple seeds, confidence intervals, or significance tests. For diffusion FID/IS/sFID and ImageNet top-1, please consider report mean±std over ≥3 seeds and specify sample counts and evaluation protocols used for FID/IS (e.g., 50k samples, classifier, resize method) to support the claims of consistent improvements .\n\n\n- Language modeling evaluation is narrow. The 0.3–0.34B model trained on 5B tokens is assessed on a small suite (ARC-c, Wino, CoPA). Please add perplexity on standard corpora and long-context stress tests to directly evaluate the purported benefits at long sequence lengths, and compare to strong linear/SSM baselines under matched budgets .\n\n\n- Complexity & memory overhead clarity. The method introduces an M×M mixing step with O(M²d²) cost. While argued negligible when M²≤N, the paper would benefit from explicit profiling of wall-clock time and peak memory versus N and M across hardware and tasks, and from reporting the realized M values used in each experiment (classification/generation/NLP) .\n\n\n- Streaming/causal claims are not empirically demonstrated. The text asserts compatibility with chunkwise parallel training and streaming/stateful execution; please provide experiments demonstrating long-context streaming inference throughput/latency versus linear attention baselines under causal masking .\n\n\n- Comparisons breadth in T2I/C2I. The DiT/DiG comparisons are strong, but additional comparisons to other recent linear or hybrid attention mechanisms in diffusion backbones would strengthen the case, especially where prior methods also raise rank or sparsify attention (the paper cites some but not all are evaluated directly) .\n\nAdditionally, there are minor typos (e.g., “diversty”, “Iamge”)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915745690,"tcdate":1761824580144,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1352/Reviewer_uTES"],"signatures":["ICLR.cc/2026/Conference/Submission1352/Reviewer_uTES"],"forum":"340QjF3jJP","number":2,"license":"CC BY 4.0","cdate":1761824580144,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1352/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915745690,"domain":"ICLR.cc/2026/Conference","replyto":"340QjF3jJP","id":"OoNFH1nojT","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"Multi-Head Linear Attention addresses the performance degradation in linear attention by preserving representational diversity through head-wise token dimension computation."},"keywords":{"value":["Linear Attention","Model Architecture","Efficiency"]},"primary_area":{"value":"generative models"},"abstract":{"value":"While the Transformer architecture dominates many fields, its quadratic self-attention complexity hinders its use in large-scale applications. **Linear attention** offers an efficient alternative, but its direct application often degrades performance, with existing fixes typically re-introducing computational overhead through extra modules (e.g., depthwise separable convolution and few self-attention blocks) that defeat the original purpose. In this work, we identify a key failure mode in these methods: **global context collapse**, where the model loses representational diversity. To address this, we propose **Multi-Head Linear Attention (MHLA)**, which preserves this diversity by computing attention within divided heads along the token dimension. We prove that MHLA maintains linear complexity while recovering much of the expressive power of softmax attention, and verify its effectiveness across multiple domains, achieving a **3.6%** improvement on ImageNet classification, a **6.3%** gain on NLP, a **12.6%** improvement in image generation tasks and a **41%** enhancement in video generation tasks with the same computational complexity."},"_bibtex":{"value":"@inproceedings{\nzhang2026mhla,\ntitle={{MHLA}: Restoring Expressivity of Linear Attention via Token-Level Multi-Head},\nauthor={Kewei Zhang and Ye Huang and Yufan Deng and Jincheng YU and Junsong Chen and Huan Ling and Enze Xie and Daquan Zhou},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=340QjF3jJP}\n}"},"title":{"value":"MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head"},"pdf":{"value":"/pdf/cd6bb3047f1271e4adc285113e13ed1cfccb2109.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|mhla_restoring_expressivity_of_linear_attention_via_tokenlevel_multihead"},"authorids":{"value":["~Kewei_Zhang2","~Ye_Huang4","~Yufan_Deng4","~Jincheng_YU1","~Junsong_Chen1","~Huan_Ling1","~Enze_Xie1","~Daquan_Zhou1"]},"authors":{"value":["Kewei Zhang","Ye Huang","Yufan Deng","Jincheng YU","Junsong Chen","Huan Ling","Enze Xie","Daquan Zhou"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Baleen, a new state space model that combines information bottleneck principles with stochastic variables to improve interpretability and robustness in SSMs. The authors propose two variants: Baleen-B, which models the state transition matrix $A_t$ as a Bernoulli RV, and Baleen-E, which models the discretization parameter $\\Delta_t$ as an exponential RV. Both variants include analytical KL divergence terms (or bounds) as regularizers. \nExperiments demonstrate improvements over Mamba-2 on language benchmarks using 370M trained from scratch on 20B tokens, and 7B finetuned on 0.5B tokens. Baleen models also show better adversarial robustness, and self-interpretation capabilities derived from the transition matrix $A_t$."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"- What is the Crystal dataset used for 7B instruction tuning? I could not find further information about it.\n\n- For the Exponential case, what is the exact gradient estimator used?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"Things that I liked in this paper include:\n\n- Casting SSMs as an information-bottlenecked sequence model (Eq. 5) is a clean way to show that MLE-trained Mamba can suffer with spurious context and is sensitive to perturbations. I think the paper connects Mamba to the special case (\\beta = 0) and builds from there. This provides a principled framework for understanding the trade-off between retention and compression in linear recurrent models.\n\n- The Exponential-gated timestep (indirectly on both (A_t) and (B_t)) is well-motivated and look less ad-hoc.\n\n- I think the adversarial robustness setting is an interesting angle to SSMs, and Baleen seems to handle that well. For example, the position bias attack experiments (Table 3) show convincing improvements, with Baleen-E maintaining a higher accuracy than Mamba-2. The prompt attack experiments also demonstrate consistent robustness improvements.\n\n- The method is easy to implement and cause re-use existing efficient kernels. This makes Baleen a easy drop-in improvement for Mamba and also other linear recurrent models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- I think that the claim that Mamba \"just does MLE\" is an overstatement. That is, the argument that Mamba = SSIB with $\\beta = 0$ is  true only for an idealized training loss, without regularization. In practice, large-scale Mamba training uses regularization (dropout, weight decay, data noise, possibly distillation) that already limits I(H;X).\n\n- For me, the Bernoulli parameterization of $A_t$ is not fully convincing. The paper says \"it is natural to model $A_t$ as Bernoulli\" because each entry decides keep/drop. But in Mamba-2 the reason $A_t \\in (0,1)^{N \\times D}$ is the specific discretization, which leads to entries being viewed as *decay factors*, not probabilities. Turning *every* entry into an i.i.d. Bernoulli with parameter $p_t$ is very a strong modeling assumption (e.g., it allows exactly zeros which implies full forgetting about the context). \n\n- For Bernoulli the paper says that Gumbel–Softmax (Binary Concrete) is used, which is fine. For the Exponential variant there is no information about the differentiability of the estimator.\n\n- The \"importance\" score simply averages $A_t$ over both the state dimension ($N$) and the feature dimension ($D$). That is exactly the kind of averaging MambaAttention/MambaLRP later criticized as washing out channel-specific effects. \n\n- For interpretability, the evaluation would be stronger if the paper includes other SSM-specific methods, such as MambaAttention [1], MambaLRP [2], and LATIM [3]. The current baselines are somewhat weak and outdated. \n\n- On 370M / 20B-token pretraining, the improvements over Mamba-2 are small and vary across tasks. On datasets like MMLU and WinoGrande at that scale the noise is high, so it is hard to attribute the gains to the IB formulation. The 7B finetuning comparison is also slightly unfair because Baleen is finetuned on the extra 0.5B Crystal data but the baseline Mamba-2 is not.\n\n[1] https://arxiv.org/abs/2403.01590 \n[2] https://arxiv.org/abs/2406.07592\n[3] https://aclanthology.org/2025.acl-long.1194/"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925364256,"tcdate":1761968016228,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15041/Reviewer_BgmA"],"signatures":["ICLR.cc/2026/Conference/Submission15041/Reviewer_BgmA"],"forum":"976tPua6Hd","number":3,"license":"CC BY 4.0","cdate":1761968016228,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15041/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925364256,"domain":"ICLR.cc/2026/Conference","replyto":"976tPua6Hd","id":"5ymOgDRFR4","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Interpretation","Robustness","Large Language Models","State-space models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"We introduce \\textit{\\textbf{Baleen}}, a family of state space models that unifies \\textbf{stochastic selection} with \\textbf{information bottleneck} to build interpretable and robust long‑context learners. Unlike Mamba/Mamba2’s deterministic gates, Baleen treats selection as a random variable and regularizes it with a closed‑form KL to a sparsity prior: (i) \\textit{\\textbf{\\bib}} samples Bernoulli state‑transition gates; (ii) \\textit{\\textbf{\\gib}} samples Exponential  time‑intervals. This yields an explicit trade‑off between retention and compression and exposes token‑level selection heatmaps at inference for self‑interpretation. On language benchmarks, \\ib  improves average accuracy over Mamba2 by +0.95 at 370M pretraining and +1.38 at 7B finetuning. Baleen delivers stronger robustness to localized perturbations and adversarial attacks: under CIFAR‑10 sequence perturbation, prefix damage falls to 0.6\\% vs 26.5\\% for Mamba2 (average under attacks 0.542 vs 0.385). Finally, Baleen’s self‑interpretations outperform IG/Grad‑CAM on average fidelity across four text classification tasks. We will release our Baleen‑7B models on Hugging Face with code, checkpoints, and an interactive selection‑heatmap demo."},"_bibtex":{"value":"@misc{\nzhu2026baleen,\ntitle={Baleen: Self\\nobreakdash-Interpretable, Robust {SSM}s with Stochastic Selective Memory},\nauthor={Jiajun Zhu and Yuehao Wang and Peihao Wang and Ruisi Cai and Siqi Miao and Liliang Ren and Pan Li and Jason D. Lee and Zhangyang Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=976tPua6Hd}\n}"},"title":{"value":"Baleen: Self‑Interpretable, Robust SSMs with Stochastic Selective Memory"},"pdf":{"value":"/pdf/651f6e0f7dafa731cf8adeb1471c685a32749a87.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhu|baleen_selfinterpretable_robust_ssms_with_stochastic_selective_memory"},"authorids":{"value":["~Jiajun_Zhu1","~Yuehao_Wang1","~Peihao_Wang1","~Ruisi_Cai1","~Siqi_Miao1","~Liliang_Ren1","~Pan_Li2","~Jason_D._Lee1","~Zhangyang_Wang1"]},"authors":{"value":["Jiajun Zhu","Yuehao Wang","Peihao Wang","Ruisi Cai","Siqi Miao","Liliang Ren","Pan Li","Jason D. Lee","Zhangyang Wang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2505.09205v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"zhang|hmamba_hyperbolic_mamba_for_sequential_recommendation"},"authorids":{"value":["~Qianru_Zhang2","https://dblp.org/search/pid/api?q=author:Honggang_Wen:","~Wei_Yuan4","https://dblp.org/search/pid/api?q=author:Crystal_Chen:","https://dblp.org/search/pid/api?q=author:Menglin_Yang_0005:","https://dblp.org/search/pid/api?q=author:Siu-Ming_Yiu:","~Hongzhi_Yin2"]},"html":{"value":"https://doi.org/10.48550/arXiv.2505.09205"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2505-09205,\n  publtype={informal},\n  author={Qianru Zhang and Honggang Wen and Wei Yuan and Crystal Chen and Menglin Yang and Siu-Ming Yiu and Hongzhi Yin},\n  title={HMamba: Hyperbolic Mamba for Sequential Recommendation},\n  year={2025},\n  month={May},\n  cdate={1746057600000},\n  journal={CoRR},\n  volume={abs/2505.09205},\n  url={https://doi.org/10.48550/arXiv.2505.09205}\n}\n"},"abstract":{"value":"Sequential recommendation systems have become a cornerstone of personalized services, adept at modeling the temporal evolution of user preferences by capturing dynamic interaction sequences. Existing approaches predominantly rely on traditional models, including RNNs and Transformers. Despite their success in local pattern recognition, Transformer-based methods suffer from quadratic computational complexity and a tendency toward superficial attention patterns, limiting their ability to infer enduring preference hierarchies in sequential recommendation data. Recent advances in Mamba-based sequential models introduce linear-time efficiency but remain constrained by Euclidean geometry, failing to leverage the intrinsic hyperbolic structure of recommendation data. To bridge this gap, we propose Hyperbolic Mamba, a novel architecture that unifies the efficiency of Mamba's selective state space mechanism with hyperbolic geometry's hierarchical representational power. Our framework introduces (1) a hyperbolic selective state space that maintains curvature-aware sequence modeling and (2) stabilized Riemannian operations to enable scalable training. Experiments across four benchmarks demonstrate that Hyperbolic Mamba achieves 3-11% improvement while retaining Mamba's linear-time efficiency, enabling real-world deployment. This work establishes a new paradigm for efficient, hierarchy-aware sequential modeling."},"title":{"value":"HMamba: Hyperbolic Mamba for Sequential Recommendation"},"authors":{"value":["Qianru Zhang","Honggang Wen","Wei Yuan","Crystal Chen","Menglin Yang","Siu-Ming Yiu","Hongzhi Yin"]}},"tmdate":1771387711889,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2505-09205"],"tcdate":1768178465764,"writers":["~"],"signatures":["~Hongzhi_Yin2"],"forum":"TwJGJ5qsNN","license":"CC BY-SA 4.0","number":743379,"cdate":1746057600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1771387711889,"domain":"DBLP.org","id":"TwJGJ5qsNN","version":2},{"content":{"venue":{"value":"IJCNN 2026"},"pdf":{"value":"/pdf/c86639bf7b9d984cfe24b2b29f050a7f130c252b.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"yue|adaptive_state_space_experts_dynamic_expert_selection_with_stateaware_routing_for_efficient_sequence_modeling"},"authorids":{"value":["~Guo_Yue1"]},"abstract":{"value":"State space models (SSMs) provide a principled route to linear-time sequence modeling and have recently reached competitive accuracy via selective, input-dependent dynamics. In parallel, mixture-of-experts (MoE) scaling decouples model capacity from per-token compute through sparse routing. Yet, existing SSM+MoE designs largely reuse token-only routers, ignoring the fact that an SSM already maintains a learned summary of the prefix in its hidden state. We introduce Adaptive State Space Experts (ASSE), a framework that couples SSM dynamics and expert selection through (i) state-conditioned routing that conditions expert choice on a compact summary of the running state, (ii) adaptive state retention that lets experts learn distinct temporal horizons, and (iii) a convergence-based early-exit signal derived from state dynamics for input-adaptive depth. On language modeling, ASSE achieves 13.04 perplexity on the Pile benchmark, outperforming both SSM+MoE baselines (13.38) and hybrid Transformer-SSM architectures (13.21), while trading merely 0.06 PPL for 30% compute reduction via early exit. On the Long Range Arena, ASSE achieves 83.01% average accuracy, outperforming Mamba (82.14%) and other SSM baselines. Analysis reveals that experts naturally specialize into distinct temporal scales, with routing patterns showing 73% contextual coherence compared to 58% for token-only baselines. Our results demonstrate that tight integration between SSM state dynamics and MoE routing yields substantial gains in both quality and efficiency."},"title":{"value":"Adaptive State Space Experts: Dynamic Expert Selection with State-Aware Routing for Efficient Sequence Modeling"},"authors":{"value":["Guo Yue"]}},"tmdate":1778227722209,"pdate":1785168000000,"tcdate":1778227722209,"writers":["~Guo_Yue1"],"signatures":["~Guo_Yue1"],"forum":"BEsV5QXEP4","license":"CC BY 4.0","number":50206,"cdate":1778227722209,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1778227722209,"domain":"OpenReview.net/Archive","id":"BEsV5QXEP4","version":2},{"content":{"summary":{"value":"This manuscript introduces Geo-Mamba, a geometric state-space model designed to address the misspecification of conventional Euclidean sequence models when analyzing functional connectivity (FC) data from fMRI, which inherently resides on the SPD manifold. Geo-Mamba employs a dual-path selective state-space design, combining hierarchical modeling with geometry-aware dimensionality reduction, and fusing their outputs via a tailored GeoMix operator. The model was evaluated on six public fMRI datasets against several baseline methods."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"- What is the rationale for sharing the query (Q) across all spatial scales?\n- Clarify the necessity and role of the self-attention layer positioned between the two SSM blocks, as the original Mamba architecture typically omits self-attention. Include ablation studies demonstrating the impact of removing either the self-attention or the SSM blocks.\n- After model fitting, analyze the learned weight alpha in the GeoMix layer to determine the relative importance of path 1 versus path 2.\n- The description of the geometry preserving core SSM component (lines 210 to 220), needs significant rephrasing and expansion for clarity.\n- There is an incorrect reference at line 330 (Adam et al. 2014)."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- Originality: The method introduces an original combination of hierarchical pyramid multi-spatial scale FC aggregates and stacks them to form a sequence fed to a state space model, which preserves geometry.\n\n- Evaluation: The authors evaluate their approach across several public datasets, encompassing both small and large scales, and compare it to several relevant baselines.\n\n- Neuroscientific Interpretation: The interpretation of the results - although limited to a subpath of the entire architecture - established meaningful connections to existing neuroscientific understanding."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Interpretation analysis was limited to path #2, omitting coverage of the SSM path.\n- The title and introduction emphasize SSMs, yet the proposed method incorporates multiple paths and components in addition to the proposed geometric SSM, making it difficult to isolate the source of performance gains.\n- The ablation analysis lacks clarity. It would be beneficial to focus on a single key metric (e.g., accuracy) and move others to the appendix. Obvious ablation candidates include removing path #1 and/or path #2 entirely. Another relevant candidate is ablating the self-attention block within path #1.\n- Table 4 (appendix) indicates the use of different hyperparameters and multiple learning rates per dataset. The authors should clarify whether they tuned the learning rate of Geo-Mamba for each dataset, how they selected the parameters, and whether the same approach was applied to the baseline methods."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922739992,"tcdate":1762724390603,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11691/Reviewer_cQhP"],"signatures":["ICLR.cc/2026/Conference/Submission11691/Reviewer_cQhP"],"forum":"7XonwHRkTS","number":5,"license":"CC BY 4.0","cdate":1762724390603,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11691/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922739992,"domain":"ICLR.cc/2026/Conference","replyto":"7XonwHRkTS","id":"p9gz99afaW","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Brain functional connectivity","Riemannian geometry","State space model","Geometric deep learning"]},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Functional magnetic resonance imaging (fMRI)–derived functional connectivity (FC) is represented as graphs and as correlation/covariance matrices that live on non-Euclidean spaces—cortical graphs and the Riemannian manifold of symmetric positive-definite (SPD) matrices—so conventional Euclidean sequence models are misspecified. To this end, we introduce *Geo-Mamba*, a geometric variant of Mamba formulated on Riemannian manifolds. *Geo-Mamba* employs a dual-path selective state-space design: *a stacked path* performs hierarchical spatial modeling by aggregating pyramid multi-scale features to capture local and global dependencies, while *a embedding path* combats redundancy in high-dimensional SPD inputs via progressive, geometry-aware dimensionality reduction (operating in the appropriate manifold spaces) to produce compact states without violating Riemannian constraints. Their complementary outputs are fused through the tailored *GeoMix* operator to yield a compact, discriminative SPD representation. *Geo-Mamba* is evaluated on six public fMRI datasets—ADNI, OASIS, PPMI, Taowu, Neurocon, and Mātai—spanning Alzheimer’s and Parkinson’s cohorts as well as multi-site normative populations with diverse acquisition protocols. Across these benchmarks, it delivers consistently competitive accuracy and robustness, supporting the value of dual-path manifold modeling for neuroimaging and its potential for clinical translation."},"_bibtex":{"value":"@misc{\ncao2026geomamba,\ntitle={Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics},\nauthor={Yuwei Cao and Tingting Dan and Yang Yang and Guorong Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=7XonwHRkTS}\n}"},"title":{"value":"Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics"},"pdf":{"value":"/pdf/1ce491d063b410272cb740d9493deafc822b3586.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cao|geomamba_dualpath_riemannian_statespace_models_for_functional_dynamics"},"authorids":{"value":["~Yuwei_Cao3","~Tingting_Dan1","~Yang_Yang12","~Guorong_Wu1"]},"authors":{"value":["Yuwei Cao","Tingting Dan","Yang Yang","Guorong Wu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes to leverage linear networks such as RWKV and Mamba to capture long-range dependencies in LiDAR-based outdoor 3D object detection, leading to relatively larger group sizes of the voxel partition. The proposed techniques include voxel merging/expanding and voxel generation. Experiments are conducted on the Waymo Open dataset and nuScenes dataset, achieving state-of-the-art performance."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- It seems like the multiple diffused voxels can occupy a single position. How do you handle this situation?\n- How many new voxels will be generated? Does the voxel generation reduce the efficiency?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- This paper is an early attempt to utilize linear RNN for outdoor LiDAR-based 3D object detection.\n- LION achieves state-of-the-art performance on the mainstream datasets.\n- LION can be built upon multiple linear RCNNs such as mamba, RetNet, and RWKV, showcasing the universality of the proposed framework."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Some claims are obscure and not well supported by evidence, such as: \"However, effectively applying linear group RNN to 3D object detection in highly sparse point clouds is not trivial due to its limitation in handling spatial modeling.\" Why is it not trivial? What is the limitation? What is spatial modeling?\n- The proposed techniques are not novel. The spatial descriptor is a common spconv-based module. Voxel merging and expanding are trivial. The voxel generation is new but also similar to some methods, such as the \"virtual voxel\" in \"fsdv2: Improving fully sparse 3d object detection with virtual voxels\". It would be better to conduct a discussion.\n- The ablation for larger group size is not sufficient. The authors should conduct experiments with different group sizes to showcase the workings of group sizes since it is a main claim.\n- Since the most significant advantage of utilizing linear RCNN is efficiency, the authors are encouraged to conduct a more detailed runtime evaluation to reveal the latency of each component and how the designs affect the efficiency."},"limitations":{"value":"The authors adequately addressed the limitations."}},"nonreaders":[],"tmdate":1730878664885,"tcdate":1720511604253,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission886/Reviewer_9fJS"],"signatures":["NeurIPS.cc/2024/Conference/Submission886/Reviewer_9fJS"],"forum":"5tGkAcY7uV","number":1,"license":"CC BY 4.0","cdate":1720511604253,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission886/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878664885,"domain":"NeurIPS.cc/2024/Conference","replyto":"5tGkAcY7uV","id":"0JTtEa5r4t","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["3D Object Detection","Linear RNN"]},"primary_area":{"value":"machine_vision"},"abstract":{"value":"The benefit of transformers in large-scale 3D point cloud perception tasks, such as 3D object detection, is limited by their quadratic computation cost when modeling long-range relationships. In contrast, linear RNNs have low computational complexity and are suitable for long-range modeling. Toward this goal, we propose a simple and effective window-based framework built on Linear group RNN (i.e., perform linear RNN for grouped features) for accurate 3D object detection, called LION. The key property is to allow sufficient feature interaction in a much larger group than transformer-based methods. However, effectively applying linear group RNN to 3D object detection in highly sparse point clouds is not trivial due to its limitation in handling spatial modeling. To tackle this problem, we simply introduce a 3D spatial feature descriptor and integrate it into the linear group RNN operators to enhance their spatial features rather than blindly increasing the number of scanning orders for voxel features. To further address the challenge in highly sparse point clouds, we propose a 3D voxel generation strategy to densify foreground features thanks to linear group RNN as a natural property of auto-regressive models. \nExtensive experiments verify the effectiveness of the proposed components and the generalization of our LION on different linear group RNN operators including Mamba, RWKV, and RetNet. Furthermore, it is worth mentioning that our LION-Mamba achieves state-of-the-art on Waymo, nuScenes, Argoverse V2, and ONCE datasets. Last but not least, our method supports kinds of advanced linear RNN operators (e.g., RetNet, RWKV, Mamba, xLSTM and TTT) on small but popular KITTI dataset for a quick experience with our linear RNN-based framework."},"_bibtex":{"value":"@inproceedings{\nliu2024lion,\ntitle={{LION}: Linear Group {RNN} for 3D Object Detection in Point Clouds},\nauthor={Zhe Liu and Jinghua Hou and Xinyu Wang and Xiaoqing Ye and Jingdong Wang and Hengshuang Zhao and Xiang Bai},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=5tGkAcY7uV}\n}"},"title":{"value":"LION: Linear Group RNN for 3D Object Detection in Point Clouds"},"pdf":{"value":"/pdf/4328f471f60186bd07dea76178ce8929385cb6e4.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"liu|lion_linear_group_rnn_for_3d_object_detection_in_point_clouds"},"authorids":{"value":["~Zhe_Liu12","~Jinghua_Hou1","~Xinyu_Wang7","~Xiaoqing_Ye1","~Jingdong_Wang1","~Hengshuang_Zhao2","~Xiang_Bai1"]},"authors":{"value":["Zhe Liu","Jinghua Hou","Xinyu Wang","Xiaoqing Ye","Jingdong Wang","Hengshuang Zhao","Xiang Bai"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Mamba-CDSP, a novel approach for time-varying counterfactual prediction (TCP) using state-space models. The key innovation lies in combining the Mamba architecture (a recent advance in state-space modeling) with a new Covariate-based Decorrelation towards Selective Parameters (CDSP) mechanism. The method addresses two major challenges in TCP: computational efficiency and the over-balancing problem. The authors demonstrate superior performance over existing methods like Causal Transformer and G-Net on both synthetic and real-world datasets."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. How sensitive is the CDSP mechanism to the choice of decorrelation threshold?\n2. Could the authors provide more insight into the computational complexity trade-offs between CDSP and traditional balancing methods?\n3. How does the method perform on extremely long sequences (e.g., >1000 timesteps)?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Technical Innovation:\n\n-Novel combination of Mamba architecture with TCP\n\n-Well-designed CDSP mechanism that addresses known limitations\n\n-Efficient implementation with linear time complexity\n\n2. Practical Value:\n\n-Better handling of long sequences\n\n-Improved computational efficiency\n\n-Real-world applicability demonstrated on MIMIC-III dataset\n\n\n3.Experimental Design:\n\n-Comprehensive ablation studies\n\n-Multiple evaluation scenarios\n\n-Reasonable baseline comparisons"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Theoretical Analysis:\n\n-Limited theoretical justification for why CDSP works better than traditional balancing\n\n-Could benefit from more formal analysis of the bias-variance trade-off\n\n2.Empirical Validation:\n\n-Could benefit from more diverse real-world datasets\n\n-Limited discussion of failure cases\n\n-More detailed hyperparameter sensitivity analysis needed"}},"nonreaders":[],"tmdate":1731427712061,"tcdate":1730706154223,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission7314/Reviewer_FCGz"],"signatures":["ICLR.cc/2025/Conference/Submission7314/Reviewer_FCGz"],"forum":"yheQRc5xWB","number":3,"license":"CC BY 4.0","cdate":1730706154223,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission7314/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427712061,"domain":"ICLR.cc/2025/Conference","replyto":"yheQRc5xWB","id":"lWR5KP2DW1","forumContent":{"TLDR":{"value":"This paper presents a novel method for counterfactual prediction over time series with state-space models."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series; State-space Models; Treatment Effect Estimation"]},"primary_area":{"value":"causal reasoning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Time-varying counterfactual prediction (TCP) from observational data supports the answer of when and how to assign multiple sequential treatments, yielding importance in various applications. Despite the progress achieved by recent advances, e.g., LSTM or Transformer based causal approaches, their capability of capturing interactions in long sequences remains to be improved in both prediction performance and running efficiency. In parallel with the development of TCP, the success of the state-space models (SSMs) has achieved remarkable progress toward long-sequence modeling with saved running time. Consequently, studying how Mamba simultaneously benefits the effectiveness and efficiency of TCP  becomes a compelling research direction. In this paper, we propose to exploit advantages of the SSMs to tackle the TCP task, by introducing a counterfactual Mamba model with Covariate-based Decorrelation towards Selective Parameters (Mamba-CDSP). Motivated by the over-balancing problem in TCP of the direct covariate balancing methods, we propose to de-correlate between the current treatment and the representation of historical covariates, treatments, and outcomes, which can mitigate the confounding bias while preserve more covariate information. In addition, we show that the overall de-correlation in TCP is equivalent to regularizing the selective parameters of Mamba over each time step, which leads our approach to be effective and lightweight. We conducted extensive experiments on both synthetic and real-world datasets, demonstrating that Mamba-CDSP not only outperforms baselines by a large margin, but also exhibits prominent running efficiency."},"_bibtex":{"value":"@inproceedings{\nwang2025effective,\ntitle={Effective and Efficient Time-Varying Counterfactual Prediction with State-Space Models},\nauthor={Haotian Wang and Haoxuan Li and Hao Zou and Haoang Chi and Long Lan and Wanrong Huang and Wenjing Yang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=yheQRc5xWB}\n}"},"title":{"value":"Effective and Efficient Time-Varying Counterfactual Prediction with State-Space Models"},"pdf":{"value":"/pdf/6543649eb8bf53c6ca75c251a0032d3e63a00c35.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wang|effective_and_efficient_timevarying_counterfactual_prediction_with_statespace_models"},"authorids":{"value":["~Haotian_Wang2","~Haoxuan_Li6","~Hao_Zou1","~Haoang_Chi1","~Long_Lan2","~Wanrong_Huang1","~Wenjing_Yang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Haotian Wang","Haoxuan Li","Hao Zou","Haoang Chi","Long Lan","Wanrong Huang","Wenjing Yang"]}},"version":2},{"content":{"comment":{"value":"> Question 3: How does mGRADE works without causal convolution on LRA to see how performant is the sequence mixing of the mGRADE? \n\nWe agree that examining the core sequence mixing capabilities of mGRADE is essential. The model without the causal convolution component reduces to the minGRU architecture. We have performed this important ablation study and refer the reviewer to Table A7, where the results are presented. The data shows that while minGRU achieves approximately on par performance on ListOps, its performance significantly lags behind mGRADE on the Image task and fails on Pathfinder, demonstrating the critical role of the causal convolution for robust sequence mixing across diverse modalities. We highlight this in the main text (last paragraph of Section 4). \n\n> Question 4: What is the difference of mGRADE's minimal GRU compared to GRFA's or Mamba2's sequence mixing? \n\nWe refer the reviewer to our answer to weakness #1. \n\n### References \n\n[1] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024 \n\n[2] Y. Sarrof et al. The expressive capacity of state space models: A formal language perspective. Neurips, 2024 \n\n[3] I. Hammouamri et al. Learning delays in spiking neural networks using dilated convolutions with learnable spacings. ICLR, 2024 \n\n[4] T. Dao and A. Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024 \n\n[5] Z. Qin et al. Hgrn2: Gated linear rnns with state expansion. Conference on Language Modeling, 2024"},"title":{"value":"Reply - Part 3 (+ references)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763662957136,"tcdate":1763662957136,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19489/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission19489/Authors"],"forum":"nqUwo7R2vv","number":6,"license":"CC BY 4.0","cdate":1763662957136,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19489/-/Official_Comment"],"mdate":1763662957136,"domain":"ICLR.cc/2026/Conference","replyto":"ZYR6ImcDD2","id":"Gm6e5ekzo0","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["RNN","sequence processing","model architecture","embedded systems"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Processing temporal data directly at the sensor source demands models that capture both short- and long-range dynamics under tight memory constraints. While State-of-the-Art (SotA) sequence models such as Transformers excel at these tasks, their quadratic memory scaling with sequence length makes them impractical for edge settings. Recurrent Neural Networks (RNNs) offer constant memory scaling, but train sequentially and slowly, and Temporal Convolutional Networks (TCNs), though efficiently trainable, also scale memory with kernel length. For more memory-efficient sequence modeling, we propose mGRADE (minimally Gated Recurrent Architecture with Delay Embedding), a hybrid-memory system that integrates a temporal convolution with learnable spacings with a gated recurrent component equivalent to a minimal Gated Recurrent Unit (minGRU). The convolution with learnable spacings can express a flexible delay embedding that captures rapid temporal variations, while the recurrent component efficiently maintains global context with minimal memory overhead. We theoretically ground and empirically validate our approach on two types of synthetic tasks, demonstrating that mGRADE effectively separates and preserves temporal features across multiple timescales. Furthermore, on the challenging Long-Range Arena (LRA) benchmark, mGRADE reduces the memory footprint by up to a factor of 8, while maintaining competitive performance compared to SotA models."},"_bibtex":{"value":"@misc{\ntorchet2026mgrade,\ntitle={m{GRADE}: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling},\nauthor={Tristan Torchet and Christian Metzner and Karthik Charan Raghunathan and Jimmy Weber and Laura Kriener and Melika Payvand},\nyear={2026},\nurl={https://openreview.net/forum?id=nqUwo7R2vv}\n}"},"title":{"value":"mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling"},"pdf":{"value":"/pdf/5c41faee958abf75002def8814a54ee2bdafa10b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"torchet|mgrade_minimal_recurrent_gating_meets_delay_convolutions_for_lightweight_sequence_modeling"},"authorids":{"value":["~Tristan_Torchet1","~Christian_Metzner1","~Karthik_Charan_Raghunathan1","~Jimmy_Weber1","~Laura_Kriener1","~Melika_Payvand1"]},"authors":{"value":["Tristan Torchet","Christian Metzner","Karthik Charan Raghunathan","Jimmy Weber","Laura Kriener","Melika Payvand"]}},"version":2},{"content":{"summary":{"value":"MambaVoiceCloning proposes a TTS system whose entire deployed conditioning path (text, rhythm/duration, prosody) is state–space only at inference, paired with a diffusion decoder for quality, yielding linear-time encoding, bounded activation memory, long‑form stability, and feasible streaming with finite look‑ahead. It uses three SSM modules: (i) a gated bidirectional Mamba text encoder, (ii) a Temporal Bi‑Mamba trained with a lightweight aligner used only during training, and (iii) an Expressive Mamba with AdaLN conditioning.\n\nThe proposed method shows statistically reliable quality and efficiency gains over StyleTTS2 and Mamba-attention hybrids under a matched mel–diffusion–vocoder setup, while confirming the diffusion decoder remains the main latency bottleneck. \n\nAll encoder modules run in O(T) with bounded activations and no global attention maps, reducing peak memory, improving encoder throughput, and enabling longer inputs/batches; the diffusion decoder still dominates latency."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. How are Mamba selective-scan hyperparameters (state size, convolution kernel, gating temperature) chosen for each encoder, and how sensitive are results to these choices across datasets and lengths?\n\n2. The Bi‑Mamba text fusion uses a learned gate plus AdaLN; is the gating stable on very long sequences or under domain shift, and are there failure cases where one direction collapses?\n\n3. Why is fusion in the Temporal Bi‑Mamba kept linear (no gating) while gating is used elsewhere; are there quantitative trade-offs or counterexamples where temporal gating would help long-form speech?\n\n4. The lightweight attention aligner is used only during training; what architecture and capacity does it have, and how robust are learned alignments if the aligner is partially mis-specified or noisy?\n\n5. How long must the reference be for stable similarity in zero-shot cloning?\n\n6. The paper claims finite look‑ahead streaming with Uni‑Mamba; what look‑ahead L is used in experiments, and how does WER/MOS vary with L for 2–6 minute inputs?\n\n7. MVC is trained on English but tested on CSS10 ES/DE/FR with shared vocab and <lang> tags; what failure modes occur in cross-lingual phoneme inventories and stress patterns, and does alignment degrade on languages with different syllable timing?"},"rating":{"value":6},"details_of_ethics_concerns":{"value":"Given the zero/few-shot cloning capability, what safeguards or watermarking were actually enabled in the released code, and how detectable is MVC synthesis under common forensic detectors?"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The proposed method provides linear‑time scans, bounded activations, lower peak memory, and higher encoder throughput (≈1.6×).\n\n2. The proposed method offers modest improvements over StyleTTS2 and Mamba‑attention hybrids in MOS/CMOS and objective metrics"},"flag_for_ethics_review":{"value":["Yes, Discrimination / bias / fairness concerns"]},"weaknesses":{"value":"1. Since the decoder model dominates the latency, the linear-time encoder does not translate into large overall RTF gains.\n\n2. On LJSpeech and LibriTTS, differences in metrics as compared to the baselines are small; the paper frames MVC as a practical encoder-side refinement, not a paradigm shift.\n\n3. The system lacks explicit emotion or fine-grained style controls beyond AdaLN conditioning, limiting targeted prosody manipulation compared to methods with richer style token or reference-attention mechanisms.\n\n4. Claims of long-form stability are positive, but focus on MOS/WER and pitch drift; cross-chunk smoothing and boundary artifacts under streaming conditions are not deeply analyzed."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360520558,"tcdate":1761974233461,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10373/Reviewer_Dek4"],"signatures":["ICLR.cc/2026/Conference/Submission10373/Reviewer_Dek4"],"forum":"0oXyMbPMtP","number":4,"license":"CC BY 4.0","cdate":1761974233461,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10373/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360520558,"domain":"ICLR.cc/2026/Conference","replyto":"0oXyMbPMtP","id":"D43WL43Mmd","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"SSM-only TTS conditioning at inference (no attention/RNN); a gated Bi-Mamba improves long-form stability/streaming and gives ~1.6× encoder speed with modest, statistically significant quality gains."},"keywords":{"value":["text-to-speech (TTS)","speech synthesis","voice cloning","Mamba","state space models (SSM)","diffusion TTS","prosody modeling","streaming/low-latency"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference—removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody—while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time $\\mathcal{O}(T)$ conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba--TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel--diffusion--vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba--attention hybrids in MOS/CMOS, F$_0$ RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by $1.6\\times$. Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability. \nCode available at: \\url{https://github.com/sahilkumar15/MVC}."},"_bibtex":{"value":"@inproceedings{\nkumar2026mambavoicecloning,\ntitle={MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control},\nauthor={Sahil Kumar and Namrataben Patel and Honggang Wang and Youshan Zhang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=0oXyMbPMtP}\n}"},"title":{"value":"MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control"},"pdf":{"value":"/pdf/86cde596cbc42add1397f867565a227aba8ab9c3.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"kumar|mambavoicecloning_efficient_and_expressive_texttospeech_via_statespace_modeling_and_diffusion_control"},"authorids":{"value":["~Sahil_Kumar1","~Namrataben_Patel1","~Honggang_Wang3","~Youshan_Zhang1"]},"authors":{"value":["Sahil Kumar","Namrataben Patel","Honggang Wang","Youshan Zhang"]}},"version":2},{"content":{"summary":{"value":"The paper introduces DSEE-MDiff, an encoder–decoder framework for temporal knowledge graph extrapolation. The encoder selects history via a dynamic–static entity selection module that fuses global semantic evolution with local structural cues. The decoder combines a ConvTransE scoring head with a Mamba-driven diffusion denoiser whose mean is a gated mixture of Transformer and Mamba paths. Training jointly optimizes the ConvTransE loss and the diffusion loss, and the final prediction aggregates the two scores additively. Experiments on ICEWS14, ICEWS18, ICEWS05-15 and GDELT report competitive performance with state-of-the-art results on three benchmarks. The paper includes ablations, sensitivity studies on historical window and diffusion sequence length, and an unseen-facts evaluation with a short case study."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1.\tHow sensitive is performance to the fixed additive fusion. Please report results when a single scalar weight multiplies the ConvTransE score and the diffusion score, and include a short calibration analysis of both heads. \n2.\tWhat does the gating variable learn across diffusion steps and datasets. Please provide summaries of the gate values, such as per-step averages and dispersion, and discuss conditions under which one path dominates. \n3.\tCould you define the time-aware metrics in the main text and confirm protocol parity with all baselines. A brief statement on filtering, ranking direction, and temporal handling would resolve ambiguity in Table two. \n4.\tWhat is the full cost profile of DSEE-MDiff. Please report parameters, FLOPs, wall-clock training time, inference latency, and peak memory on each dataset, and compare these numbers with at least one diffusion baseline and one contrastive baseline. \n5.\tCan you analyze the GDELT gap in more detail. A breakdown by relation family and a study of performance versus sequence length or snapshot density would help determine whether the diffusion head or the selector is the bottleneck. \n6.\tWould you visualize diffusion dynamics. Plots of representation trajectories or similarity to clean targets across steps, and a comparison of Transformer-only versus Mamba-only denoisers, would clarify how the selective state-space path contributes beyond depth."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1.\tThe method design is coherent. The paper specifies the diffusion reverse process and the gated combination of Transformer and Mamba, then connects these to ConvTransE decoding and a joint objective. This facilitates reproducibility. \n2.\tThe aggregation rule is explicit. The final probability is defined as the sum of ConvTransE and diffusion scores, which clarifies inference behavior. \n3.\tEvaluation breadth. Four datasets are covered with a large baseline set, and the main table uses time-aware metrics for MRR and Hits. \n4.\tInformative ablations. Removal of the dynamic selector causes a larger drop than removal of the static selector, and removing either Mamba or Transformer degrades performance to a similar degree. The decoder removal further confirms the contribution of ConvTransE. \n5.\tSensitivity and unseen-facts analysis. The paper varies history window and diffusion sequence length, and constructs unseen-facts test splits for ICEWS14 and ICEWS18 with comparisons to two representative baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tFusion calibration and design space remain under-explored. The paper fixes aggregation to a simple sum of Sct and Sdiff, without reporting variants with a learned weight or confidence calibration of either head. This limits understanding of whether the improvement comes from complementary information or uncalibrated score addition. A small study that tunes a scalar weight or reports calibration errors would clarify this point. \n2.\tBehavior of the gating variable is not analyzed. The denoiser mean uses a learnable gate that mixes Transformer and Mamba outputs, yet the paper does not report statistics of the gate across diffusion steps or datasets, nor whether it saturates to one path in specific regimes. Summaries such as per-step averages or histograms would illuminate how the gate allocates responsibility. \n3.\tDefinition and protocol of time-aware metrics are not explained in the main text. The main results table states that time-aware metrics are used, but the manuscript does not define the metric computation in the main body or verify parity with baselines there. A concise definition and protocol confirmation in the main section would remove ambiguity. \n4.\tResource profile is not presented. The experiments section reports accuracy but does not include parameter counts, FLOPs, wall-clock training time, inference latency, or memory consumption per dataset. Given the additional diffusion head and dual-path denoiser, such metrics would help assess practical efficiency. \n5.\tAnalysis of the GDELT gap is brief. The text notes that performance on GDELT is slightly below a diffusion baseline, yet there is no breakdown by relation type or sequence length to explain the gap. A diagnosis connected to dataset properties such as snapshot density or sequence length would make the result more actionable. \n6.\tDiffusion dynamics are only indirectly discussed. The paper studies sequence length but does not visualize how representations evolve along diffusion steps or how the denoiser removes noise. Trajectory plots or cosine-similarity traces across steps would provide concrete insight into stability and information retention."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916349500,"tcdate":1761833575877,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2729/Reviewer_YTPf"],"signatures":["ICLR.cc/2026/Conference/Submission2729/Reviewer_YTPf"],"forum":"gWyYeO0LWZ","number":3,"license":"CC BY 4.0","cdate":1761833575877,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2729/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916349500,"domain":"ICLR.cc/2026/Conference","replyto":"gWyYeO0LWZ","id":"emm3nO2Jbq","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Temporal knowledge graph","Temporal knowledge graph reasoning","Selective State Space Model","Diffusion"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Temporal Knowledge Graph (TKG) reasoning aims to predict future missing facts based on historical evidence. Prior studies on graph learning and logical rules often overlook global latent semantics and struggle with long-range dependencies, particularly under sparse or unseen facts. To address these limitations, we propose DSEE-MDiff, which frames TKG reasoning as selecting informative history and denoising future signals. Specifically, a Dynamic–Static Entity Selection encoder captures global semantic evolution alongside local structural cues, while a Mamba-based diffusion module injects and removes noise with a selective state-space model to better recover long-range dependencies and mitigate sparsity. The two outputs are fused for prediction through a ConvTransE decoder. Experiments on four public datasets demonstrate that DSEE-MDiff achieves state-of-the-art performance across multiple metrics, validating the effectiveness of the proposed approach."},"_bibtex":{"value":"@misc{\ngeng2026dynamicstatic,\ntitle={Dynamic{\\textendash}Static Representation Learning with Mamba-Enhanced Diffusion for Temporal Knowledge Graph Reasoning},\nauthor={Rushan Geng and Cuicui Luo},\nyear={2026},\nurl={https://openreview.net/forum?id=gWyYeO0LWZ}\n}"},"title":{"value":"Dynamic–Static Representation Learning with Mamba-Enhanced Diffusion for Temporal Knowledge Graph Reasoning"},"pdf":{"value":"/pdf/00bcae00b45c2be35021b53c192d4691538865ca.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"geng|dynamicstatic_representation_learning_with_mambaenhanced_diffusion_for_temporal_knowledge_graph_reasoning"},"authorids":{"value":["~Rushan_Geng1","~Cuicui_Luo1"]},"authors":{"value":["Rushan Geng","Cuicui Luo"]}},"version":2},{"content":{"summary":{"value":"## Learning frequency-selective spectral representations using state space models\n\n- The paper proposes FSSM (Frequency Selective State Space Models) for learning spectral representations from audio data, as a direct challenger of FFT-based spectral representations and establishes their performance on Radar object detection and Keyword spotting tasks.\n- This is done through learning a *frontend* fitted with 2-d modelling using the proposed FSSM modules along samples and chirps for RADAR data, whereas it uses the proposed FSSM followed by bidirectional-Mamba for keyword spotting."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"No direct questions for now, kindly address the weaknesses."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- Novel framework for spectral representation learning through frequency-selective state spaces.\n- The proposed methods perform well against baselines on radar object detection on the RADIal dataset and keyword spotting (Speech Commands v2) datasets."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"## Weakness 1: Why Keyword Spotting on Speech Commands?\n- The use of keyword spotting on Speech Commands as a representative for audio recognition performance is not well motivated. \n- The proposed FSSM touts frequency-selectivity as its main strength; however, I don't believe Keyword Spotting is a challenging enough task to put that to a true test. Speech Commands v2 has 35-keyword classes and is an English-only dataset, and is rather limited.\n- A more comprehensive audio recognition task, like multi-label audio classification on AudioSet or FSD50k would have been a better test-bed for testing frequency-selectivity of the proposed FSSM frontend.\n- In essence, to be of wider applicability and to prove that FSSM is a good learnable frontend, a better pairing of auxiliary task is needed alongside radar-based object detection.\n\n---\n\n## Weakness 2: Figures\n- Maybe I'm splitting hairs here, but all the figures in the paper are subpar. They have poor resolution and quality, and the text is too small (especially Figure 3), making it difficult to read in print. \n- In Figure 5, there are so many empty bins it is not possible to clearly see the difference between the FFT and FSSM visualizations. There is also an outline to the figure which looks weird.\n\n---\n\n## Weakness 3: Frequency-selectivity\n\n- The frequency selectivity of the proposed approach is not well demonstrated. Only Figure 5 shines a light on it, that too poorly.\n- A more comprehensive empirical/exploratory investigative analysis is necessary to ascertain that FSSM indeed does better at being frequency selective.\n\n---\n\n## Weakness 4: Empirical analysis\n\n- Alongside the use of Speech Commands as the auxiliary task, the overall quality of empirical analysis in the paper is lacking.\n- There are no confidence intervals, and the precision used in reported results (significant digits) is insufficient, especially for RADIal analysis, where margins between the evaluated approaches are quite small. For instance, the mAP for FFT-RadNet and the proposed FSSM-FFTRadNet is 0.97 and 0.98, respectively.\n\n---\n\n## Weakness 5: Spelling and grammatical errors\n- There are several spelling and grammatical errors throughout the paper. for instance *represetation*, *segentation*. The paper needs more  proofreading and polish.\n\n---\n\n# OVERALL\n\n- It's a good idea, but the execution is lacking. It needs significant work before it is ready for publishing."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942581962,"tcdate":1761815613155,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23270/Reviewer_QdMU"],"signatures":["ICLR.cc/2026/Conference/Submission23270/Reviewer_QdMU"],"forum":"SyTuymSbBJ","number":2,"license":"CC BY 4.0","cdate":1761815613155,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23270/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942581962,"domain":"ICLR.cc/2026/Conference","replyto":"SyTuymSbBJ","id":"bCwRSwu8lu","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We propose a state-space model for learning spectral representations of time series data inspired by discrete Fourier transforms."},"keywords":{"value":["frequency spectrum","state space models","spectral operators","band-selective kernels","radar object detection","audio keyword recognition","sequence models"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"We introduce the first state space model (FSSM) with frequency selective spectral operators, parameterizing a family of stable, causal, band-selective kernels whose spectral weights are conditioned on the end task. This yields a representation that adapts its characteristics per task domain while retaining linear-time inference and memory. The key novelty is the trainable spectral front-end through which the model can adapt frequency weighting and inter-bin window size. We show the effectiveness of our learned spectral representations on two independent domains: radar object detection and speech keyword recognition, outperforming state of the art frequency based methods in both domains while maintaining competitive throughput and computational overhead. We further show the robustness of our approach under input perturbations, demonstrating the value of stabilized sequential operators in spectral representation learning."},"_bibtex":{"value":"@misc{\nsen2026fssm,\ntitle={{FSSM}: Frequency-Selective State Space Models for Spectral Representation Learning},\nauthor={Anuvab Sen and Mir Sayeed Mohammad and Saibal Mukhopadhyay},\nyear={2026},\nurl={https://openreview.net/forum?id=SyTuymSbBJ}\n}"},"title":{"value":"FSSM: Frequency-Selective State Space Models for Spectral Representation Learning"},"pdf":{"value":"/pdf/c79f7107ce97de50e6f86b3e654576f0e1200e0c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"sen|fssm_frequencyselective_state_space_models_for_spectral_representation_learning"},"authorids":{"value":["~Anuvab_Sen1","~Mir_Sayeed_Mohammad1","~Saibal_Mukhopadhyay2"]},"authors":{"value":["Anuvab Sen","Mir Sayeed Mohammad","Saibal Mukhopadhyay"]}},"version":2},{"content":{"summary":{"value":"This paper introduces the Graph Mamba Operator (GRAMO), a new architecture integrating State-Space Models (SSMs) with graph neural networks to learn the dynamics of particle-based systems. The model consists of a temporal SSM (Mamba) backbone to capture long-range time dependencies and a novel SSM-parameterized spatial graph layer for spatial interactions. The authors claim this S2T (state-to-trajectory) model achieves state-of-the-art results on several benchmarks, including MD17 and Mocap, and provide a theoretical analysis of its properties, such as permutation equivariance and stability.\n\nHowever, due to a potential data leakage issue in 2 (out of 4) datasets, and a weird fixed/hardcoded starting frame issue in N-body dataset, I am very suspicious of the validity of their empirical reports. The seemingly unrelated proofs (that belong to generic GNN or Mamba paper) is another distraction point in paper writing. I am issuing reject now; if authors can fix the dataset bug and reproduce the affected experiments, also remove redundant proofs, I will re-consider."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1.  Can the authors correct all the \"random\" and \"in-trajectory\" splitting methodology to be \"by-trajectory\" splits for datasets (MD17, MDAnalysis), and re-run the corresponding experiments? This is essential to validate the main empirical claims of the paper.\n2.  Please verify the derivation in Remark 4.1. Given $B=0$ and $C=0$, the output $Y(t+1)$ becomes zero. How does this recover the GCN message-passing rule as claimed?\n3.  According to Table 5, the EGNN-spatial + GRAMO-temporal model (28.22) outperforms the full GRAMO model (34.12). Does this not imply that the novel spatial operator is the weakest part of the contribution and actually degrades performance compared to a baseline?\n4.  Can the authors clarify what, precisely, is theoretically novel in Section 4.2? The analysis of permutation equivariance and the spectral properties of the normalized adjacency matrix are standard in the GNN literature.\n5. Can you remove the abundant proof that belongs to previous Mamba works and only focus on your novel contributions?"},"rating":{"value":2},"details_of_ethics_concerns":{"value":"NA"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1.  The central idea of integrating modern SSMs (like Mamba) with geometric graph learning is a compelling and timely research direction. Tackling long-range temporal dependencies in complex graph-structured physical systems is a significant and valuable problem for the community.\n2.  The proposed architecture, which uses an SSM to parameterize the spatial message-passing layer, is an interesting concept, even if its empirical validation is flawed in the current manuscript."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper suffers from several critical flaws that undermine its central claims.\n\n1.  The paper's headline results are derived from an invalid dataloader setup. Checking the codes in the anonymous repository reveals that:\n    * For continuous trajectory datasets (MD17, MDAnalysis), the authors use \"random initial points\" and \"in-trajectory\" splitting. This is a severe form of data leakage. For sequential data, if a test sample (e.g., starting with initial time $30$) is close/adjacent to a training sample (e.g., at time $28$); assuming time step gap is $2$, your training sequence will contain $28,30,32,34,36,...$, while the test sequence will contain $30,32,34,36,...$. This setup allows the model to memorize short-term transitions from leaked data rather than learn the true long-range dynamics.\n    * All results, extrapolation studies, and physics-conservation analyses on these datasets (MD17, MDAnalysis) are consequently rendered unreliable.\n    * The N-Body is correctly splited by trajectories, but this one seems to hard code the initial frame to be $6,20,30$ for three variants, all of which discard many beginning frames for training. This is a very suspicious choice that needs justification. The results of this dataset also only shows a marginal **4.61\\%** gain, adding on to the suspicion that the other datasets' gains are due to leakage.\n\n3.  The \"Theoretical Analysis\" (Section 4.2) presented as a core contribution is not novel.\n    * Proposition 4.3 (Permutation Equivariance) is a \"sanity check\" or a minimum requirement for any GNN, not a new discovery.\n    * Lemma 4.2 (Spectral properties of $A$) is a textbook result from spectral graph theory and the basis of the original GCN, not a new finding.\n    * The stability analysis (Prop 4.4) is a standard calculation for a simplified linear system, which ignores the actual non-linear \"selective\" nature of Mamba, hence not particularly insightful here.\n\n4.  More mathematical errors:\n    * The paper claims that setting $B=0$ and $C=0$ in the spatial operator ($S(t+1)=A^S(t)W+X(t)B$, $Y(t+1)=S(t+1)C$) recovers the GCN rule $Y=\\sigma(A^XW)$.\n    * A simple substitution shows this is false. If $C=0$, the output $Y(t+1)$ is always zero."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926648321,"tcdate":1761994673170,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16568/Reviewer_2qBh"],"signatures":["ICLR.cc/2026/Conference/Submission16568/Reviewer_2qBh"],"forum":"6CO8dwHl4F","number":2,"license":"CC BY 4.0","cdate":1761994673170,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16568/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926648321,"domain":"ICLR.cc/2026/Conference","replyto":"6CO8dwHl4F","id":"2N52gnS0xT","forumContent":{"TLDR":{"value":"Introduces a new graph neural operator that leverages state-space models (SSMs) for long range modelling."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Graph Neural Network","Neural Operator","State Space Models"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"abstract":{"value":"Modelling complex 3D dynamics---from molecular conformations to particle interactions and human motion---requires capturing dependencies spanning long temporal horizons and non-local spatial interactions. Graph neural networks (GNNs) have shown promise in spatio-temporal settings but often suffer from instability and degraded accuracy in long-range forecasting. We propose the Graph Mamba Operator (GraMO), a neural operator that integrates state-space models (SSMs) with geometric learning to capture spatio-temporal correlations jointly. To jointly model complex dynamics, GraMO integrates a stable, SSM-based temporal backbone with an SSM-parameterized graph update to capture long-range spatial dependencies. Unlike stepwise predictors that accumulate errors over time, GraMO learns entire trajectories in a single forward pass. Across diverse benchmarks ranging from molecular dynamics to human motion capture, GraMO shows notable improvements in trajectory fidelity, stability, and robustness over strong baselines with relative improvements of over 26.3\\% on motion capture benchmarks and 25.2\\% on MD17 final-state prediction. Ablation studies reveal that temporal SSM components consistently improve performance, while spatial SSM updates show task-dependent benefits---helping with long-range interactions in large molecules but potentially hindering performance on systems with primarily local dependencies. Altogether, these results suggest that selective integration of SSM components with graph neural networks can improve performance on particle-based systems, with applications in molecular simulations, articulated rigid body dynamics, and particle systems."},"_bibtex":{"value":"@misc{\ntiwari2025graph,\ntitle={Graph Mamba Operator for Learning the Dynamics of Particle-based Systems},\nauthor={Karn Tiwari and Niladri Dutta and Prathosh AP and N M Anoop Krishnan},\nyear={2025},\nurl={https://openreview.net/forum?id=6CO8dwHl4F}\n}"},"title":{"value":"Graph Mamba Operator for Learning the Dynamics of Particle-based Systems"},"pdf":{"value":"/pdf/2f19ba46da22220db7180ecf5ababb98f9ad45eb.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"tiwari|graph_mamba_operator_for_learning_the_dynamics_of_particlebased_systems"},"authorids":{"value":["~Karn_Tiwari1","~Niladri_Dutta1","~Prathosh_AP1","~N_M_Anoop_Krishnan1"]},"authors":{"value":["Karn Tiwari","Niladri Dutta","Prathosh AP","N M Anoop Krishnan"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Brain functional connectivity","Riemannian geometry","State space model","Geometric deep learning"]},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Functional magnetic resonance imaging (fMRI)–derived functional connectivity (FC) is represented as graphs and as correlation/covariance matrices that live on non-Euclidean spaces—cortical graphs and the Riemannian manifold of symmetric positive-definite (SPD) matrices—so conventional Euclidean sequence models are misspecified. To this end, we introduce *Geo-Mamba*, a geometric variant of Mamba formulated on Riemannian manifolds. *Geo-Mamba* employs a dual-path selective state-space design: *a stacked path* performs hierarchical spatial modeling by aggregating pyramid multi-scale features to capture local and global dependencies, while *a embedding path* combats redundancy in high-dimensional SPD inputs via progressive, geometry-aware dimensionality reduction (operating in the appropriate manifold spaces) to produce compact states without violating Riemannian constraints. Their complementary outputs are fused through the tailored *GeoMix* operator to yield a compact, discriminative SPD representation. *Geo-Mamba* is evaluated on six public fMRI datasets—ADNI, OASIS, PPMI, Taowu, Neurocon, and Mātai—spanning Alzheimer’s and Parkinson’s cohorts as well as multi-site normative populations with diverse acquisition protocols. Across these benchmarks, it delivers consistently competitive accuracy and robustness, supporting the value of dual-path manifold modeling for neuroimaging and its potential for clinical translation."},"_bibtex":{"value":"@misc{\ncao2026geomamba,\ntitle={Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics},\nauthor={Yuwei Cao and Tingting Dan and Yang Yang and Guorong Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=7XonwHRkTS}\n}"},"title":{"value":"Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics"},"pdf":{"value":"/pdf/1ce491d063b410272cb740d9493deafc822b3586.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cao|geomamba_dualpath_riemannian_statespace_models_for_functional_dynamics"},"authorids":{"value":["~Yuwei_Cao3","~Tingting_Dan1","~Yang_Yang12","~Guorong_Wu1"]},"authors":{"value":["Yuwei Cao","Tingting Dan","Yang Yang","Guorong Wu"]}},"tmdate":1770804964745,"tcdate":1758203102161,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11691/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission11691/Authors"],"forum":"7XonwHRkTS","license":"CC BY 4.0","number":11691,"cdate":1758203102161,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission11691/-/Full_Submission","ICLR.cc/2026/Conference/Submission11691/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804964745,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"7XonwHRkTS","version":2},{"content":{"summary":{"value":"The paper introduces a novel post-hoc optimization strategy for existing Vision Mamba architectures, termed Vision Mamba Mender, aimed at enhancing the performance of Mamba models in visual recognition tasks. The authors seek to identify and rectify flaws in the Mamba model’s mechanisms from both external and internal hidden state perspectives, proposing a state correlation analysis method. Through this analysis, the authors pinpoint critical modules requiring repair and subsequently design a state correlation constraint method. The authors conduct extensive experiments on several Vision Mamba architectures to validate the efficacy of the proposed optimization strategy."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. In Eqns. (8) and (10), the two terms are multiplied together rather than averaged. What is the rationale behind this design choice?\n2. In Figure 4, why are both the ViM and VMamba structures adopted to obtain these observations?\n3. The loss function defined in Eq. (11) seems incorrect. Could the authors clarify this?\n4. In Table 1, it appears that the performance of \"Internal Flaw Repair\" is better than that of \"External Flaw Repair.\" Are the authors aware of this, and do they have any thoughts on why this is the case?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper proposes a novel and intriguing method to enhance state space models from a post-hoc perspective for Mamba. The state correlation analysis used to identify flaws in Mamba, and the subsequent state correlation constraint employed to rectify these flaws, are both new explorations for this kind of technique.\n2. The proposed method for identifying and rectifying flaws in Mamba is systematic to some degree. Moreover, the experiments are clearly designed and well-executed, demonstrating that the proposed method can be applied to mainstream Mamba architectures, further improving their performance.\n3. The paper is well-written, with clear language and a well-structured presentation. The background and motivation are clearly articulated and convincing. Each part of the method is presented step-by-step and in detail, allowing readers to effectively grasp the core ideas.\n4. The paper also provides some insights into the working mechanisms of Mamba, having the potential to inspire future research on the explanation of Mamba models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Before reading the methodology section, the description of \"certain modules of Mamba\" (Lines 65-67) may be confusing to readers. The authors should avoid using such vague terms.\n2. The results in Table 1 show that the proposed method improves the performance of Mamba by 1% to 5%. For the sake of rigor, the description in Line 71, stating that the performance is \"enhanced by 3% to 5%,\" does not seem very precise.\n3. In Lines 224-229, the authors claim that the states $c_n^{(l)}$ and $s_n^{(l)}$ in the shallowest blocks are crucial for influencing the model’s anomalous decisions. However, this cannot be observed from Fig. 4, as $c_n^{(l)}$ and $s_n^{(l)}$ do not appear to vary most significantly from the simple sample (Fig. 4a) to the hard sample (Fig. 4b).\n4. In addition to comparing the state correlation scores before and after flaw repair (Fig. 5), the authors should provide more visual comparisons of the state correlation (like Fig. 2 and Fig. 3), which would be more intuitive for the readers.\n5. Only analyzing and enhancing state space models from a post-hoc perspective for Mamba may not be enough."},"limitations":{"value":"The limitations appear to have been discussed in Section 6 by the authors. Moreover, there is no societal impact on the work performed."}},"nonreaders":[],"tmdate":1730879147799,"tcdate":1720062802566,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission7275/Reviewer_JDBm"],"signatures":["NeurIPS.cc/2024/Conference/Submission7275/Reviewer_JDBm"],"forum":"9VnevS2YoR","number":1,"license":"CC BY 4.0","cdate":1720062802566,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission7275/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879147799,"domain":"NeurIPS.cc/2024/Conference","replyto":"9VnevS2YoR","id":"BJUl1GA5iv","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","State Space Models","Computer Vision"]},"supplementary_material":{"value":"/attachment/231f16a7aedf14ace21ad17af99f0f84a5a06ee6.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Mamba, a state-space model with selective mechanisms and hardware-aware architecture, has demonstrated outstanding performance in long sequence modeling tasks, particularly garnering widespread exploration and application in the field of computer vision. While existing works have mixed opinions of its application in visual tasks, the exploration of its internal workings and the optimization of its performance remain urgent and worthy research questions given its status as a novel model. Existing optimizations of the Mamba model, especially when applied in the visual domain, have primarily relied on predefined methods such as improving scanning mechanisms or integrating other architectures, often requiring strong priors and extensive trial and error. In contrast to these approaches, this paper proposes the Vision Mamba Mender, a systematic approach for understanding the workings of Mamba, identifying flaws within, and subsequently optimizing model performance. Specifically, we present methods for predictive correlation analysis of Mamba's hidden states from both internal and external perspectives, along with corresponding definitions of correlation scores, aimed at understanding the workings of Mamba in visual recognition tasks and identifying flaws therein. Additionally, tailored repair methods are proposed for identified external and internal state flaws to eliminate them and optimize model performance. Extensive experiments validate the efficacy of the proposed methods on prevalent Mamba architectures, significantly enhancing Mamba's performance."},"_bibtex":{"value":"@inproceedings{\nhu2024vision,\ntitle={Vision Mamba Mender},\nauthor={Jiacong Hu and Anda Cao and Zunlei Feng and Shengxuming Zhang and Yi Wang and Lingxiang Jia and Mingli Song},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=9VnevS2YoR}\n}"},"title":{"value":"Vision Mamba Mender"},"pdf":{"value":"/pdf/8a5b7efdabc5e5544e1b8f3b86095c84a05b91e1.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"hu|vision_mamba_mender"},"authorids":{"value":["~Jiacong_Hu1","~Anda_Cao1","~Zunlei_Feng1","~Shengxuming_Zhang1","~Yi_Wang56","~Lingxiang_Jia1","~Mingli_Song1"]},"authors":{"value":["Jiacong Hu","Anda Cao","Zunlei Feng","Shengxuming Zhang","Yi Wang","Lingxiang Jia","Mingli Song"]}},"version":2},{"content":{"summary":{"value":"The paper presents a geometric extension of the Mamba selective state space model and applies it for the analysis of fMRI functional connectivity (FC) data. The connectivity measures are represented as symmetric positive definite (SPD) matrices on Riemannian manifolds due to its natural correlational structure. Additionally, the method uses a pyramid-based multi-scale geometric feature extraction from the original SPD matrices and also ensures that the sub-blocks of matrices at multiple levels themselves satisfy the SPD constraint. The paper uses a dual-path architecture (proposed before) that uses a a stacked path for hierarchical temporal modeling and a distillation path for progressive, manifold-preserving dimensionality reduction. They further use the geomix operator (which also has been proposed before) for manifold-consistent fusion. The authors show experimental results on six public fMRI datasets (ADNI, OASIS, PPMI, Taowu, Neurocon, Matai) and also show comparisons and performance improvement over existing methods including the original Mamba model, SPDNet, and BrainGNN."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Why aren't the results in Table 1 starred? Were they not significant? \n\nIn the first step following the pearson correlation calculation to construct the connectivity matrices, why not just keep the significant correlations? Depending upon the dataset (ADNI) for e.g., the matrix may be sparse (significance threshold) to begin with. Will it make the signal more noisy?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The main novelty of the paper is to ensure that the SPD manifold constraints at each stage of the selective state space model evolution are satisfied. \n\nIt extends the selective state-space models to SPD manifolds with explicit log-exp mapping and orthogonality constraints.\n\n\nThe authors perform comprehensive experimental validation. The method is tested on six datasets spanning Alzheimer’s, Parkinson’s, and other population with contact sports. \n\nThe ablation studies are somewhat comprehensive, although there's a few weaknesses in their actual implementation (see below)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The main weakness of the paper is that it combines multiple methods instead of making an original novel contribution. \n\n\nSeveral ideas (except the pyramidal projections of connectivity and satisfying SPD constraints) have been proposed before individually. Even the DimMap projection operator that is at the core of the method has been proposed before. Theoretical and computational advancements  for optimization on SPD manifolds both using classical and deep learning approaches have been proposed before. This paper combines several such ideas and applies it to fMRI data. Thus the novelty of the method is low. \n\nThe GeoMix operator is simply an interpolation between SPD matrices on the manifold. The authors introduce it as a new idea but it has been heavily used and studied before. \n\n\nBesides SPD manifold optimization, techniques such as dual path stacking have also been proposed before. For e.g. Brain-Mamba Behrouz & Hashemi (2024) performs a temporal and spatial stacking, while FST-Mamba Wei et al. (2025) has proposed a hierarchical stacking formulation. The authors have cited these methods. \n\n\nStep (iv) bypassing the Path #1, which disables the initial manifold-aware temporal evolution and\neliminating the sequence fusion under SSM and relying solely on the average time-step matrix for\nsubsequent computations. Too many things removed. Just disbale the manifold-ware temporal evolution\n\n\n“While the authors claim a substantial decline when Path #1 is removed, Figure 3 shows only marginal quantitative changes (mostly ≤ 3 % in Accuracy / F1). The benefit of the geometric temporal constraint thus appears more in preserving SPD validity and smooth dynamics than in producing large performance gains.”\n\nIn the ablation studies, step iv) is over ablated. To truly test the geometric constraint, the authors should keep Path #1 intact but replace the log–exp manifold operations with Euclidean updates, holding all other factors fixed. The step iv) ablation mixes removal of geometric constraints with also removal of recurrence (SSM), thus weakening the main argument about geometry-aware SSM, which is the main contribution of the paper. Thus it is unclear whether the modest performance drop which is observed (2–4 %) denoted by purple colors, may simply reflect losing temporal recurrence and not necessarily arise from avoiding the geometry constraint itself. A fairer test would retain the same SSM structure but operate in Euclidean space to quantify the direct impact of manifold geometry.\n\nIn experimental evaluation, the results of the method should ideally be compared with FST-Mamba, BNMamba, BTMamba to really separate the geometric-constraint contribution and whether the projection steps and the dynamic update steps are truly required for performance. They are certainly required to ensure the appropriate constraints, so that is not under question.\n\n\n\nOther manifold temporal models (e.g., Riemannian LSTM, SPD recurrent networks) or covariance-based DPL frameworks could also provide appropriate comparisons (although this is not required)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922741691,"tcdate":1762032349962,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11691/Reviewer_NBvP"],"signatures":["ICLR.cc/2026/Conference/Submission11691/Reviewer_NBvP"],"forum":"7XonwHRkTS","number":3,"license":"CC BY 4.0","cdate":1762032349962,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11691/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922741691,"domain":"ICLR.cc/2026/Conference","replyto":"7XonwHRkTS","id":"sgKCxLGyYa","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Brain functional connectivity","Riemannian geometry","State space model","Geometric deep learning"]},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Functional magnetic resonance imaging (fMRI)–derived functional connectivity (FC) is represented as graphs and as correlation/covariance matrices that live on non-Euclidean spaces—cortical graphs and the Riemannian manifold of symmetric positive-definite (SPD) matrices—so conventional Euclidean sequence models are misspecified. To this end, we introduce *Geo-Mamba*, a geometric variant of Mamba formulated on Riemannian manifolds. *Geo-Mamba* employs a dual-path selective state-space design: *a stacked path* performs hierarchical spatial modeling by aggregating pyramid multi-scale features to capture local and global dependencies, while *a embedding path* combats redundancy in high-dimensional SPD inputs via progressive, geometry-aware dimensionality reduction (operating in the appropriate manifold spaces) to produce compact states without violating Riemannian constraints. Their complementary outputs are fused through the tailored *GeoMix* operator to yield a compact, discriminative SPD representation. *Geo-Mamba* is evaluated on six public fMRI datasets—ADNI, OASIS, PPMI, Taowu, Neurocon, and Mātai—spanning Alzheimer’s and Parkinson’s cohorts as well as multi-site normative populations with diverse acquisition protocols. Across these benchmarks, it delivers consistently competitive accuracy and robustness, supporting the value of dual-path manifold modeling for neuroimaging and its potential for clinical translation."},"_bibtex":{"value":"@misc{\ncao2026geomamba,\ntitle={Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics},\nauthor={Yuwei Cao and Tingting Dan and Yang Yang and Guorong Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=7XonwHRkTS}\n}"},"title":{"value":"Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics"},"pdf":{"value":"/pdf/1ce491d063b410272cb740d9493deafc822b3586.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cao|geomamba_dualpath_riemannian_statespace_models_for_functional_dynamics"},"authorids":{"value":["~Yuwei_Cao3","~Tingting_Dan1","~Yang_Yang12","~Guorong_Wu1"]},"authors":{"value":["Yuwei Cao","Tingting Dan","Yang Yang","Guorong Wu"]}},"version":2},{"content":{"summary":{"value":"This paper extends the state space model Mamba into the multi-modal domain. The authors propose utilizing separate Mamba blocks to process each modality and suggest conditioning the state of each modality on the others to facilitate modality interaction. They further introduce a parallelism technique for the coupled Mamba transition. The framework is evaluated on three benchmark datasets."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. In Table 9, does 'average (concat) fusion' means averaging (concating) the features from all modality and then go through a single mamba block?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The attempt to extending Mamba model to multi-modal domain is a motivating direction, which may facilitate the efficiency of future multimodal foundation model.\n\n2. The model achieved good evaluation results on 3 benchmark datasets, while reducing the GPU memory usage to linear cost.\n\n3. Detailed mathematical derivation of the new global convolutional kernel is provided."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The demonstration of the parallelism is not clearly illustrated. The original Mamba model's essence lies in the selection mechanism and the hardware-aware parallel scan. The parallelism achieved through the global convolution kernel in S4 is no longer as effective in the Mamba framework, given that the transition matrices at each time step are conditioned on the input. This inspired the introduction of the parallel scan method in the Mamba paper. While the authors show that coupled Mamba can be computed using a global convolution, the question remains, how is the parallelism achieved in the context of Mamba's scanning process?\n\n2. The idea of using a sequential model to model each modality and then employing a simple interaction scheme (in this paper, sum) to enable modality fusion is straightforward, but maybe too simple to model the complex modality interaction.\n\n3. Comparison with existing multi-modal Mamba models, for instance, [1-2], should be presented.\n\n[1] VL-Mamba: Exploring State Space Models for Multimodal Learning\n[2] Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference"},"limitations":{"value":"Yes"}},"nonreaders":[],"tmdate":1730878771338,"tcdate":1719567931112,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2377/Reviewer_nXDG"],"signatures":["NeurIPS.cc/2024/Conference/Submission2377/Reviewer_nXDG"],"forum":"UXEo3uNNIX","number":1,"license":"CC BY 4.0","cdate":1719567931112,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2377/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878771338,"domain":"NeurIPS.cc/2024/Conference","replyto":"UXEo3uNNIX","id":"77qpdapT09","forumContent":{"TLDR":{"value":"Enhanced Multimodal Fusion with Coupled State Space Model"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","multi-modal fusion","multi-modal sentiment analysis"]},"supplementary_material":{"value":"/attachment/1af5552642a4f1fdcea9c586b54a5450f6c0c432.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"The essence of multi-modal fusion lies in exploiting the complementary information inherent in diverse modalities.However, most prevalent fusion methods rely on traditional neural architectures and are inadequately equipped to capture the dynamics of interactions across modalities, particularly in presence of complex intra- and inter-modality correlations.Recent advancements in State Space Models (SSMs), notably exemplified by the Mamba model, have emerged as promising contenders. Particularly, its state evolving process implies stronger modality fusion paradigm, making multi-modal fusion on SSMs an appealing direction. However, fusing multiple modalities is challenging for SSMs due to its hardware-aware parallelism designs. To this end, this paper proposes the Coupled SSM model, for coupling state chains of multiple modalities while maintaining independence of intra-modality state processes. Specifically, in our coupled scheme, we devise an inter-modal hidden states transition scheme, in which the current state is dependent on the states of its own chain and that of the neighbouring chains at the previous time-step. To fully comply with the hardware-aware parallelism, we obtain the global convolution kernel by deriving the state equation while introducing the historical state.Extensive experiments on CMU-MOSEI, CH-SIMS, CH-SIMSV2 through multi-domain input verify the effectiveness of our model compared to current state-of-the-art methods, improved F1-Score by 0.4%, 0.9%, and 2.3% on the three datasets respectively, 49% faster inference and 83.7% GPU memory save. The results demonstrate that Coupled Mamba model is capable of enhanced multi-modal fusion."},"_bibtex":{"value":"@inproceedings{\nli2024coupled,\ntitle={Coupled Mamba: Enhanced Multimodal Fusion with Coupled State Space Model},\nauthor={Wenbing Li and Hang Zhou and Junqing Yu and Zikai Song and Wei Yang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=UXEo3uNNIX}\n}"},"title":{"value":"Coupled Mamba: Enhanced Multimodal Fusion with Coupled State Space Model"},"pdf":{"value":"/pdf/c0967468999c3d38c161a8dd93682b7d1707ad5e.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"li|coupled_mamba_enhanced_multimodal_fusion_with_coupled_state_space_model"},"authorids":{"value":["~Wenbing_Li1","~Hang_Zhou12","~Junqing_Yu1","~Zikai_Song1","~Wei_Yang3"]},"authors":{"value":["Wenbing Li","Hang Zhou","Junqing Yu","Zikai Song","Wei Yang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.07487v4"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"zhang|motion_mamba_efficient_and_long_sequence_motion_generation_with_hierarchical_and_bidirectional_selective_ssm"},"authorids":{"value":["","https://dblp.org/search/pid/api?q=author:Akide_Liu:","https://dblp.org/search/pid/api?q=author:Ian_D._Reid_0001:","~Richard_Hartley1","~Bohan_Zhuang1","https://dblp.org/search/pid/api?q=author:Hao_Tang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.07487"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-07487,\n  publtype={informal},\n  author={Zeyu Zhang and Akide Liu and Ian D. Reid and Richard Hartley and Bohan Zhuang and Hao Tang},\n  title={Motion Mamba: Efficient and Long Sequence Motion Generation with Hierarchical and Bidirectional Selective SSM},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.07487},\n  url={https://doi.org/10.48550/arXiv.2403.07487}\n}\n"},"abstract":{"value":"Human motion generation stands as a significant pursuit in generative computer vision, while achieving long-sequence and efficient motion generation remains challenging. Recent advancements in state space models (SSMs), notably Mamba, have showcased considerable promise in long sequence modeling with an efficient hardware-aware design, which appears to be a promising direction to build motion generation model upon it. Nevertheless, adapting SSMs to motion generation faces hurdles since the lack of a specialized design architecture to model motion sequence. To address these challenges, we propose Motion Mamba, a simple and efficient approach that presents the pioneering motion generation model utilized SSMs. Specifically, we design a Hierarchical Temporal Mamba (HTM) block to process temporal data by ensemble varying numbers of isolated SSM modules across a symmetric U-Net architecture aimed at preserving motion consistency between frames. We also design a Bidirectional Spatial Mamba (BSM) block to bidirectionally process latent poses, to enhance accurate motion generation within a temporal frame. Our proposed method achieves up to 50% FID improvement and up to 4 times faster on the HumanML3D and KIT-ML datasets compared to the previous best diffusion-based method, which demonstrates strong capabilities of high-quality long sequence motion modeling and real-time human motion generation. See project website https://steve-zeyu-zhang.github.io/MotionMamba/"},"title":{"value":"Motion Mamba: Efficient and Long Sequence Motion Generation with Hierarchical and Bidirectional Selective SSM"},"authors":{"value":["Zeyu Zhang","Akide Liu","Ian D. Reid","Richard Hartley","Bohan Zhuang","Hao Tang"]}},"tmdate":1767793104985,"pdate":1704067200000,"tcdate":1727455842772,"writers":["~"],"signatures":["~Zeyu_Zhang11"],"forum":"BL8v5xdGAk","license":"CC BY-SA 4.0","number":90156,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1767793104985,"domain":"DBLP.org","id":"BL8v5xdGAk","version":2},{"content":{"summary":{"value":"This paper introduces \"Factorization Memory,\" a novel recurrent neural network (RNN) architecture designed to combine the strengths of Transformers and existing RNNs like Mamba-2. The core innovation lies in its ability to handle long-context language modeling tasks efficiently while maintaining competitive performance on short-context tasks. Factorization Memory achieves this by employing a 2-dimensional recurrent state with sparse updates, selectively activating only a subset of memory states at each time step. This sparse formulation aims to reduce computational and memory overhead during training and inference, allowing for larger recurrent states without sacrificing efficiency. The paper presents empirical analyses comparing Factorization Memory to Transformer and Mamba-2 architectures, highlighting its superior generalization in long-context scenarios and higher inference efficiency due to sparse updates."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"see Weaknesses"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"1. Factorization Memory offers a genuinely novel approach to address the long-standing challenge of long-context understanding in RNNs. By introducing a 2D memory state with sparse updates, it attempts to reconcile the trade-off between memory capacity and computational efficiency.\n2. The concept of sparse memory activation, where only a subset of recurrent states are updated at each step, is a significant strength. This directly tackles the computational and memory bottlenecks of dense recurrent states, especially as models scale. The paper explicitly states that this is \"the first RNN architecture that successfully combines sparse memory activation with competitive performance.\"\n3. The empirical results (e.g., Figure 4) show that Factorization Memory exhibits better extrapolation capabilities in long-context settings compared to both Transformer and Mamba-2, which is a critical advantage for handling very long sequences.\n4. The paper demonstrates that Factorization Memory achieves higher inference efficiency, particularly on long contexts, outperforming both Transformer (due to its quadratic complexity) and Mamba-2 (due to its sparse updates). The release of optimized CUDA/Triton kernels further supports this claim and aids reproducibility."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. \"MoM: Linear Sequence Modeling with Mixture-of-Memories\" is a good work also focus on sparsely expanding RNN memory states, but this paper did not compare with it. I recommend the authors to compare their algorithm difference and experimental performance.\n2. The primary limitation stated by the authors themselves is the constraint of computational resources, leading to investigations on \"relatively small-scale models with a low FLOPS budget\" (60-70 million parameters for memory scaling, 1B parameters for main results). While the findings provide insights, the generalization to much larger models (e.g., tens or hundreds of billions of parameters) and more complex evaluations remains an open question.\n3. While Figure 1 (Right) illustrates \"only selected top-k states are updated\" and Section 3.2 mentions k as a configurable parameter, there isn't an extensive ablation study or deeper analysis on the optimal choice of k or its sensitivity for various tasks and model sizes beyond the general statement that \"in the ideal case we want k as small as possible.\" Figure 5 only focuses on m and τ.\n4. While the mathematical formulations are provided, a clearer, more intuitive explanation or a block diagram detailing how the \"sparse mask\" is applied and how at acts as a \"router\" for both memory read and write could enhance understanding for readers less familiar with the specific mathematical notation.\n5. While the paper mentions Mixture of Experts (MoE) as orthogonal and touches upon memory-augmented Transformers, a more explicit comparison of Factorization Memory's sparse update mechanism to other sparsity-inducing or memory-aware RNNs (e.g., those using external memory) would strengthen the discussion of its unique advantages.\n6. The paper highlights inference efficiency as a strength. However, a more direct comparison of training time/costs between Factorization Memory, Transformer, and Mamba-2, beyond just FLOPS, would be beneficial. While sparse updates reduce FLOPs, the overhead of selecting top-k states could introduce other costs.\n7. Figure 3a indicates that Factorization Memory requires \"slightly more compute to achieve comparable test loss on 1024-token window\" compared to Transformer and Mamba-2. While it surpasses them in extrapolation, this initial \"cost\" for comparable performance needs a more thorough explanation, especially in light of the overall efficiency claims.\n8. The term \"Factorization Memory\" implies a particular structure or operation. While at being a conditional probability distribution that \"factorizes the input into a few selected states\" is mentioned, further elaboration on the \"factorization\" aspect and how it contributes to the model's capabilities would be useful."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926940805,"tcdate":1761991287677,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16914/Reviewer_fb5C"],"signatures":["ICLR.cc/2026/Conference/Submission16914/Reviewer_fb5C"],"forum":"LPqjfqw2OI","number":4,"license":"CC BY 4.0","cdate":1761991287677,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16914/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926940805,"domain":"ICLR.cc/2026/Conference","replyto":"LPqjfqw2OI","id":"oqXNnuzwEw","forumContent":{"TLDR":{"value":"We propose Factorization Memory (FM), a novel and efficient recurrent neural network (RNN) architecture that achieves performance comparable to Transformer models."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["llm","rnn","factorization memory"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"We propose Factorization Memory, an efficient recurrent neural network (RNN) architecture that achieves performance comparable to Transformer models on short-context language modeling tasks while also demonstrating superior generalization in long-context scenarios. Our model builds upon Mamba-2, enabling Factorization Memory to exploit parallel computations during training while preserving constant computational and memory complexity during inference. To further optimize model efficiency and representational capacity, we develop a sparse formulation of Factorization Memory that updates only a subset of recurrent states at each step while preserving the strong performance of its dense counterpart. To our knowledge, this represents the first RNN architecture that successfully combines sparse memory activation with competitive performance across both short and long-context settings. This work provides a systematic empirical analysis of Factorization Memory in comparison to Transformer and Mamba-2 architectures."},"_bibtex":{"value":"@misc{\nxiong2026language,\ntitle={Language Modeling With Factorization Memory},\nauthor={Lee Xiong and Maksim Tkachenko and Johanes Effendi and Ting Cai},\nyear={2026},\nurl={https://openreview.net/forum?id=LPqjfqw2OI}\n}"},"title":{"value":"Language Modeling With Factorization Memory"},"pdf":{"value":"/pdf/08058ce05792fde07379de57f8529a6f25058ca1.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"xiong|language_modeling_with_factorization_memory"},"authorids":{"value":["~Lee_Xiong1","~Maksim_Tkachenko1","~Johanes_Effendi1","~Ting_Cai2"]},"authors":{"value":["Lee Xiong","Maksim Tkachenko","Johanes Effendi","Ting Cai"]}},"version":2},{"content":{"TLDR":{"value":"We propose a new algorithm for deep generative modeling of sequence data in continuous spaces based on a novel adaptation of operator theory for probabilistic dynamical systems."},"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["sequence modeling","time series","hidden Markov models","mean embeddings","linear operators","maximum mean discrepancy","gradient flows"]},"primary_area":{"value":"deep_learning"},"abstract":{"value":"A key question in sequence modeling with neural networks is how to represent and learn highly nonlinear and probabilistic state dynamics. Operator theory views such dynamics as linear maps on Hilbert spaces containing mean embedding vectors of distributions, offering an appealing but currently overlooked perspective. We propose a new approach to sequence modeling based on an operator-theoretic view of a hidden Markov model (HMM). Instead of materializing stochastic recurrence, we embed the full sequence distribution as a tensor in the product Hilbert space. A generative process is then defined as maximum mean discrepancy (MMD) gradient flow in the space of sequences. To overcome challenges with large tensors and slow sampling convergence, we introduce spectral mean flows, a novel tractable algorithm integrating two core concepts. First, we propose a new neural architecture by leveraging spectral decomposition of linear operators to derive a scalable tensor network decomposition of sequence mean embeddings. Second, we extend MMD gradient flows to time-dependent Hilbert spaces and connect them to flow matching via the continuity equation, enabling simulation-free learning and faster sampling. We demonstrate competitive results on a range of time-series modeling datasets."},"_bibtex":{"value":"@inproceedings{\nkim2025sequence,\ntitle={Sequence Modeling with Spectral Mean Flows},\nauthor={Jinwoo Kim and Max Beier and Petar Bevanda and Nayun Kim and Seunghoon Hong},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=e2WesV6Voe}\n}"},"title":{"value":"Sequence Modeling with Spectral Mean Flows"},"pdf":{"value":"/pdf/e2da96c07da31b393b4375f92e15fe992d60a772.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"kim|sequence_modeling_with_spectral_mean_flows"},"authorids":{"value":["~Jinwoo_Kim4","~Max_Beier1","~Petar_Bevanda1","~Nayun_Kim1","~Seunghoon_Hong2"]},"authors":{"value":["Jinwoo Kim","Max Beier","Petar Bevanda","Nayun Kim","Seunghoon Hong"]}},"tmdate":1783627311231,"pdate":1758217326161,"tcdate":1746973134902,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission21718/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission21718/Authors"],"forum":"e2WesV6Voe","license":"CC BY 4.0","number":21718,"cdate":1746973134902,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission21718/-/Full_Submission","NeurIPS.cc/2025/Conference/Submission21718/-/Supplementary_Material","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission21718/-/Camera_Ready_Revision"],"mdate":1783627311231,"odate":1761705015026,"domain":"NeurIPS.cc/2025/Conference","id":"e2WesV6Voe","version":2},{"content":{"TLDR":{"value":"We investigate the theoretical properties of selective state-space layers via multivariate polynomials and demonstrate that they have superior expressivity and generalization compared to transformers, particularly for long-range tasks."},"venue":{"value":"AISTATS 2026 Poster"},"abstract":{"value":"Recent advances in efficient sequence modeling have introduced selective state-space layers, a key component of the Mamba architecture, which have demonstrated remarkable success in a wide range of NLP and vision tasks. While Mamba’s empirical performance has matched or surpassed SoTA transformers on such diverse benchmarks, the theoretical foundations underlying its powerful representational capabilities remain less explored. In this work, we investigate the expressivity of selective state-space layers using multivariate polynomials, and prove that they surpass linear transformers in expressiveness. Consequently, our findings reveal that Mamba offers superior representational power over linear attention-based models for long-sequences, while not sacrificing their generalization. Our theoretical insights are validated by a comprehensive set of empirical experiments on various datasets."},"_bibtex":{"value":"@inproceedings{\ncohen-karlik2026on,\ntitle={On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach},\nauthor={Edo Cohen-Karlik and Itamar Zimerman and Liane Galanti and Ido Andrew Atad and Amir Globerson and Lior Wolf},\nbooktitle={The 29th International Conference on Artificial Intelligence and Statistics},\nyear={2026},\nurl={https://openreview.net/forum?id=9vVYKBQF1N}\n}"},"title":{"value":"On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach"},"pdf":{"value":"/pdf/fa0f6b06c5da4788b21fb1796b188d95be817fa5.pdf"},"Signed_copyright_form":{"value":"/attachment/612030573d2c56f960897d5402d17d1e5a777522.pdf"},"venueid":{"value":"aistats.org/AISTATS/2026/Conference"},"paperhash":{"value":"cohenkarlik|on_the_expressivity_of_selective_statespace_layers_a_multivariate_polynomial_approach"},"authorids":{"value":["~Edo_Cohen-Karlik1","~Itamar_Zimerman1","~Liane_Galanti1","~Ido_Andrew_Atad1","~Amir_Globerson1","~Lior_Wolf1"]},"format_confirmation":{"value":"I agree that I have read and followed the formatting instructions for the camera ready version."},"code_dataset_promise":{"value":"No"},"authors":{"value":["Edo Cohen-Karlik","Itamar Zimerman","Liane Galanti","Ido Andrew Atad","Amir Globerson","Lior Wolf"]}},"tmdate":1784233588767,"pdate":1770123824099,"tcdate":1757281626854,"writers":["aistats.org/AISTATS/2026/Conference","aistats.org/AISTATS/2026/Conference/Submission61/Authors"],"signatures":["aistats.org/AISTATS/2026/Conference/Submission61/Authors"],"forum":"9vVYKBQF1N","license":"CC BY 4.0","number":61,"cdate":1757281626854,"readers":["everyone"],"invitations":["aistats.org/AISTATS/2026/Conference/-/Submission","aistats.org/AISTATS/2026/Conference/-/Post_Submission","aistats.org/AISTATS/2026/Conference/Submission61/-/Full_Submission","aistats.org/AISTATS/2026/Conference/-/PC_Revision","aistats.org/AISTATS/2026/Conference/-/Edit","aistats.org/AISTATS/2026/Conference/Submission61/-/Camera_Ready_Revision"],"mdate":1784233588767,"odate":1770123824099,"domain":"aistats.org/AISTATS/2026/Conference","id":"9vVYKBQF1N","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2502.02209v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"cohenkarlik|on_the_expressivity_of_selective_statespace_layers_a_multivariate_polynomial_approach"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Edo_Cohen-Karlik:","https://dblp.org/search/pid/api?q=author:Itamar_Zimerman:","https://dblp.org/search/pid/api?q=author:Liane_Galanti:","https://dblp.org/search/pid/api?q=author:Ido_Atad:","~Amir_Globerson1","https://dblp.org/search/pid/api?q=author:Lior_Wolf:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2502.02209"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2502-02209,\n  publtype={informal},\n  author={Edo Cohen-Karlik and Itamar Zimerman and Liane Galanti and Ido Atad and Amir Globerson and Lior Wolf},\n  title={On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach},\n  year={2025},\n  month={February},\n  cdate={1738368000000},\n  journal={CoRR},\n  volume={abs/2502.02209},\n  url={https://doi.org/10.48550/arXiv.2502.02209}\n}\n"},"abstract":{"value":"Recent advances in efficient sequence modeling have introduced selective state-space layers, a key component of the Mamba architecture, which have demonstrated remarkable success in a wide range of NLP and vision tasks. While Mamba's empirical performance has matched or surpassed SoTA transformers on such diverse benchmarks, the theoretical foundations underlying its powerful representational capabilities remain less explored. In this work, we investigate the expressivity of selective state-space layers using multivariate polynomials, and prove that they surpass linear transformers in expressiveness. Consequently, our findings reveal that Mamba offers superior representational power over linear attention-based models for long sequences, while not sacrificing their generalization. Our theoretical insights are validated by a comprehensive set of empirical experiments on various datasets."},"title":{"value":"On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach"},"authors":{"value":["Edo Cohen-Karlik","Itamar Zimerman","Liane Galanti","Ido Atad","Amir Globerson","Lior Wolf"]}},"tmdate":1742480376834,"pdate":1735689600000,"tcdate":1742480363409,"writers":["~"],"signatures":["~Amir_Globerson1"],"forum":"t6ccHXACbH","license":"CC BY-SA 4.0","number":370426,"cdate":1738368000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1742480376834,"domain":"DBLP.org","id":"t6ccHXACbH","version":2},{"content":{"venue":{"value":"TIME 2018"},"pdf":{"value":"https://drops.dagstuhl.de/storage/00lipics/lipics-vol120-time2018/LIPIcs.TIME.2018.6/LIPIcs.TIME.2018.6.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2018"},"paperhash":{"value":"bozzelli|results_on_alternatingtime_temporal_logics_with_linear_past"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Laura_Bozzelli:","~Aniello_Murano1","https://dblp.org/search/pid/api?q=author:Loredana_Sorrentino:"]},"html":{"value":"https://doi.org/10.4230/LIPIcs.TIME.2018.6"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/BozzelliMS18,\n  author={Laura Bozzelli and Aniello Murano and Loredana Sorrentino},\n  title={Results on Alternating-Time Temporal Logics with Linear Past},\n  year={2018},\n  cdate={1514764800000},\n  pages={6:1-6:22},\n  url={https://doi.org/10.4230/LIPIcs.TIME.2018.6},\n  booktitle={TIME},\n  crossref={conf/time/2018}\n}\n"},"abstract":{"value":"We investigate the succinctness gap between two known equally-expressive and different linear-past extensions of standard CTL^* (resp., ATL^*). We establish by formal non-trivial arguments that the \"memoryful\" linear-past extension (the history leading to the current state is taken into account) can be exponentially more succinct than the standard \"local\" linear-past extension (the history leading to the current state is forgotten). As a second contribution, we consider the ATL-like fragment, denoted ATL_{lp}, of the known \"memoryful\" linear-past extension of ATL^{*}. We show that ATL_{lp} is strictly more expressive than ATL, and interestingly, it can be exponentially more succinct than the more expressive logic ATL^{*}. Moreover, we prove that both satisfiability and model-checking for the logic ATL_{lp} are Exptime-complete."},"title":{"value":"Results on Alternating-Time Temporal Logics with Linear Past"},"authors":{"value":["Laura Bozzelli","Aniello Murano","Loredana Sorrentino"]}},"tmdate":1747299225904,"pdate":1514764800000,"tcdate":1747299214238,"writers":["~"],"signatures":["~Aniello_Murano1"],"forum":"vnpiT9DDvU","license":"CC BY-SA 4.0","number":480343,"cdate":1514764800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747299225904,"domain":"DBLP.org","id":"vnpiT9DDvU","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.15083v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"wang|fourierkanmamba_a_novel_statespace_equation_approach_for_timeseries_anomaly_detection"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xiancheng_Wang:","https://dblp.org/search/pid/api?q=author:Lin_Wang:","https://dblp.org/search/pid/api?q=author:Rui_Wang:","https://dblp.org/search/pid/api?q=author:Zhibo_Zhang:","~Minghang_Zhao1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2511.15083"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-15083,\n  publtype={informal},\n  author={Xiancheng Wang and Lin Wang and Rui Wang and Zhibo Zhang and Minghang Zhao},\n  title={Fourier-KAN-Mamba: A Novel State-Space Equation Approach for Time-Series Anomaly Detection},\n  year={2025},\n  month={November},\n  cdate={1761955200000},\n  journal={CoRR},\n  volume={abs/2511.15083},\n  url={https://doi.org/10.48550/arXiv.2511.15083}\n}\n"},"abstract":{"value":"Time-series anomaly detection plays a critical role in numerous real-world applications, including industrial monitoring and fault diagnosis. Recently, Mamba-based state-space models have shown remarkable efficiency in long-sequence modeling. However, directly applying Mamba to anomaly detection tasks still faces challenges in capturing complex temporal patterns and nonlinear dynamics. In this paper, we propose Fourier-KAN-Mamba, a novel hybrid architecture that integrates Fourier layer, Kolmogorov-Arnold Networks (KAN), and Mamba selective state-space model. The Fourier layer extracts multi-scale frequency features, KAN enhances nonlinear representation capability, and a temporal gating control mechanism further improves the model's ability to distinguish normal and anomalous patterns. Extensive experiments on MSL, SMAP, and SWaT datasets demonstrate that our method significantly outperforms existing state-of-the-art approaches."},"title":{"value":"Fourier-KAN-Mamba: A Novel State-Space Equation Approach for Time-Series Anomaly Detection"},"authors":{"value":["Xiancheng Wang","Lin Wang","Rui Wang","Zhibo Zhang","Minghang Zhao"]}},"tmdate":1769476255994,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2511-15083"],"tcdate":1769476248577,"writers":["~"],"signatures":["~Minghang_Zhao1"],"forum":"Epp2XViHUs","license":"CC BY-SA 4.0","number":805415,"cdate":1761955200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769476255994,"domain":"DBLP.org","id":"Epp2XViHUs","version":2},{"content":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"supplementary_material":{"value":"/attachment/6ffbb90cd1e7825ee4bef566f610eccabfe566c2.zip"},"abstract":{"value":"The typical selective state-space model (SSM) of Mamba addresses several limitations of Transformers, such as quadratic computational complexity with sequence length and significant inference-time memory requirements due to the key-value cache. However, the growing size of Mamba models continues to pose training and deployment challenges and raises concerns due to considerable training and inference compute consumption. In this work, we introduce Bi-Mamba, a scalable and powerful 1-bit Mamba architecture designed for more efficient large language models with multiple sizes across 780M, 1.3B, and 2.7B. Bi-Mamba models are trained from scratch on data volume as regular LLM pertaining using an autoregressive distillation loss. Extensive experimental results on language modeling demonstrate that Bi-Mamba achieves performance comparable to its full-precision counterparts (e.g., FP16 or BF16) and much better accuracy than post-training-binarization (PTB) Mamba and binarization-aware training (BAT) Transformer baselines, while significantly reducing memory footprint and computational consumption compared to the original Mamba model. Our study pioneers a new linear computational complexity LLM framework under low-bit representation and facilitates future design of specialized hardware tailored for efficient 1-bit Mamba-based LLMs. Our code is provided in Supplementary Material and the pre-trained weights are available anonymously at https://drive.google.com/drive/folders/1jfk_TlDzFbER84ITvU2hOX2VyPC9H4MA?usp=sharing"},"_bibtex":{"value":"@article{\ntang2025bimamba,\ntitle={Bi-Mamba: Towards Accurate 1-Bit State Space Model},\nauthor={Shengkun Tang and Liqun Ma and Haonan Li and Mingjie Sun and Zhiqiang Shen},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=CKQ4AgoRQm},\nnote={}\n}"},"title":{"value":"Bi-Mamba: Towards Accurate 1-Bit State Space Model"},"changes_since_last_submission":{"value":"We have carefully addressed all concerns in our revised manuscript. These include:\n\n- *Performance comparisons with stronger baselines* such as SqueezeLLM and AQLM, as shown in Table 3.\n\n- *Additional results on more downstream tasks*, including HumanEval and GSM8K, as shown in Table 4.\n\n- *Analysis of actual speedup, memory, and energy consumption*, presented in Table 8 and Section 5.5.\n\n- *Complexity Analysis* Section added in method.\n\n- *Fairer experimental comparisons with FBI-LLM*, included in Table 3.\n\n- *Training cost analysis of Bi-Mamba objectives*, provided in Table 6.\n\n- *Weight distribution visualization for different training objectives*, depicted in Figure 7.\n\n- Clarification of *data types used during training, inference, and storage of Bi-Mamba*, elaborated in Section 5.4."},"pdf":{"value":"/pdf/d0fe04ae19b7e5c4af82d49797dc33d8146d38ff.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"tang|bimamba_towards_accurate_1bit_state_space_model"},"authorids":{"value":["~Shengkun_Tang1","~Liqun_Ma1","~Haonan_Li2","~Mingjie_Sun1","~Zhiqiang_Shen1"]},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=HxU0wSMZ0n"},"assigned_action_editor":{"value":"~Vladimir_Braverman1"},"authors":{"value":["Shengkun Tang","Liqun Ma","Haonan Li","Mingjie Sun","Zhiqiang Shen"]}},"tmdate":1760971895313,"pdate":1760971895266,"tcdate":1747916679780,"writers":["TMLR"],"signatures":["TMLR/Paper4910/Authors"],"forum":"CKQ4AgoRQm","license":"CC BY 4.0","number":4910,"cdate":1747916679780,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/Paper4910/-/Revision","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/Paper4910/-/Camera_Ready_Revision","TMLR/-/Accepted"],"mdate":1760971895313,"odate":1750090950627,"domain":"TMLR","id":"CKQ4AgoRQm","version":2},{"content":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Rejected by TMLR"},"abstract":{"value":"The typical selective state-space model (SSM) of Mamba addresses several limitations of Transformers, such as quadratic computational complexity with sequence length and significant inference-time memory requirements due to the key-value cache. However, the growing size of Mamba models continues to pose training and deployment challenges and raises environmental concerns due to considerable energy consumption. In this work, we introduce Bi-Mamba, a scalable and powerful 1-bit Mamba architecture designed for more efficient large language models with multiple sizes across 780M, 1.3B, and 2.7B. Bi-Mamba models are trained from scratch on data volume as regular LLM pertaining using an autoregressive distillation loss. Extensive experimental results on language modeling demonstrate that Bi-Mamba achieves performance comparable to its full-precision counterparts (e.g., FP16 or BF16) and much better accuracy than post-training-binarization (PTB) Mamba and binarization-aware training (BAT) Transformer baselines, while significantly reducing memory footprint and energy consumption compared to the original Mamba model. Our study pioneers a new linear computational complexity LLM framework under low-bit representation and facilitates  future design of specialized hardware tailored for efficient 1-bit Mamba-based LLMs. Our code is provided in supplementary material and the pre-trained weights are available anonymously at https://drive.google.com/drive/folders/1jfk_TlDzFbER84ITvU2hOX2VyPC9H4MA?usp=sharing."},"_bibtex":{"value":"@article{\nanonymous2025bimamba,\ntitle={Bi-Mamba: Towards Accurate 1-Bit State Space Model},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2025},\nurl={https://openreview.net/forum?id=HxU0wSMZ0n},\nnote={Rejected}\n}"},"title":{"value":"Bi-Mamba: Towards Accurate 1-Bit State Space Model"},"pdf":{"value":"/pdf/aa4aabcb131c45723166043f52a58bc07fb9deea.pdf"},"venueid":{"value":"TMLR/Rejected"},"assigned_action_editor":{"value":"~Binhang_Yuan1"}},"tmdate":1745248182288,"tcdate":1739285208391,"writers":["TMLR","TMLR/Paper4185/Authors"],"signatures":["TMLR/Paper4185/Authors"],"forum":"HxU0wSMZ0n","license":"CC BY 4.0","number":4185,"cdate":1739285208391,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/Paper4185/-/Revision","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/-/Rejected"],"mdate":1745248182288,"odate":1739321889055,"domain":"TMLR","id":"HxU0wSMZ0n","version":2},{"content":{"summary":{"value":"The paper proposes **WARP**, a framework that performs *linear recurrence in weight space* rather than in hidden-state space. At each step, the parameters of a small decoder network evolve linearly:\n\n$$\\omega_t = A \\omega_{t-1} + B(x_t - x_{t-1}), \\qquad y_t = \\text{MLP}_{\\omega_t}(\\varepsilon)$$\n\nwhere \\(A, B\\) are learned transition matrices and $\\varepsilon$ encodes position or context.  \n\nConceptually, this shifts recurrence from feature dynamics to parameter dynamics. It blends (i) the efficiency of linear RNNs/SSMs (e.g., S4, Mamba), (ii) the expressivity of nonlinear decoders, and (iii) in-context or gradient-free adaptation through weight evolution. The model is tested across image completion, time-series forecasting, dynamical-system reconstruction, and classification."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"• Include direct comparisons to LinOSS, FACTS, and Griffin.\n\n• Explore structured or low-rank A, B for scale.\n\n• Add runtime and memory tables.\n\n• Include a brief stability/spectral analysis.\n\n• Clarify what fundamentally distinguishes this from hypernetworks and fast-weight RNNs."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"S1. Original framing. The move to perform recurrence directly in parameter space is novel\nand quite elegant. It reads as a middle ground between hypernetworks and fast-weight\nRNNs, but with the analytical simplicity of a linear transition.\n\nS2. Range of results. The experiments span diverse domains - MNIST/CelebA completion,\nETT and PEMS forecasting, DSR, and UEA time-series classification. The UEA section\nis particularly strong: comparisons include modern SSMs like S5, Mamba, S6, NRDE,\nand NCDE, with WARP performing competitively across most tasks.\n\nS3. Interpretability and analogy. The weight updates via input differences evoke synaptic-\nplasticity rules, which gives the method a neat biological parallel and some explanatory\nappeal.\n\nS4. Presentation. The paper is clear, visually well-organized, and balances theory with\nintuition. Figures showing progressive reconstruction genuinely help convey how the recurrence behaves."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. Benchmark depth. While broad, the benchmark is missing some of the newer SSMs\nthat define the current frontier. In particular, LinOSS (Rusch & Rus, 2024)—an oscilla-\ntory, long-sequence SSM—is cited but not compared. Given that LinOSS, FACTS, and\nGriffin all outperform S4 and Mamba on long forecasting tasks, excluding them makes\nthe SoTA claim weaker.\n\nW2. Scalability. The transition matrix $A \\in \\mathbb{R}^{D_\\omega \\times D_\\omega}$ scales quadratically with the size of the decoder, which will quickly become impractical. No structured or low-rank variants are\nexplored.\n\nW3. Theory gap. The paper is mostly empirical. There’s no discussion of spectral properties,\nstability, or representational capacity of the linear map in weight space.\n\nW4. Domain imbalance. Some domains (especially physics and image experiments) use\nsmall or older baselines (ConvCNP, GRU, Transformer). More recent adaptive or physics-\ninformed baselines like Neural Context Flows (ICLR 2025) or ZEBRA (2024) would\nstrengthen those sections."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917573426,"tcdate":1762517894861,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4784/Reviewer_T7rz"],"signatures":["ICLR.cc/2026/Conference/Submission4784/Reviewer_T7rz"],"forum":"zHdKaF3ZM7","number":5,"license":"CC BY 4.0","cdate":1762517894861,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4784/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917573426,"domain":"ICLR.cc/2026/Conference","replyto":"zHdKaF3ZM7","id":"lmMAYKkBmE","forumContent":{"TLDR":{"value":"We introduce a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["physics-informed machine learning","weight-space learning","meta-learning","deep sequence modeling","linear recurrence","test-time training"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"We introduce WARP (**W**eight-space **A**daptive **R**ecurrent **P**rediction), a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into fixed-dimensional hidden states, WARP explicitly parametrizes its hidden state as the weights and biases of a distinct auxiliary neural network, and uses input differences to drive its recurrence. This brain-inspired formulation enables efficient gradient-free adaptation of the auxiliary network at test-time, in-context learning abilities, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state-of-the-art baselines on diverse classification tasks, featuring in the top three in 4 out of 6 real-world challenging datasets. Furthermore, extensive experiments across sequential image completion, multivariate time series forecasting, and dynamical system reconstruction demonstrate its expressiveness and generalization capabilities. Remarkably, a physics-informed variant of our model outperforms the next best model by more than 10x. Ablation studies confirm the architectural necessity of key components, solidifying weight-space linear RNNs as a transformative paradigm for adaptive machine intelligence."},"_bibtex":{"value":"@inproceedings{\nnzoyem2026weightspace,\ntitle={Weight-Space Linear Recurrent Neural Networks},\nauthor={Roussel Desmond Nzoyem and Nawid Keshtmand and Enrique Crespo-Fernandez and Idriss TSAYEM and Raul Santos-Rodriguez and David A.W. Barton and Tom Deakin},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zHdKaF3ZM7}\n}"},"title":{"value":"Weight-Space Linear Recurrent Neural Networks"},"pdf":{"value":"/pdf/c01f4c1dbdaf1c6e821cde510623f871a07c3404.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"nzoyem|weightspace_linear_recurrent_neural_networks"},"authorids":{"value":["~Roussel_Desmond_Nzoyem1","~Nawid_Keshtmand1","~Enrique_Crespo-Fernandez1","~Idriss_TSAYEM1","~Raul_Santos-Rodriguez1","~David_A.W._Barton1","~Tom_Deakin1"]},"authors":{"value":["Roussel Desmond Nzoyem","Nawid Keshtmand","Enrique Crespo-Fernandez","Idriss TSAYEM","Raul Santos-Rodriguez","David A.W. Barton","Tom Deakin"]}},"version":2},{"content":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"supplementary_material":{"value":"/attachment/62e66b6a21ab636b3ab233ff3c733762ba104f8b.pdf"},"abstract":{"value":"Selective State-Space Models (SSMs) such as Mamba have become central to long-sequence modeling. Still, their stability is poorly understood: their state-space coefficients are modulated online by a token-dependent gating signal, making the recurrence neither linear time-invariant nor classically nonlinear. We study continuous-time selective SSMs through passivity, dissipativity, and Input-to-State Stability (ISS), explicitly separating the selection signal $x(\\cdot)$ from the driving input $u(\\cdot)$. We obtain four results: exponential forgetting under strict dissipativity; a canonical $\\mathrm{AUC}_{\\mathrm{loc}}$ quadratic storage for the frozen-selection subsystem that accommodates discontinuous gating; a parametric LMI together with universal kernel constraints and \"irreversible forgetting\" under universal quadratic storage; and sufficient conditions for global ISS uniformly over admissible selection schedules. We then bridge to practice by deriving a sampled block LMI for the Mamba selective-scan core, which is used as a differentiable training-time regularizer. Across seven standard time-series datasets and four prediction horizons, the regularizer reduces sampled Mamba-core LMI violations by roughly 92% in 28/28 pairs at a clean-MSE cost of less than 0.018%. It improves internal Mamba passivity and state-norm diagnostics under injected perturbations. Our results turn classical control-theoretic tools into verifiable structural and training criteria for selective SSMs, while honestly scoping which guarantees transfer to a deep selective-scan architecture."},"_bibtex":{"value":"@article{\nzubic2026regularity,\ntitle={Regularity and Stability Properties of Selective {SSM}s with Discontinuous Gating},\nauthor={Nikola Zubic and Davide Scaramuzza},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=7Vav53cDeN},\nnote={}\n}"},"title":{"value":"Regularity and Stability Properties of Selective SSMs with Discontinuous Gating"},"changes_since_last_submission":{"value":"Accepted after rebuttal revision \"as is\"."},"pdf":{"value":"/pdf/348c094a1371853757a1a70f1e2b4549fe4adfb2.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"zubic|regularity_and_stability_properties_of_selective_ssms_with_discontinuous_gating"},"authorids":{"value":["~Nikola_Zubic1","~Davide_Scaramuzza1"]},"assigned_action_editor":{"value":"~Yoshinobu_Kawahara1"},"authors":{"value":["Nikola Zubic","Davide Scaramuzza"]}},"tmdate":1783581452283,"pdate":1783581452220,"tcdate":1771903568937,"writers":["TMLR"],"signatures":["TMLR/Paper7659/Authors"],"forum":"7Vav53cDeN","license":"CC BY 4.0","number":7659,"cdate":1771903568937,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/Paper7659/-/Revision","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/Paper7659/-/Camera_Ready_Revision","TMLR/-/Accepted"],"mdate":1783581452283,"odate":1773444923769,"domain":"TMLR","id":"7Vav53cDeN","version":2},{"content":{"summary":{"value":"This paper claims to provide the first theoretical study of the learning and generalization dynamics of the Mamba architecture, a selective state space model (SSM) that has recently emerged as an efficient alternative to Transformers. The authors analyze a simplified single-layer Mamba block with input-dependent gating followed by a two-layer MLP trained via gradient descent under two structured data regimes—majority-voting and locality-structured sequences containing both class-relevant and irrelevant tokens. They prove non-asymptotic sample complexity and convergence guarantees, showing that Mamba generalizes efficiently when class-relevant signals are strong or locally concentrated, and that its gating vector naturally aligns with informative features while suppressing noisy ones, formalizing its feature-selection role. Synthetic experiments align with theoretical insights."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Please address weaknesses above."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"The paper’s primary strength lies in its original theoretical treatment of Mamba, offering the first formal generalization and convergence analysis for selective state space models—an area that has so far been dominated by empirical studies. In terms of originality, it introduces a novel analytical framework for gated architectures trained with gradient descent, bridging ideas from feature-learning theory and state-space modeling, and extending prior analyses of Transformers to a fundamentally different recurrence-based mechanism. In terms of technical quality, the authors provide rigorous, non-asymptotic sample complexity and convergence bounds under well-defined data models, supported by clear mathematical reasoning and alignment with empirical trends. The paper could benefit from an improved exposition of the theoretical results. In general, the paper is well structured, with intuitive explanations preceding formal results. Regarding significance, the work fills a key theoretical gap by formalizing how Mamba’s gating acts as a feature selector that prioritizes class-relevant signals, thus deepening understanding of why such models generalize efficiently and under what data conditions they outperform attention-based architectures. While it is unclear whether this specific architecture will stand the test of time, this reviewer celebrates the introduction of rigorous analysis of architectural components."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While the paper makes a valuable theoretical contribution, several weaknesses limit its impact and clarity. First, the analysis is restricted to a highly simplified single-layer Mamba block and may not generalize to deeper or multi-head architectures used in practice; extending the framework to multi-layer dynamics or coupling between blocks would strengthen its relevance. While this is hard to do, authors could comment on whether these results extend to multi-layer settings. Second, the structured data assumptions (majority-voting and locality-structured) are somewhat idealized and may not capture the complexity of real-world sequences; providing evidence that these regimes approximate practical scenarios (e.g., via ablations on natural datasets) would improve the paper’s applicability. Third, the empirical validation is minimal—limited to synthetic data—and lacks comparisons to other theoretically analyzed architectures (e.g., gated linear attention), making it hard to assess the distinct benefits of Mamba’s gating mechanism. Fourth, while the paper claims to establish generalization guarantees, the proofs rely on strong assumptions such as balanced datasets, small token noise, and specific initialization schemes, which could limit robustness. Finally, the connection to prior theoretical work on Transformers and SSMs could be deepened by discussing how Mamba’s multiplicative gating fundamentally changes training dynamics beyond attention-style weighting. Strengthening these aspects would make the work more comprehensive and impactful."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931500626,"tcdate":1761975491469,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19648/Reviewer_rcjx"],"signatures":["ICLR.cc/2026/Conference/Submission19648/Reviewer_rcjx"],"forum":"hvpKqEYJjj","number":2,"license":"CC BY 4.0","cdate":1761975491469,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19648/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931500626,"domain":"ICLR.cc/2026/Conference","replyto":"hvpKqEYJjj","id":"DA2HQIkZBY","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Training Dynamics","Feature Learning","Generalization","Mamba","Learning Theory","Selective State Space Models"]},"supplementary_material":{"value":"/attachment/c9fcb2ad298f51c4efaa9f51d0c97c0844014e27.zip"},"primary_area":{"value":"learning theory"},"abstract":{"value":"The recent empirical success of Mamba and other selective state space models (SSMs) has renewed interest in non-attention architectures for sequence modeling, yet their theoretical foundations remain underexplored. We present a first-step analysis of generalization and learning dynamics for a simplified but representative Mamba block: a single-layer, single-head selective SSM with input-dependent gating, followed by a two-layer MLP trained via gradient descent (GD). Our study adopts a structured data model with tokens that include both class-relevant and class-irrelevant patterns under token-level noise and examines two canonical regimes: majority-voting and locality-structured data sequences. We prove that the model achieves guaranteed generalization by establishing non-asymptotic sample complexity and convergence rate bounds, which improve as the effective signal increases and the noise decreases.\nFurthermore, we show that the gating vector aligns with class-relevant features while ignoring irrelevant ones, thereby formalizing a feature-selection role similar to attention but realized through selective recurrence. Numerical experiments on synthetic data justify our theoretical results. Overall, our results provide principled insight into when and why Mamba-style selective SSMs learn efficiently, offering a theoretical counterpoint to Transformer-centric explanations."},"_bibtex":{"value":"@inproceedings{\nshandirasegaran2026a,\ntitle={A Theoretical Analysis of Mamba{\\textquoteright}s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models},\nauthor={Mugunthan Shandirasegaran and Hongkang Li and Songyang Zhang and Meng Wang and Shuai Zhang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=hvpKqEYJjj}\n}"},"title":{"value":"A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models"},"pdf":{"value":"/pdf/7ff125c33f63bfb0017d10d5e63ca5d19e737a02.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"shandirasegaran|a_theoretical_analysis_of_mambas_training_dynamics_filtering_relevant_features_for_generalization_in_state_space_models"},"authorids":{"value":["~Mugunthan_Shandirasegaran1","~Hongkang_Li1","~Songyang_Zhang2","~Meng_Wang4","~Shuai_Zhang6"]},"authors":{"value":["Mugunthan Shandirasegaran","Hongkang Li","Songyang Zhang","Meng Wang","Shuai Zhang"]}},"version":2},{"content":{"summary":{"value":"Sequence models, crucial for tasks like information extraction and reasoning, face a trade-off between recall quality (ability to use long-sequence information) and memory efficiency. Linear attention models, in theory, can expand this trade-off frontier beyond other architectures like softmax attention and state-space models, but are limited by hardware bottlenecks, particularly regarding the size of their recurrent states. Current linear attention implementations struggle to scale beyond a few megabytes due to register memory exhaustion and expensive register spills, leading to significant slowdowns.\n\nCYLON addresses this by partitioning the recurrent state across multiple GPU streaming multiprocessors (SMs) and asynchronously combining these partitions. This design leverages modern GPU features such as Tensor Memory Accelerators (TMA) for asynchronous I/O and reductions, and Tensor Cores for asynchronous multiplication. By ensuring each partition remains register-resident within an SM, CYLON maximizes locality and enables significantly larger effective state sizes. The paper demonstrates that this approach achieves up to 3x higher throughput compared to prior linear attention algorithms on both Hopper and Blackwell GPUs, and unlocks state sizes (e.g., ≥ 131 MB) that were previously inaccessible. CYLON-partitioned models maintain or exceed the quality of default linear attention models on language modeling and NLU tasks, effectively expanding the Pareto frontier for recall-efficiency."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"see Weaknesses"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper tackles a well-identified and significant challenge in linear attention: the inability to scale recurrent state sizes due to hardware memory limitations. This is a crucial problem for improving the recall quality of sequence models.\n2. CYLON is meticulously designed with a deep understanding of modern GPU architectures, leveraging specific features like TMAs and Tensor Cores for asynchronous operations. This hardware-software co-design is key to its performance gains.\n3. The reported 3x higher throughput and the ability to unlock much larger state sizes (≥ 131 MB) represent a substantial leap forward in the practical application of linear attention, particularly on modern GPUs.\n4. Despite the architectural changes for efficiency, CYLON-partitioned models match or exceed the quality of non-partitioned linear attention models on various language modeling and NLU tasks, indicating that efficiency gains do not come at the cost of performance.\n5. The paper includes theoretical proofs (Theorems 1, 2, 3) demonstrating that partitioning the state does not reduce the expressivity of linear attention and that the decomposition can be computationally efficient."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The asynchronous nature and distributed state management across SMs, while efficient, likely introduce significant complexity in implementation and debugging compared to traditional linear attention kernels.\n2. While the paper discusses using learned MLPs for feature maps to ensure diversity, the specifics of how these distinct feature maps are learned and their impact on model training dynamics could be explored further.\n3. The reliance on features like TMA and Warp-group matrix multiply means that CYLON's direct applicability might be limited to NVIDIA's Hopper and Blackwell architectures. While these are prevalent, performance on other hardware (e.g., AMD, custom AI accelerators) is not discussed.\n4. The argument for \"increasingly asynchronous machines\" is based on current high-end NVIDIA GPUs. While this trend is likely to continue, future architectural shifts could potentially alter the optimal strategies for handling recurrent states.\n5. While the paper evaluates language modeling and NLU tasks, a broader set of applications where long-context recall is paramount could further solidify CYLON's benefits.\n6. While the paper mentions \"long-latency roundtrips and inter-SM synchronization overhead,\" a more detailed breakdown of the specific overheads that CYLON successfully mitigates, perhaps with microbenchmark results for each component, could further strengthen the argument."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942901842,"tcdate":1761987562549,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission24023/Reviewer_9j4n"],"signatures":["ICLR.cc/2026/Conference/Submission24023/Reviewer_9j4n"],"forum":"va4FWa5TDN","number":3,"license":"CC BY 4.0","cdate":1761987562549,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission24023/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942901842,"domain":"ICLR.cc/2026/Conference","replyto":"va4FWa5TDN","id":"uL6sQRwHeD","forumContent":{"TLDR":{"value":"Hardware-aware linear attention algorithm for large state sizes."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["systems","recurrent","architectures","kernels","cuda"]},"supplementary_material":{"value":"/attachment/e29454619ea90326bcf1a9a03536ce2321ccf44c.zip"},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Sequence models face stark tradeoffs between recall quality and memory efficiency. Recall -- the ability to use information over long sequences -- is critical for sequence modeling tasks ranging from information extraction to reasoning. \nPrior work has shown that in theory, \\textit{linear} attention models with sufficient recurrent state sizes can expand the Pareto frontier of the recall-memory tradeoff space beyond alternative architectures such as softmax attention and state space models.\nHowever, it is difficult to scale the linear attention state size due to hardware bottlenecks. I/O aware algorithms store the linear attention states in thread registers, however state sizes beyond $\\approx 3$ megabytes exhaust register memory and trigger expensive register spills. \nIn this work, we introduce CYLON, a hardware-aware strategy for partitioning linear attention's recurrent state across the registers of multiple GPU processors and asynchronously combining the partitions. When applying CYLON to popular architectures, such as Hedgehog and Mamba-2, we unlock 3$\\times$ higher throughput compared to prior linear attention algorithms for these architectures on both Hopper and Blackwell GPUs. Finally, CYLON makes large states available to model designers by unlocking sizes (e.g., $\\geq$ 131 MB) that are not achievable by the existing linear attention kernels."},"_bibtex":{"value":"@misc{\nwaitz2026cylon,\ntitle={Cylon: Asynchronous Linear Attention},\nauthor={Alexander Waitz and Benjamin Frederick Spector and Christopher Re and Simran Arora},\nyear={2026},\nurl={https://openreview.net/forum?id=va4FWa5TDN}\n}"},"title":{"value":"Cylon: Asynchronous Linear Attention"},"pdf":{"value":"/pdf/7f2eee22004cc8e3675440d39926a9b5f5d263d2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"waitz|cylon_asynchronous_linear_attention"},"authorids":{"value":["~Alexander_Waitz1","~Benjamin_Frederick_Spector1","~Christopher_Re1","~Simran_Arora1"]},"authors":{"value":["Alexander Waitz","Benjamin Frederick Spector","Christopher Re","Simran Arora"]}},"version":2},{"content":{"comment":{"value":"> Would state-space-models like Mamba also be applicable in this area of applications?\n\n**Response**: Previous works on State-Space Models typically assume evenly spaced observations. In recent years, SSMs such as Mamba are a promising class of architectures for continuous/irregular time series modeling. By parameterizing a latent continuous dynamical system and using fast convolution-style implementations, they can model long-range temporal dependencies with good computational efficiency. In principle, they could be applied to the analysis of astrophysical data. Although still in its early age, prior work has shown results on challenging sequence tasks and several time-series tasks [1,2]. **However, these works still develop in a one-model-per-task trend.** Our focus in this work is not on proposing or comparing new sequence architectures trained from scratch, but on evaluating existing, large pre-trained time-series foundation models in a strict zero-shot setting on real astrophysical data. **At present, there does not exist a broadly pre-trained, publicly available Mamba-based TSFM that could be used in exactly the same “drop-in zero-shot embedding” fashion as Chronos or Moirai.** Applying Mamba would therefore require substantial additional model development and large-scale training, which is orthogonal to our main goal of assessing cross-domain transfer of current TSFMs.\n\nConceptually, our benchmark is architecture-agnostic: any future foundation model, whether Transformer-, SSM-, or hybrid-based, could be evaluated on StarEmbed by producing embeddings for our light curves and plugging them into the same downstream heads and metrics. We thus view state-space models like Mamba as an interesting direction for future work, and our benchmark is designed precisely so that such models can be fairly compared once suitable pre-trained versions become available.\n\n---\n\n[1] Gu, A., Goel, K., & Ré, C. (2021). Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396.\n\n[2] Zhang, M., Saab, K. K., Poli, M., Dao, T., Goel, K., & Ré, C. (2023). Effectively modeling time series with simple discrete state spaces."},"title":{"value":"Part 5 of 6: Relevance of State-space Models to StarEmbed"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763850878808,"tcdate":1763848536201,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22350/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission22350/Authors"],"forum":"ZXRlSzHoDE","number":5,"license":"CC BY 4.0","cdate":1763848536201,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22350/-/Official_Comment"],"mdate":1763850878808,"domain":"ICLR.cc/2026/Conference","replyto":"fGB7PzErF7","id":"mWjKkuHFNj","forumContent":{"TLDR":{"value":"We propose the first benchmark for time series foundation model on stellar time series observations."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["benchmark","astrophysics","light curve","photometrics","ZTF","periodic variables","time series foundation models","embedding models","classifcation"]},"supplementary_material":{"value":"/attachment/5ed8901a09a1b3dc28ac0f6f0e210397b6085140.zip"},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"Time series foundation models (TSFMs) are increasingly being adopted as highly-capable general-purpose time series representation learners. Although their training corpora are vast, they exclude astronomical time series data. Observations of stars produce peta-scale time series with unique challenges including irregular sampling and heteroskedasticity. We introduce $\\texttt{StarEmbed}$, the first public benchmark for rigorous and standardized evaluation of state-of-the-art TSFMs on stellar time series observations (``light curves''). We benchmark on three scientifically-motivated downstream tasks: unsupervised clustering, supervised classification, and out-of-distribution source detection. $\\texttt{StarEmbed}$ integrates a catalog of expert-vetted labels with multi-variate light curves from the Zwicky Transient Facility, yielding $\\sim$40k hand-labeled light curves spread across seven astrophysical classes. We evaluate the zero-shot representation capabilities of three TSFMs ($\\texttt{Moirai}$, $\\texttt{Chronos}$, $\\texttt{Chronos-Bolt}$) and a domain-specific transformer ($\\texttt{Astromer}$) against handcrafted feature extraction, the long-standing baseline in the astrophysics literature. Our results demonstrate that these TSFMs, especially the $\\texttt{Chronos}$ models, which are trained on data completely unlike the astronomical observations, can outperform established astrophysics-specific baselines in some tasks and effectively generalize to entirely new data. In particular, TSFMs deliver state-of-the-art performance on our out-of-distribution source detection benchmark. With the first benchmark of TSFMs on astronomical time series data, we test the limits of their generalization and motivate a paradigm shift in time-domain astronomy from using task-specific, fully supervised pipelines toward adopting generic foundation model representations for the analysis of peta-scale datasets from forthcoming observatories."},"_bibtex":{"value":"@misc{\nli2026starembed,\ntitle={StarEmbed: Benchmarking Time Series Foundation Models on Astronomical Observations of Variable Stars},\nauthor={Weijian Li and Hong-Yu Chen and Nabeel Rehemtulla and Ved G. Shah and Dennis Wu and Qinjie Lin and Adam A. Miller and Han Liu},\nyear={2026},\nurl={https://openreview.net/forum?id=ZXRlSzHoDE}\n}"},"title":{"value":"StarEmbed: Benchmarking Time Series Foundation Models on Astronomical Observations of Variable Stars"},"pdf":{"value":"/pdf/3d312977b9820b5a93034479a79f3f37417ce88f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"li|starembed_benchmarking_time_series_foundation_models_on_astronomical_observations_of_variable_stars"},"authorids":{"value":["~Weijian_Li2","~Hong-Yu_Chen1","~Nabeel_Rehemtulla1","~Ved_G._Shah1","~Dennis_Wu1","~Qinjie_Lin1","~Adam_A._Miller1","~Han_Liu4"]},"authors":{"value":["Weijian Li","Hong-Yu Chen","Nabeel Rehemtulla","Ved G. Shah","Dennis Wu","Qinjie Lin","Adam A. Miller","Han Liu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a novel backbone named Voxel Mamba for point cloud 3D detection. Different from previous methods that group the voxels into fixed-length sequences through padding, this paper adopt an interesting group-free strategy to sort all voxels into a single sequence through the space-filling curve. The paper also proposes the Asymmetrical State Space Models (ASSMs) and Implicit Window-based Position Embedding (IWPE) to enlarge the receptive fields of Voxel SSMs. \nExperiments are mostly based on challenging outdoor datasets, the Waymo Open Dataset and nuScene. Experiments demonstrates Voxel Mamba surpasses previous methods."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please see Weaknesses above. Overall, the proposed group-free strategy has better efficiency than previous group-based operations without token padding. Voxel Mamba also demonstrates better performance than existing sparse convolution or transformer structures. As such, I recommend acceptance at this stage."},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1.\tThe paper presents a novel approach to deal with irregularity problem of point cloud detection. By taking the advantage of linear complexity of SSM, this paper employ a group-free strategy which is more efficient and deployment-friendly than previous group-based methods.\n2.\tASSMs and IWPE are well motivated. The multi-scale design and implicit window partition are integrated under group-free design in an economical way. Besides, the figures in this paper are clear and greatly aid in the comprehension of the proposed methods.\n3.\tThe proposed Voxel Mamba exhibits a notably larger ERF than group-based methods DSVT.  The visualization effectively supports their claim.\n4.\tThe experiments are thorough and convincingly demonstrate the effectiveness of the proposed method."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tThe experiments of model architectures are a bit insufficient. In this paper, the authors extend Mamba to bidirectional Mamba (line 68) and further to Voxel Mamba (line 74). However, there are no experiments demonstrating the performance gains from these upgrades. As Voxel Mamba is based on Mamba, it would be necessary to report the accuracy of apply group-free bidirectional Mamba and Mamba with the same SSM parameters or layers.\n2.\tGiven that the group-free strategy is a significant contribution of this paper, it is necessary to include more analysis and comparisons between group-free and group-based operations. For example, the author cloud provide a detailed latency comparison between window-based grouping (such as dsvt and sst) and HIL. Besides, on Line 188, the authors mention that the mapping processes takes 0.7ms. However, the latency is likely to vary with different sequence lengths. Please provide a clarification regarding the timing.\n3.\tLack of comparison with non-group-free Mamba. Although this work provides quantitative and visualization comparison between group-free and group-based (dsvt) methods. It is necessary to provide the ERFs and detection accuracy for non-group-free Voxel Mamba. As a linear RNN models, what difference does Mamba introduce compared to Transformers.\n4.\tThe analysis of the ablation on ASSM and HIL is a little vague. Specifically, it is unclear why performances drop under the downsampling setting {2,2,2} and {4,4,4} in Table 5(b). Besides, the window sweep, Z-order, and Hilbert methods in Table 5(d) exhibit similar performance. Can I infer that the form of the space-filling curve does not significantly affect the accuracy? Please provide a discussion on them, which would be helpful to improve the quality of this paper.\n5.\tHilbert Input Layer (HIL) sort the voxels by querying the space-filling curve template. As presented in Sec. 3.3, form my understanding, the template needs cover the whole scenes. My concern is the amount of GPU memory required by the template under the current voxel size setting and its contribution to the total memory consumption. And for a more fine-grained size such as (0.1m, 0.1m, 0.15m) in CenterPoint."},"limitations":{"value":"Limitations have been included"}},"nonreaders":[],"tmdate":1730878757713,"tcdate":1720526384029,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2210/Reviewer_2cxV"],"signatures":["NeurIPS.cc/2024/Conference/Submission2210/Reviewer_2cxV"],"forum":"gHYhVSCtDH","number":3,"license":"CC BY 4.0","cdate":1720526384029,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2210/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878757713,"domain":"NeurIPS.cc/2024/Conference","replyto":"gHYhVSCtDH","id":"nnrtxASFNJ","forumContent":{"TLDR":{"value":"A simple and effective 3D backbone for point cloud based 3D object detection."},"venue":{"value":"NeurIPS 2024 spotlight"},"keywords":{"value":["Point Cloud based 3D Object Detection","State Space Models","Group-free"]},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Serialization-based methods, which serialize the 3D voxels and group them into multiple sequences before inputting to Transformers, have demonstrated their effectiveness in 3D object detection. However, serializing 3D voxels into 1D sequences will inevitably sacrifice the voxel spatial proximity. Such an issue is hard to be addressed by enlarging the group size with existing serialization-based methods due to the quadratic complexity of Transformers with feature sizes. Inspired by the recent advances of state space models (SSMs), we present a Voxel SSM, termed as Voxel Mamba, which employs a group-free strategy to serialize the whole space of voxels into a single sequence. The linear complexity of SSMs encourages our group-free design, alleviating the loss of spatial proximity of voxels. To further enhance the spatial proximity, we propose a Dual-scale SSM Block to establish a hierarchical structure, enabling a larger receptive field in the 1D serialization curve, as well as more complete local regions in 3D space. Moreover, we implicitly apply window partition under the group-free framework by positional encoding, which further enhances spatial proximity by encoding voxel positional information. Our experiments on Waymo Open Dataset and nuScenes dataset show that Voxel Mamba not only achieves higher accuracy than state-of-the-art methods, but also demonstrates significant advantages in computational efficiency. The source code is available at https://github.com/gwenzhang/Voxel-Mamba."},"_bibtex":{"value":"@inproceedings{\nzhang2024voxel,\ntitle={Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection},\nauthor={Guowen Zhang and Lue Fan and Chenhang HE and Zhen Lei and Zhaoxiang Zhang and Lei Zhang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=gHYhVSCtDH}\n}"},"title":{"value":"Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection"},"pdf":{"value":"/pdf/0cd37d7602d988cb025ea9d85f62abe79ef5139d.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"zhang|voxel_mamba_groupfree_state_space_models_for_point_cloud_based_3d_object_detection"},"authorids":{"value":["~Guowen_Zhang1","~Lue_Fan1","~Chenhang_HE1","~Zhen_Lei2","~Zhaoxiang_Zhang3","~Lei_Zhang2"]},"authors":{"value":["Guowen Zhang","Lue Fan","Chenhang HE","Zhen Lei","Zhaoxiang Zhang","Lei Zhang"]}},"version":2},{"content":{"summary":{"value":"In this paper, the authors propose the TIMBA model for multivariate time series imputation, integrating the Diffusion Probabilistic Model (DDPM) with State-Space Models (SSM), Graph Neural Networks (GNN), and other advanced techniques. By replacing time-oriented Transformers with SSMs and embedding the Selective State Space Models (S6) architecture within Mamba blocks, TIMBA effectively combines SSMs, GNNs, and node-based Transformers to enhance spatiotemporal representations."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1  Why Choose SSM over Time-Oriented Transformers?: Why does the model replace time-oriented transformers with State-Space Models (SSMs)? What makes SSMs better suited for time modeling compared to transformers?\n\n2  Overfitting Concerns: In the results from the PEMS-Bay dataset, the CSDI model using only the diffusion mechanism slightly outperforms the proposed model, which includes additional techniques such as GNN. Does this suggest that the proposed model might be overfitting to the spatio-temporal datasets?\n\n3  Ablation Study: The ablation study focuses only on TIMBA-Uni and TIMBA. What about the contributions of other components of the model? How do they impact performance individually?\n\n4  Latest Works in Time Series Imputation: In 2024, many papers on time series imputation have been published. I recommend incorporating more recent works in the introduction to provide a more comprehensive and up-to-date background.\nImputeFormer: Low rankness-induced transformers for generalizable spatiotemporal imputation; SADI: Similarity-Aware Diffusion Model-Based Imputation for Incomplete Temporal EHR Data; Higher-order Spatio-temporal Physics-incorporated Graph Neural Network for Multivariate Time Series Imputation; and other recent works.\n\n5  Training Strategy and Result Variations: Could the differences in results from prior works (e.g., PriSTI) be due to different training strategies, or are there other factors that could explain these variations?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"In this paper, the authors propose TIMBA, a novel model that replaces time-oriented transformers in diffusion models with bi-directional Mamba blocks. Through extensive benchmarking on three real-world datasets, TIMBA consistently outperforms or matches state-of-the-art models. The paper further assesses the model's robustness through an ablation study, sensitivity analysis to varying missing rates, and downstream task evaluation after data imputation."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1  Unclear Motivation: The paper combines several machine learning methods (e.g., Transformers, SSMs, diffusion models, GNNs), but the motivation behind each choice is unclear. There is little to no theoretical analysis to support the use of these models or to explain why this specific combination of techniques is beneficial.\n\n2  Lack of Comparison to Relevant Baselines: The experiments do not compare the proposed model to other state-of-the-art diffusion-based models like SSSD or diffusion with RNNs. This omission leaves uncertainty about whether the proposed TIMBA models truly offer better performance than existing methods.\n\n3 Results Discrepancies: Some of the results reported in the paper differ from those in previous works (e.g., PriSTI). However, the authors do not provide an explanation for these discrepancies. This raises concerns about whether the differences arise from the training strategy, model implementation, or some other factor.\n\n4 Inadequate Coverage of Time Series Imputation: While the paper claims to focus on time series imputation, the experiments primarily involve spatio-temporal datasets. There are no experiments on non-spatio-temporal datasets like clinical healthcare, which limits the generalizability of the model. This mismatch between the title and the scope of experiments weakens the overall contribution.\n\n5  Format Issues: Some tables in the paper are out of range, indicating formatting problems that need to be addressed to improve readability and presentation quality."}},"nonreaders":[],"tmdate":1731428682890,"tcdate":1729971035919,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6555/Reviewer_6mih"],"signatures":["ICLR.cc/2025/Conference/Submission6555/Reviewer_6mih"],"forum":"Sw10tbj0gM","number":2,"license":"CC BY 4.0","cdate":1729971035919,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6555/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428682890,"domain":"ICLR.cc/2025/Conference","replyto":"Sw10tbj0gM","id":"LSAvMDOG6I","forumContent":{"TLDR":{"value":"We introduce TIMBA, a novel model using Mamba blocks in place of Transformers for superior multivariate time series imputation across various benchmarks and datasets."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Multivariate Time Series Imputation","Diffusion models","Mamba","Space State Models","Generative Models","Graph Neural Networks","Transformers"]},"supplementary_material":{"value":"/attachment/00fc8f6bdb0e5659e4d633c09941288f8fbce065.zip"},"primary_area":{"value":"generative models"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The problem of imputing multivariate time series spans a wide range of fields, from clinical healthcare to multi-sensor systems. Initially, Recurrent Neural Networks (RNNs) were employed for this task; however, their error accumulation issues led to the adoption of Transformers, leveraging attention mechanisms to mitigate these problems. Concurrently, the promising results of diffusion models in capturing original distributions have positioned them at the forefront of current research, often in conjunction with Transformers. In this paper, we propose replacing time-oriented Transformers with State-Space Models (SSM), which are better suited for temporal data modeling. Specifically, we utilize the latest SSM variant, S6, which incorporates attention-like mechanisms. By embedding S6 within Mamba blocks, we develop a model that integrates SSM, Graph Neural Networks, and node-oriented Transformers to achieve enhanced spatiotemporal representations. Implementing these architectural modifications, previously unexplored in this field, we present Time series Imputation with Bi-directional mamba blocks and diffusion models (TIMBA). TIMBA achieves superior performance in almost all benchmark scenarios and performs comparably in others across a diverse range of missing value situations and three real-world datasets. We also evaluate how the performance of our model varies with different amounts of missing values and analyse its performance on downstream tasks. In addition, we provide the original code to replicate the results."},"_bibtex":{"value":"@misc{\nsol{\\'\\i}s-garc{\\'\\i}a2024timba,\ntitle={{TIMBA}: Time series Imputation with Bi-directional Mamba Blocks and Diffusion models},\nauthor={Javier Sol{\\'\\i}s-Garc{\\'\\i}a and Bel{\\'e}n Vega-M{\\'a}rquez and Juan A. Nepomuceno and Isabel A. Nepomuceno-Chamorro},\nyear={2024},\nurl={https://openreview.net/forum?id=Sw10tbj0gM}\n}"},"title":{"value":"TIMBA: Time series Imputation with Bi-directional Mamba Blocks and Diffusion models"},"pdf":{"value":"/pdf/d22f8aa876bfcdf7a12b5d82cb657b4429e0290c.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"solísgarcía|timba_time_series_imputation_with_bidirectional_mamba_blocks_and_diffusion_models"},"authorids":{"value":["~Javier_Solís-García1","~Belén_Vega-Márquez1","~Juan_A._Nepomuceno1","~Isabel_A._Nepomuceno-Chamorro1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Javier Solís-García","Belén Vega-Márquez","Juan A. Nepomuceno","Isabel A. Nepomuceno-Chamorro"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2501.15461v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"he|mambabased_graph_convolutional_networks_tackling_oversmoothing_with_selective_state_space"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xin_He_0003:","https://dblp.org/search/pid/api?q=author:Yili_Wang_0004:","~Wenqi_Fan1","https://dblp.org/search/pid/api?q=author:Xu_Shen_0002:","https://dblp.org/search/pid/api?q=author:Xin_Juan:","https://dblp.org/search/pid/api?q=author:Rui_Miao_0003:","https://dblp.org/search/pid/api?q=author:Xin_Wang_0035:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2501.15461"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2501-15461,\n  publtype={informal},\n  author={Xin He and Yili Wang and Wenqi Fan and Xu Shen and Xin Juan and Rui Miao and Xin Wang},\n  title={Mamba-Based Graph Convolutional Networks: Tackling Over-smoothing with Selective State Space},\n  year={2025},\n  month={January},\n  cdate={1735689600000},\n  journal={CoRR},\n  volume={abs/2501.15461},\n  url={https://doi.org/10.48550/arXiv.2501.15461}\n}\n"},"abstract":{"value":"Graph Neural Networks (GNNs) have shown great success in various graph-based learning tasks. However, it often faces the issue of over-smoothing as the model depth increases, which causes all node representations to converge to a single value and become indistinguishable. This issue stems from the inherent limitations of GNNs, which struggle to distinguish the importance of information from different neighborhoods. In this paper, we introduce MbaGCN, a novel graph convolutional architecture that draws inspiration from the Mamba paradigm-originally designed for sequence modeling. MbaGCN presents a new backbone for GNNs, consisting of three key components: the Message Aggregation Layer, the Selective State Space Transition Layer, and the Node State Prediction Layer. These components work in tandem to adaptively aggregate neighborhood information, providing greater flexibility and scalability for deep GNN models. While MbaGCN may not consistently outperform all existing methods on each dataset, it provides a foundational framework that demonstrates the effective integration of the Mamba paradigm into graph representation learning. Through extensive experiments on benchmark datasets, we demonstrate that MbaGCN paves the way for future advancements in graph neural network research."},"title":{"value":"Mamba-Based Graph Convolutional Networks: Tackling Over-smoothing with Selective State Space"},"authors":{"value":["Xin He","Yili Wang","Wenqi Fan","Xu Shen","Xin Juan","Rui Miao","Xin Wang"]}},"tmdate":1768448125081,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2501-15461"],"tcdate":1768448019756,"writers":["~"],"signatures":["~Wenqi_Fan1"],"forum":"kaPXz0edmc","license":"CC BY-SA 4.0","number":754675,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1768448125081,"domain":"DBLP.org","id":"kaPXz0edmc","version":2},{"content":{"summary":{"value":"The paper proposes the Visual 2-Dimensional Mamba (V2M) framework, an innovative model for image representation learning. Unlike traditional Mamba models that process image data in a 1D sequence, V2M adapts a 2D state space model to retain spatial structure, preserving local relationships between pixels. This 2D adaptation allows V2M to process image patches as 2D entities directly, thereby enhancing locality and coherence in visual representation."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Could you add more efficiency experiments and ablation studies to this paper and provide a more thorough discussion based on those findings? \n\nAdditionally, could you offer a more detailed analysis and explanation of the simplifications mentioned in Section 3.2?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"This paper innovatively introduces a 2D state space to address the spatial coherence issues faced by SSM models in the visual domain, achieving promising results."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.Novelty\n\nThe novelty is moderate, as there has already been substantial research, such as VMamba[1], exploring how sequence models can preserve 2D structural information in visual tasks.\n\n2.Implementation\n\nWhile this paper introduces a theory of 2D state space to address coherence issues in visual tasks,  its implementation relies on excessive simplification, lacking a detailed explanation of the rationale behind this approach and its potential implications.\n\n3.Performance\n\nThe performance demonstrated in this paper shows only minor gains over early baselines like Vim[2] and VMamba[1], without advantages over more recent models like EfficientVMamba[3].\n\n4.Experiment\n\nIt is foreseeable that the proposed method will impact speed and memory usage; however, the paper lacks sufficient efficiency tests, memory usage tests, and ablation studies, resulting in limited experimental support.\n\n5.Presentation\n\nThe expression of the formulas is somewhat confusing, such as the approach used to split Equation 7 into Equations 9 and 10, and the unclear logical progression from Equation 11 to Equation 12.\n\n[1]VMamba: Visual State Space Model. Proceedings of the 38th Conference on Neural Information Processing Systems.\n\n[2]Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model. Proceedings of the 41st International Conference on Machine Learning.\n\n[3]EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba. arXiv preprint arXiv:2403.09977."}},"nonreaders":[],"tmdate":1731427657932,"tcdate":1730649650643,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2798/Reviewer_7MsE"],"signatures":["ICLR.cc/2025/Conference/Submission2798/Reviewer_7MsE"],"forum":"FowFLhUTgO","number":2,"license":"CC BY 4.0","cdate":1730649650643,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2798/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427657932,"domain":"ICLR.cc/2025/Conference","replyto":"FowFLhUTgO","id":"fROVe3R8hX","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Visual Backbone","Representation Learning"]},"supplementary_material":{"value":"/attachment/5af7971476d880f0927c713ccfbd3afb094e976e.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has garnered widespread attention due to its flexible design and efficient hardware performance to process 1D sequences based on the state space model (SSM). Recent studies have attempted to apply Mamba to the visual domain by flattening 2D images into patches and then regarding them as a 1D sequence. To compensate for the 2D structure information loss (e.g., local similarity) of the original image, most existing methods focus on designing different orders to sequentially process the tokens, which could only alleviate this issue to some extent. In this paper, we propose a Visual 2-Dimensional Mamba (V2M) model as a complete solution, which directly processes image tokens in the 2D space. We first generalize SSM to the 2-dimensional space which generates the next state considering two adjacent states on both dimensions (e.g., columns and rows). We then construct our V2M based on the 2-dimensional SSM formulation and incorporate Mamba to achieve hardware-efficient parallel processing. The proposed V2M effectively incorporates the 2D locality prior yet inherits the efficiency and input-dependent scalability of Mamba. Extensive experimental results on ImageNet classification and downstream visual tasks including object detection and instance segmentation on COCO and semantic segmentation on ADE20K demonstrate the effectiveness of our V2M compared with other visual backbones."},"_bibtex":{"value":"@misc{\nwang2025vm,\ntitle={V2M: Visual 2-Dimensional Mamba for Image Representation Learning},\nauthor={Chengkun Wang and Wenzhao Zheng and Yuanhui Huang and Jie Zhou and Jiwen Lu},\nyear={2025},\nurl={https://openreview.net/forum?id=FowFLhUTgO}\n}"},"title":{"value":"V2M: Visual 2-Dimensional Mamba for Image Representation Learning"},"pdf":{"value":"/pdf/e5c0ef44b4174b06f68eb3a20282f4958621aa79.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"wang|v2m_visual_2dimensional_mamba_for_image_representation_learning"},"authorids":{"value":["~Chengkun_Wang1","~Wenzhao_Zheng1","~Yuanhui_Huang1","~Jie_Zhou3","~Jiwen_Lu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chengkun Wang","Wenzhao Zheng","Yuanhui Huang","Jie Zhou","Jiwen Lu"]}},"version":2},{"content":{"summary":{"value":"The paper introduces the first scheme for post-training quantization (PTQ) of Mamba models. In contrast to transformers, which are well studied in the context of PTQ, Mamba models have large state tensors that pose unique quantization challenges, as they can take a significant portion of memory, especially after quantizing weights.\nThey tackle this issue by focusing on the quantization of the states cache to low bits, down to 4 bits. They begin by examining the activation tensors and find that they have large outliers across both channel and state dimensions, making quantization difficult. They propose two methods to address this: \n\n1. Decoupled Scale Quantization (DSQ) that uses different scales and quantization schemes along the channel and state dimensions. This flattens outliers across both dimensions\n\n2. Efficient Selective Reconstruction (ESG), which is a local fine-tuning technique for certain SSM parameters ($B$, $C$ and $\\Delta$).  \n\nBy applying SmoothQuant to the rest of the linear layers of the network and ESG+DSQ to states the quantize the Mamba-2 models to W8A8H4. Specifically for Mamba2-2.7B they achieve 50% memory reduction with only 2.17% drop in accuracy degradation for zero-shot tasks"},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"* Many relevant references relate to LLM quantization in section 2.2, such as OPTQ, OmniQuant, OutlierSuppersion+, and etc.\n* Figure 1: What exactly is learnable at block $N$ ? $\\Delta$ is mentioned in the caption but is missing from the graph.\n* Figure 2: is this the memory consumption during pre-fill or generation?\n* Theorem 1 is very hard to follow. What does the split equation operator do? It’s never introduced. Where did the weight matrices ($W_x$ & $W_b$) come from? Please make this section more readable. \n* Figure 6, the legend hides the graph\n* Figure 9, the caption does not explain what the different colours are.\n* It would be very useful to add a graph of the Mamba block highlighting the final precision of each tensor and the precision of each compute. Look at Figure 6  of SmoothQuant.\n## Section 5.2: \n  * you never introduce an equation of what block-wise reconstruction means. I understand from the references it is the $\\ell_2$ loss between quantized and unquantized values but that's because I happen to know one of the cited papers well. \n * What do you learn exactly? Where is the quantization? Do you use a straight-through estimator? Do you also learn the quantization scales?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"* The authors are the first to tackle the quantization of Mamba models and achieve impressive compression of the state tensors that can take a large proportion of memory. They also show the efficacy of their work by showing the memory reduction and latency improvement in Figure 6.\n\n* The authors systematically study the state tensors and the presence of outliers to motivate Decoupled Scale Quantization with good visualization in Fig. 4 & Fig 8.\n\n* The profiling of Mamba inference and memory requirements supported by figures (Fig, 2 & Fig. 8)\n\n* An easy-to-read paper with good ablation studies"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Unfortunately, the paper lacks innovation from a quantization perspective even if the application of PTQ is quite novel, recycling ideas that are already well established in existing quantization literature. I believe that this paper would make an excellent workshop paper but its contributions are not enough for an ICLR paper. In addition, there is a lot of crucial information missing to understand and reproduce both Decoupled Scale Quantization & Efficient Selective Reconstruction. \n\n## Decoupled Scale Quantization\nAccording to the SMM equation in (1), the channels are independent from each other. I am not sure why then quantizing the values along the channel dimension would make any difference in the case of multiplication with $A$ & $C$. Per-state quantization of these matrices and of the states should lead to optimal quantization performance IF these matrix multiplications are in fact done in lower precision. Or the quantization scheme only for compression and matrix multiplications are done in higher precision. There are a lot of quantization details missing here that would explain the merits of the scheme.\n\n## Efficient Selective Reconstruction\nA lot of information missing here. Is this a local QAT scheme similar to OmniQuant for transformers? Are the learnt matrices just adjusted to the quantized states? Or they are also quantized using straight-through?"}},"nonreaders":[],"tmdate":1733156151195,"tcdate":1730456281271,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2045/Reviewer_UHAm"],"signatures":["ICLR.cc/2025/Conference/Submission2045/Reviewer_UHAm"],"forum":"AY1S52vr0a","number":3,"license":"CC BY 4.0","cdate":1730456281271,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2045/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733156151195,"domain":"ICLR.cc/2025/Conference","replyto":"AY1S52vr0a","id":"HLkSZKnLIF","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Quantization"]},"supplementary_material":{"value":"/attachment/0924a0296e7a1a9ad7158edce07c33777608192e.pdf"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSMs), such as Mamba, have recently demonstrated the potential to match or even surpass Transformers in language understanding tasks, making them a promising alternative for designing Large Language Models (LLMs). \nConcurrently, model quantization, especially Post-Training Quantization (PTQ), has been proven effective in reducing memory usage and inference latency in LLMs.\nIn this paper, we explore post-training quantization for Mamba (\\textbf{Q-Mamba}) by turning both linear projections and state caches into low-bit integers for efficient inference. \nAfter a theoretical analysis of the causes of outliers in states, we propose \\textbf{Decoupled Scale Quantization (DSQ)}, which mitigates outliers in both the state and channel dimensions by applying separate quantization scales.\nTo preserve the selective ability of quantized Mamba, we introduce \\textbf{Efficient Selectivity Reconstruction (ESR)}, a block-wise reconstruction method that involves a novel quantization simulation scheme, enabling fast parallel scan algorithms with the non-linear quantization function.\nWe demonstrate the effectiveness of Q-Mamba across various quantization settings, model sizes, and both generation and zero-shot tasks. \nIn particular, for Mamba2-2.7B with W8A8H4 quantization, Q-Mamba achieves a 50\\% reduction in memory consumption with only a 2.13\\% average accuracy degradation on zero-shot tasks."},"_bibtex":{"value":"@misc{\ntianqi2025qmamba,\ntitle={Q-Mamba: Towards more efficient Mamba models via Post-Training Quantization},\nauthor={Chen Tianqi and Yuanteng Chen and Weixiang Xu and Zeyu Zhu and Peisong Wang and Jian Cheng},\nyear={2025},\nurl={https://openreview.net/forum?id=AY1S52vr0a}\n}"},"title":{"value":"Q-Mamba: Towards more efficient Mamba models via Post-Training Quantization"},"pdf":{"value":"/pdf/1c7e10c8b02e5d50be6957f056d845dde2575456.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"tianqi|qmamba_towards_more_efficient_mamba_models_via_posttraining_quantization"},"authorids":{"value":["~Chen_Tianqi1","~Yuanteng_Chen1","~Weixiang_Xu2","~Zeyu_Zhu1","~Peisong_Wang1","~Jian_Cheng7"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chen Tianqi","Yuanteng Chen","Weixiang Xu","Zeyu Zhu","Peisong Wang","Jian Cheng"]}},"version":2},{"content":{"summary":{"value":"The paper studies hybrid language model architectures that mix Transformer attention with state-space models (SSMs), especially Mamba. It compares two ways to combine them: inter-layer (alternating Transformer and Mamba blocks) and intra-layer (splitting attention heads within a layer between Transformer and Mamba, then fusing the outputs). The paper compares quality, efficiency, and scaling, and gives design guidance on block ratios and intra-layer choices."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"See weaknesses."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":1},"strengths":{"value":"+ Clear comparisons and practical guidance: The paper compares inter-layer vs intra-layer hybrids alongside Transformer, Mamba, and sliding-window attention, providing practical design insights.\n\n+ Thorough intra-layer ablations: The paper studies normalization, fusion choices, gating, output projection, and dimension splits for intra-layer hybrid design.\n\n+ Efficiency analysis: The paper shows that lower FLOPs lead to lower step time, higher throughput, and a smaller KV cache than a Transformer at the same scale."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Novelty is limited: Similar hybrid ideas appear in prior work, and they provide a similar recipe and conclusion. Please clarify what is new here beyond existing hybrids and head-wise designs (e.g., what part of the architecture, training recipe, analysis, or design rules is truly novel).\n\n#### references\n- Griffin: Mixing Gated Linear Recurrences with Local Attention\n- MambaFormer: Hybrid Architecture for In-Context Learning\n- An Empirical Study of Mamba-based Language Models\n- Jamba: Hybrid transformer-mamba language models\n- Jamba-1.5: Hybrid transformer-mamba models at scale\n- Hymba: A hybrid-head architecture for small language models\n\n2. Scale: Results focus on 1B-parameter models, 8K training context, and fixed budgets (~60–73B tokens). Several prior hybrid papers (see above references) studied at larger scales (>1B). It would be helpful to demonstrate whether the design rules (e.g., block ratios, placement) still hold for larger models (e.g., 3B–7B) and for longer contexts (e.g., 32K+). Some scaling-law figure would strengthen claims.\n\n3. Hybrid variants not covered: The paper mainly studies head-wise intra-layer fusion, which is already explored in prior work. More experiments with other linear modules (e.g., RWKV, linear attention) or other fusion styles (e.g., sequence-wise fusion) would support the claim that the proposed “best recipe” generalizes across hybrid families.\n4. Scope of tasks: The main quality metrics are NLL and a small few-shot set. Downstream long-context tasks (e.g., QA, retrieval, summarization) are limited. Adding these would better validate the method."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926039327,"tcdate":1761764712389,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15809/Reviewer_xUqE"],"signatures":["ICLR.cc/2026/Conference/Submission15809/Reviewer_xUqE"],"forum":"1aAFc4OANr","number":2,"license":"CC BY 4.0","cdate":1761764712389,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15809/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926039327,"domain":"ICLR.cc/2026/Conference","replyto":"1aAFc4OANr","id":"kEFN2AFFPc","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Hybrid Architecture","Transformer","Mamba","Language Modeling","Efficiency"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Recent progress in large language models demonstrates that hybrid architectures--combining self-attention mechanisms with structured state space models like Mamba--can achieve a compelling balance between modeling quality and computational efficiency, particularly for long-context tasks. While these hybrid models show promising performance, systematic comparisons of hybridization strategies and analyses on the key factors behind their effectiveness have not been clearly shared to the community. In this work, we present a holistic evaluation of hybrid architectures based on inter-layer (sequential) or intra-layer (parallel) fusion. We evaluate these designs from a variety of perspectives: language modeling performance, long-context capabilities, scaling analysis, and training and inference efficiency. By investigating the core characteristics of their computational primitive, we identify the most critical elements for each hybridization strategy and further propose optimal design recipes for both hybrid models. Our comprehensive analysis provides practical guidance and valuable insights for developing hybrid language models, facilitating the optimization of architectural configurations."},"_bibtex":{"value":"@misc{\nbae2026hybrid,\ntitle={Hybrid Architectures for Language Models: Systematic Analysis and Design Insights},\nauthor={Sangmin Bae and Bilge Acun and Haroun Habeeb and Seungyeon Kim and Chien-Yu Lin and Liang Luo and Junjie Wang and Carole-Jean Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=1aAFc4OANr}\n}"},"title":{"value":"Hybrid Architectures for Language Models: Systematic Analysis and Design Insights"},"pdf":{"value":"/pdf/0e89b5aa27c2bea5f31c5069d50f1ff9ac97c543.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"bae|hybrid_architectures_for_language_models_systematic_analysis_and_design_insights"},"authorids":{"value":["~Sangmin_Bae1","~Bilge_Acun1","~Haroun_Habeeb2","~Seungyeon_Kim1","~Chien-Yu_Lin1","~Liang_Luo2","~Junjie_Wang17","~Carole-Jean_Wu2"]},"authors":{"value":["Sangmin Bae","Bilge Acun","Haroun Habeeb","Seungyeon Kim","Chien-Yu Lin","Liang Luo","Junjie Wang","Carole-Jean Wu"]}},"version":2},{"content":{"comment":{"value":"### **(W4, Q4) Design Choices**\nWe appreciate the reviewer’s request for clarification on our architectural design choices. We provide the rationale for each component below:\n\n**1. Choice of Mamba (SSM) over Transformers**  \nWe selected the Mamba framework primarily for its suitability for **continuous signal modeling**. Structured SSMs have recently emerged as a powerful paradigm for extending standard state-space models to deep learning, showing significant promise in time-series forecasting. Unlike Transformers, which treat time steps as discrete tokens, SSMs are rooted in continuous-time differential equations. This formulation offers a framework that naturally aligns with the continuous biological fluctuations inherent in BOLD signals (fMRI), facilitating the modeling of temporal evolution.\n\n**2. Two-stream Design and Gating Mechanism**  \nThe two-stream architecture (x-branch and z-branch) with multiplicative gating is adopted from the standard Mamba block design [1], functioning as a robust selection mechanism. The gating mechanism allows the model to selectively propagate information, thereby controlling the information flow through the network.\n\nRegarding the comparison with attention, while attention mechanisms also provide selectivity, they incur high computational costs due to global pairwise calculations. In contrast, the gating mechanism achieves input-dependent selectivity via efficient **element-wise operations** (following linear projections), which significantly lowers computational overhead. This approach has been proven both efficient and effective in prior studies [1].\n\n**3. Unified Parameterization across Feature Dimensions**  \nStandard SSM implementations often apply independent SSMs to each feature dimension. However, given the multi-channel nature of fMRI data (many ROIs), applying independent dynamics to every feature dimension would significantly increase model size. Therefore, as described in **Section 4.2.1**, we employ a unified parameterization (using projection matrices $\\mathbf{B}_r \\in \\mathbb{R}^{N \\times D}$) to integrate the entire $D$-dimensional ROI embedding into the latent state. This design choice aims to achieve parameter efficiency while enabling selectivity at the ROI level, effectively preserving the distinct dynamics of each brain region.\n\n### Reference\n> [1] Gu et al., \"Mamba: Linear-time sequence modeling with selective state spaces,\" COLM, 2024."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764752190805,"tcdate":1764752190805,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8714/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8714/Authors"],"forum":"522R7hU5J0","number":19,"license":"CC BY 4.0","cdate":1764752190805,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8714/-/Official_Comment"],"mdate":1764752190805,"domain":"ICLR.cc/2026/Conference","replyto":"cS5f3VIWRt","id":"p04jOCfCbR","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Resting-state fMRI","State-space model","Brain state dynamics"]},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Brain states, observable through resting-state functional magnetic resonance imaging (rs-fMRI), represent dynamic transitions between recurring connectivity patterns and are closely linked to neurological and psychiatric conditions. Therefore, developing a computational model for dynamic brain state estimation has been a long-lasting research interest. Among existing approaches, state-space models (SSMs) provide a principled framework for modeling these dynamics. However, existing methods face key limitations: they fail to preserve the brain's spatial architecture, and they model temporal dynamics without considering co-evolving spatial patterns.\nTo address these limitations, we propose $\\textbf{DyBraSS}$ ($\\textbf{Dy}$namic $\\textbf{Bra}$in $\\textbf{S}$tate-$\\textbf{S}$pace model), a novel structured SSM that unifies spatial and temporal modeling within a single framework, enhancing ROI-level modeling capacity and interpretability through a clustering-based global aggregation module.\nThis module respects the brain's network topology by integrating information from all regions during each local update, and represents evolving brain states as interpretable clusters.\nComprehensive experiments on multiple fMRI datasets demonstrate that our method consistently outperforms state-of-the-art baselines in diagnostic performance across diverse metrics. \nAdditionally, individual- and group-level brain state analyses reveal that the learned dynamics align with known neurobiological alterations, providing clinically relevant insights for understanding neural dysfunction and developing diagnostic biomarkers."},"_bibtex":{"value":"@misc{\njeong2026dybrass,\ntitle={DyBra{SS}: Dynamic Brain State Modeling with State-Space Model},\nauthor={Ah-Yeong Jeong and Seungwoo Jeong and Da-Woon Heo and Heung-Il Suk},\nyear={2026},\nurl={https://openreview.net/forum?id=522R7hU5J0}\n}"},"title":{"value":"DyBraSS: Dynamic Brain State Modeling with State-Space Model"},"pdf":{"value":"/pdf/7ec6c7264a37bbfe18a74f079c78222663cc7d50.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"jeong|dybrass_dynamic_brain_state_modeling_with_statespace_model"},"authorids":{"value":["~Ah-Yeong_Jeong1","~Seungwoo_Jeong2","~Da-Woon_Heo1","~Heung-Il_Suk1"]},"authors":{"value":["Ah-Yeong Jeong","Seungwoo Jeong","Da-Woon Heo","Heung-Il Suk"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2405.14022v4"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"atli|i2imamba_multimodal_medical_image_synthesis_via_selective_state_space_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Omer_F._Atli:","https://dblp.org/search/pid/api?q=author:Bilal_Kabas:","https://dblp.org/search/pid/api?q=author:Fuat_Arslan:","https://dblp.org/search/pid/api?q=author:Mahmut_Yurt:","~Onat_Dalmaz1","https://dblp.org/search/pid/api?q=author:Tolga_Çukur:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2405.14022"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2405-14022,\n  publtype={informal},\n  author={Omer F. Atli and Bilal Kabas and Fuat Arslan and Mahmut Yurt and Onat Dalmaz and Tolga Çukur},\n  title={I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2405.14022},\n  url={https://doi.org/10.48550/arXiv.2405.14022}\n}\n"},"abstract":{"value":"In recent years, deep learning models comprising transformer components have pushed the performance envelope in medical image synthesis tasks. Contrary to convolutional neural networks (CNNs) that use static, local filters, transformers use self-attention mechanisms to permit adaptive, non-local filtering to sensitively capture long-range context. However, this sensitivity comes at the expense of substantial model complexity, which can compromise learning efficacy particularly on relatively modest-sized imaging datasets. Here, we propose a novel adversarial model for multi-modal medical image synthesis, I2I-Mamba, that leverages selective state space modeling (SSM) to efficiently capture long-range context while maintaining local precision. To do this, I2I-Mamba injects channel-mixed Mamba (cmMamba) blocks in the bottleneck of a convolutional backbone. In cmMamba blocks, SSM layers are used to learn context across the spatial dimension and channel-mixing layers are used to learn context across the channel dimension of feature maps. Comprehensive demonstrations are reported for imputing missing images in multi-contrast MRI and MRI-CT protocols. Our results indicate that I2I-Mamba offers superior performance against state-of-the-art CNN- and transformer-based methods in synthesizing target-modality images."},"title":{"value":"I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling"},"authors":{"value":["Omer F. Atli","Bilal Kabas","Fuat Arslan","Mahmut Yurt","Onat Dalmaz","Tolga Çukur"]}},"tmdate":1738180781408,"pdate":1704067200000,"tcdate":1738180771120,"writers":["~"],"signatures":["~Onat_Dalmaz1"],"forum":"YBMJEBuGtK","license":"CC BY-SA 4.0","number":287043,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1738180781408,"domain":"DBLP.org","id":"YBMJEBuGtK","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Linear Attention","Linear Sequence Modeling","Efficient Architectures"]},"supplementary_material":{"value":"/attachment/cc98fbb09e0f6da1325d9323fa1345ceb3cd504b.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Linear recurrent models compress history into a fixed-size state, forcing the same finite memory to accommodate an ever-growing sequence of writes. Existing updates handle this pressure through superposition, forgetting, or replacement, which can make earlier associations difficult to recover. We introduce CyFA (Cyclic Flow Attention), which instead reuses memory by cyclically transporting\nthe surviving history as new information arrives. Earlier writes move away from the age-zero end of the memory while new key--value pairs enter there, producing a relative-time-partitioned state in which multiple historical associations can coexist across the cycle. An exact change of coordinates reduces this moving memory to two scalar-decay linear attention recurrences, enabling efficient chunk-wise training. Across 400M--1.4B pretraining experiments, CyFA consistently outperforms linear attention models on recall-intensive tasks under matched recurrent state sizes. At 400M, it reaches $42.60$ on FDA versus $26.07$ for KDA, while maintaining strong language modeling and higher training throughput than models with fine-grained gating or Delta Rule updates."},"_bibtex":{"value":"@inproceedings{\nanonymous2026cyfa,\ntitle={Cy{FA}: Linear Sequence Modeling with Relative-Time-Partitioned Memory},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=MrNCcG3NCj},\nnote={under review}\n}"},"title":{"value":"CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory"},"pdf":{"value":"/pdf/9f02f8b4f6831a34acb41d7f2fe5d1ceb10f6abe.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791228444337,"tcdate":1789385265619,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission17871/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission17871/Authors"],"forum":"MrNCcG3NCj","license":"CC BY 4.0","number":17871,"cdate":1789385265619,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission17871/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791228444337,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"MrNCcG3NCj","version":2},{"content":{"summary":{"value":"The paper presents Mamba-IVP, a novel model that integrates a Mamba-based autoencoder to effectively handle time-series with block-wise missingness and improve robustness to noisy measurements. Mamba-IVP is evaluated on time-series classification and forecasting tasks, demonstrating a better performance/efficiency trade-off compared to IVP-VAE."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please address the concerns in Weaknesses"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"* Mamba-IVP shows better performance than IVP-VAE across different tasks and demonstrates improved robustness under varying noise and missingness levels.\n\n* The paper provides theoretical analysis that supports the use of Mamba blocks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Contributions**\n\n* Mamba-IVP is basically an IVF-VAE with a Mamba autoencoder which seems an incremental technical contribution.\n\n**Evaluation**\n\n* Mamba-IVP achieves better performance than IVP-VAE but uses ~3× more parameters. Although Mamba-IVP remains ~40% faster, a comparison under the equal number of parameters is important to determine whether the gains are due to the Mamba architecture or simply more parameters.\n\n* Mamba-IVP is compared against GRU-D, while its counterparts (BRITS[1] and SAITS[2]) show stronger performance in prior work. Also, there are no comparisons to diffusion-based imputation methods [3,4], where [4] also uses a state-space architecture. More recent generative-based methods [5,6] are also not discussed or compared.\n\n&nbsp; &nbsp; &nbsp; The authors claim that these methods exhibit prohibitive computation cost, struggle to handle structured missingness and have limited robustness, but I could not find any evidence for these claims either in Mamba-IVP or IVP-VAE.\n\n* I can see that Mamba-IVP, as well as IVP-VAE and GRU-D, perform classification on top of the latent representations while many imputation methods, including [1-6], explicitly impute missing values and then train a downstream classifier on the filled data. Moreover, if Mamba-IVP follows the IVP-VAE training setup, the classifier is trained jointly with the main model. I believe it is an important detail to ensure fair comparisons.\n\n&nbsp; &nbsp; &nbsp; Are all other baselines also train classifiers jointly in the feature space? Could the authors compare unsupervised Mamba-IVP with s.o.t.a. baselines under the explicit imputation setup?\n\n**Writing**\n\n* The paper claims a 7.3% MSE reduction and 40× speedup, but the results show up to 4% MSE reduction and 40% speedup compared to IVP-VAE. Framing the gains relative to the weakest baseline sounds like an overclaim.\n \n* Figure 1 is somewhat confusing; I suggest refining the visualization and extending the caption to better explain the scheme.\n\n* The training procedure is not described. I assume it uses a reconstruction or VAE loss, but this should be explicitly stated. Also, it is unclear if the classification loss is used jointly with the main loss. \n\n---\n\n[1] BRITS: Bidirectional Recurrent Imputation for Time Series. NeuriPS2018\n\n[2] Saits: Self-attention-based imputation for time series. ESWA2023\n\n[3] CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. NeuriPS2021\n\n[4] Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models. TMLR2022\n\n[5] Frequency-aware Generative Models for Multivariate Time Series Imputation. NeurIPS2025\n\n[6] SADI: Similarity-Aware Diffusion Model-Based Imputation for Incomplete Temporal EHR Data. AISTATS2024"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764359889741,"tcdate":1761864484406,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4935/Reviewer_YMkW"],"signatures":["ICLR.cc/2026/Conference/Submission4935/Reviewer_YMkW"],"forum":"GG9OqLP0vO","number":2,"license":"CC BY 4.0","cdate":1761864484406,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4935/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764359889741,"domain":"ICLR.cc/2026/Conference","replyto":"GG9OqLP0vO","id":"4tNtScYylS","forumContent":{"TLDR":{"value":"Mamba-IVP is a lightweight state-space generative model that robustly imputes clinical time series under noise and missingness, achieving SOTA accuracy with superior efficiency."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Clinical Time Series","Mask-Aware Dual-Mamba Encoder","Mamba-Hybrid Decoder","Initial Value Problem","Denoising Proof"]},"supplementary_material":{"value":"/attachment/6409f6ce08476099e0c6d5a23ed37d73c571f39b.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Missing clinical time series is a critical bottleneck in intensive care units (ICUs). In large-scale ICU electronic health record datasets such as MIMIC-IV, missing rates exceed 90% due to sensor failures, monitor degradation, and systemic outages, while aging devices inject unstable noise that makes reliable modeling nearly impossible. Existing methods remain unsafe for deployment: statistical heuristics distort missingness, deep models collapse under block-wise gaps and noise, and ODE- or diffusion-based approaches demand prohibitive computation. To overcome these limitations, we propose Mamba-IVP, a state-space generative model with a Mask-Aware Dual-Mamba Encoder (MADME) to handle block-wise missingness and a Mamba-Hybrid Decoder (MHD) to denoise continuous-time reconstructions. We validate our method through 61 experiments across two tasks: time series forcatsing and node classification. Our experiments involve 7 classic and state-of-the-art target models and 3 publicly available datasets: (1) it achieves state-of-the-art accuracy, reducing MSE by 3.0%, improving AUROC by 3.0%, and enhancing AUPRC by 3.9%; and (2) it remains robust under noise and block-wise missingness up to 12h, where other models degrade sharply."},"_bibtex":{"value":"@misc{\nfang2026mambaivp,\ntitle={Mamba-{IVP}: A Denoising State-Space Initial Value Problem Framework for {SOTA} Clinical Time Series, Healthcare Alternative},\nauthor={Jiayu Fang and Zhiqi Shao and Hongwei Ma and S T Boris Choy and Junbin Gao},\nyear={2026},\nurl={https://openreview.net/forum?id=GG9OqLP0vO}\n}"},"title":{"value":"Mamba-IVP: A Denoising State-Space Initial Value Problem Framework for SOTA Clinical Time Series, Healthcare Alternative"},"pdf":{"value":"/pdf/c2f5ef7880e75bc7edf617da1c90ed9c557976d5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"fang|mambaivp_a_denoising_statespace_initial_value_problem_framework_for_sota_clinical_time_series_healthcare_alternative"},"authorids":{"value":["~Jiayu_Fang1","~Zhiqi_Shao1","~Hongwei_Ma1","~S_T_Boris_Choy1","~Junbin_Gao1"]},"authors":{"value":["Jiayu Fang","Zhiqi Shao","Hongwei Ma","S T Boris Choy","Junbin Gao"]}},"version":2},{"content":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Withdrawn by Authors"},"abstract":{"value":"Self-supervised contrastive learning offers a compelling route to transferable time-series representations in label-scarce settings.\nYet existing frameworks face a persistent trade-off between preserving fine-grained local dynamics at high temporal resolution and scaling to long sequences under practical compute constraints.\nConvolutional encoders often require deep stacks to retain rapid transitions, whereas Transformers incur quadratic cost in sequence length, making high-resolution long-context training expensive.\nRecent selective state-space models such as \\emph{Mamba} enable linear-time ($O(L)$) sequence modeling and offer a promising path to mitigate this bottleneck.\nHowever, their potential for \\emph{general-purpose} time-series representation learning remains underexplored;\nto our knowledge, prior Mamba-based contrastive learners have not been evaluated on the full UCR 2018 archive (128 datasets) under a unified protocol.\nWe propose \\textbf{DeltaSM} (\\emph{Delta-selective Mamba}), a self-supervised framework for univariate time series that reconciles efficiency and expressivity.\nDeltaSM integrates (i) a lightweight Mamba backbone, (ii) token-budget-constrained training, and (iii) a $\\Delta$-level contrastive objective that counterbalances Mamba's smoothing tendency.\nSpecifically, we apply curvature-adaptive weighting to first-order differences of the latent sequence, encouraging the encoder to emphasize informative local transitions without increasing computational cost.\nAt inference time, we further augment the learned time-domain embeddings with explicitly extracted frequency-domain descriptors from the raw signal to improve expressivity at negligible overhead.\nAcross all 128 UCR datasets, under \\textbf{Protocol A}---a unified compute setting with a fixed number of optimization steps and a standardized downstream classifier---DeltaSM converges in seconds and achieves classification accuracy comparable to or better than strong baselines such as TS-TCC, TS2Vec, and TimesURL, using a single global configuration and a fixed pretraining-step budget (300 optimization updates per dataset).\nOn a focused subset that includes long-sequence datasets under \\textbf{Protocol B}---where baselines are allowed their recommended training budgets and hyperparameters while DeltaSM remains fixed as in Protocol A---DeltaSM reduces pretraining time by up to $184\\times$ while remaining competitive.\nExtensive ablations confirm that curvature-based weighting is crucial for suppressing noise while capturing local dynamics, and that inference-time frequency integration provides complementary gains with minimal additional cost."},"_bibtex":{"value":"@article{\nanonymous2026deltasm,\ntitle={Delta{SM}: Delta-Level Contrastive Learning with Mamba for Time-Series Representation},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2026},\nurl={https://openreview.net/forum?id=3S1kAqCiAS},\nnote={Withdrawn}\n}"},"title":{"value":"DeltaSM: Delta-Level Contrastive Learning with Mamba for Time-Series Representation"},"pdf":{"value":"/pdf/9badc13e59b4a5eb784f32d5bd8ffdca71528128.pdf"},"venueid":{"value":"TMLR/Withdrawn_Submission"},"assigned_action_editor":{"value":"~John_Timothy_Halloran1"}},"tmdate":1774257009067,"tcdate":1767886631937,"writers":["TMLR","TMLR/Paper6914/Authors"],"signatures":["TMLR/Paper6914/Authors"],"forum":"3S1kAqCiAS","license":"CC BY 4.0","number":6914,"cdate":1767886631937,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/-/Withdrawn"],"mdate":1774257009067,"odate":1769752536217,"domain":"TMLR","id":"3S1kAqCiAS","version":2},{"content":{"summary":{"value":"This paper shows a one-layer Mamba block can implement in-context learning (ICL) on stylized binary tasks, and its nonlinear gate is the key: linear attention selects helpful examples; the gate suppresses outliers and induces a local bias. Compared to a linear-attention Transformer (same setting), Mamba may train a bit slower but stays accurate under a much higher fraction of corrupted examples—even approaching 100% under their assumptions."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. The test outlier is a positive combination of train outliers (Condition (11)). How realistic is this in language tasks? What happens for combinations with mixed signs or novel directions?\n\n2. Can you extend beyond binary classification to multiclass or regression within the same framework? Where do the proofs fail?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The theoretical claims seem to align with empirical evidence (on the high level), e.g., the paper’s core takeaway—Mamba’s gate makes ICL more robust to corrupted prompts than a pure linear-attention Transformer—is directionally consistent with what the community has observed.\n\n- Clear mechanistic story: linear attention aligns to the query’s relevant pattern; gating both suppresses outliers and biases to nearby positions."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper studies a stylized binary, pattern-based ICL task where each input contains exactly one relevant and one irrelevant orthogonal pattern of equal norm; labels depend only on the relevant pattern. This is a clean but **unnatural** abstraction for real data. Results derived on this task may not transfer to more general tasks studies in practice. \n\n2. Outlier modeling is also narrow/artificial: Outliers are injected additively, orthogonal to both relevant and irrelevant patterns during training, and at test time are linear combinations of the training-time outlier directions with a positivity constraint.\n\n3. Transformer comparator is a very specific “linear attention” variant (no softmax, multi-heads...). So the headline advantage (“Mamba tolerates higher outlier fractions”) is established against a simplified comparator that may be unrepresentative of modern Transformers. This undermines the breadth of the claim."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919929155,"tcdate":1762280243034,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7901/Reviewer_okLR"],"signatures":["ICLR.cc/2026/Conference/Submission7901/Reviewer_okLR"],"forum":"tswBfpkwHn","number":4,"license":"CC BY 4.0","cdate":1762280243034,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7901/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919929155,"domain":"ICLR.cc/2026/Conference","replyto":"tswBfpkwHn","id":"8vQz25BpCf","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"This paper provide the first theoretical analysis of the training dynamics of a one-layer Mamba model and characterizes its in-context learning generaization ability with additive outliers in the prompt"},"keywords":{"value":["Mamba","in-context learning","Transformer","generalization","training dynamics","deep learning theory"]},"primary_area":{"value":"learning theory"},"abstract":{"value":"The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks. Like Transformers, Mamba exhibits in-context learning (ICL) capabilities, i.e.,  making predictions for new tasks based on a prompt containing input-label pairs and a query, without requiring fine-tuning.\nDespite its empirical success, the theoretical understanding of Mamba remains limited, largely due to the nonlinearity introduced by its gating mechanism. To the best of our knowledge, this paper presents the first theoretical analysis of the training dynamics of a one-layer Mamba model, which consists of a linear attention component followed by a nonlinear gating layer, and its ICL generalization on unseen binary classification tasks, even when the prompt includes additive outliers. Our analysis shows that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layer to suppress the influence of outliers. By establishing and comparing to the analysis of linear Transformers under the same setting, we show that although Mamba may require more training iterations to converge, it maintains accurate predictions even when the proportion of outliers exceeds the threshold that a linear Transformer can tolerate. These theoretical findings are supported by empirical experiments."},"_bibtex":{"value":"@misc{\nli2026can,\ntitle={Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis},\nauthor={Hongkang Li and Songtao Lu and Xiaodong Cui and Pin-Yu Chen and Meng Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=tswBfpkwHn}\n}"},"title":{"value":"Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis"},"pdf":{"value":"/pdf/771e925a0afdf6afc4b8a950bc4b771d6afabad8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"li|can_mamba_learn_in_context_with_outliers_a_theoretical_generalization_analysis"},"authorids":{"value":["~Hongkang_Li1","~Songtao_Lu1","~Xiaodong_Cui1","~Pin-Yu_Chen1","~Meng_Wang4"]},"authors":{"value":["Hongkang Li","Songtao Lu","Xiaodong Cui","Pin-Yu Chen","Meng Wang"]}},"version":2},{"content":{"summary":{"value":"The paper introduces TIMBA, an architecture for time series imputation. The main idea behind the proposed approach is to replace the Transformer layers of an existing architecture (PriSTI) with bidirectional Mamba blocks. Empirical performance matches or even surpasses the state of the art."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please comment on the highlighted weaknesses."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":1},"strengths":{"value":"* **Relevant problem**: Integrating SSMs into time series models is an important direction for future work.\n* **Somewhat strong empirical results**: Good performance on relevant benchmarks, though improvements are limited, especially considering the reported standard deviations. Furthermore, the ablation analysis is limited (see weaknesses)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Major weaknesses**\n\n* **No clear motivation**: The authors argue that the proposed Mamba blocks should be used instead of Transformers as they are more tailored to processing time series given their inductive biases. However, the same could be said for models like RNNs and TCNs and for specialized Transformer architectures for time series. While SSMs have proven particularly useful for processing long sequences, the sequences processed here are a maximum of 36 time steps. It's unclear what specific benefits the proposed architectural modification offers. I suspect that the performance gap (often not particularly significant given the reported deviations) relative to existing models could be eliminated with better hyperparameter selection.\n\n* **Limited novelty**: The paper suggests replacing the sequence modeling part of an existing architecture with another (existing) sequence modeling layer. The contributions, therefore, appear limited by ICLR standards.\n\n* **Limited analysis**: Since the contribution is primarily an architectural modification, this should be the focus of additional ablation studies. Simply showing that the bidirectional version of the model achieves better results is not sufficient. One possibility would be to replace the S6 layers with RNNs/TCNs and/or replace the Mamba block entirely with other sequence modeling blocks.\n\n**Minor comments**\n* Some relevant missing references, e.g.,\n    - [1] SPIN, an attention-based graph neural network for imputation, which mitigates the error accumulation seen in GRIN.\n    - [2] SAITS, a popular attention-based imputation architecture.\n\n#### References\n\n[1] Marisca et al., \"Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observations\", NeurIPS 2022\\\n[2] Du et al., “SAITS: Self-Attention-based Imputation for Time Series”, Expert Systems with Applications, 2022"}},"nonreaders":[],"tmdate":1731428683116,"tcdate":1729087131758,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6555/Reviewer_G76U"],"signatures":["ICLR.cc/2025/Conference/Submission6555/Reviewer_G76U"],"forum":"Sw10tbj0gM","number":1,"license":"CC BY 4.0","cdate":1729087131758,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6555/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428683116,"domain":"ICLR.cc/2025/Conference","replyto":"Sw10tbj0gM","id":"O753spvByx","forumContent":{"TLDR":{"value":"We introduce TIMBA, a novel model using Mamba blocks in place of Transformers for superior multivariate time series imputation across various benchmarks and datasets."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Multivariate Time Series Imputation","Diffusion models","Mamba","Space State Models","Generative Models","Graph Neural Networks","Transformers"]},"supplementary_material":{"value":"/attachment/00fc8f6bdb0e5659e4d633c09941288f8fbce065.zip"},"primary_area":{"value":"generative models"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The problem of imputing multivariate time series spans a wide range of fields, from clinical healthcare to multi-sensor systems. Initially, Recurrent Neural Networks (RNNs) were employed for this task; however, their error accumulation issues led to the adoption of Transformers, leveraging attention mechanisms to mitigate these problems. Concurrently, the promising results of diffusion models in capturing original distributions have positioned them at the forefront of current research, often in conjunction with Transformers. In this paper, we propose replacing time-oriented Transformers with State-Space Models (SSM), which are better suited for temporal data modeling. Specifically, we utilize the latest SSM variant, S6, which incorporates attention-like mechanisms. By embedding S6 within Mamba blocks, we develop a model that integrates SSM, Graph Neural Networks, and node-oriented Transformers to achieve enhanced spatiotemporal representations. Implementing these architectural modifications, previously unexplored in this field, we present Time series Imputation with Bi-directional mamba blocks and diffusion models (TIMBA). TIMBA achieves superior performance in almost all benchmark scenarios and performs comparably in others across a diverse range of missing value situations and three real-world datasets. We also evaluate how the performance of our model varies with different amounts of missing values and analyse its performance on downstream tasks. In addition, we provide the original code to replicate the results."},"_bibtex":{"value":"@misc{\nsol{\\'\\i}s-garc{\\'\\i}a2024timba,\ntitle={{TIMBA}: Time series Imputation with Bi-directional Mamba Blocks and Diffusion models},\nauthor={Javier Sol{\\'\\i}s-Garc{\\'\\i}a and Bel{\\'e}n Vega-M{\\'a}rquez and Juan A. Nepomuceno and Isabel A. Nepomuceno-Chamorro},\nyear={2024},\nurl={https://openreview.net/forum?id=Sw10tbj0gM}\n}"},"title":{"value":"TIMBA: Time series Imputation with Bi-directional Mamba Blocks and Diffusion models"},"pdf":{"value":"/pdf/d22f8aa876bfcdf7a12b5d82cb657b4429e0290c.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"solísgarcía|timba_time_series_imputation_with_bidirectional_mamba_blocks_and_diffusion_models"},"authorids":{"value":["~Javier_Solís-García1","~Belén_Vega-Márquez1","~Juan_A._Nepomuceno1","~Isabel_A._Nepomuceno-Chamorro1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Javier Solís-García","Belén Vega-Márquez","Juan A. Nepomuceno","Isabel A. Nepomuceno-Chamorro"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.08070v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"tang|mambamim_pretraining_mamba_with_state_space_tokeninterpolation"},"authorids":{"value":["","~Bingkun_Nian2","https://dblp.org/search/pid/api?q=author:Yingtai_Li:","~Jie_Yang84","https://dblp.org/search/pid/api?q=author:Liu_Wei:","https://dblp.org/search/pid/api?q=author:S._Kevin_Zhou:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.08070"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-08070,\n  publtype={informal},\n  author={Fenghe Tang and Bingkun Nian and Yingtai Li and Jie Yang and Liu Wei and S. Kevin Zhou},\n  title={MambaMIM: Pre-training Mamba with State Space Token-interpolation},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.08070},\n  url={https://doi.org/10.48550/arXiv.2408.08070}\n}\n"},"abstract":{"value":"Generative self-supervised learning demonstrates outstanding representation learning capabilities in both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). However, there are currently no generative pre-training methods related to selective state space models (Mamba) that can handle long-range dependencies effectively. To address this challenge, we introduce a generative self-supervised learning method for Mamba (MambaMIM) based on Selective Structure State Space Sequence Token-interpolation (S6T), a general-purpose pre-training method for arbitrary Mamba architectures. Our method, MambaMIM, incorporates a bottom-up 3D hybrid masking strategy in the encoder to maintain masking consistency across different architectures. Additionally, S6T is employed to learn causal relationships between the masked sequence in the state space. MambaMIM can be used on any single or hybrid Mamba architectures to enhance the Mamba long-range representation capability. Extensive downstream experiments reveal the feasibility and advancement of using Mamba for pre-training medical image tasks. The code is available at: https://github.com/FengheTan9/MambaMIM"},"title":{"value":"MambaMIM: Pre-training Mamba with State Space Token-interpolation"},"authors":{"value":["Fenghe Tang","Bingkun Nian","Yingtai Li","Jie Yang","Liu Wei","S. Kevin Zhou"]}},"tmdate":1791950968798,"pdate":1704067200000,"tcdate":1730988529412,"writers":["~"],"signatures":["~FengheTang1"],"forum":"bWoV0d8auR","license":"CC BY-SA 4.0","number":173927,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1791950968798,"domain":"DBLP.org","id":"bWoV0d8auR","version":2},{"content":{"summary":{"value":"This paper introduces SILA, which enhances linear attention retrieval through improvements in both recurrence modeling and training objectives. Its core idea is to allow the model to selectively disregard irrelevant tokens.\n\nThe authors begin with an analysis of the NIAH task and establish a more reliable evaluation setting. Motivated by the need for selective memory writing, SILA incorporates several architectural innovations: it decouples the “read” and “write” operations (i.e., read-before-write) and introduces state-dependent gates. In addition, the authors employ a progressive selective weighted loss, based on a pretrained transformer, to improve the efficiency of long-context training.\n\nExperimental results under the Transformer-to-RNN paradigm demonstrate that SILA improves the retrieval capability of linear attention models while maintaining competitive performance on general short-context modeling."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. In Table 1, the in-context retrieval capability appears correlated with training data scale. Since Qwen3 was trained on 36T high-quality tokens while other linear models were significantly undertrained, would a Transformer trained on a comparable corpus (e.g., hundreds of billions of tokens) still demonstrate similarly robust NIAH-Word performance?\n\n2. Compared to linear attention baselines such as GatedDeltaNet, how do SILA’s computational and parameter costs change? What are the shapes of $W_{\\gamma}$ and $W_{\\beta}$? \n\n3. What is the rationale for the chosen functional form of the forget gate $\\phi(x)$ on the negative axis in Eq.10, which seems relatively uncommon?\n\n4. As the memory-dependent gates introduce nonlinearity in the recurrence via the hidden state, does this hurt parallelism or training efficiency, particularly for long-context sequences? From an implementation standpoint, is it necessary to materialize all intermediate states across time steps?\n\n5. Regarding the weighted loss, how were the threshold and scaling factor determined? Were these hyperparameters tuned through systematic sweeps?\n\n6. What are the NIAH results for SILA (standard loss)? Only commonsense reasoning results are reported.\n\n7. To support the claim that the improvement stems from selective ignoring, it would be helpful to visualize and analyze the gate values (${\\gamma}_t$ and ${\\beta}_t$) and compare them against those of GatedDeltaNet."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The motivation and proposed approach of this work are well-reasoned and coherent. The paper identifies a key deficiency of linear attention models under NIAH-style tasks and demonstrates that this issue can be alleviated through selective ignoring.\n\n2. The analysis of the NIAH benchmark is logical and convincing, ensuring the reliability of the experimental evaluation.\n\n3. The improvements in both modeling and training are essential and directly address the need for selective ignoring."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Some of the conclusions are not thoroughly or fairly validated. For example, the passkey retrieval extrapolation results of Qwen3 and GatedDelta are not shown; since the transferred Qwen3 checkpoints already support 32k context without ntk scaling, further clarification is needed here. In addition, in Table 5, only SILA is a transferred model, making it difficult to fairly compare the effectiveness of the proposed method with other selective writing approaches such as LongMamba.\n\n2. The paper lacks ablation studies and detailed analyses for the proposed improvements. It remains unclear how each component—such as the training strategy and the two architectural modifications—specifically contributes to retrieval performance or NIAH results.\n\n3. The experimental setting is overly constrained. The model scale (0.6B), training length (1k/4k), and evaluation scenario (Transformer-to-RNN transfer) are all highly specialized. Under such a narrow setup, it is difficult to disentangle how much of the observed gain arises from the proposed method itself rather than from others like parameter increases or tuning. Moreover, there are concerns regarding the scalability of the approach."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942785819,"tcdate":1761746586390,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23737/Reviewer_PcX2"],"signatures":["ICLR.cc/2026/Conference/Submission23737/Reviewer_PcX2"],"forum":"csZndfXLpC","number":1,"license":"CC BY 4.0","cdate":1761746586390,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23737/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942785819,"domain":"ICLR.cc/2026/Conference","replyto":"csZndfXLpC","id":"PC5XtPSJBw","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["language model","linear attention","long-context retrieval"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Linear attention models have recently emerged as computationally efficient alternatives to Transformers.\nDespite competitive performance on general commonsense tasks, they still struggle to match Transformers on long-context retrieval tasks.\nIn this work, we re-examine linear attention models from the perspective of memory writing.\nWe propose that enabling linear attention models to learn selective ignoring provides a promising approach to addressing long-context retrieval tasks under fixed memory capacity.\nGuided by this principle, we demonstrate how to interpret and intervene in the behavior of linear attention models, thereby revealing the true retrieval capabilities of popular models.\nInformed by these observations, we introduce Selective Ignoring Linear Attention (SILA), which incorporates a redesigned memory architecture and a weighted loss training strategy to encourage selective memory writing.\nSILA exhibits remarkable long-context retrieval capabilities, achieving 20$\\times$ context length extrapolation on the Passkey Retrieval task, and demonstrating superior memory utilization efficiency on the Needle-in-a-Haystack benchmark."},"_bibtex":{"value":"@misc{\nzhang2026sila,\ntitle={{SILA}: Enhancing Long-Context Retrieval Capability of Linear Attention via Selective Ignoring},\nauthor={Jiayu Zhang and Shuyang Chen and Qian Zheng and Rui Yan and Gang Pan and Huajin Tang},\nyear={2026},\nurl={https://openreview.net/forum?id=csZndfXLpC}\n}"},"title":{"value":"SILA: Enhancing Long-Context Retrieval Capability of Linear Attention via Selective Ignoring"},"pdf":{"value":"/pdf/243f8c9a4f4095799cdb837c604bda50e0844c89.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|sila_enhancing_longcontext_retrieval_capability_of_linear_attention_via_selective_ignoring"},"authorids":{"value":["~Jiayu_Zhang10","~Shuyang_Chen1","~Qian_Zheng5","~Rui_Yan9","~Gang_Pan1","~Huajin_Tang1"]},"authors":{"value":["Jiayu Zhang","Shuyang Chen","Qian Zheng","Rui Yan","Gang Pan","Huajin Tang"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a time series imputation method using Mamba state space model as the backbone of diffusion model. The SSM-based blocks aim to improve inter-variable and bidirectional dependency modeling. Experimental results show this approach achieves state-of-the-art performance across multiple datasets and varying missing scenarios and ratios."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please refer to the weakness. Here my main concern is with the motivation of this paper. It appears to be an exploratory study investigating a good model combination without clearly defining a technical, theoretical, or practical research question. Vanilla CNN, Transformer, and SSM are just model architectures; using them in an appropriate way does not, in itself, demonstrate significance. Overall, the contribution of this paper is limited, and a more comprehensive literature review and clearer positioning within the research field are needed."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"1) The proposed method focuses on modeling channel dependencies in time series and the the backbone selection in diffusion models. This is a novel and interesting idea.\n2) The conditional diffusion model should allow exploitation of useful information in the data for accurate reconstruction. This is a strength.\n3) There are good experimental evaluations and comparisons presented in the paper."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1) The main motivation of model compatibility in diffusion-based imputation is not well-stated. Seems like authors would investigate which backbone is more suitable for diffusion model. It does not clear what certain limitation in using other models as the backbone, since CNN, Transformer, and SSM are channel independent and unidirectional. \n2) Authors claim the proposed method is channel-dependent and has low complexity. But it seems like using SSM in a different way, i.e., modeling across channel correlation. I do not agree this point can be stated as a drawback. If capturing inter-channel relationships is one of the key motivations for this paper, I would like to know why these relationships are essential for the imputation task and what issues might arise from ignoring them. But, this discussion is missing from the paper.\n3) In the original SSSD paper, ECG data was also used for experiments, and the model was evaluated based on imputation accuracy and visualization. The ECG experiments were significant in the SSSD paper, yet it appears the authors did not include this experiment in their current paper. Given that the authors replicated other experiments from the SSSD paper, omitting this key experiment seems unusual. Without it, the authors only tested their method on two datasets for the imputation task. Including additional datasets could enhance the reliability of the results.\n4) Have the authors tried other approaches for modeling channel-wise dependencies? As noted in the paper, there is no clear order among channels in multivariate time series, so using an order-agnostic attention mechanism for inter-channel relationships is intuitive. However, the motivation for using the Mamba model to capture inter-channel relationships is unclear, as state space models typically assume a temporal order in the data. Could the authors provide further explanation to clarify the rationale behind the channel Mamba block? Additionally, an ablation study on channel modeling would be beneficial."}},"nonreaders":[],"tmdate":1731427590346,"tcdate":1730775186680,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2397/Reviewer_pCfq"],"signatures":["ICLR.cc/2025/Conference/Submission2397/Reviewer_pCfq"],"forum":"j1OucVFZMJ","number":4,"license":"CC BY 4.0","cdate":1730775186680,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2397/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427590346,"domain":"ICLR.cc/2025/Conference","replyto":"j1OucVFZMJ","id":"cT2hPrbVQn","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["diffusion","time series impuation","state space model"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Probabilistic time series imputation has been widely applied in real-world scenarios due to its ability to estimate uncertainty of imputation results. Meanwhile, denoising diffusion probabilistic models (DDPMs) have achieved great success in probabilistic time series imputation tasks with its power to model complex distributions. However, current DDPM-based probabilistic time series imputation methodologies are confronted with two types of challenges: 1) \\textit{ The backbone modules of the denoising parts are not capable of achieving sequence modeling with low time complexity.} 2) \\textit{ The architecture of denoising modules can not handle the inter-variable and bidirectional dependencies in the time series imputation problem effectively.} To address the first challenge, we integrate the computational efficient state space model, namely Mamba, as the backbone denosing module for DDPMs. To tackle the second challenge, we carefully devise several SSM-based blocks for bidirectional modeling and inter-variable relation understanding. Experimental results demonstrate that our approach can achieve state-of-the-art time series imputation results on multiple datasets, different missing scenarios and missing ratios."},"_bibtex":{"value":"@misc{\ngao2025diffimp,\ntitle={DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone},\nauthor={Hongfan Gao and Wangmeng Shen and Xiangfei Qiu and Ronghui Xu and Jilin Hu and Bin Yang},\nyear={2025},\nurl={https://openreview.net/forum?id=j1OucVFZMJ}\n}"},"title":{"value":"DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone"},"pdf":{"value":"/pdf/26faf0d7208528fa5a326d36fadf37ed808e0f38.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"gao|diffimp_efficient_diffusion_model_for_probabilistic_time_series_imputation_with_bidirectional_mamba_backbone"},"authorids":{"value":["~Hongfan_Gao1","~Wangmeng_Shen1","~Xiangfei_Qiu1","~Ronghui_Xu2","~Jilin_Hu1","~Bin_Yang4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hongfan Gao","Wangmeng Shen","Xiangfei Qiu","Ronghui Xu","Jilin Hu","Bin Yang"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a parameter-free modification to the transformer attention mechanism called Selective Attention, which reduces the attention to irrelevant or unneeded tokens in a sequence. This approach enhances performance in language modeling tasks by reducing memory usage and computational costs without compromising model quality. Experiments show that transformers with Selective Attention achieve comparable performance to larger models with traditional attention mechanisms, providing improved efficiency and effectiveness across various model sizes and tasks."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Have you considered evaluating Selective Attention on other architectures, such as encoder or encoder-decoder transformers? If so, could you share any preliminary results or insights?\n\n2. How sensitive is the performance of Selective Attention to the memory budget per layer? Could you offer guidelines on tuning this budget for tasks beyond language modeling?\n\n3. Have you tested Selective Attention on tasks other than language modeling, such as question answering or text summarization? How might the approach handle tasks with varied context and attention needs?\n\n4. Could you provide further comparisons between Selective Attention and other efficient attention methods, such as sparse or linear attention, to clarify the unique advantages of Selective Attention?\n\n5. How does context pruning with Selective Attention affect tasks requiring long-range dependencies? Are certain types of tokens or information more prone to being pruned?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. **Efficient Memory Management**: The Selective Attention mechanism effectively prunes unneeded tokens, significantly reducing memory usage during inference without degrading model performance. This efficiency gain is particularly valuable for scaling transformers in resource-constrained environments.\n2. **No Additional Parameters**: Selective Attention operates without introducing new parameters or significantly increasing computational overhead, which preserves the simplicity of the transformer architecture.\n3. **Theoretical Motivation and Experimental Rigor**: The paper provides a thorough theoretical motivation for selective attention, supporting it with extensive experimental results on benchmarks like HellaSwag and C4, which demonstrate the effectiveness of the approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Limited Scope of Model Architectures**: The experiments are primarily conducted on decoder-only transformer models. Further analysis is needed to verify if Selective Attention can similarly benefit encoder models or encoder-decoder models used in other tasks, such as translation or summarization.\n2. **Potential Over-Reliance on Hyperparameter Tuning**: Selective Attention’s performance may depend on optimal memory budget settings per layer, which could complicate deployment in different tasks or models. Although a memory reduction process is described, further tuning could be required in practical applications.\n3. **Dataset and Task Diversity**: While Selective Attention shows improvements in language modeling tasks, testing it on a wider range of tasks (e.g., text generation or long document understanding) would strengthen the case for its generalizability and adaptability to diverse applications.\n4. **Comparison with Similar Methods**: The paper could benefit from a more detailed comparison with other recent efficient attention mechanisms, such as sparse attention or adaptive attention approaches, to highlight any relative advantages Selective Attention may have."}},"nonreaders":[],"tmdate":1732676001653,"tcdate":1730749529778,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2232/Reviewer_L9tT"],"signatures":["ICLR.cc/2025/Conference/Submission2232/Reviewer_L9tT"],"forum":"v0FzmPCd1e","number":4,"license":"CC BY 4.0","cdate":1730749529778,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2232/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732676001653,"domain":"ICLR.cc/2025/Conference","replyto":"v0FzmPCd1e","id":"AnVLc5er9k","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["selective attention","attention","transformer","llm","language model"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Unneeded elements in the attention’s context degrade performance. We introduce Selective Attention, a simple parameter-free change to the standard attention mechanism which reduces attention to unneeded elements. Selective attention consistently improves language modeling and downstream task performance in a variety of model sizes and context lengths. For example, transformers trained with the language modeling objective on C4 with selective attention perform language modeling equivalently to standard transformers with ~2X more heads and parameters in their attention modules. Selective attention also allows decreasing the size of the attention’s context buffer, leading to meaningful reductions in the memory and compute requirements during inference. For example, transformers trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and 47X less memory for their attention module, respectively, when equipped with selective attention, as those without selective attention, with the same validation perplexity."},"_bibtex":{"value":"@inproceedings{\nleviathan2025selective,\ntitle={Selective Attention Improves Transformer},\nauthor={Yaniv Leviathan and Matan Kalman and Yossi Matias},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=v0FzmPCd1e}\n}"},"title":{"value":"Selective Attention Improves Transformer"},"pdf":{"value":"/pdf/736a82addd28e9cf5eaa18ed9de4a08a7ae98a3d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"leviathan|selective_attention_improves_transformer"},"authorids":{"value":["~Yaniv_Leviathan1","~Matan_Kalman1","~Yossi_Matias2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yaniv Leviathan","Matan Kalman","Yossi Matias"]}},"version":2},{"content":{"summary":{"value":"This work addresses meta-continual learning using a state space model Mamba. It performs comprehensive experiments across various CL benchmarks and reports several interesting results, including comparison with Transformers and extension to Mamba mixture-of-experts."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please see the Weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. It proposes MambaCL as a strong sequential approach to meta-continual learning. \n\n2. It performs thorough experiments and discover multiple interesting observations.\n- The use of Mamba may be more helpful for generalization over Transformers as discussed in Fig.3.\n- MambaCL is particularly effective in on fine-grained recognition tasks as shown in Table 3.\n- Integration of Mamba with MoE improves the MCL performance as reported in Table 6."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The technical novelty is limited.\n- This work is largely based on the work of (Lee et al., 2024), which first formulates the MCL problem as a sequent modeling.\n- This work simply replaces Transformers of (Lee et al., 2024) with a state space model Mamba. \n- Except this replacement, there is little novelty as its application is rather straightforward, following (Lee et al., 2024). \n\n2. The use of Mamba instead of Transformers leads to little performance improvement as reported in Table 1-5. \n- The main benefit of Mamba over Transformer lies in fewer parameters and increased processing speed as shown in Table 7.\n\n3. Implementation details are missing. \n- Appendix is too sketchy to fully understand how the MambaCL is implemented. \n- The code is not provided."}},"nonreaders":[],"tmdate":1731427182477,"tcdate":1730551061480,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9/Reviewer_bJ6d"],"signatures":["ICLR.cc/2025/Conference/Submission9/Reviewer_bJ6d"],"forum":"1TXDtnDIsV","number":2,"license":"CC BY 4.0","cdate":1730551061480,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427182477,"domain":"ICLR.cc/2025/Conference","replyto":"1TXDtnDIsV","id":"7EKJG9grUw","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Continual Learning","Sequence Modelling"]},"primary_area":{"value":"transfer learning, meta learning, and lifelong learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Continual learning (CL) aims to efficiently learn and accumulate knowledge from a data stream with different distributions. By formulating CL as a sequence prediction task, meta-continual learning (MCL) enables to meta-learn an efficient continual learner based on the recent advanced sequence models, e.g., Transformers. Although attention-free models (e.g., Linear Transformers) can ideally match CL's essential objective and efficiency requirements, they usually perform not well in MCL. Considering that the attention-free Mamba achieves excellent performances matching Transformers' on general sequence modeling tasks, in this paper, we aim to answer a question -- Can attention-free Mamba perform well on MCL? By formulating Mamba with a selective state space model (SSM) for MCL tasks, we propose to meta-learn Mamba as a continual learner, referred to as MambaCL. By incorporating a selectivity regularization, we can effectively train MambaCL. Through comprehensive experiments across various CL tasks, we also explore how Mamba and other models perform in different MCL scenarios. Our experiments and analyses highlight the promising performance and generalization capabilities of Mamba in MCL."},"_bibtex":{"value":"@misc{\nzhao2025learning,\ntitle={Learning Mamba as a Continual Learner},\nauthor={Chongyang Zhao and Dong Gong},\nyear={2025},\nurl={https://openreview.net/forum?id=1TXDtnDIsV}\n}"},"title":{"value":"Learning Mamba as a Continual Learner"},"pdf":{"value":"/pdf/65bb972e176ce4ef1a4d56629ad29b0e78847ac1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhao|learning_mamba_as_a_continual_learner"},"authorids":{"value":["~Chongyang_Zhao1","~Dong_Gong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chongyang Zhao","Dong Gong"]}},"version":2},{"content":{"summary":{"value":"The paper presents S7, a simplified state-space model (SSM) designed for long-range sequence tasks. Building on the S5 model, it introduces input-dependence to allow dynamic adjustments to state transitions based on input content. The paper claims S7 achieves stable reparameterization and efficient performance across diverse tasks, including neuromorphic event-based datasets and Long Range Arena (LRA) benchmarks, without requiring the complexity of models like S4 or S6 (Mamba). The proposed model is argued to balance computational simplicity with adaptive filtering and content-based reasoning."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. **Could the authors clarify the novelty of the reparameterization?** How does it differ from StableSSM’s reparameterization framework?\n\n2. **Why was the S5 model chosen as the basis for S7?** Given that S6 with the StableSSM reparameterization might provide similar benefits, what informed this design choice?\n\n3. **Could the authors specify if the neuromorphic-specific design applies solely to neuromorphic tasks or to all benchmarks?** This would improve clarity regarding the model's consistency across different tasks.\n\n4. **What is the author’s perspective on improving the model's performance on data whre time-dependance is note relevant?** Given S7’s limited success on LRA, is there a feasible modification that could address these challenges while preserving input-dependence?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"*Input-Dependent Dynamics:* S7’s adaptation of the  S5 model to be input-dependent is a promising approach. This could enhance the model’s responsiveness to input variability, a significant issue in long-range sequence tasks.\n\n*Stable Reparameterization:* The model claims to maintain gradient stability over long sequences, addressing gradient explosion/vanishing issues commonly faced in deep learning. This feature has potential benefits for training efficiency and stability.\n\n*Broad Applicability:* S7’s successful application across various domains, from physical time series to neuromorphic vision, suggests it may generalize well to different task types."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Limited Novelty:** The paper is introducing an input-dependent update mechanism (already introduced by S6)  stabilized through the reparameterization framework and key equations (Eqs. 6, 7, and 8) borrowed directly from StableSSM [1], raising concerns about the originality of the contributions. The paper only states that it was “inspired” by stable reparameterization, yet much of the core methodology relies on prior work.\n\n\n**Inconsistent Notation:** The notation for $𝐴_k$ is unclear, with dependency on input appearing inconsistently (e.g., it appears in Eq. 5 and line 267 -- e.g. $𝐴_k(u_k, \\theta_m)$ but is omitted elsewhere -- e.g. $𝐴_k(\\theta_m)$. This lack of uniformity in notation undermines the model’s theoretical presentation.\n\n**Weak Justification for S5 Model Selection:** S5 is mentioned as the basis for S7, but no rationale is provided for not using S6 (Mamba) and the reparameterization technique from StableSSM. Moreover, no connection or description of the S5 model is given (MIMO approach etc.)\n\n**Assumptions Clarity:** Assumptions (3.1, 3.2, and 3.3) are not well justified or examined for feasibility, and the text lacks guidance on implementing or verifying these assumptions. This leaves important theoretical aspects of the model unaddressed.\n\n**Unclear Contribution of Neuromorphic Design:** The neuromorphic-specific design choices in Section 3.4 seem disconnected from the rest of the model’s development (no other mention on the first part of the paper). It’s unclear whether these additions (Eq. 11, 12) apply exclusively to neuromorphic tasks or extend to other benchmark tasks.\n\n**Lack of Benchmark Justification:** The paper does not clarify why specific datasets were chosen. For instance, given the input-dependent nature of S7, benchmarks used by similar models like Mamba (e.g., Selective Copy or Induction Heads or other similar benchmarks -- see Section 7/Table 4 of the thematic survey [2]) might have been more appropriate for comparison.\n\n**Poor Performance on LRA Benchmarks:** S7’s subpar performance on LRA benchmarks raises concerns about its applicability to heterogeneous data. The authors provide only a brief discussion, without substantial insight or proposed solutions for improving performance on these challenging tasks.\n\n\n[1] Wang, Shida, and Qianxiao Li. \"Stablessm: Alleviating the curse of memory in state-space models through stable reparameterization.\" arXiv preprint arXiv:2311.14495 (2023).\n\n[2] Tiezzi, Matteo, et al. \"State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era.\" arXiv preprint arXiv:2406.09062 (2024)."}},"nonreaders":[],"tmdate":1731427551424,"tcdate":1730482202956,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2167/Reviewer_wghn"],"signatures":["ICLR.cc/2025/Conference/Submission2167/Reviewer_wghn"],"forum":"4wtcXV0kbi","number":2,"license":"CC BY 4.0","cdate":1730482202956,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2167/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427551424,"domain":"ICLR.cc/2025/Conference","replyto":"4wtcXV0kbi","id":"d9eoP37RTA","forumContent":{"TLDR":{"value":"A new sequence model for long sequence modeling employed on various tasks."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["state space models","neural network architectures","deep learning architectures","sequence modeling","event-based vision","event cameras","neural odes"]},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"A central challenge in sequence modeling is efficiently handling tasks with extended contexts. While recent state-space models (SSMs) have made significant progress in this area, they often lack input-dependent filtering or require substantial increases in model complexity to handle input variability. We address this gap by introducing S7, a simplified yet powerful SSM that can handle input dependence while incorporating stable reparameterization and specific design choices to dynamically adjust state transitions based on input content, maintaining efficiency and performance. We prove that this reparameterization ensures stability in long-sequence modeling by keeping state transitions well-behaved over time. Additionally, it controls the gradient norm, enabling efficient training and preventing issues like exploding or vanishing gradients. S7 significantly outperforms baselines across various sequence modeling tasks, including neuromorphic event-based datasets, Long Range Arena benchmarks, and various physical and biological time series. Overall, S7 offers a more straightforward approach to sequence modeling without relying on complex, domain-specific inductive biases, achieving significant improvements across key benchmarks."},"_bibtex":{"value":"@misc{\nsoydan2024s,\ntitle={S7: Selective and Simplified State Space Layers for Sequence Modeling},\nauthor={Taylan Soydan and Nikola Zubic and Nico Messikommer and Siddhartha Mishra and Davide Scaramuzza},\nyear={2024},\nurl={https://openreview.net/forum?id=4wtcXV0kbi}\n}"},"title":{"value":"S7: Selective and Simplified State Space Layers for Sequence Modeling"},"pdf":{"value":"/pdf/1fffab6aa0f02c9b5326306d49f2faa5435961a9.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"soydan|s7_selective_and_simplified_state_space_layers_for_sequence_modeling"},"authorids":{"value":["~Taylan_Soydan1","~Nikola_Zubic1","~Nico_Messikommer1","~Siddhartha_Mishra1","~Davide_Scaramuzza1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Taylan Soydan","Nikola Zubic","Nico Messikommer","Siddhartha Mishra","Davide Scaramuzza"]}},"version":2},{"content":{"summary":{"value":"The paper introduces Mamba-IVP, a state-space generative framework designed for clinical time-series modeling under extreme missingness and noise conditions. The model integrates a Mask-Aware Dual-Mamba Encoder (MADME) to jointly process observed values and missingness masks, ensuring robustness under block-wise gaps, and a Mamba-Hybrid Decoder (MHD) to reconstruct and denoise continuous trajectories through parallelizable latent evolution. Mamba-IVP is evaluated on three large-scale ICU datasets  (MIMIC-IV, PhysioNet 2012, and eICU) for forecasting and mortality classification tasks. The model achieves state-of-the-art performance, outperforming GRU-D, mTAN, and IVP-VAE in both accuracy and efficiency. It also provides a theoretical analysis of Mamba’s variance-contraction property, demonstrating stability under noisy and missing conditions.."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- **Model capacity**: Could the authors provide details on the number of parameters in Mamba-IVP (and optionally compare it to other baselines)? This would help assess the model’s complexity.\n\n- **Addressing concerns**: I would be glad to revise my evaluation if the authors can improve upon some of the points mentioned above, particularly by clarifying the scope of datasets, improving the presentation of Section 3, and expanding the discussion or comparisons with diffusion-based and consistency-based models."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- **Clinically important problem**: The paper tackles the early detection and modeling of sepsis, a highly impactful clinical task where accurate forecasting can have direct consequences for patient outcomes. The focus on handling severe missingness reflects a realistic and pressing challenge in ICU time-series data.\n\n- **Mathematically sound formulation**: The proposed Mamba-IVP framework is well formulated and theoretically motivated. The dual design of the encoder–decoder structure (MADME and MHD) is clearly presented, and the mathematical derivations appear consistent and rigorous.\n\n- **Strong empirical results**: The model achieves competitive or superior performance across multiple benchmark datasets (MIMIC-IV, PhysioNet 2012, and eICU), demonstrating both its robustness to missing data and its practical relevance for clinical forecasting and classification tasks"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Limited diversity of clinical data**: If I understand correctly, the three datasets used in the study (MIMIC-IV, PhysioNet 2012, and eICU) contain only ICU patients diagnosed with sepsis. Is there a specific reason for restricting the analysis to this condition? Expanding the evaluation to other diseases or broader ICU cohorts could better demonstrate the model’s generalization capacity and support its use.\n\n- **Clarity and structure of the method section**: While the proposed framework is mathematically sound, the explanation in Section 3 is somewhat difficult to follow. The presentation could benefit from a clearer structure, helping readers connect the MADME and MHD components more intuitively.\n\n- **Comparison with diffusion-based baselines**: The paper briefly mentions diffusion models such as CSDI and argues that they are less suited for this domain due to computational cost. However, including quantitative results for CSDI or PriSTI [1], even in a limited setting, would provide a clearer picture of where Mamba-IVP stands in terms of accuracy–efficiency trade-offs.\n\n- **Missing discussion on consistency models**: Beyond diffusion approaches, it would also be interesting to include a brief discussion on consistency models [2], which can be viewed as distilled, discrete formulations of diffusion models that emphasize faster inference. In the context of time-series imputation, the recently proposed CoSTI model [2] follows this idea and has also been applied to PhysioNet data. Understanding how Mamba-IVP compares to these diffusion-like but more efficient paradigms would clarify its position in the broader generative modeling landscape.\n\n[1] Liu, M., Huang, H., Feng, H., Sun, L., Du, B., & Fu, Y. (2023, April). Pristi: A conditional diffusion framework for spatiotemporal imputation. In 2023 IEEE 39th International Conference on Data Engineering (ICDE) (pp. 1927-1939). IEEE. https://arxiv.org/abs/2302.09746\n\n[2] Javier Solís-García, Belén Vega-Márquez, Juan A. Nepomuceno, and Isabel A. Nepomuceno-Chamorro. Costi: Consistency models for (a faster) spatio-temporal imputation. Knowledge-Based Systems, 327:114117, 2025. https://arxiv.org/abs/2501.19364"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764359889420,"tcdate":1760882497414,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4935/Reviewer_E9kx"],"signatures":["ICLR.cc/2026/Conference/Submission4935/Reviewer_E9kx"],"forum":"GG9OqLP0vO","number":1,"license":"CC BY 4.0","cdate":1760882497414,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4935/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764359889420,"domain":"ICLR.cc/2026/Conference","replyto":"GG9OqLP0vO","id":"KBbIwqXGok","forumContent":{"TLDR":{"value":"Mamba-IVP is a lightweight state-space generative model that robustly imputes clinical time series under noise and missingness, achieving SOTA accuracy with superior efficiency."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Clinical Time Series","Mask-Aware Dual-Mamba Encoder","Mamba-Hybrid Decoder","Initial Value Problem","Denoising Proof"]},"supplementary_material":{"value":"/attachment/6409f6ce08476099e0c6d5a23ed37d73c571f39b.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Missing clinical time series is a critical bottleneck in intensive care units (ICUs). In large-scale ICU electronic health record datasets such as MIMIC-IV, missing rates exceed 90% due to sensor failures, monitor degradation, and systemic outages, while aging devices inject unstable noise that makes reliable modeling nearly impossible. Existing methods remain unsafe for deployment: statistical heuristics distort missingness, deep models collapse under block-wise gaps and noise, and ODE- or diffusion-based approaches demand prohibitive computation. To overcome these limitations, we propose Mamba-IVP, a state-space generative model with a Mask-Aware Dual-Mamba Encoder (MADME) to handle block-wise missingness and a Mamba-Hybrid Decoder (MHD) to denoise continuous-time reconstructions. We validate our method through 61 experiments across two tasks: time series forcatsing and node classification. Our experiments involve 7 classic and state-of-the-art target models and 3 publicly available datasets: (1) it achieves state-of-the-art accuracy, reducing MSE by 3.0%, improving AUROC by 3.0%, and enhancing AUPRC by 3.9%; and (2) it remains robust under noise and block-wise missingness up to 12h, where other models degrade sharply."},"_bibtex":{"value":"@misc{\nfang2026mambaivp,\ntitle={Mamba-{IVP}: A Denoising State-Space Initial Value Problem Framework for {SOTA} Clinical Time Series, Healthcare Alternative},\nauthor={Jiayu Fang and Zhiqi Shao and Hongwei Ma and S T Boris Choy and Junbin Gao},\nyear={2026},\nurl={https://openreview.net/forum?id=GG9OqLP0vO}\n}"},"title":{"value":"Mamba-IVP: A Denoising State-Space Initial Value Problem Framework for SOTA Clinical Time Series, Healthcare Alternative"},"pdf":{"value":"/pdf/c2f5ef7880e75bc7edf617da1c90ed9c557976d5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"fang|mambaivp_a_denoising_statespace_initial_value_problem_framework_for_sota_clinical_time_series_healthcare_alternative"},"authorids":{"value":["~Jiayu_Fang1","~Zhiqi_Shao1","~Hongwei_Ma1","~S_T_Boris_Choy1","~Junbin_Gao1"]},"authors":{"value":["Jiayu Fang","Zhiqi Shao","Hongwei Ma","S T Boris Choy","Junbin Gao"]}},"version":2},{"content":{"summary":{"value":"The paper addresses the challenge of multivariate time series modeling using a neural architecture based on a variation of two-dimensional state-space models (SSMs), referred to as Chimera. This approach features a stack of 2D SSMs combined with nonlinearities, a decomposition of time series into trend and seasonal components, various discretization processes, time-variant parameters, and a novel two-dimensional parallel selective scan for rapid training with linear complexity. The authors demonstrate that Chimera can encompass multiple existing time series models, including SARIMA, SpaceTime, and S4nd. Experiments across various tasks, such as time series forecasting, classification, and anomaly detection, validate the effectiveness of Chimera."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See weaknesses."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- State-space models (SSMs) have proven to be very effective and efficient for modeling temporal data compared to Transformers. Designing new efficient SSM architectures for multivariate time series is an important and challenging problem.\n\n- Chimera allows the state space to vary in two dimensions (time and variate). It also combines multiple components to build a specific architecture for multivariate time series.\n\n- Chimera is compared to multiple baselines across various datasets for tasks such as time series forecasting, classification, and anomaly detection."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Motivation**\n  - Two-dimensional state-space models (2D-SSMs) have been considered for multivariate time series with specific structures, such as S4ND and 2DSSM. The authors discuss the discretization process and data-independent parameters as new challenges. However, I found lines 113-128 unconvincing regarding the need for a new 2D-SSM for multivariate time series.\n\n  I believe this paper proposes yet another deep model for time series, while it is known that after removing strong seasonality, multivariate time series from real-world phenomena often exhibit weak time and variate dependencies/signals, which do not require such complex models. This is very different from text and image data.\n\n- **Contributions**\n  - The paper essentially combines multiple existing ideas: 2D-SSM, bi-directional models [46], selection mechanisms [32], companion structures [27], convolutional forms [33], etc. There are no substantial or specific contributions. The authors claim that one of their main technical contributions is having input-dependent parameters, which is a relatively weak contribution.\n\n- **Experiments**\n  - Even considering the experimental results, many (simpler) methods achieve similar results to the proposed approach (see Tables 8-11).\n\n  - I also found that the paper tends to overstate its claims, using phrases like \"outstanding performance\" and \"significantly outperforms recurrent models, including very recent Mamba-based architectures.\"\n\n  - I question whether a fair comparison was made between the methods. For example, Chimera involves a time series decomposition (lines 218-219, 236-237), whereas other baselines do not. Can the authors comment on this?\n\n  Additionally, there is a lack of naive baselines, such as methods that treat the multivariate series as multiple univariate series.\n\n  It would also be useful to visualize some time series decompositions of Chimera to validate its accuracy.\n\n  - The authors write, \"We attribute the outstanding performance of Chimera, specifically compared to SpaceTime [27], to its ability to capture seasonal patterns and its input-dependent parameters, resulting in dynamically learned dependencies.\" Why didn't you confirm this by analyzing the results?\n\n  - Given that the authors combine multiple existing ideas/components, I found the ablation study relatively weak.\n    - \"The results show that all the components of Chimera contribute to its performance.\" I do not agree with this statement. Table 6 only shows that input-independence reduces accuracy (significantly).\n    - Why did you only choose ETT datasets?\n\n  - For the forecasting problems, it would be useful to provide the results per horizon or group multiple horizons together to analyze performance over the forecast horizon. Does the improvement come from the first few horizons?\n\n\n\n- **Data**\n  - I question the validity of the benchmark datasets used.\n    - For time series forecasting, there is the Monash Time Series Forecasting Repository (https://forecastingdata.org/). I think the multivariate time series datasets considered in the paper have been overly used and overfitted by the community.\n    - For anomaly detection, I encourage the authors to check the following paper: Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology (https://arxiv.org/abs/2308.13068)\n\n- **Other comments**\n  - I think Theorems 1-5 should be labeled as Propositions.\n  - Seasonal autoregressive process: The authors introduce the AR(p) model with a multivariate time series (x_k in R^d, not R^1). This is confusing as there is also the VAR(p) model with a different formulation. The AR(p) presented uses the same coefficients for each dimension.\n  - Table 7: What do you mean by \"series length\"? M4 series do not have six observations. Also, it is not long-term vs short-term but rather multivariate vs univariate time series. This should also be changed in the main text. Does this mean that Chimera has been used with both univariate and multivariate time series? What are the implications? Do you still need the second dimension in that case?\n  - The paper uses different terminologies: \"intervariate information flow,\" \"inter-variate dependencies,\" \"complicated dependencies,\" \"dynamically model the dependencies of variate and time dimensions,\" \"cannot capture the dynamics of dependencies.\" Please be consistent and define these terms clearly. There are a lot of \"buzz\" words without clear definitions.\n\n  - The authors write: \"Note that anomaly detection can be seen as a binary classification task.\" This is supervised anomaly detection, which is rare as labeled anomalies are hard to find. How did you handle the imbalance during training?\n\n- **Typos**\n  - Line 665: \"These matrices can be decompose\"\n  - Line 695: \"recursively calculate the the outputs\"\n\n- Please provide standard errors in your tables."},"limitations":{"value":"The limitations have not been discussed in the paper."}},"nonreaders":[],"tmdate":1730879687017,"tcdate":1720698618011,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission14135/Reviewer_hxax"],"signatures":["NeurIPS.cc/2024/Conference/Submission14135/Reviewer_hxax"],"forum":"ncYGjx2vnE","number":3,"license":"CC BY 4.0","cdate":1720698618011,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission14135/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879687017,"domain":"NeurIPS.cc/2024/Conference","replyto":"ncYGjx2vnE","id":"BcCvfWI8C4","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Multivariate Time Series","Time Series Modeling","Time Series Forecasting","Time Series Classification"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Modeling multivariate time series is a well-established problem with a wide range of applications from healthcare to financial markets. It, however, is challenging as it requires methods to (1) have high expressive power of representing complicated dependencies along the time axis to capture both long-term progression and seasonal patterns, (2) capture the inter-variate dependencies when it is informative, (3) dynamically model the dependencies of variate and time dimensions, and (4) have efficient training and inference for very long sequences. Traditional State Space Models (SSMs) are classical approaches for univariate time series modeling due to their simplicity and expressive power to represent linear dependencies. They, however, have fundamentally limited expressive power to capture non-linear dependencies, are slow in practice, and fail to model the inter-variate information flow. Despite recent attempts to improve the expressive power of SSMs by using deep structured SSMs, the existing methods are either limited to univariate time series, fail to model complex patterns (e.g., seasonal patterns), fail to dynamically model the dependencies of variate and time dimensions, and/or are input-independent.  We present Chimera, an expressive variation of the 2-dimensional SSMs with careful design of parameters to maintain high expressive power while keeping the training complexity linear. Using two SSM heads with different discretization processes and input-dependent parameters, Chimera is provably able to learn long-term progression, seasonal patterns, and desirable dynamic autoregressive processes. To improve the efficiency of complex 2D recurrence, we present a fast training using a new 2-dimensional parallel selective scan. Our experimental evaluation shows the superior performance of Chimera on extensive and diverse benchmarks, including ECG and speech time series classification, long-term and short-term time series forecasting, and time series anomaly detection."},"_bibtex":{"value":"@inproceedings{\nbehrouz2024chimera,\ntitle={Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models},\nauthor={Ali Behrouz and Michele Santacatterina and Ramin Zabih},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=ncYGjx2vnE}\n}"},"title":{"value":"Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models"},"pdf":{"value":"/pdf/293e7ef70612d586ad3576a085191e54b2c0eb16.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"behrouz|chimera_effectively_modeling_multivariate_time_series_with_2dimensional_state_space_models"},"authorids":{"value":["~Ali_Behrouz1","~Michele_Santacatterina2","~Ramin_Zabih1"]},"authors":{"value":["Ali Behrouz","Michele Santacatterina","Ramin Zabih"]}},"version":2},{"content":{"summary":{"value":"This paper presents Hybrid Mamba (HM), a method that combines the Mamba model and Transformer to enhance reinforcement learning (RL) performance. HM leverages Mamba to generate high-value sub-goals, which then condition the transformer, leading to significant improvements in online testing efficiency and task-solving capabilities."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see weakness. If I have misunderstood some parts of the paper, I welcome corrections and further discussion."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper is well-written and clear to read.\n2. HM significantly accelerates testing speed, achieving up to 28 times faster results than baseline methods.\n3. HM demonstrates superior performance across various benchmarks, such as D4RL, Grid World, and Tmaze."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. This paper claims to present a in-context RL approach. The motivation of this paper is concerned with the problems encountered with the no-gradient updates in-context approach (line 28), where the policy network does not require parameter updates. However, this paper uses a global update approach, which is closer to gradient-based and conditional-based offline RL. It seems to contradict the original intention of this paper. \n\n2. HM benefits from using a powerful subgoal encoder (Mamba in this case) and conditioning the policy network with subgoals. The performance improvement is expected and unsurprising due to the advantages inherent in conditional-based RL algorithms. Hence, it is necessary to further explain the unique contributions of combining Mamba and causal transformer in this paper.\n\n3. If the sub-goal encoder are replaced with other advanced and efficient sequence encoders (e.g., flash-attention1/2 [1,2], x-lstm [3]), would it also yield better or more efficient performance? \n\n4. The experiments demonstrating HW's efficacy in capturing long-term dependencies are unconvincing. Achieving good results in tasks with an arbitrarily horizon (e.g., Tmaze) does not necessarily prove effective long-term memory embedding. It is crucial to test the stability and performance of HM with varying horizon lengths or other length-based settings. For example, Mamba’s original paper [4] demonstrated the ability to capture long-term dependencies through the scaling laws.\n\n5. Could the authors clarify in which specific aspects HM's training time is faster than DT's? Since HM appears to be a combination of Mamba and DT.\n\n6. There are parts of the paper that are not clearly explained. For instance, in lines 228-233, it is mentioned that the transformer predicts a c-step action sequence (named $a_1$ here) through the sub-goal $z_t$ and another c-step action sequence (named $a_2$) through valuable sub-goals from offline data. How are $a_1$ and $a_2$ subsequently updated or processed? \n\n7. (minor) The paper contains some typos and inconsistencies in tense usage. For example, in the related work section, the section on Mamba uses the present tense, while the section on in-context RL uses the past tense. These should be corrected for consistency. In addition, what's the meaning of the different gaussian distribution figures in Figure 1?\n\n*Reference:*\n\n[1] Dao T, Fu D, Ermon S, et al. Flashattention: Fast and memory-efficient exact attention with io-awareness. NeurIPS 2022.\n\n[2] Dao T. Flashattention-2: Faster attention with better parallelism and work partitioning. ICLR 2024.\n\n[3] Beck M, Pöppel K, Spanring M, et al. xLSTM: Extended Long Short-Term Memory. arXiv 2024.\n\n[4] Gu A, Dao T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv 2023."},"limitations":{"value":"The authors raise some limitaions, for example, how to control the setting of hyperparameter $c$, which is not addressed in this paper but is claimed to be solved in the future."}},"nonreaders":[],"tmdate":1730879033050,"tcdate":1720442270481,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission5823/Reviewer_YUg4"],"signatures":["NeurIPS.cc/2024/Conference/Submission5823/Reviewer_YUg4"],"forum":"wFzIMbTsY7","number":2,"license":"CC BY 4.0","cdate":1720442270481,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission5823/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879033050,"domain":"NeurIPS.cc/2024/Conference","replyto":"wFzIMbTsY7","id":"0U27B892i7","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["In-context Reinforcement Learning","Mamba"]},"primary_area":{"value":"reinforcement_learning"},"abstract":{"value":"Recent works have shown the remarkable superiority of transformer models in reinforcement learning (RL), where the decision-making problem is formulated as sequential generation. Transformer-based agents could emerge with self-improvement in online environments by providing task contexts, such as multiple trajectories, called in-context RL. However, due to the quadratic computation complexity of attention in transformers, current in-context RL methods suffer from huge computational costs as the task horizon increases. In contrast, the Mamba model is renowned for its efficient ability to process long-term dependencies, which provides an opportunity for in-context RL to solve tasks that require long-term memory. To this end, we first implement Decision Mamba (DM) by replacing the backbone of Decision Transformer (DT). Then, we propose a Decision Mamba-Hybrid (DM-H) with the merits of transformers and Mamba in high-quality prediction and long-term memory. Specifically, DM-H first generates high-value sub-goals from long-term memory through the Mamba model. Then, we use sub-goals to prompt the transformer, establishing high-quality predictions. Experimental results demonstrate that DM-H achieves state-of-the-art in long and short-term tasks, such as D4RL, Grid World, and Tmaze benchmarks. Regarding efficiency, the online testing of DM-H in the long-term task is 28$\\times$ times faster than the transformer-based baselines."},"_bibtex":{"value":"@inproceedings{\nhuang2024decision,\ntitle={Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling},\nauthor={Sili Huang and Jifeng Hu and Zhejian Yang and Liwei Yang and Tao Luo and Hechang Chen and Lichao Sun and Bo Yang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=wFzIMbTsY7}\n}"},"title":{"value":"Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling"},"pdf":{"value":"/pdf/f79cbc369ef6968176c7cc958c79839cb99e59b0.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"huang|decision_mamba_reinforcement_learning_via_hybrid_selective_sequence_modeling"},"authorids":{"value":["~Sili_Huang1","~Jifeng_Hu1","~Zhejian_Yang1","~Liwei_Yang2","~Tao_Luo2","~Hechang_Chen2","~Lichao_Sun1","~Bo_Yang6"]},"authors":{"value":["Sili Huang","Jifeng Hu","Zhejian Yang","Liwei Yang","Tao Luo","Hechang Chen","Lichao Sun","Bo Yang"]}},"version":2},{"content":{"venue":{"value":"AAAI 2026"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/39317/43278"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"he|dual_mamba_for_nodespecific_representation_learning_tackling_oversmoothing_with_selective_state_space_modeling"},"html":{"value":"https://doi.org/10.1609/aaai.v40i26.39317"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/HeWDW26,\n  author={Xin He and Yili Wang and Yiwei Dai and Xin Wang},\n  title={Dual Mamba for Node-Specific Representation Learning: Tackling Over-Smoothing with Selective State Space Modeling},\n  year={2026},\n  cdate={1767225600000},\n  pages={21672-21680},\n  url={https://doi.org/10.1609/aaai.v40i26.39317},\n  booktitle={AAAI},\n  crossref={conf/aaai/2026}\n}\n"},"abstract":{"value":"Over-smoothing remains a fundamental challenge in deep Graph Neural Networks (GNNs), where repeated message passing causes node representations to become indistinguishable. While existing solutions, such as residual connections and skip layers, alleviate this issue to some extent, they fail to explicitly model how node representations evolve in a node-specific and progressive manner across layers. Moreover, these methods do not take global information into account, which is also crucial for mitigating the over-smoothing problem. To address the aforementioned issues, in this work, we propose a Dual Mamba-enhanced Graph Convolutional Network (DMbaGCN), which is a novel framework that integrates Mamba into GNNs to address over-smoothing from both local and global perspectives. DMbaGCN consists of two modules: the Local State-Evolution Mamba (LSEMba) for local neighborhood aggregation and utilizing Mamba’s selective state space modeling to capture node-specific representation dynamics across layers, and the Global Context-Aware Mamba (GCAMba) that leverages Mamba’s global attention capabilities to incorporate global context for each node. By combining these components, DMbaGCN enhances node discriminability in deep GNNs, thereby mitigating over-smoothing. Extensive experiments on multiple benchmarks demonstrate the effectiveness and efficiency of our method."},"title":{"value":"Dual Mamba for Node-Specific Representation Learning: Tackling Over-Smoothing with Selective State Space Modeling"},"authors":{"value":[{"fullname":"Xin He","username":""},{"fullname":"Yili Wang","username":""},{"fullname":"Yiwei Dai","username":"~Yiwei_Dai1"},{"fullname":"Xin Wang","username":""}]}},"tmdate":1790324113882,"pdate":1798675200000,"externalIds":["dblp:conf/aaai/HeWDW26"],"tcdate":1790324106810,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Yiwei_Dai1"],"forum":"Vlsa2mvWTv","license":"CC BY-SA 4.0","number":156429,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1790324113882,"domain":"OpenReview.net/Public_Article","id":"Vlsa2mvWTv","version":2},{"content":{"summary":{"value":"The paper introduces KambaAD, a model designed to improve anomaly detection in time series by integrating the Kolmogorov - Arnold Network (KAN) with attention mechanisms and the Selective Structured State Space Model (MAMBA). Specifically, the model incorporates different components (KAN, attention mechanisms, and MAMBA) into an encoder - decoder framework to detect anomalies. The paper conducts experiments on multivariate time series datasets, demonstrating a 5% improvement in F1 score over existing baselines."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- Could the authors elaborate on the unique advantages of combining KAN, attention, and MAMBA specifically for time series anomaly detection? What specific problems does each component address, and why is this integration particularly effective?\n\n- How does KambaAD perform in terms of computational efficiency and scalability, especially for high-dimensional or large-scale time series data? \n\n- How does KambaAD handle noisy, sparse, or incomplete data? \n\n- How sensitive is KambaAD to hyperparameter tuning? Are there recommended settings for its main components, and how would performance vary with suboptimal hyperparameters?\n\n- Given the complexity of the model, how can users interpret KambaADs detection results? Is there a way to identify which component contributes most to identifying an anomaly? Could the authors provide an ablation study to evaluate the impact of each component independently?\n\n- How does KambaAD compare with more recent listed models above? Could the authors include these models in their evaluation?\n\n- How does the performance of KambaAD vary with different sequence lengths or time window sizes? How to choose these parameters?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"- The paper considers both global and local anomalies, with the attention mechanism captures long range dependencies, while MAMBA focuses on local temporal dynamics. \n- The integration of KAN into encoder-decoder structure is relatively novel, but has limitations.\n- The model performance improves over existing baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper has some fundamental limitations:\n\n- Although the integration of KAN, attention mechanism and MAMBA are relatively novel, all those individual components are established techniques. The paper could benefit from a deeper exploration and justification on how this specific combination uniquely addresses challenges in anomaly detection. Without this theoretical-based exploration to provide a thorough understanding, the paper is seemingly a typical applied paper that tries to combine existing components to handle an existing task, which does not have not enough novelties and contributions for a conference like ICLR. \n\n- The paper needs a more thorough comparison with state of the art models, such as TranAD [1], GDN [2], TadGAN [3], MTAD-GAT [4], to assert the advantages and potential limitations of proposed model, and make clearer its position in comparison with the current SoA methods.\n\n- The paper lacks a detailed analysis of scalability where it examines high-dimensional datasets.\n\n- While KambaAD achieves strong results, the paper does not thoroughly analyze the sensitivity of key parameters (e.g., window size, patch size, and model layers) on model performance. Without understanding how sensitive KambaAD is to parameter choices, its robustness and adaptability to diverse datasets or unseen conditions remain unclear. A detailed sensitivity analysis would help practitioners better configure the model for specific applications.\n\n- KambaAD lacks the discussion of interpretability and explainability, especially, which component is responsible for detecting specific types of anomalies. The paper would benefit from visualizing or explaining the anomaly detection process by showing the relative contributions of KAN, attention, and MAMBA components in making the prediction.\n\n[1] https://arxiv.org/abs/2201.07284\n[2] https://arxiv.org/abs/2101.03804\n[3] https://arxiv.org/abs/2009.07769\n[4] https://arxiv.org/abs/2009.02040"}},"nonreaders":[],"tmdate":1732639464529,"tcdate":1730590670076,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4490/Reviewer_zjAh"],"signatures":["ICLR.cc/2025/Conference/Submission4490/Reviewer_zjAh"],"forum":"XmnTfSX5Az","number":3,"license":"CC BY 4.0","cdate":1730590670076,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4490/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732639464529,"domain":"ICLR.cc/2025/Conference","replyto":"XmnTfSX5Az","id":"5CmOkmogVS","forumContent":{"TLDR":{"value":"We introduce KambaAD, a framework combining MAMBA and KAN to enhance time series anomaly detection, outperforming state-of-the-art models across multiple datasets."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Anomaly Detection","Mamba","KAN","Time series"]},"supplementary_material":{"value":"/attachment/a7877f9569c65354bff9efad6c6e06a7477733f9.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Time series anomaly detection is critical in numerous practical applications, yet existing deep learning methods often fall short of real-world demands. These models fail to swiftly filter out physically implausible anomalies, insufficiently address distributional shifts, and lack a comprehensive approach that integrates both global and local perspectives for anomaly detection. Moreover, most successful models rely on channel-dependent methods that tend to treat all features at the same timestamp as a single token and then focus on finding relationships between these tokens. This approach overlooks the unique periodicities, trends, and lagged relationships between different features, leading to suboptimal performance. To address these limitations, we propose KambaAD, a model comprised of an Encoder and Reconstructor. The Encoder integrates the strengths of the Kolmogorov-Arnold Network (KAN), the attention mechanism, and the Selective Structured State Space Model (MAMBA). Specifically, KAN is employed to swiftly enforce data consistency, enabling rapid detection of anomalies that violate physical laws. The attention mechanism ensures balanced processing of global information while enhancing the representation of key data characteristics. We leverage MAMBA's capability as a sequence model to capture anomalies caused by local variations. Additionally, its internal selection mechanism allows the model to effectively handle distribution shifts, ensuring robustness and adaptability in the presence of changing data distributions. Additionally, the framework incorporates a time-series-specific Reconstructor, which reduces computational complexity through patch-based operations that exploit local consistency in time series data. It also employs channel-independent linear reconstruction to prevent interference between different features. Through extensive experiments on multiple multivariate datasets, KambaAD consistently outperforms state-of-the-art models, demonstrating its superior performance in anomaly detection."},"_bibtex":{"value":"@misc{\nchen2025kambaad,\ntitle={Kamba{AD}: Enhancing State Space Models with Kolmogorov{\\textendash}Arnold for time series Anomaly Detection},\nauthor={Yiyun Chen and Jia Guo and Xinchun Yu and Defu Cao and Xiao-Ping Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=XmnTfSX5Az}\n}"},"title":{"value":"KambaAD: Enhancing State Space Models with Kolmogorov–Arnold for time series Anomaly Detection"},"pdf":{"value":"/pdf/91e241263b9d771776b2d96ac63471b17016dc63.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chen|kambaad_enhancing_state_space_models_with_kolmogorovarnold_for_time_series_anomaly_detection"},"authorids":{"value":["~Yiyun_Chen2","~Jia_Guo9","~Xinchun_Yu1","~Defu_Cao1","~Xiao-Ping_Zhang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yiyun Chen","Jia Guo","Xinchun Yu","Defu Cao","Xiao-Ping Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper explores the quantization of the weight matrices in Orthogonal Recurrent Neural Nets (ORNNs). Authors introduce and compare two strategies for constructing ORNNs or approximately orthogonal RNNs with quantized weights (QORNNs): STE-projUNN and STE-Björck. These strategies are extensions of two methods of constructing full-precision ORNNs, respectively projUNN and Björck, that employ quantization-aware training with Straight-through assumption (STE).\n\nQORNNs are evaluated on a synthetic copy-task, PTB and few variants of MNIST (pMNIST, sMNIST) datasets and compared against their floating-point variants and other families of RNNs, including LSTM, GRU, fastRNN, fastGRNN. The most efficient models achieve results similar to state-of-the-art full-precision ORNN, LSTM and FastRNN on a variety of standard benchmarks, even with 3-bits quantization."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"* In Tables 1-2, could you explain what do you mean with “NU” (‘Not useful because of other figures’)?\n* What is the memory increase during training for the proposed methods compared to FastRNN, LSTM and vanilla RNN?\n* In Table 1, it would be nice if authors could also include accuracy for copy-task, as it is easier to interpret the differences compared to cross-entropy numbers.\n* Any reasoning why STE-Björk is faster than FastRNN and FastGRNN on pMNIST in Table 14? It seems quite surprising given that at every optimizer step Björck procedure requires running 15 iterations of the recursion with subsequent backpropagation through this procedure. While it's not the case for some other tasks, providing a detailed analysis of the computational complexity or runtime breakdown for STE-Björk vs. FastRNN/FastGRNN might be quite insightful."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"* Seems fairly easy to implement.\n* Authors thoroughly motivate and explain challenges of constructing and training ORNNs & instabilities caused by quantization. \n* A comparison of model sizes in resulting QORNN models vs. other methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The proposed methods seem very straightforward: combine the strategy of constructing ORNNs with STE, which is a standard and well-known technique in the quantization literature.\n* L286-L300: authors derive bounds for approximate orthogonality of q(W) which they themselves state are too loose to be useful in practice. It would be quite insightful to track and report $||W_q W_q^T – I ||$ during training, to see if the reason of proposed method working well in practice is due to $q(W)$ being fairly close to being orthogonal or not.\n* L079: authors claimed SotA results on pMNIST, however they did not compare against other 4-bit sequence-to-sequence models, for instance SSM models such as Mamba [1], transformer models such as LLaMA-3 [2] etc. \n* Considered benchmarks are very small by today standards. While most of them seem standard in ORNN literature, it would make a story more convincing if authors included some of the more recent real-world datasets and benchmarks. For instance, it would be insightful to evaluate the proposed methods on some of common reasoning language tasks (MMLU, HellaSwag, Winogrande).\n\n[1] Gu et al., “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”. ArXiV: 2312.00752\n\n[2] Dubey et al., “The Llama 3 Herd of Models”. ArXiV: 2407.21783"}},"nonreaders":[],"tmdate":1731427669501,"tcdate":1730479726695,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4641/Reviewer_LdC4"],"signatures":["ICLR.cc/2025/Conference/Submission4641/Reviewer_LdC4"],"forum":"zuKrRYM3Tg","number":2,"license":"CC BY 4.0","cdate":1730479726695,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4641/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427669501,"domain":"ICLR.cc/2025/Conference","replyto":"zuKrRYM3Tg","id":"zqOF6qqfrz","forumContent":{"TLDR":{"value":"This paper explores the quantization of orthogonal recurrent neural networks, and analyses the impact of the quantizer bitwidth on model performance."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["ecurrent neural networks","neural network quantization","orthogonal recurrent neural networks","quantization bitwidth"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"In recent years, Orthogonal Recurrent Neural Networks (ORNNs) have gained popularity due to their ability to manage tasks involving long-term dependencies, such as the copy task, and their linear complexity. However, existing ORNNs utilize full precision weights and activations, which prevents their deployment on compact devices.\n\nIn this paper, we explore the quantization of the weight matrices in ORNNs, leading to Quantized approximately Orthogonal RNNs (QORNNs). The construction of such networks remained an open problem, acknowledged for its inherent instability. We propose and investigate two strategies to learn QORNN by combining quantization-aware training (QAT) and orthogonal projections. We also study post-training quantization of the activations for pure integer computation of the recurrent loop. The most efficient models achieve results similar to state-of-the-art full-precision ORNN, LSTM and FastRNN on a variety of standard benchmarks, even with 3-bits quantization."},"_bibtex":{"value":"@misc{\nfoucault2025quantized,\ntitle={Quantized Approximately Orthogonal Recurrent Neural Networks},\nauthor={Armand Foucault and Francois Malgouyres and Franck Mamalet},\nyear={2025},\nurl={https://openreview.net/forum?id=zuKrRYM3Tg}\n}"},"title":{"value":"Quantized Approximately Orthogonal Recurrent Neural Networks"},"pdf":{"value":"/pdf/6ad5969f13e1f6c3b7e019a5072c629170e11449.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"foucault|quantized_approximately_orthogonal_recurrent_neural_networks"},"authorids":{"value":["~Armand_Foucault1","~Francois_Malgouyres1","~Franck_Mamalet2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Armand Foucault","Francois Malgouyres","Franck Mamalet"]}},"version":2},{"content":{"venue":{"value":"IJCAI 2025"},"pdf":{"value":"https://www.ijcai.org/proceedings/2025/0595.pdf"},"venueid":{"value":"dblp.org/conf/IJCAI/2025"},"paperhash":{"value":"he|mambabased_graph_convolutional_networks_tackling_oversmoothing_with_selective_state_space"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xin_He_0003:","https://dblp.org/search/pid/api?q=author:Yili_Wang_0004:","~Wenqi_Fan1","https://dblp.org/search/pid/api?q=author:Xu_Shen_0002:","https://dblp.org/search/pid/api?q=author:Xin_Juan:","https://dblp.org/search/pid/api?q=author:Rui_Miao_0003:","https://dblp.org/search/pid/api?q=author:Xin_Wang_0035:"]},"html":{"value":"https://doi.org/10.24963/ijcai.2025/595"},"_bibtex":{"value":"@inproceedings{DBLP:conf/ijcai/00030FSJ0025,\n  author={Xin He and Yili Wang and Wenqi Fan and Xu Shen and Xin Juan and Rui Miao and Xin Wang},\n  title={Mamba-Based Graph Convolutional Networks: Tackling Over-smoothing with Selective State Space},\n  year={2025},\n  cdate={1735689600000},\n  pages={5345-5353},\n  url={https://doi.org/10.24963/ijcai.2025/595},\n  booktitle={IJCAI},\n  crossref={conf/ijcai/2025}\n}\n"},"abstract":{"value":"Graph Neural Networks (GNNs) have shown great success in various graph-based learning tasks. However, it often faces the issue of over-smoothing as the model depth increases, which causes all node representations to converge to a single value and become indistinguishable. This issue stems from the inherent limitations of GNNs, which struggle to distinguish the importance of information from different neighborhoods. In this paper, we introduce MbaGCN, a novel graph convolutional architecture that draws inspiration from the Mamba paradigm—originally designed for sequence modeling. MbaGCN presents a new backbone for GNNs, consisting of three key components: the Message Aggregation Layer, the Selective State Space Transition Layer, and the Node State Prediction Layer. These components work in tandem to adaptively aggregate neighborhood information, providing greater flexibility and scalability for deep GNN models. While MbaGCN may not consistently outperform all existing methods on each dataset, it provides a foundational framework that demonstrates the effective integration of the Mamba paradigm into graph representation learning. Through extensive experiments on benchmark datasets, we demonstrate that MbaGCN paves the way for future advancements in graph neural network research. Our code is in https://github.com/hexin5515/MbaGCN."},"title":{"value":"Mamba-Based Graph Convolutional Networks: Tackling Over-smoothing with Selective State Space"},"authors":{"value":["Xin He","Yili Wang","Wenqi Fan","Xu Shen","Xin Juan","Rui Miao","Xin Wang"]}},"tmdate":1768448130219,"pdate":1735689600000,"externalIds":["dblp:conf/ijcai/00030FSJ0025"],"tcdate":1768448019756,"writers":["~"],"signatures":["~Wenqi_Fan1"],"forum":"Dp43rl3yzc","license":"CC BY-SA 4.0","number":754680,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1768448130219,"domain":"DBLP.org","id":"Dp43rl3yzc","version":2},{"content":{"TLDR":{"value":"We propose LION, a framework for extending Linear Transformers to the bidirectional setting by providing three theoretically equivalent representations: full attention, bidirectional RNN, and chunkwise parallel form."},"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["Transformer","RNN","Bidirectionality","SSM","Linear Attention","Inference","Training"]},"supplementary_material":{"value":"/attachment/4169741b64da3a01f7d6f6eca3a47def1bcc05d7.zip"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Linear Transformers and State Space Models have emerged as efficient alternatives\nto softmax Transformers for causal sequence modeling, enabling parallel training\nvia matrix multiplication and efficient RNN-style inference. However, despite their\nsuccess in causal tasks, no unified framework exists for applying Linear Transformers to bidirectional sequence modeling. We introduce LION, the first framework to\nsystematically extend Linear Transformers to the bidirectional setting. LION generalizes three core representations commonly used in the causal case—full Linear\nAttention , bidirectional RNN, and chunkwise parallel form—to the bidirectional\nsetting. These forms are theoretically equivalent and enable models to exploit the\nstrengths of each during training and inference. We prove that a broad class of\nLinear Transformers can be extended using LION and validate our framework via\nthree core examples based on the choice of decay type: LION-LIT, the bidirectional\nextension of [25]; LION-D, based on [44]; and LION-S, a variant using selective\ndecay [34, 13]. Across standard bidirectional tasks, LION enables models to match\nor exceed the performance of softmax Transformers, while offering significantly\nfaster training and more efficient inference than existing State Space Models."},"_bibtex":{"value":"@inproceedings{\nafzal2025linear,\ntitle={Linear Attention for Efficient Bidirectional Sequence Modeling},\nauthor={Arshia Afzal and Elias Abad Rocamora and Leyla Naz Candogan and Pol Puigdemont and Francesco Tonin and Yongtao Wu and Mahsa Shoaran and Volkan Cevher},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=Ar62cqTduE}\n}"},"title":{"value":"Linear Attention for Efficient Bidirectional Sequence Modeling"},"pdf":{"value":"/pdf/3086b4c96b928c2816b1182de45669b2fb8cb21f.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"afzal|linear_attention_for_efficient_bidirectional_sequence_modeling"},"authorids":{"value":["~Arshia_Afzal1","~Elias_Abad_Rocamora1","~Leyla_Naz_Candogan1","~Pol_Puigdemont1","~Francesco_Tonin1","~Yongtao_Wu1","~Mahsa_Shoaran1","~Volkan_Cevher1"]},"authors":{"value":["Arshia Afzal","Elias Abad Rocamora","Leyla Naz Candogan","Pol Puigdemont","Francesco Tonin","Yongtao Wu","Mahsa Shoaran","Volkan Cevher"]}},"tmdate":1783627295797,"pdate":1758217402120,"tcdate":1746992782765,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission23483/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission23483/Authors"],"forum":"Ar62cqTduE","license":"CC BY 4.0","number":23483,"cdate":1746992782765,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission23483/-/Full_Submission","NeurIPS.cc/2025/Conference/Submission23483/-/Supplementary_Material","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission23483/-/Camera_Ready_Revision"],"mdate":1783627295797,"odate":1761705045573,"domain":"NeurIPS.cc/2025/Conference","id":"Ar62cqTduE","version":2},{"content":{"summary":{"value":"the paper MamKO leverages Mamba architecture to generate koopman operators  with projected benefits towards model predication capability and adaptability relative to koopman models with constant koopman operators.  MamKO generates the Koopman Operator from realtime data making the model learn nonlinear behavior over time. To demonstrate capability the MamKO model is applied to model predictive control (MPC).  The modeling and control performance of the proposed method is evaluated through experiments on benchmark time-invariant and time-varying systems.\nFour claims are made as advancing the state of the art\n1. Matrix generation using Mamba structure\n2. approach can handle unstable and time varying systems\n3. Computationally efficient MPC is developed based on the MamKO approach\n4. Experiments are provided in support. \nOverall, the paper is well written and the developments are clear and error free.  While the authors have not demonstrated the results against any real world practical system, they do provide adequate development for a practitioner to adopt into real world problems and study applicability.  In that regard I think this work is worth publishing so the SOTA in Kooman based approaches   can continue to be explored for real world use."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Recommended fixes:\nwhich provides enhanced model predictability and adaptability, as compared to Koopman models with constant Koopman operators -- you mean prediction capability  ?\nSince time scales are important , figures should include time , instead of time steps ( or epochs) , Fig 8 just has  \"t\" with no units,  Please provide unit son all figure axes where possible   \n\nClaims requiring more evidence:  \nMamKO achieves good balance between advanced modeling capabilities and real-time control implementation efficiency --> \nThis claim in RT implementation needs more support hence actual time units are needed.  The RSCP problem has a scale of hours and is therefore generally very slow.  It will be good to see clearly the range of problems in time scale and frequency scale to understanding capability against system dynamics. \n\nTheoretical:  The paper could use more rigorous theoretical analysis or proofs to back the claims of stability and effectiveness in capturing time-varying dynamics. For example, formal proofs of the convergence properties of the Mamba-based Koopman operators or bounds on the error in approximating nonlinear systems would greatly increase the paper’s credibility.\n\nEquation (5): The transition from a time-invariant to a time-varying state-space model is theoretically justified but lacks a rigorous mathematical proof. The paper relies heavily on empirical evidence, perhaps some theoretical discussion may be help support eth generalizability of the method.\n\nCELU Activation: The use of the negative CELU function is an interesting choice to handle instability. The mathematical justification provided for this choice seems reasonable, but more in-depth analysis of how it impacts model stability across different systems would strengthen the claim."},"rating":{"value":8},"details_of_ethics_concerns":{"value":"no concerns on ethics"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"the paper has several strengths\n1. Leveraging Mamba for generating time-varying Koopman operators to address the limitations of fixed linear state-space models is innovative.\n2. The authors  effectively incorporate MamKO into an MPC framework and demonstrate practical applications.  ability to formulate the control problem as a convex optimization task adds to the framework's appeal for real-time applications.\n3. The experimental validation (although all simulations)  is based on multiple benchmark systems (e.g., CartPole, Gene Regulatory Networks, and CSTRs). Results demonstrate the advantages of MamKO over traditional methods like Deep Koopman Operator (DKO) and MLP-based models. \n4. Real-Time Adaptability: By generating matrices in real-time, MamKO addresses the challenge of modeling time-varying systems efficiently. The proposed method's performance on rapidly changing systems highlights its practical utility.\n5. Supplemental material allows reproducibility"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The framework assumes that the Mamba-generated matrices adequately capture the dynamics of the underlying system. This assumption may not hold for  highly nonlinear systems where matrix representations could become unstable or inaccurate.\n\nExperimental results do support the claims of improved performance. However, the paper could enhance its rigor by including statistical analyses (e.g., confidence intervals or hypothesis tests) to demonstrate the significance of the results.\n\nAuthors do acknowledge that their method may not generalize to all real-world systems, but a more explicit discussion on the boundaries of applicability (e.g., systems with extremely high-frequency dynamics) would add value.\n\nComparison: The benchmark methods used have all been solved using classical/modern control methods, yet eth author did not include any non data-driven solutions for comparison.  This may be a useful addition to the paper."}},"nonreaders":[],"tmdate":1731428340619,"tcdate":1730561145809,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10895/Reviewer_A4oR"],"signatures":["ICLR.cc/2025/Conference/Submission10895/Reviewer_A4oR"],"forum":"hNjCVVm0EQ","number":3,"license":"CC BY 4.0","cdate":1730561145809,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10895/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428340619,"domain":"ICLR.cc/2025/Conference","replyto":"hNjCVVm0EQ","id":"LtsGkdYmuz","forumContent":{"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Koopman operator; model predictive control; nonlinear systems"]},"supplementary_material":{"value":"/attachment/0bf4c76c44568ae8539ede6166bb6a47f0d2be3a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The Koopman theory, which enables the transformation of nonlinear systems into linear representations, is a powerful and efficient tool to model and control nonlinear systems. However, the ability of the Koopman operator to model complex systems, particularly time-varying systems, is limited by the fixed linear state-space representation. To address the limitation, the large language model, Mamba, is considered a promising strategy for enhancing modeling capabilities while preserving the linear state-space structure.\nIn this paper, we propose a new framework, the Mamba-based Koopman operator (MamKO), which provides enhanced model prediction capability and adaptability, as compared to Koopman models with constant Koopman operators. Inspired by the Mamba structure, MamKO generates Koopman operators from online data; this enables the model to effectively capture the dynamic behaviors of the nonlinear system over time. A model predictive control system is then developed based on the proposed MamKO model. The modeling and control performance of the proposed method is evaluated through experiments on benchmark time-invariant and time-varying systems. The experimental results demonstrate the superiority of the proposed approach. Additionally, we perform ablation experiments to test the effectiveness of individual components of MamKO. This approach unlocks new possibilities for integrating large language models with control frameworks, and it achieves a good balance between advanced modeling capabilities and real-time control implementation efficiency."},"_bibtex":{"value":"@inproceedings{\nli2025mamko,\ntitle={Mam{KO}: Mamba-based Koopman operator for modeling and predictive control},\nauthor={ZHAOYANG LI and Minghao Han and Xunyuan Yin},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=hNjCVVm0EQ}\n}"},"title":{"value":"MamKO: Mamba-based Koopman operator for modeling and predictive control"},"pdf":{"value":"/pdf/49097a18837a65cc1415fac755f61c717559a104.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"li|mamko_mambabased_koopman_operator_for_modeling_and_predictive_control"},"authorids":{"value":["~ZHAOYANG_LI12","~Minghao_Han2","~Xunyuan_Yin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["ZHAOYANG LI","Minghao Han","Xunyuan Yin"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2412.08160v4"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"yuan|dgmamba_robust_and_efficient_dynamic_graph_structure_learning_with_selective_state_space_models"},"html":{"value":"https://doi.org/10.48550/arXiv.2412.08160"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-08160,\n  publtype={informal},\n  author={Haonan Yuan and Qingyun Sun and Zhaonan Wang and Xingcheng Fu and Cheng Ji and Yongjian Wang and Bo Jin and Jianxin Li},\n  title={DG-Mamba: Robust and Efficient Dynamic Graph Structure Learning with Selective State Space Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2412.08160},\n  url={https://doi.org/10.48550/arXiv.2412.08160}\n}\n"},"abstract":{"value":"Dynamic graphs exhibit intertwined spatio-temporal evolutionary patterns, widely existing in the real world. Nevertheless, the structure incompleteness, noise, and redundancy result in poor robustness for Dynamic Graph Neural Networks (DGNNs). Dynamic Graph Structure Learning (DGSL) offers a promising way to optimize graph structures. However, aside from encountering unacceptable quadratic complexity, it overly relies on heuristic priors, making it hard to discover underlying predictive patterns. How to efficiently refine the dynamic structures, capture intrinsic dependencies, and learn robust representations, remains under-explored. In this work, we propose the novel DG-Mamba, a robust and efficient Dynamic Graph structure learning framework with the Selective State Space Models (Mamba). To accelerate the spatio-temporal structure learning, we propose a kernelized dynamic message-passing operator that reduces the quadratic time complexity to linear. To capture global intrinsic dynamics, we establish the dynamic graph as a self-contained system with State Space Model. By discretizing the system states with the cross-snapshot graph adjacency, we enable the long-distance dependencies capturing with the selective snapshot scan. To endow learned dynamic structures more expressive with informativeness, we propose the self-supervised Principle of Relevant Information for DGSL to regularize the most relevant yet least redundant information, enhancing global robustness. Extensive experiments demonstrate the superiority of the robustness and efficiency of our DG-Mamba compared with the state-of-the-art baselines against adversarial attacks."},"title":{"value":"DG-Mamba: Robust and Efficient Dynamic Graph Structure Learning with Selective State Space Models"},"authors":{"value":[{"fullname":"Haonan Yuan","username":""},{"fullname":"Qingyun Sun","username":"~Qingyun_Sun2"},{"fullname":"Zhaonan Wang","username":""},{"fullname":"Xingcheng Fu","username":""},{"fullname":"Cheng Ji","username":""},{"fullname":"Yongjian Wang","username":""},{"fullname":"Bo Jin","username":""},{"fullname":"Jianxin Li","username":""}]}},"tmdate":1784621890655,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2412-08160"],"tcdate":1784621875115,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Qingyun_Sun2"],"forum":"zBJu6dRlJC","license":"CC BY-SA 4.0","number":80850,"cdate":1704067200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784621890655,"domain":"OpenReview.net/Public_Article","id":"zBJu6dRlJC","version":2},{"content":{"venue":{"value":"AAAI 2025"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/34382/36537"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"yuan|dgmamba_robust_and_efficient_dynamic_graph_structure_learning_with_selective_state_space_models"},"html":{"value":"https://doi.org/10.1609/aaai.v39i21.34382"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/YuanS0F0WJ025,\n  author={Haonan Yuan and Qingyun Sun and Zhaonan Wang and Xingcheng Fu and Cheng Ji and Yongjian Wang and Bo Jin and Jianxin Li},\n  title={DG-Mamba: Robust and Efficient Dynamic Graph Structure Learning with Selective State Space Models},\n  year={2025},\n  cdate={1735689600000},\n  pages={22272-22280},\n  url={https://doi.org/10.1609/aaai.v39i21.34382},\n  booktitle={AAAI},\n  crossref={conf/aaai/2025}\n}\n"},"abstract":{"value":"Dynamic graphs exhibit intertwined spatio-temporal evolutionary patterns, widely existing in the real world. Nevertheless, the structure incompleteness, noise, and redundancy result in poor robustness for Dynamic Graph Neural Networks (DGNNs). Dynamic Graph Structure Learning (DGSL) offers a promising way to optimize graph structures. However, aside from encountering unacceptable quadratic complexity, it overly relies on heuristic priors, making it hard to discover underlying predictive patterns. How to efficiently refine the dynamic structures, capture intrinsic dependencies, and learn robust representations, remains under-explored. In this work, we propose the novel DG-Mamba, a robust and efficient Dynamic Graph structure learning framework with the Selective State Space Models (Mamba). To accelerate the spatio-temporal structure learning, we propose a kernelized dynamic message-passing operator that reduces the quadratic time complexity to linear. To capture global intrinsic dynamics, we establish the dynamic graph as a self-contained system with State Space Model. By discretizing the system states with the cross-snapshot graph adjacency, we enable the long-distance dependencies capturing with the selective snapshot scan. To endow learned dynamic structures more expressive with informativeness, we propose the self-supervised Principle of Relevant Information for DGSL to regularize the most relevant yet least redundant information, enhancing global robustness. Extensive experiments demonstrate the superiority of the robustness and efficiency of our DG-Mamba compared with the state-of-the-art baselines against adversarial attacks."},"title":{"value":"DG-Mamba: Robust and Efficient Dynamic Graph Structure Learning with Selective State Space Models"},"authors":{"value":[{"fullname":"Haonan Yuan","username":""},{"fullname":"Qingyun Sun","username":"~Qingyun_Sun2"},{"fullname":"Zhaonan Wang","username":""},{"fullname":"Xingcheng Fu","username":""},{"fullname":"Cheng Ji","username":""},{"fullname":"Yongjian Wang","username":""},{"fullname":"Bo Jin","username":""},{"fullname":"Jianxin Li","username":""}]}},"tmdate":1784621889488,"pdate":1767139200000,"externalIds":["dblp:conf/aaai/YuanS0F0WJ025"],"tcdate":1784621874644,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Qingyun_Sun2"],"forum":"KIbyW7Oo7U","license":"CC BY-SA 4.0","number":80834,"cdate":1735689600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784621889488,"domain":"OpenReview.net/Public_Article","id":"KIbyW7Oo7U","version":2},{"content":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Efficient/Low-Resource Methods for NLP","Linear Sequence Modeling","Machine Learning for NLP"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Linear sequence modeling methods, such as linear attention, state space modeling, and linear RNNs, offer significant efficiency improvements by reducing the complexity of training and inference. However, these methods typically compress the entire input sequence into a single fixed-size memory state, which leads to suboptimal performance on recall-intensive tasks. To address this limitation, we introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes multiple independent memory states, with a router network directing input tokens to specific memory states. This approach greatly enhances the overall memory capacity while minimizing memory interference. MoM serves as a general framework that can be seamlessly combined with diverse memory update mechanisms across linear models. As a result, MoM performs exceptionally well on recall-intensive tasks, surpassing existing linear sequence modeling techniques. Despite incorporating multiple memory states, the computation of each memory state remains linear in complexity, allowing MoM to retain the linear-complexity advantage during training, while constant-complexity during inference. Our experimental results show that MoM outperforms current linear sequence models on downstream language tasks, particularly recall-intensive tasks, and even achieves performance comparable to Transformer models."},"_bibtex":{"value":"@inproceedings{\ndu2026mom,\ntitle={MoM: Linear Sequence Modeling with Mixture-of-Memories},\nauthor={Jusen Du and Weigao Sun and Disen Lan and Jiaxi Hu and Tao Zhang and Yu Cheng},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=3PdOq8Rgue}\n}"},"title":{"value":"MoM: Linear Sequence Modeling with Mixture-of-Memories"},"pdf":{"value":"/pdf/6d8ebeff897bedb6faacfae3dae56e0c956cf7ca.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"du|mom_linear_sequence_modeling_with_mixtureofmemories"},"authorids":{"value":["~Jusen_Du1","~Weigao_Sun1","~Disen_Lan1","~Jiaxi_Hu1","~Tao_Zhang9","~Yu_Cheng1"]},"authors":{"value":["Jusen Du","Weigao Sun","Disen Lan","Jiaxi Hu","Tao Zhang","Yu Cheng"]}},"tmdate":1775877028666,"pdate":1769435940804,"tcdate":1758086908294,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8491/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8491/Authors"],"forum":"3PdOq8Rgue","license":"CC BY 4.0","number":8491,"cdate":1758086908294,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission8491/-/Full_Submission","ICLR.cc/2026/Conference/Submission8491/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission8491/-/Camera_Ready_Revision"],"mdate":1775877028666,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"3PdOq8Rgue","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.07145v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"chen|stuffed_mamba_state_collapse_and_state_capacity_of_rnnbased_longcontext_modeling"},"authorids":{"value":["~Yingfa_Chen1","https://dblp.org/search/pid/api?q=author:Xinrong_Zhang:","https://dblp.org/search/pid/api?q=author:Shengding_Hu:","~Xu_Han2","https://dblp.org/search/pid/api?q=author:Zhiyuan_Liu_0001:","~Maosong_Sun1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.07145"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-07145,\n  publtype={informal},\n  author={Yingfa Chen and Xinrong Zhang and Shengding Hu and Xu Han and Zhiyuan Liu and Maosong Sun},\n  title={Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.07145},\n  url={https://doi.org/10.48550/arXiv.2410.07145}\n}\n"},"abstract":{"value":"One essential advantage of recurrent neural networks (RNNs) over transformer-based language models is their linear computational complexity concerning the sequence length, which makes them much faster in handling long sequences during inference. However, most publicly available RNNs (e.g., Mamba and RWKV) are trained on sequences with less than 10K tokens, and their effectiveness in longer contexts remains largely unsatisfying so far. In this paper, we study the cause of the inability to process long context for RNNs and suggest critical mitigations. We examine two practical concerns when applying state-of-the-art RNNs to long contexts: (1) the inability to extrapolate to inputs longer than the training length and (2) the upper bound of memory capacity. Addressing the first concern, we first investigate *state collapse* (SC), a phenomenon that causes severe performance degradation on sequence lengths not encountered during training. With controlled experiments, we attribute this to overfitting due to the recurrent state being overparameterized for the training length. For the second concern, we train a series of Mamba-2 models on long documents to empirically estimate the recurrent state capacity in language modeling and passkey retrieval. Then, three SC mitigation methods are proposed to improve Mamba-2's length generalizability, allowing the model to process more than 1M tokens without SC. We also find that the recurrent state capacity in passkey retrieval scales exponentially to the state size, and we empirically train a Mamba-2 370M with near-perfect passkey retrieval accuracy on 256K context length. This suggests a promising future for RNN-based long-context modeling."},"title":{"value":"Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling"},"authors":{"value":["Yingfa Chen","Xinrong Zhang","Shengding Hu","Xu Han","Zhiyuan Liu","Maosong Sun"]}},"tmdate":1759858747896,"pdate":1704067200000,"tcdate":1747711825509,"writers":["~"],"signatures":["~Xu_Han2"],"forum":"qkPmoStFzB","license":"CC BY-SA 4.0","number":532041,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1759858747896,"domain":"DBLP.org","id":"qkPmoStFzB","version":2},{"content":{"summary":{"value":"This paper proposes a new method to extend the context length of Mamba. The authors start with an explanation of why Mamba in its original form cannot extend the context length. They propose viewing the S6 block as applying an “attention operation” (Section 2) and based on these attention weights, they compute the Mamba mean distance to measure the effective receptive field of Mamba (Section 3). They basically show that the ERF decreases as the context length at inference time increases and that the main culprit is the fast decrease of the sum of discretization steps. To tackle this issue, they introduce Decimamba, a filtering mechanism that discards tokens of lesser importance (Section 4). They finally show on multiple information retrieval benchmarks that the method better performs on long contexts than Mamba."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"I asked my questions in the weakness section."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"Overall, I like this paper, I think that Mamba is a very appealing method due to its low inference cost and getting methods that allow extending the context length for Mamba is a very important question. I appreciate the fact that the authors considered diverse benchmarks and the gap between Mamba and Decimamba is pretty consistent in some cases. I also appreciated the scientific approach in the paper that consists in isolating  the problematic component in Mamba and proposing a method to alleviate the issue."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I have a few concerns regarding this paper that I list below: \n\n- **Hyperparameter choices**: I agree with the fact that the fast decay of the sum of the discrete time steps may explain the lack of length generalization. However, the approach looks a bit hacky in that it introduces multiple novel hyperparameters: the decay factor, the maximal length of the sequence after the first decimating later and the number of layers to decimate. And it does not seem very clear how to make these choices without a gridsearch?\n- **Ablations**: have you tried to do an ablation with respect to the number of layers to be decimated or with respect to the decay rate? \nWriting should be improved: I found the decimation strategy in Decimamba (Section 4) pretty hard to follow. I think that this should be better explained. \n- **Tasks where Decimamba offers much higher gains?**: we see that Decimamba leads sometimes to big improvements (Squad and Passkey retrieval) but on Multi-Document QA, the decay of Mamba seems to also be important. Do you have an explanation/intuition of the type of tasks where Decimamba leads to substantial improvements?\n\n\n\nMinor points:\n\n**Improving the introduction**: I think the author should maybe better explain the Decimamba method in the introduction (and maybe add Figure 5 to the introduction?). \n- **Lack of length generalization of Mamba**: I was a bit confused about the fact that the authors were surprised by the lack of length generalization of Mamba. I agree with the authors that compared to Transformers, the context cache (the hidden states) do not grow with the context length and thus Mamba can scale the context length to infinity. However, this state is **bounded** and thus the model cannot store all the information it may need . In an information retrieval setting, when the number of documents is large, it is easy to imagine that Mamba has trouble deciding which tokens to store. Anyway, this all to say that the authors should maybe highlight that one of the factors explaining the poor length generalization is the small state space size. I think that this is aligned with the method proposed by the authors in that it looks to decimate some non-important tokens that may be captured in the hidden state.\n- **Influence of hidden state size on length generalization**: one thing that i am curious about is: have you tried to study the length generalization of models of similar scale but with one of the models having a bigger state space?\n- **Comparison with Transformers**: have you tried to run Transformers in the benchmarks of Section 5? Just curious to see the gap of DeciMamba with Transformers.\n- **Out of memory**: Maybe I would add a comment saying that the OOM happens in Figure 6 because the complexity at **training time** is O(L). Most people may have in mind the complexity of Mamba at inference time, that is independent of the context length and may be confused by the OOM you encounter. \n- **Typo**: In equation (8), i think the letters \"j\" and \"L\" were swapped. It should be $d(L,j)$."}},"nonreaders":[],"tmdate":1731428295515,"tcdate":1730657024471,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5066/Reviewer_PVqe"],"signatures":["ICLR.cc/2025/Conference/Submission5066/Reviewer_PVqe"],"forum":"iWSl5Zyjjw","number":4,"license":"CC BY 4.0","cdate":1730657024471,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5066/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428295515,"domain":"ICLR.cc/2025/Conference","replyto":"iWSl5Zyjjw","id":"ME8gzctzal","forumContent":{"TLDR":{"value":"We explore Mamba's length extrapolation potential, identifying its limitation due to a restricted effective receptive field, and introduce DeciMamba, the first context-extension method specifically designed for Mamba."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Long context","Context extension","Long-range language modeling"]},"supplementary_material":{"value":"/attachment/7b340eb64654c7225cf43f5c6138ec972b453804.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrates high performance and achieves Transformer-level capabilities while requiring substantially fewer computational resources. In this paper we explore the length-generalization capabilities of Mamba, which we find to be relatively limited. Through a series of visualizations and analyses we identify that the limitations arise from a restricted effective receptive field, dictated by the sequence length used during training. To address this constraint, we introduce DeciMamba, a context-extension method specifically designed for Mamba. This mechanism, built on top of a hidden filtering mechanism embedded within the S6 layer, enables the trained model to extrapolate well even without additional training. Empirical experiments over real-world long-range NLP tasks show that DeciMamba can extrapolate to context lengths that are significantly longer than the ones seen during training, while enjoying faster inference. We will release our code and models."},"_bibtex":{"value":"@inproceedings{\nben-kish2025decimamba,\ntitle={DeciMamba: Exploring the Length Extrapolation Potential of Mamba},\nauthor={Assaf Ben-Kish and Itamar Zimerman and Shady Abu-Hussein and Nadav Cohen and Amir Globerson and Lior Wolf and Raja Giryes},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=iWSl5Zyjjw}\n}"},"title":{"value":"DeciMamba: Exploring the Length Extrapolation Potential of Mamba"},"pdf":{"value":"/pdf/b2e18164fc1679d810d80cd5fdb09a2cc1a6003f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"benkish|decimamba_exploring_the_length_extrapolation_potential_of_mamba"},"authorids":{"value":["~Assaf_Ben-Kish1","~Itamar_Zimerman1","~Shady_Abu-Hussein1","~Nadav_Cohen1","~Amir_Globerson1","~Lior_Wolf1","~Raja_Giryes1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Assaf Ben-Kish","Itamar Zimerman","Shady Abu-Hussein","Nadav Cohen","Amir Globerson","Lior Wolf","Raja Giryes"]}},"version":2},{"content":{"summary":{"value":"This paper introduces DRAMA, a model-based reinforcement learning (MB-RL) agent that leverages the Mamba-2 architecture, a state space model (SSM), as its core dynamics architecture. Traditional MB-RL approaches often rely on recurrent neural networks (RNNs) or transformers for world modelling, which suffer from issues like vanishing gradients, difficulty in capturing long-term dependencies, and quadratic scaling of computational complexity with sequence length. DRAMA addresses these challenges by utilizing Mamba-2, which achieves linear computational and memory complexity while effectively capturing long-term dependencies.\n\nAdditionally, the authors propose a novel dynamic frequency-based sampling (DFS) method to mitigate the suboptimality arising from imperfect world models during early training stages. They evaluate DRAMA on the Atari100k benchmark, demonstrating that it achieves performance comparable to state-of-the-art algorithms using a significantly smaller world model (7 million parameters) that can be trained on standard hardware. The paper also includes ablation studies comparing Mamba-1 and Mamba-2, highlighting the superior performance of Mamba-2 despite its constrained expressive power."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- **Evidence of Long-Term Dependency Capture:** You mention the ability of DRAMA to capture long-term dependencies. Could you provide more direct evidence or experiments that demonstrate this capability? For example, have you considered tasks that specifically require long-term memory or conducted analyses that quantify the effective memory length of the model?\n- **Scaling DRAMA for Direct Comparison:** Have you considered scaling up DRAMA to match the model size of DreamerV3 for a direct comparison? If not, could you explain the limitations (e.g., hardware constraints) that prevented this? Testing DRAMA with larger models on a few games could provide insights into its scalability and performance limits.  \n- **Effectiveness of DFS:** Could you elaborate on how the dynamic frequency-based sampling (DFS) method compares to other sampling strategies in terms of its impact on learning efficiency and final performance? Including quantitative comparisons or ablation studies would be helpful.\n - **Hyperparameter Sensitivity and Trade-offs:** Can you provide insights into the trade-offs between model size and performance in DRAMA? Specifically, how does increasing the size of the Mamba-2 model or the autoencoder affect results across different games?\n- **Code and Reproducibility:** Is there a plan to release the code and pretrained models for DRAMA to facilitate reproducibility and further research in this area?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- **Originality:** The paper introduces the novel application of Mamba-2 SSMs within MB-RL, specifically as the dynamics model in the world model. This is a new approach that addresses the limitations of existing architectures like typical RNNs and transformers and it makes a lot of sense in my opinion.\n- **Quality:** The authors provide thorough experimental evaluations on the Atari100k benchmark, demonstrating that DRAMA achieves competitive performance with significantly fewer parameters (at least in the tasks and model sizes tested). The inclusion of ablation studies comparing Mamba-1 and Mamba-2, as well as the impact of DFS, strengthens the empirical results.\n- **Clarity:** The paper is well-written and structured, providing clear explanations of the methodology, including detailed descriptions of Mamba-2 and how it is integrated into the world model. The figures and tables effectively support the textual content although I would make some further stylistic improvements in Figure 1 to maximise clarity.\n- **Practical impact:** By achieving comparable performance to state-of-the-art methods with a smaller and more computationally efficient model, the DRAMA method contributes to making MB-RL more accessible and practical, particularly in resource-constrained environments."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Unsupported claims about capturing long-term dependencies:** While the authors claim repeatedly that Mamba-2 effectively handling long-term dependencies, the paper provides limited direct evidence or analysis to demonstrate this capability. Including experiments or analyses that specifically test and showcase the ability to capture long-term dependencies would strengthen the paper. For instance, a task designed to require long-term memory or metrics that quantify the model's ability to capture dependencies over long sequences could be included. In the current form of the paper, it is not clear to me what \"long\" really means.\n- **Limited Comparison with Scaled Models:** The comparison with DreamerV3 is somewhat limited. While the authors emphasize parameter efficiency, it would be valuable to see how DRAMA performs when scaled up to match the model size of DreamerV3, even on a subset of games. This would help understand the limits and potential of their approach and whether the advantages of Mamba-2 persist at larger scales. If hardware limitations prevented this, a discussion of these constraints would be very helpful.\n- **Marginal Performance Gains:** While DRAMA achieves comparable performance to existing methods, the improvements are not substantial across all games. Demonstrating scenarios where DRAMA significantly outperforms other approaches would strengthen the claims about its effectiveness.\n- **DFS Method Clarity:** The explanation of the dynamic frequency-based sampling method could be more detailed. Providing more comprehensive comparisons with other sampling strategies would help in understanding its effectiveness. Additionally, including ablation studies that isolate the impact of DFS would clarify its contribution.\n- **Hyperparameter Sensitivity:** The paper mentions that increasing the model size leads to better performance but does not deeply explore this aspect. Though I understand that extensive hyperparameter search might be too difficult given computational constraints, some analysis of how sensitive DRAMA is to hyperparameter choices, including model size, sequence length, and learning rates, would be very valuable for understanding its practical applicability, otherwise the paper looks relatively incomplete."}},"nonreaders":[],"tmdate":1732734955561,"tcdate":1730665744883,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission7606/Reviewer_3CFw"],"signatures":["ICLR.cc/2025/Conference/Submission7606/Reviewer_3CFw"],"forum":"7XIkRgYjK3","number":4,"license":"CC BY 4.0","cdate":1730665744883,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission7606/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732734955561,"domain":"ICLR.cc/2025/Conference","replyto":"7XIkRgYjK3","id":"Thwrl17mCr","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Model based reinforcement learning","Atari100k","Mamba-2"]},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Model-based reinforcement learning (RL) offers a solution to the data inefficiency that plagues most model-free RL algorithms. However, learning a robust world model often requires complex and deep architectures, which are computationally expensive and challenging to train. Within the world model, sequence models play a critical role in accurate predictions, and various architectures have been explored, each with its own challenges. Currently, recurrent neural network (RNN)-based world models struggle with vanishing gradients and capturing long-term dependencies. Transformers, on the other hand, suffer from the quadratic memory and computational complexity of self-attention mechanisms, scaling as $O(n^2)$, where $n$ is the sequence length.\n\nTo address these challenges, we propose a state space model (SSM)-based world model, Drama, specifically leveraging Mamba, that achieves $O(n)$ memory and computational complexity while effectively capturing long-term dependencies and enabling efficient training with longer sequences. We also introduce a novel sampling method to mitigate the suboptimality caused by an incorrect world model in the early training stages. Combining these techniques, Drama achieves a normalised score on the Atari100k benchmark that is competitive with other state-of-the-art (SOTA) model-based RL algorithms, using only a 7 million-parameter world model. Drama is accessible and trainable on off-the-shelf hardware, such as a standard laptop. Our code is available at https://github.com/realwenlongwang/Drama.git."},"_bibtex":{"value":"@inproceedings{\nwang2025drama,\ntitle={Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient},\nauthor={Wenlong Wang and Ivana Dusparic and Yucheng Shi and Ke Zhang and Vinny Cahill},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=7XIkRgYjK3}\n}"},"title":{"value":"Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient"},"pdf":{"value":"/pdf/2fd478f42038a3b2375180a33d90544518a2805f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wang|drama_mambaenabled_modelbased_reinforcement_learning_is_sample_and_parameter_efficient"},"authorids":{"value":["~Wenlong_Wang4","~Ivana_Dusparic2","~Yucheng_Shi5","~Ke_Zhang30","~Vinny_Cahill1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Wenlong Wang","Ivana Dusparic","Yucheng Shi","Ke Zhang","Vinny Cahill"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.04320v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"behrouz|chimera_effectively_modeling_multivariate_time_series_with_2dimensional_state_space_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Ali_Behrouz:","~Michele_Santacatterina2","https://dblp.org/search/pid/api?q=author:Ramin_Zabih:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.04320"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-04320,\n  publtype={informal},\n  author={Ali Behrouz and Michele Santacatterina and Ramin Zabih},\n  title={Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.04320},\n  url={https://doi.org/10.48550/arXiv.2406.04320}\n}\n"},"abstract":{"value":"Modeling multivariate time series is a well-established problem with a wide range of applications from healthcare to financial markets. Traditional State Space Models (SSMs) are classical approaches for univariate time series modeling due to their simplicity and expressive power to represent linear dependencies. They, however, have fundamentally limited expressive power to capture non-linear dependencies, are slow in practice, and fail to model the inter-variate information flow. Despite recent attempts to improve the expressive power of SSMs by using deep structured SSMs, the existing methods are either limited to univariate time series, fail to model complex patterns (e.g., seasonal patterns), fail to dynamically model the dependencies of variate and time dimensions, and/or are input-independent. We present Chimera that uses two input-dependent 2-D SSM heads with different discretization processes to learn long-term progression and seasonal patterns. To improve the efficiency of complex 2D recurrence, we present a fast training using a new 2-dimensional parallel selective scan. We further present and discuss 2-dimensional Mamba and Mamba-2 as the spacial cases of our 2D SSM. Our experimental evaluation shows the superior performance of Chimera on extensive and diverse benchmarks, including ECG and speech time series classification, long-term and short-term time series forecasting, and time series anomaly detection."},"title":{"value":"Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models"},"authors":{"value":["Ali Behrouz","Michele Santacatterina","Ramin Zabih"]}},"tmdate":1738781836051,"pdate":1704067200000,"tcdate":1738781832903,"writers":["~"],"signatures":["~Michele_Santacatterina2"],"forum":"Rx4r2NviRP","license":"CC BY-SA 4.0","number":296424,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1738781836051,"domain":"DBLP.org","id":"Rx4r2NviRP","version":2},{"content":{"pdf":{"value":"/pdf/c7b305bef1d7ee17e98f690153f2910293b7dc9e.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"behrouz|chimera_effectively_modeling_multivariate_time_series_with_2dimensional_state_space_models"},"authorids":{"value":["~Ali_Behrouz1","~Michele_Santacatterina2","~Ramin_Zabih1"]},"abstract":{"value":"Modeling multivariate time series is a well-established problem with a wide range of applications from healthcare to financial markets. Traditional State Space Models (SSMs) are classical approaches for univariate time series modeling due to their simplicity and expressive power to represent linear dependencies. They, however, have fundamentally limited expressive power to capture non-linear dependencies, are slow in practice, and fail to model the inter-variate information flow. Despite recent attempts to improve the expressive power of SSMs by using deep structured SSMs, the existing methods are either limited to univariate time series, fail to model complex patterns (e.g., seasonal patterns), fail to dynamically model the dependencies of variate and time dimensions, and/or are input-independent. We present Chimera that uses two input-dependent 2-D SSM heads with different discretization processes to learn long-term progression and seasonal patterns. To improve the efficiency of complex 2D recurrence, we present a fast training using a new 2-dimensional parallel selective scan. We further present and discuss 2-dimensional Mamba and Mamba-2 as the spacial cases of our 2D SSM. Our experimental evaluation shows the superior performance of Chimera on extensive and diverse benchmarks, including ECG and speech time series classification, long-term and short-term time series forecasting, and time series anomaly detection."},"title":{"value":"Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models"},"authors":{"value":["Ali Behrouz","Michele Santacatterina","Ramin Zabih"]}},"tmdate":1725553830481,"pdate":1717694944281,"tcdate":1725553781387,"writers":["~Ali_Behrouz1","~Michele_Santacatterina2","~Ramin_Zabih1"],"signatures":["~Ali_Behrouz1"],"forum":"UJvZPkCzgq","license":"CC BY 4.0","number":28132,"cdate":1725553781387,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1725553830481,"domain":"OpenReview.net/Archive","id":"UJvZPkCzgq","version":2},{"content":{"summary":{"value":"This paper introduces a novel architecture for diffusion models that leverages spatial and frequency information to emphasize local features in image generation. It integrates wavelet transformation into the state-space networks, such as Mamba, to enhance the awareness of local structures in visual inputs. The outputs are fused with original Mamba outputs using a cross-attention fusion layer to optimize order awareness, crucial for image generation quality. A globally-shared transformer is added to boost Mamba's performance. The method achieves state-of-the-art results on standard benchmarks, with faster training convergence and high-quality outputs."},"soundness":{"value":4},"confidence":{"value":5},"questions":{"value":"as above"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":4},"strengths":{"value":"- Overall, I appreciate this paper. It selects the most promising technical stacks—combining flow matching and Mamba—to address generative tasks.\n- The paper is well-organized, featuring valuable methodology design and experimental ablations.\n- It's interesting to see that the community has started considering state exploration in Mamba, as indicated in Line245. The initialization of the state is critical. We need to examine the code to understand how to implement it efficiently. See https://github.com/state-spaces/mamba/issues/101 for more details."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- In Line 106, I'm unsure why sigma brings extra computation when the scan path is larger, given that the computation is always fixed.\n- In Line105, it's important to note that flow matching has been utilized in various domains to capture the reader's interest. E.g., boosting diffusion[1], depth estimation[2], motiom[3], even text generation[4].\n- In Line 176, I'm curious as to why only B, C, and delta are time-dependent. Why isn't A?\n- In Line183, One paper is missed. [5]\n- In Fig 3a, what does the notation on ZigMa mean?\n- Regarding line 261, what's the rationale behind sharing those parameters of transformers? Are there any previous works indicating its effectiveness?\n- In Table 1, it would be better to display training steps. Researchers of generative models tend to compare using training steps rather than epochs.\n- In Line342, for sweep4, considering four forwards in a single-forward usually leads to a worse training speed (iter/s) and increased memory usage. It would be interesting to compare these two aspects with other methods.\n- In Fig5, is there any reference to the jpeg-8 scan path? How does it defined?\n\nSuggestions:\n\n- The previously mentioned figure or table should be placed first.\n\n[1]. Boosting Latent Diffusion with Flow Matching\n\n[2]. DepthFM: Fast Monocular Depth Estimation with Flow Matching\n\n[3]. Motion Flow Matching for Human Motion Synthesis and Editing\n\n[4]. Flow Matching for Conditional Text Generation in a Few Sampling Steps\n\n[5], Latent Space Editing in Transformer-based Flow Matching\n\nOverall, I think the exploration of using Mamba and flow matching is the right way to go, while I have some technical concerns about it. I am eager to hear the authors’ feedback."},"limitations":{"value":"as above"}},"nonreaders":[],"tmdate":1730879645860,"tcdate":1719786336549,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission13587/Reviewer_nhSz"],"signatures":["NeurIPS.cc/2024/Conference/Submission13587/Reviewer_nhSz"],"forum":"KqbLzSIXkm","number":2,"license":"CC BY 4.0","cdate":1719786336549,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission13587/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879645860,"domain":"NeurIPS.cc/2024/Conference","replyto":"KqbLzSIXkm","id":"KBlK9fcS2b","forumContent":{"TLDR":{"value":"A new Mamba architecture for image diffusion models"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Diffusion models","Mamba","Wavelet transformation"]},"supplementary_material":{"value":"/attachment/1281fd9115978629e9ed30829686033a665680cb.zip"},"primary_area":{"value":"generative_models"},"abstract":{"value":"We introduce a novel state-space architecture for diffusion models, effectively harnessing spatial and frequency information to enhance the inductive bias towards local features in input images for image generation tasks. While state-space networks, including Mamba, a revolutionary advancement in recurrent neural networks, typically scan input sequences from left to right, they face difficulties in designing effective scanning strategies, especially in the processing of image data. Our method demonstrates that integrating wavelet transformation into Mamba enhances the local structure awareness of visual inputs and better captures long-range relations of frequencies by disentangling them into wavelet subbands, representing both low- and high-frequency components. These wavelet-based outputs are then processed and seamlessly fused with the original Mamba outputs through a cross-attention fusion layer, combining both spatial and frequency information to optimize the order awareness of state-space models which is essential for the details and overall quality of image generation. Besides, we introduce a globally-shared transformer to supercharge the performance of Mamba, harnessing its exceptional power to capture global relationships. Through extensive experiments on standard benchmarks, our method demonstrates superior results compared to DiT and DIFFUSSM, achieving faster training convergence and delivering high-quality outputs. The codes and pretrained models are released at https://github.com/VinAIResearch/DiMSUM.git."},"_bibtex":{"value":"@inproceedings{\nphung2024dimsum,\ntitle={Di{MSUM}: Diffusion Mamba - A Scalable and Unified Spatial-Frequency Method for Image Generation},\nauthor={Hao Phung and Quan Dao and Trung Tuan Dao and Hoang Phan and Dimitris N. Metaxas and Anh Tuan Tran},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=KqbLzSIXkm}\n}"},"title":{"value":"DiMSUM: Diffusion Mamba - A Scalable and Unified Spatial-Frequency Method for Image Generation"},"pdf":{"value":"/pdf/0ae8bfdeeec0ac6c1b9be00728313d0eee7040d2.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"phung|dimsum_diffusion_mamba_a_scalable_and_unified_spatialfrequency_method_for_image_generation"},"authorids":{"value":["~Hao_Phung1","~Quan_Dao1","~Trung_Tuan_Dao1","~Hoang_Phan1","~Dimitris_N._Metaxas1","~Anh_Tuan_Tran2"]},"authors":{"value":["Hao Phung","Quan Dao","Trung Tuan Dao","Hoang Phan","Dimitris N. Metaxas","Anh Tuan Tran"]}},"version":2},{"content":{"summary":{"value":"This paper proposes MambaMatcher, a Mamba-based approach to establish semantic correspondences. MambaMatcher attempts to model inter-image correlations by treating multi-level correlation scores at each position in the correlation map as a state in a state-space model. Specifically, the multi-level features obtained by the frozen extractor are first enhanced through feature aggregation. The enhanced features are used to construct a multi-level correlation map. Then, the correlation map is refined with the proposed similarity-aware selective scan mechanism. Experimental results demonstrate that MambaMatcher achieves a good balance between accuracy and efficiency in establishing semantic correspondences."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please provide more discussions and experimental results to address the above weaknesses.\n\n-------------After the Discussion Period---------------\n\nThank the authors for the clear and detailed responses to my questions. As mentioned in the strengths I concluded, I consider this work as a meaningful exploration. I also agree with the authors that MambaMatcher can complement some existing components. However, my concern in W1 is mainly about the significance of this work, which is not addressed well in the rebuttal. According to the responses, the core Mamba-based module provides only modest improvements compared with some other parts like feature extractors and single-image feature aggregations. These results indicate that the significance of this work is relatively small. Considering both such strengths and weaknesses, I prefer to maintain my initial score. Thanks.\n\nMoreover, I can understand the responses for W2.1. However, in my opinion, the statements about “high resolution” in the manuscript are confusing because the authors did not clarify that the resolution is discussed for the 4D correlation map rather than the original 2D feature map."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":2},"strengths":{"value":"1. Mamba is a potential architecture to balance the accuracy and efficiency of establishing semantic correspondences. It is meaningful to explore how to apply the Mamba structure in this task.\n2. The organization and presentation of this paper are clear."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**1. The core design of MambaMatcher, i.e., treating the correlation scores at each position in the correlation map as a state in a state-space model, seems to provide only a minor benefit.**\n\nAccording to the results in Table 4 and Table 5, the combination of a DINOv2 feature extractor, a 2D-Conv_{k=5} feature aggregation and the existing correlation aggregation processes (based on 4D-Conv or FastFormer) provide a similar accuracy compared with the proposed MambaMatcher. This phenomenon indicates that the selection of feature extractors and the design of single-image feature aggregation may be more significant problems in establishing semantic correspondences. Replacing the existing 4D-Conv or FastFormer with the Mamba structure in the correlation aggregation process just gives a small improvement. Such a performance decreases the significance of the motivation and the core design in this paper.\n\n**2. Some claims in this paper are not discussed clearly.**\n\n2.1. In Line 48, the authors claim that one problem of the existing correlation-metric approaches is that they “severely limit the feature map resolution”. However, the proposed MambaMatcher only processes the 30x30 feature map. Could we consider the 30x30 size as a large resolution? Maybe more experiments are required to validate MambaMatcher’s superiority in handling larger feature maps.\n\n2.2. In Line 69, the authors claim that “MambaMatcher seamlessly integrates feature-metric and correlation-metric approaches into a unified pipeline.” It indicates that some parts of MambaMatcher are feature-metric-based while some are correlation-metric-based. However, there is no discussion to clarify such a claim.\n\n**3. The introduction of the proposed “similarity-aware selective scan mechanism” is not clear enough.**\n\nThe first reason of this problem is that Figure 2 is not cited in the main text. Besides, some equations may make the process of “similarity-aware selective scan mechanism” clearer."}},"nonreaders":[],"tmdate":1733286127069,"tcdate":1730779187565,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5072/Reviewer_Xgjj"],"signatures":["ICLR.cc/2025/Conference/Submission5072/Reviewer_Xgjj"],"forum":"GmE8ovvXaJ","number":4,"license":"CC BY 4.0","cdate":1730779187565,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5072/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733286127069,"domain":"ICLR.cc/2025/Conference","replyto":"GmE8ovvXaJ","id":"OY5mcPEIXU","forumContent":{"TLDR":{"value":"We introduce MambaMatcher, which formulates Mamba layers to process the correlation map between image pairs to establish accurate semantic correspondences with improved performance and efficiency."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["semantic correspondence","state-space models","correlation aggregation","Mamba","feature aggregation","Similarity-aware Selective Scan"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Establishing semantic correspondences between images is a fundamental yet challenging task in computer vision. Traditional feature-metric methods enhance visual features but may miss complex inter-image relationships, while recent correlation-metric approaches attempt to model these relationships but are hindered by high computational costs due to processing 4D correlation maps. We introduce MambaMatcher, a novel method that overcomes these limitations by efficiently modeling high-dimensional correlations using selective state-space models (SSMs), treating multi-level correlation scores as states. By implementing a similarity-aware selective scan mechanism adapted from Mamba’s linear-complexity algorithm, MambaMatcher refines the 4D correlation tensor effectively without compromising feature map resolution or receptive field. Experiments on standard semantic correspondence benchmarks demonstrate that MambaMatcher achieves state-of-the-art performance without relying on large input images or computationally expensive diffusion-based feature extractors, effectively capturing rich inter-image correlations while maintaining computational efficiency."},"_bibtex":{"value":"@misc{\nkim2025selective,\ntitle={Selective State-Space Modeling of Correlation Maps for Semantic Correspondence},\nauthor={Seungwook Kim and Minsu Cho},\nyear={2025},\nurl={https://openreview.net/forum?id=GmE8ovvXaJ}\n}"},"title":{"value":"Selective State-Space Modeling of Correlation Maps for Semantic Correspondence"},"pdf":{"value":"/pdf/904fad05fa77d7d08d5defc1a4f7abaade7973d6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"kim|selective_statespace_modeling_of_correlation_maps_for_semantic_correspondence"},"authorids":{"value":["~Seungwook_Kim2","~Minsu_Cho1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seungwook Kim","Minsu Cho"]}},"version":2},{"content":{"summary":{"value":"This paper presents MambaTS, an LTSF model addressing Transformers' self-attention complexity and bias by using causal relationships for global dependency modeling. The author designs variable-aware scan along time to get variable causal relationships and also Temporal Mamba Block to avoid causal convolution. The experimental results show that MambaTS outperforms several state-of-the-art models."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. In proposition 1, the assumption is that the causal graph exists. What if it doesn't exist? And is there any support on the random walk without return is a promising approach to estimate causal links? \n2. What is the definition of cost C and how we can get/set it empirically (e.g., the cost from node i to node j)?\n3. Proposition 2 indicates that theoretically the causal relationships can be estimated without infinite random walks with return. I would like to ask how many walks are required empirically. And also the time spent? \n4. In Eq (6), how we can get the p^{(0)}? \n5. In my view, another major difference between MambaTS and iTransformer is that MambaTS model the dependencies on both time and variables, while iTransformer mainly on variables. I would like to know how this contributes to eventual performance. Because UniTST [1] is also modeling the dependencies on both time and variables dimensions, but with Transformer architecture. How does MambaTS compare with UniTST? \n\nReference:\n\n[1] UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"1. It is interesting to see another new work on mamba for time series forecasting. In my view, some properties of mamba are fit for time series and it's an interesting direction to explore more. \n2. The authors propose several designs to tailor mamba for time series application, which has its merits."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The clarity of the paper needs to be improved. In some parts, I cannot fully understand, such as the cost of a random walk without return. Also what is the cost from node i to node j, and how we can get this in the first iteration\n2. Some claims have no support/evidence. For example, the authors mention that the random walk without return is a promising approach to estimate causal links. I would like to know the reason, e.g., any citations/proofs. \n3. The experiments seem not comprehensive. The authors only compare MambaTS with 7 baselines. There are a few more after iTransformer, which are worth to be compared. E.g., ModernTCN [1], UniTST [2], TSLANet [3]. \n\n\nReferences:\n\n[1] ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis. \n\n[2] UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting. \n\n[3] TSLANet: Rethinking Transformers for Time Series Representation Learning"}},"nonreaders":[],"tmdate":1732872171092,"tcdate":1730274887256,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10330/Reviewer_pAtf"],"signatures":["ICLR.cc/2025/Conference/Submission10330/Reviewer_pAtf"],"forum":"vEtDApqkNR","number":1,"license":"CC BY 4.0","cdate":1730274887256,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10330/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732872171092,"domain":"ICLR.cc/2025/Conference","replyto":"vEtDApqkNR","id":"qP5b7lknfI","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"This paper questions the necessity of self-attention in long-term sequence forecasting and introduces MambaTS, which models global dependencies across time and variables by leveraging causal relationships through a single linear scan."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forcasting; State Space Model"]},"supplementary_material":{"value":"/attachment/dc191874f3a708459705ff4c9016637ffa6cb3d7.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"In recent years, Transformers have become the de-facto architecture for long-term sequence forecasting (LTSF), yet they face challenges associated with the self-attention mechanism, including quadratic complexity and permutation invariant bias. This raises an important question: \\emph{do we truly need the self-attention mechanism to establish long-range dependencies in LTSF?} Recognizing the significance of causal relationships in multivariate LTSF, we propose MambaTS, which leverages causal relationships to model global dependencies across time and variables through a single linear scan. However, causal graphs are often unknown. To address this, we introduce variable-aware scan along time (VAST), which dynamically discovers variable relationships during training and decodes the optimal variable scan order by solving the shortest path visiting all nodes problem during inference. MambaTS employs the latest Mamba model as its backbone. We suggest that the causal convolution in Mamba is unnecessary due to the presence of independent variables, leading to the development of the Temporal Mamba Block (TMB). To mitigate model overfitting, we further incorporate a dropout mechanism for selective parameters in TMB. Extensive experiments conducted on eight public datasets demonstrate that MambaTS achieves new state-of-the-art performance."},"_bibtex":{"value":"@misc{\ncai2025mambats,\ntitle={Mamba{TS}: Improved Selective State Space Models for Long-term Time Series Forecasting},\nauthor={Xiuding Cai and Yaoyao Zhu and Xueyao Wang and Yu Yao},\nyear={2025},\nurl={https://openreview.net/forum?id=vEtDApqkNR}\n}"},"title":{"value":"MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting"},"pdf":{"value":"/pdf/d5d8697d5fd93b05c6b17f60b0f8962287f5c5f2.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"cai|mambats_improved_selective_state_space_models_for_longterm_time_series_forecasting"},"authorids":{"value":["~Xiuding_Cai1","~Yaoyao_Zhu2","~Xueyao_Wang1","~Yu_Yao4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xiuding Cai","Yaoyao Zhu","Xueyao Wang","Yu Yao"]}},"version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Mamba","Dynamic Fractal","Renormalization Group (RG) flow","Time series","Spatial Transcriptomics","sequenal mode"]},"supplementary_material":{"value":"/attachment/5b95aefda54bcae337133554349e3d0440980807.zip"},"_bibtex":{"value":"@inproceedings{\nfang2026dynamic,\ntitle={Dynamic Fractal Mamba: A Neural Renormalization Group Flow for Scale-Invariant Sequence Modeling},\nauthor={Shenglei Fang and Xianfang Sun and You Zhou},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=L8a9GRfoly}\n}"},"title":{"value":"Dynamic Fractal Mamba: A Neural Renormalization Group Flow for Scale-Invariant Sequence Modeling"},"paperhash":{"value":"fang|dynamic_fractal_mamba_a_neural_renormalization_group_flow_for_scaleinvariant_sequence_modeling"},"originally_submitted_PDF":{"value":"/pdf/2b1931e3496387cda841468e75ae8ce3402d9373.pdf"},"primary_area":{"value":"deep_learning->sequential_models_time_series"},"abstract":{"value":"Sequence models typically operate at a fixed temporal or spatial scale and struggle to generalize to substantially longer horizons or higher resolutions without retraining. Existing hierarchical architectures expand receptive fields but rely on scale-specific parameters and lack mechanisms to enforce consistent dynamics across scales. We propose Dynamic Fractal Mamba (DF-Mamba), a recursive state-space model that applies a single shared operator across multiple scales. By sharing parameters across recursion depths and exponentially scaling the effective time step, DF-Mamba achieves an exponentially expanding receptive field while preserving linear computational complexity. A learned content-aware coarse-graining module aggregates representations across scales. Auxiliary reconstruction and cross-scale consistency objectives stabilize recursive training. We evaluate DF-Mamba on long-range time-series forecasting, spatial transcriptomics, and computational pathology. Across all tasks, DF-Mamba consistently outperforms Transformers and flat Mamba baselines while using fewer parameters and maintaining linear-time scalability. Importantly, models trained on short sequences or low-resolution inputs generalize in a zero-shot manner to substantially larger temporal and spatial scales unseen during training. These results demonstrate that recursive parameter sharing provides an effective inductive bias for learning scale-consistent and efficient sequence representations."},"pdf":{"value":"/pdf/1143826b22f9dc2877354dc25db29d1d2a98cc5e.pdf"},"lay_summary":{"value":"Many real-world data problems involve patterns that appear at different levels of detail. For example, weather can change hour by hour but also follow longer seasonal trends; medical images may contain both small local structures and larger tissue-level patterns. Most machine learning models are trained to work at one fixed level of detail, so their performance can drop when the input is viewed at a different resolution.\n\nIn this paper, we introduce DF-Mamba, a new model designed to learn from data across multiple levels of detail. The key idea is to let the model reuse the same learned rules when looking at both fine and coarse versions of the data, rather than learning separate rules for each scale. This encourages the model to capture stable patterns that remain meaningful even when the data is compressed, enlarged, or viewed at a different resolution.\n\nWe test DF-Mamba on several types of data, including time-series forecasting, spatial biological data, and medical image classification. Across these tasks, the model performs well not only at the original training scale but also when evaluated at coarser scales without additional training. These results suggest that building scale-aware structure into machine learning models can make them more reliable and efficient for real-world applications where data resolution often varies."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Shenglei_Fang1","~Xianfang_Sun1","~You_Zhou9"]},"authors":{"value":["Shenglei Fang","Xianfang Sun","You Zhou"]}},"tmdate":1790067257776,"pdate":1777576140171,"tcdate":1767874094375,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission1016/Authors"],"signatures":["ICML.cc/2026/Conference/Submission1016/Authors"],"forum":"L8a9GRfoly","license":"CC BY 4.0","number":1016,"cdate":1767874094375,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission1016/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission1016/-/Camera_Ready_Revision"],"mdate":1790067257776,"odate":1782341894667,"domain":"ICML.cc/2026/Conference","id":"L8a9GRfoly","version":2},{"content":{"comment":{"value":"Dear Reviewer jrwp,\n\nWe greatly appreciate the reviewer seeking clarification, and thank them again for participating in the discussion.  The following is the reviewer's comment regarding Mamba models being equivalent to linear RNNs:\n\n> I respectfully disagree. Linear RNN can be unfolded to obtain \"attention\" matrix with attention entries being cumulative product of decay from j-th position to i-th position. The only requirement is that decay should be smaller than 1, so the cumulative product won't explode, regardless of whether decay is data-dependent or not. Your proof heavily reply on the fact that \n, which is equivalent to decay < 1, as I mentioned. Mamba is a special case of linear RNN, so it should be applicable to Mamba as well. Do you agree with this? If not, please specify reasons. If so, I find the proof's theoretical significance less valuable, which hinders this work's contribution.\n\nThere are several leaps in logic in the above, and, in particular, the following statement is incorrect: \"Mamba is a special case of linear RNN.\" \n\nPlease allow us to go through this very slowly, without any jumping to conclusions.  From [1], the following are the state equations for linear RNNs:\n\n$x_k = A x_{k-1} + B u_k, y_k = C x_k + D u_k$\n\nThe following are the state equations for Mamba SSMs:\n\n$x_k = A_k x_{k-1} + B_k u_k, y_k = C_k x_k$\n\nNotice, the matrices in the linear RNN's state equations are time-invariant whereas the matrices in the Mamba state equations are time-varying, i.e., $A, B, C, D$ versus $A_k, B_k, C_k$.\n\nClearly, if $A_k, B_k, C_k$ vary for values of $k$, the Mamba state-space equations cannot be represented by the linear RNNs state equations.  **Thus, Mamba is not a special case of linear RNNs.**\n\nAgain, we sincerely appreciate the reviewer seeking clarification and engaging in the discussion.\n\n# References\n[1] Orvieto, Antonio, et al. \"Resurrecting recurrent neural networks for long sequences.\" International Conference on Machine Learning. PMLR, 2023."},"title":{"value":"Mamba is not a special case of linear RNNs"}},"tmdate":1732731951446,"tcdate":1732731951446,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9040/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission9040/Authors"],"forum":"i9RTCC6whL","number":18,"license":"CC BY 4.0","cdate":1732731951446,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9040/-/Official_Comment"],"mdate":1732731951446,"domain":"ICLR.cc/2025/Conference","replyto":"OCYy3pczoU","id":"8xMPEMQwb5","forumContent":{"TLDR":{"value":"Fine-tuning Mamba LLMs is theoretically stable, and much more stable than Transformer LLMs in practice."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Mamba SSMs","SSMs","LLMs","PEFT","LoRA","Transformers"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba state-space models (SSMs) were recently shown to outperform state-of-the-art (SOTA) Transformer large language models (LLMs) across various tasks.  Despite subsequent widespread adaptation, little work has focused on Mamba LLMs' amenability for fine-tuning frameworks ubiquitously used for Transformer-based LLMs, e.g., mixed-precision fine-tuning (MPFT) and parameter-efficient fine-tuning (PEFT).  For the former, it currently remains an open question whether Mamba's recurrent dynamics are robust to small input changes, such as those encountered during MPFT.  Using dynamical systems theory (in particular, Lyapunov exponents), we answer this question in the affirmative.  We empirically validate this result through several experiments, showing that Mamba SSMs are significantly more stable to changes introduced by mixed-precision than comparable Transformers, even when both MPFT and PEFT are combined.  For PEFT, we show how targeting specific memory buffers in Mamba's customized CUDA kernels for low-rank adaptation regularizes SSM parameters, thus providing both parameter efficient learning and computational savings. Finally, with both MPFT and PEFT enabled, we explore the impact of instruction tuning Mamba SSMs for in-context learning (ICL) on natural language tasks.  While pretrained Mamba and Mamba-2 models only achieve 38% and 82% (respectively) of the ICL improvements of comparable Transformer-based LLMs, we show that instruction tuning allows Mamba models to narrow this gap to 81% and Mamba-2 models to skyrocket over this gap to 132%."},"_bibtex":{"value":"@misc{\nhalloran2025mamba,\ntitle={{MAMBA} {STATE}-{SPACE} {MODELS} {ARE} {LYAPUNOV}-{STABLE} {LEARNERS}},\nauthor={John Timothy Halloran and Manbir S Gulati and Paul F Roysdon},\nyear={2025},\nurl={https://openreview.net/forum?id=i9RTCC6whL}\n}"},"title":{"value":"MAMBA STATE-SPACE MODELS ARE LYAPUNOV-STABLE LEARNERS"},"pdf":{"value":"/pdf/c8197bfc3ebf6e57772b5572cc4c46a84fd0d994.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"halloran|mamba_statespace_models_are_lyapunovstable_learners"},"authorids":{"value":["~John_Timothy_Halloran1","~Manbir_S_Gulati1","~Paul_F_Roysdon1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["John Timothy Halloran","Manbir S Gulati","Paul F Roysdon"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2502.13685v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"du|mom_linear_sequence_modeling_with_mixtureofmemories"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jusen_Du:","https://dblp.org/search/pid/api?q=author:Weigao_Sun:","~Disen_Lan1","~Jiaxi_Hu1","https://dblp.org/search/pid/api?q=author:Yu_Cheng:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2502.13685"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2502-13685,\n  publtype={informal},\n  author={Jusen Du and Weigao Sun and Disen Lan and Jiaxi Hu and Yu Cheng},\n  title={MoM: Linear Sequence Modeling with Mixture-of-Memories},\n  year={2025},\n  month={February},\n  cdate={1738368000000},\n  journal={CoRR},\n  volume={abs/2502.13685},\n  url={https://doi.org/10.48550/arXiv.2502.13685}\n}\n"},"abstract":{"value":"Linear sequence modeling methods, such as linear attention, state space modeling, and linear RNNs, offer significant efficiency improvements by reducing the complexity of training and inference. However, these methods typically compress the entire input sequence into a single fixed-size memory state, which leads to suboptimal performance on recall-intensive downstream tasks. Drawing inspiration from neuroscience, particularly the brain's ability to maintain robust long-term memory while mitigating \"memory interference\", we introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes multiple independent memory states, with a router network directing input tokens to specific memory states. This approach greatly enhances the overall memory capacity while minimizing memory interference. As a result, MoM performs exceptionally well on recall-intensive tasks, surpassing existing linear sequence modeling techniques. Despite incorporating multiple memory states, the computation of each memory state remains linear in complexity, allowing MoM to retain the linear-complexity advantage during training, while constant-complexity during inference. Our experimental results show that MoM significantly outperforms current linear sequence models on downstream language tasks, particularly recall-intensive tasks, and even achieves performance comparable to Transformer models. The code is released at https://github.com/OpenSparseLLMs/MoM and is also released as a part of https://github.com/OpenSparseLLMs/Linear-MoE."},"title":{"value":"MoM: Linear Sequence Modeling with Mixture-of-Memories"},"authors":{"value":["Jusen Du","Weigao Sun","Disen Lan","Jiaxi Hu","Yu Cheng"]}},"tmdate":1759839519011,"pdate":1735689600000,"tcdate":1747644441842,"writers":["~"],"signatures":["~Disen_Lan1"],"forum":"sPIY9IQmOc","license":"CC BY-SA 4.0","number":525556,"cdate":1738368000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1759839519011,"domain":"DBLP.org","id":"sPIY9IQmOc","version":2},{"content":{"comment":{"value":"> [W2] The evaluation of **Mamba’s modeling capability on CD** is insufficient. \nWe believe our evaluation of **Mamba’s capability to model CD** is sufficiently supported by the following experiments:\n- **[Table 7]** compares **MLP, Mamba, and Transformer** architectures—each tailored to capture either TD or CD—across **13 datasets** and **4 forecasting horizons**, demonstrating Mamba’s effectiveness in modeling CD.\n- **[Table 4]** further reports results from various architectures employing Mamba for TD/CD modeling, where representative **Mamba-based CD models** (e.g., Bi-Mamba+, S-Mamba, FSMamba (Ours)) consistently achieve strong performance.\n\nFurthermore, **Mamba’s capability in capturing CD** has also been demonstrated in several prior works (**[A–H]**). \n\nNonetheless, if the reviewer still finds any aspect insufficient, we would be happy to include additional experiments to further reinforce our analysis!\n\n&nbsp;\n\n- [A] Aobo Liang, Xingguo Jiang, Yan Sun, and Chang Lu. Bi-mamba+: Bidirectional mamba for time series forecasting. arXiv 2024.\n- [B] ​​Zixuan Weng, Jindong Han, Wenzhao Jiang, and Hao Liu. Simplified mamba with disentangled dependency encoding for long-term time series forecasting. arXiv 2024.\n- [C] Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv 2024.\n- [D] Md Atik Ahamed and Qiang Cheng. Timemachine: A time series is worth 4 mambas for long-term forecasting. In ECAI, 2024.\n- [E] Xiuding Cai, Yaoyao Zhu, Xueyao Wang, and Yu Yao. Mambats: Improved selective state space models for long-term time series forecasting. arXiv 2024.\n- [F] Shusen Ma, Yu Kang, Peng Bai, and Yun-Bo Zhao. Fmamba: Mamba based on fast-attention for multivariate time-series forecasting. arXiv 2024.\n- [G] Sijie Xiong, Shuqing Liu, Cheng Tang, Fumiya Okubo, Haoling Xiong, and Atsushi Shimada. Attention mamba: Time series modeling with adaptive pooling acceleration and receptive field enhancements. arXiv 2025.\n- [H] Zihan Wang, Fanheng Kong, Shi Feng, Ming Wang, Xiaocui Yang, Han Zhao, Daling Wang, and Yifei Zhang. Is mamba effective for time series forecasting? Neurocomputing 2025.\n\n\n&nbsp;\n\n---\n&nbsp;\n\n> [Q1] The paper proposes a regularization method that minimizes the distance between two embedding vectors generated with reversed channel orders. However, this seems to prevent Mamba, as a **”sequence” modeling** method, from capturing **”sequential” information**. I do not understand the rationale behind this design. The final architecture appears equivalent to using Selective SSM **to capture CD**. Please explain why this design is necessary and why it works. \n\n\nWe believe that the essence of Mamba’s core SSSM lies ***not in its sequential nature***, but in its ability to **efficiently replace** the ***(quadratic-complexity) attention mechanism*** with a ***(linear-complexity) selective-scan mechanism***.\n\n\nAs shown in **Figure E.1**, a Mamba block combines 1) a lightweight 1D convolution for local context with the **2) SSSM**, the key module enabling **”linear-complexity” token dependency modeling**. Our regularization preserves this essence — it maintains the **selective-scan mechanism** while removing the bias from fixed scan order, enabling Mamba to capture CD effectively. \n\n\nRegarding the concern about applying Mamba’s **sequential** design to **non-sequential** data, we believe this issue is largely superficial. Just as (non-sequential) **Transformers** became suitable for **sequential** tasks like natural language after introducing **positional encodings**, Mamba’s sequential property merely arises from its scan direction. By regularizing it, we remove that sequential order bias while retaining Mamba’s original efficiency and design principles.\n\n\nFor clarity, the originality of our approach can be summarized as follows:\n| Model | Bias | Capture Token Dependencies | Complexity |\n|--------|------|-----------------------------|-------------|\n| Transformer | ✗ | ✓ | Quadratic |\n| + Positional Encoding | Adds bias | ✓ | Quadratic |\n| Mamba | ✓ | ✓ | Linear |\n| + Regularization (Ours) | **Reduces bias** | ✓ | Linear |\n\n\nIn short, our method retains Mamba’s core strength—**efficient dependency modeling**—while reducing its inherent directional bias, making it more suitable for capturing CD rather than enforcing TD. \n\n\nMoreover, our **extensive experimental results** across **13 datasets and 4 horizons** consistently support this rationale, empirically confirming the validity and effectiveness of the proposed design."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763688565374,"tcdate":1763688565374,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23977/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission23977/Authors"],"forum":"N6MIqE11eF","number":2,"license":"CC BY 4.0","cdate":1763688565374,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23977/-/Official_Comment"],"mdate":1763688565374,"domain":"ICLR.cc/2026/Conference","replyto":"8ht4NKW9aq","id":"Yz8kCmF4Wr","forumContent":{"TLDR":{"value":"We propose FSMamba, a time series forecasting method that uses Mamba with regularization strategy to address the sequential order bias in capturing channel dependencies."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Time Series Forecasting","Mamba"]},"supplementary_material":{"value":"/attachment/d0aef3ef8995051541105dd0e448b12055805dee.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Mamba is a sequential model that has recently emerged as a promising alternative to Transformers, offering near-linear complexity.\nHowever, although channels in time series (TS) data generally lack a sequential order, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this, prior works have adopted bidirectional Mamba to scan channels in both forward and reverse orders. In this paper, we show that unidirectional Mamba can effectively replace the bidirectional Mamba with simple strategies. To this end, we propose FSMamba, a TS forecasting method employing a unidirectional Mamba that incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order. Furthermore, we introduce channel similarity modeling, a pretraining task to preserve similarities between channels from the data space to the latent space to enhance the ability to capture CD. Extensive experiments demonstrate the efficacy of our method, achieving state-of-the-art performance on diverse datasets."},"_bibtex":{"value":"@misc{\nlee2026is,\ntitle={Is Bidirectionality Necessary in Mamba for Time Series Forecasting?},\nauthor={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\nyear={2026},\nurl={https://openreview.net/forum?id=N6MIqE11eF}\n}"},"title":{"value":"Is Bidirectionality Necessary in Mamba for Time Series Forecasting?"},"pdf":{"value":"/pdf/bf428f2b7b338e6732b17e2ed317a213028a6cfe.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"lee|is_bidirectionality_necessary_in_mamba_for_time_series_forecasting"},"authorids":{"value":["~Seunghan_Lee1","~Juri_Hong1","~Kibok_Lee1","~Taeyoung_Park1"]},"authors":{"value":["Seunghan Lee","Juri Hong","Kibok Lee","Taeyoung Park"]}},"version":2},{"content":{"summary":{"value":"This paper addresses the long-context generalization failure of hybrid Mamba-Transformer models and proposes a training-free context length extension (CLE) method called Universal Position Interpolation (UPI). UPI unifies Mamba’s cumulative decay and Transformer’s rotary positional encoding (RoPE) scaling, enabling hybrid/pure Mamba models to extend context length from 4K/8K to up to 64K tokens without sacrificing short-context accuracy."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. Rationale for the 20% Top-K Threshold for Mamba Heads. \nWhat is the systematic basis for choosing 20% specifically? Is it derived from theoretical analysis (e.g., the proportion of unstable heads in hybrid models) or just empirical tuning on the tested benchmarks? Does this threshold generalize across different hybrid model architectures (e.g., models with more Transformer layers than Mamba layers) or task types (e.g., code completion vs. multi-document QA)? If the threshold needs adjustment for new models/tasks, what guidelines can be provided?\n\n\n2. Generalization to Longer Contexts and Unseen Model Configurations\nThe paper extends context lengths from 2K/4K/8K to 64K, but key gaps remain: Have the authors tested UPI on longer contexts beyond 64K (e.g., 128K or 256K tokens)? If not, does the authors anticipate performance degradation, and what mechanisms might limit scalability?\n\n3. Comparison with More Relevant Baselines for Hybrid Models. \nThe baseline comparison uses LongMamba (for pure Mamba) + YaRN (for Transformer), but this is a \"component-wise combination\" rather than a dedicated hybrid CLE method. Have the authors considered baselines that jointly optimize Transformer and Mamba layers for hybrid models?\n\n4. Applicability to Pure Transformer Models and Adaptive Head Selection. The paper frames UPI as \"universal,\" but its utility for pure Transformer models is unaddressed. Can UPI be adapted to pure Transformer models? Since UPI unifies RoPE scaling and Mamba gate dynamics, would applying its core logic (e.g., selective dimension-wise scaling based on ERF-like metrics) outperform existing Transformer CLE methods (e.g., YaRN, LongRoPE)?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This paper provides the first systematic investigation of long-context failure in hybrid Mamba-Transformer models, addressing a critical gap, prior work focused exclusively on pure Transformer or Mamba architectures, ignoring the unique challenges of hybrid designs. The identification of \"uncontrolled state growth\" and \"uneven receptive field contributions\" as root causes is a foundational insight.  \n\n2. It balances mathematical rigor (e.g., state magnitude explosion derivation, UPI closed-form proof) and empirical validation. ERF analysis and state growth characterization provide concrete evidence for failure modes, while ablation studies (selective head adjustment, Top-K thresholding) validate the method’s components.  \n\n\n3. Hybrid Mamba-Transformer models are emerging as efficient alternatives to Transformers, but their long-context limitation hinders real-world use (e.g., document understanding, multi-turn dialogue). UPI extends their context window from 4K to 64K tokens without performance loss, making these models viable for long-sequence tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Subjectivity and Non-Generality of the Top-20% High ERF Head Threshold: UPI relies on the rule of \"selecting the Top-20% of high ERF heads for adjustment,\" but this threshold is determined solely based on experimental experience from benchmarks such as PG-19 and RULER, lacking systematic validation. The selection logic for \"20%\" is not explained, nor is its adaptability to different model architectures (e.g., models with different ratios of Mamba/Transformer layers) and task types (e.g., code generation, multi-turn dialogue) verified.\n\n2. Insufficiently Comprehensive Baselines: Only \"LongMamba + YaRN\" was selected as the baseline. However, LongMamba is a non-specific baseline designed for pure Mamba and is not compared with \"CLE methods specifically optimized for hybrid models\" (e.g., Hymba's hybrid head adjustment logic, Samba's state space co-extension). Furthermore, \"fine-tuning CLE methods\" (e.g., long context fine-tuning for hybrid models) are not compared, making it impossible to quantify the actual value of UPI's \"no-training\" advantage (e.g., fine-tuning methods may have higher performance; is UPI's deployment advantage sufficient to compensate for the performance gap?)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764359647857,"tcdate":1761960249420,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission9337/Reviewer_5ThG"],"signatures":["ICLR.cc/2026/Conference/Submission9337/Reviewer_5ThG"],"forum":"MjmORKLHUI","number":2,"license":"CC BY 4.0","cdate":1761960249420,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission9337/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764359647857,"domain":"ICLR.cc/2026/Conference","replyto":"MjmORKLHUI","id":"kVf3UhPe0S","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["LLM","Hybrid Models","Mamba","Transformer","Long Context"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Hybrid Mamba-Transformer models have emerged as promising alternatives to pure Transformers, offering efficiency and competitive performance. However, they struggle to generalize beyond their training context windows, collapsing on long-context tasks. We provide the first systematic analysis of this failure, showing that it arises from uncontrolled state growth and uneven receptive field contributions across the hybrid architecture. Guided by this understanding, we introduce Universal Position Interpolation (UPI), a closed-form, training-free scaling method that unifies Mamba's cumulative decay with Transformer rotary frequency scaling. UPI selectively stabilizes unstable Mamba dynamics while rescaling Transformer encodings, controlling state growth and enabling reliable long-context generalization, with only a few auxiliary forward passes.  Evaluation shows that UPI extends multiple state-of-the-art hybrid and pure Mamba models from 4K to up to 64K tokens on PG-19 perplexity, LongBench and RULER benchmarks, without sacrificing short-context accuracy. These findings establish the first principled bridge between Transformers and state-space models and open a new direction for training-free context extension methods for emerging hybrid models."},"_bibtex":{"value":"@inproceedings{\nshen2026from,\ntitle={From Collapse to Control: Understanding and Extending Context Length in Emerging Hybrid Models via Universal Position Interpolation},\nauthor={Haochen Shen and Davis Wertheimer and Zheng Wang and Garrett Goon and Derrick Liu and Naigang Wang and Mudhakar Srivatsa and Raghu K. Ganti and Minjia Zhang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=MjmORKLHUI}\n}"},"title":{"value":"From Collapse to Control: Understanding and Extending Context Length in Emerging Hybrid Models via Universal Position Interpolation"},"pdf":{"value":"/pdf/9483e67954400d884f2d97231c2175a8a30522d8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"shen|from_collapse_to_control_understanding_and_extending_context_length_in_emerging_hybrid_models_via_universal_position_interpolation"},"authorids":{"value":["~Haochen_Shen1","~Davis_Wertheimer1","~Zheng_Wang38","~Garrett_Goon1","~Derrick_Liu1","~Naigang_Wang1","~Mudhakar_Srivatsa1","~Raghu_K._Ganti1","~Minjia_Zhang1"]},"authors":{"value":["Haochen Shen","Davis Wertheimer","Zheng Wang","Garrett Goon","Derrick Liu","Naigang Wang","Mudhakar Srivatsa","Raghu K. Ganti","Minjia Zhang"]}},"version":2},{"content":{"summary":{"value":"The paper presents FLDmamba, a novel time series prediction framework combining Fourier and Laplace transformations with the Mamba architecture to improve accuracy and robustness for long-term predictions. The Fourier transform aims to capture multi-scale periodicity and reduce noise, while the Laplace transform enhances the model’s ability to capture transient dynamics. Through ablation studies and benchmark comparisons, FLDmamba demonstrates state-of-the-art performance on several time series prediction datasets. The authors also evaluate the model's robustness, efficiency, and sensitivity to hyperparameters, contributing to a well-rounded evaluation."},"soundness":{"value":4},"confidence":{"value":3},"questions":{"value":"1. Since both FFT and Discrete Cosine Transform (DCT) are effective for frequency-domain analysis, could the authors clarify why they selected FFT over DCT? DCT, for instance, has shown advantages in signal compression and noise reduction and might benefit time series forecasting by emphasizing low-frequency components. Further insight on this choice would help clarify the design decision.\n\n3. Deep learning models with linear layers can often approximate linear transformations, including FFT. Could the authors elaborate on the specific necessity of explicitly embedding Fourier and Laplace transforms rather than relying on the model's intrinsic capacity to learn these linear relationships? This would clarify whether these transformations improve interpretability, robustness, or training efficiency in ways that the network alone might not achieve."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":4},"strengths":{"value":"1. Innovative Framework: Integrating Fourier and Laplace transformations into the Mamba model is novel in the context of time series forecasting. This combination allows FLDmamba to address core challenges in time series data—multi-scale periodicity, noise reduction, and transient dynamics.\n\n2. Solid Performance Gains: FLDmamba consistently outperforms other models on key benchmarks, particularly in scenarios involving noisy data or long lookback lengths, demonstrating that the model effectively generalizes across diverse datasets."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While this paper is generally strong, there are a few minor weaknesses that could be addressed to further strengthen the contribution:\n\n1. Incomplete Justification for RBF Kernel: Although the RBF kernel is presented as an effective data-smoothing technique, its choice is not empirically validated. A comparison with other kernel functions or a focused ablation study would help verify this choice and ensure that RBF is the optimal choice.\n\n2. Unclear Necessity of FFT-IFFT Sequence: The FMamba block employs an FFT followed by an IFFT without a clear explanation of any specific frequency-domain manipulations before reconstructing the signal in the time domain. If this process is meant to filter specific frequencies or reduce noise, the details of such operations should be specified. Otherwise, the sequence could appear redundant, as it may be feasible for the neural network to approximate frequency characteristics without explicitly embedding FFT.\n\n3. Limited Explanation of Explicit Transformations: While the inclusion of Fourier and Laplace transforms is well-motivated theoretically, it remains unclear why these explicit transformations are necessary. Neural networks, particularly those with linear layers, can approximate operations like FFT. A clearer discussion on the unique advantages of explicitly integrating these transforms would strengthen the architectural justification.\n\n4. Incomplete Complexity Analysis: The complexity analysis, which estimates FLDmamba’s time complexity as $𝑂(𝐵𝐿𝑉𝑁)$, does not fully account for the computational costs of FFT, IFFT, and inverse Laplace transforms. Each of these operations introduces additional costs (e.g., $O(BLNlogL)$ for FFT)) that may not scale efficiently for large datasets. This makes the current complexity analysis potentially optimistic, particularly given that working in the complex domain could introduce additional memory and processing overhead. Wall-clock inference times compared to baseline models would better validate FLDmamba's practical efficiency and help justify the complexity of the FFT and Laplace operations.\n\n5. Incomplete Citation of S-Mamba: While S-Mamba is frequently referenced as a baseline, it lacks a formal citation in the main text. Adding this citation would improve the academic rigor and proper attribution within the paper.\n\n6. Clarity of Figures: Some figures could benefit from clearer axis labels to improve interpretability. For example, in Figure 1, the y-axis label is ambiguous, and the x-axis label as “Time of Day” is potentially misleading since it exceeds 24 hours. Clarifying these points would improve the readability of the time series prediction results."}},"nonreaders":[],"tmdate":1731428015575,"tcdate":1730314518999,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4022/Reviewer_8Y8C"],"signatures":["ICLR.cc/2025/Conference/Submission4022/Reviewer_8Y8C"],"forum":"9EiWIyJMNi","number":2,"license":"CC BY 4.0","cdate":1730314518999,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4022/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428015575,"domain":"ICLR.cc/2025/Conference","replyto":"9EiWIyJMNi","id":"ZEWqCyjO6A","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Time Series Prediction"]},"supplementary_material":{"value":"/attachment/1a7a0dbb359aaeb4b73874c96c7e6b288ffeb4cd.pdf"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Time series prediction, a crucial task across various domains, faces significant challenges due to the inherent complexities of time series data, including non-stationarity, multi-scale periodicity, and transient dynamics, particularly when tackling long-term predictions. While Transformer-based architectures have shown promise, their quadratic complexity with sequence length hinders their efficiency for long-term predictions. Recent advancements in State-Space Models, such as Mamba, offer a more efficient alternative for long-term modeling, but they lack the capability to capture multi-scale periodicity and transient dynamics effectively. Meanwhile, they are susceptible to the data noise issue in time series. This paper proposes a novel framework, FLDmamba (Fourier and Laplace Transform Decomposition Mamba), addressing these limitations. FLDmamba leverages the strengths of both Fourier and Laplace transforms to effectively capture both multi-scale periodicity, transient dynamics within time series data, and improve the robustness of the model to the data noise issue. By integrating Fourier analysis into Mamba, FLDmamba enhances its ability to capture global-scale properties, such as multi-scale periodicity patterns, in the frequency domain. Meanwhile, the Fourier Transform aids in isolating underlying patterns or trends from noise in time series data by emphasizing key frequency components, thereby enabling the model to mitigate noise effects. Additionally, incorporating Laplace analysis into Mamba improves its capacity to capture local correlations between neighboring data points, leading to a more accurate representation of transient dynamics. Our extensive experiments demonstrate that FLDmamba achieves superior performance on time series prediction benchmarks, outperforming both Transformer-based and other Mamba-based architectures. This work offers a computationally efficient and effective solution for long-term time series prediction, paving the way for its application in real-world scenarios. To promote the reproducibility of our method, we have made both the code and data accessible via the following URL: \\href{https://anonymous.4open.science/r/FLambas-AD7E/README.md}{https://anonymous.4open.science/r/FLDmamba}"},"_bibtex":{"value":"@misc{\nzhang2025fldmamba,\ntitle={{FLD}mamba:  Integrating Fourier and Laplace Transform Decomposition with Mamba for Enhanced Time Series Prediction},\nauthor={Qianru Zhang and Chenglei Yu and Haixin Wang and Yudong Yan and Yuansheng Cao and Hongzhi Yin and Siu Ming Yiu and Tailin Wu},\nyear={2025},\nurl={https://openreview.net/forum?id=9EiWIyJMNi}\n}"},"title":{"value":"FLDmamba:  Integrating Fourier and Laplace Transform Decomposition with Mamba for Enhanced Time Series Prediction"},"pdf":{"value":"/pdf/5dc7cf776ca2eac09c1caf0e8a4b4195707a1e3b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|fldmamba_integrating_fourier_and_laplace_transform_decomposition_with_mamba_for_enhanced_time_series_prediction"},"authorids":{"value":["~Qianru_Zhang2","~Chenglei_Yu1","~Haixin_Wang3","~Yudong_Yan1","~Yuansheng_Cao1","~Hongzhi_Yin2","~Siu_Ming_Yiu1","~Tailin_Wu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Qianru Zhang","Chenglei Yu","Haixin Wang","Yudong Yan","Yuansheng Cao","Hongzhi Yin","Siu Ming Yiu","Tailin Wu"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their positive and insightful assessment of our work. We appreciate their recognition of our novel approach in modeling B&B as a sequential decision-making process using a state-space model (Mamba), the well-designed methodology combining contrastive learning with sequential modeling, and the compelling experimental results demonstrated across diverse benchmark instances.\n\nBelow, we provide a detailed explanation regarding the weaknesses and questions.\n\n> **Weakness 1 (Concern 1): The reason why expert actions are included in the input -- common practice in the autoregressive paradigm**\n\nIn fact, this approach is a standard practice within the autoregressive paradigm in the field of Natural Language Processing (NLP). It has been widely and successfully applied in models like GPT and contemporary Large Language Models (LLMs).\n\nDuring training, ground-truth tokens provide context; during inference, the model uses its own predictions. \nThis creates a known train-test discrepancy, but the remarkable success of LLMs validates this approach. Our core contribution is not solving this fundamental challenge but applying sequence modeling to branching decisions.\n\n> **Weakness 1 (Concern 2): Statistics on average sequence lengths in training/inference**\n\nFirst, we would like to point out that **Mamba is capable of handling long sequences**:\n\n- In the context of audio modeling tasks, **the original Mamba paper [1]** states : *“Figure 7 evaluates the effect of increasing training sequence lengths from 2^13 = 8192 to **2^20 ≈ 10^6**.”* \n- For DNA modeling tasks, **the original text [1]** notes: *“We pretrain models on sequence lengths 2^10 = 1024, 2^12 = 4096, 2^14 = 16384, 2^16 = 65536, 2^18 = 262144, **2^20 = 1048576**.”*\n\nThus, it is evident that **Mamba possesses the ability to process extremely long sequences**.\n\nSubsequently, we provide information regarding the average sequence length (i.e., the actual number of tokens input to Mamba) during the training and testing (inference) of Mamba-Branching, as shown in Table 1.\n\n| Stage | Branching Path Length | Average Sequence Length (Actual Token Num)  |\n| :---- | :-------------------: | :-----------------: |\n| Train | 100 | 30745.65 |\n| Inference | 25 | 4132.02 |\n\nTable 1: The average sequence length (number of input tokens) for Mamba-Branching on MILP-S during training and inference. For the training phase, the value is computed by averaging the sequence length within each batch across all batches; for the inference phase, it is obtained by averaging over all test instances.\n\nBased on the above, the average sequence length during training for Mamba-Branching is **30,745.65**, while the maximum sequence length handled in the original Mamba paper is **1,048,576**. It is therefore **not surprising that Mamba performs well in our scenario**. \n\nFurthermore, in our response to **Reviewer 7jpx regarding \"Weakness 3: Sequence length is chosen empirically,\"** we conducted training with even longer branching path lengths. The results confirm that further increasing the sequence length indeed leads to performance degradation that Mamba cannot effectively handle, which is fully consistent with your viewpoint. This demonstrates that **our chosen setting (T=99) represents a validated performance peak** -- it **sufficiently leverages the sequential nature** of the branch-and-bound tree while **remaining within Mamba's effective** operating range before encountering the performance bottleneck associated with excessively long sequences.\n\n### Reference\n\n[1] Gu, A., & Dao, T. (2024). Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling. https://openreview.net/forum?id=tEYskw1VY2"},"title":{"value":"Summary of Advantages and Response to Weakness 1 (contains 2 Concerns)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763623531796,"tcdate":1763623531796,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12016/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission12016/Authors"],"forum":"iQKzHeLDrN","number":11,"license":"CC BY 4.0","cdate":1763623531796,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12016/-/Official_Comment"],"mdate":1763623531796,"domain":"ICLR.cc/2026/Conference","replyto":"XQllddFcBO","id":"50PLm4thSx","forumContent":{"TLDR":{"value":"We propose a branching policy that utilizes the sequential branching path of the branch-and-bound tree to guide current decision."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Mixed-Integer Linear Programming","Branch-and-Bound","Branching Variable Selection Policy","Generalization","Mamba"]},"supplementary_material":{"value":"/attachment/92e991cc82f9ad41529abd7aef31e00b1f3c69c9.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"The branch-and-bound (B\\&B) method is a dominant exact algorithm for solving Mixed-Integer Linear Programming problems (MILPs). While recent deep learning approaches have shown promise in learning branching policies using instance-independent features, they often struggle to capture the sequential decision-making nature of B\\&B, particularly over long horizons with complex inter-step dependencies and intra-step variable interactions. To address these challenges, we propose Mamba-Branching, a novel learning-based branching policy that leverages the Mamba architecture for efficient long-sequence modeling, enabling effective capture of temporal dynamics across B\\&B steps. Additionally, we introduce a contrastive learning strategy to pre-train discriminative embeddings for candidate branching variables, significantly enhancing Mamba's performance. Experimental results demonstrate that Mamba-Branching outperforms all previous neural branching policies on real-world MILP instances and achieves superior computational efficiency compared to the advanced open-source solver SCIP. The source code can be accessed at https://github.com/doctor-watson626/Mamba-Branching/."},"_bibtex":{"value":"@inproceedings{\nzhang2026towards,\ntitle={Towards Better Branching Policies: Leveraging the Sequential Nature of Branch-and-Bound Tree},\nauthor={Ce Zhang and Bin Zhang and Guoliang Fan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=iQKzHeLDrN}\n}"},"title":{"value":"Towards Better Branching Policies: Leveraging the Sequential Nature of Branch-and-Bound Tree"},"pdf":{"value":"/pdf/252a128e120299bcb68a697f9200fb14e8927006.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|towards_better_branching_policies_leveraging_the_sequential_nature_of_branchandbound_tree"},"authorids":{"value":["~Ce_Zhang10","~Bin_Zhang12","~Guoliang_Fan3"]},"authors":{"value":["Ce Zhang","Bin Zhang","Guoliang Fan"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2401.04081v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"pióro|moemamba_efficient_selective_state_space_models_with_mixture_of_experts"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Maciej_Pióro:","https://dblp.org/search/pid/api?q=author:Kamil_Ciebiera:","https://dblp.org/search/pid/api?q=author:Krystian_Król:","~Jan_Ludziejewski1","~Sebastian_Jaszczur1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2401.04081"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2401-04081,\n  publtype={informal},\n  author={Maciej Pióro and Kamil Ciebiera and Krystian Król and Jan Ludziejewski and Sebastian Jaszczur},\n  title={MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2401.04081},\n  url={https://doi.org/10.48550/arXiv.2401.04081}\n}\n"},"abstract":{"value":"State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms both Mamba and baseline Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in $2.35\\times$ fewer training steps while preserving the inference performance gains of Mamba against Transformer."},"title":{"value":"MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts"},"authors":{"value":["Maciej Pióro","Kamil Ciebiera","Krystian Król","Jan Ludziejewski","Sebastian Jaszczur"]}},"tmdate":1747229668900,"pdate":1704067200000,"tcdate":1738247457062,"writers":["~"],"signatures":["~Sebastian_Jaszczur1"],"forum":"TsmM8pgv53","license":"CC BY-SA 4.0","number":287844,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747229668900,"domain":"DBLP.org","id":"TsmM8pgv53","version":2},{"content":{"review":{"value":"## Summary\n\nThis paper explores the application of the Mamba architecture to protein sequence prediction, representing an innovative interdisciplinary practice in computational biology. The study leverages Mamba’s selective state-space updates, which offer advantages in computational efficiency and handling long-range dependencies, making this approach both innovative and feasible. The paper presents a comprehensive framework, including model evaluation metrics and a detailed experimental design, demonstrating a thoughtful approach to achieving high-performance protein sequence modeling.\n\n## Strengths\n\n- **Innovative Exploration of Mamba Architecture**: The application of Mamba architecture in protein sequence prediction is creative and shows promise for advancing the field. The selective state-space updates provide an efficient means to model protein sequences, which could be highly valuable for resource-constrained settings.\n\n- **Importance of the Research Problem**: Protein sequence prediction is a vital area of study with broad implications for biological research and practical applications. Addressing this challenge could aid in advancing drug discovery and understanding protein functions, underlining the relevance of this research.\n\n- **Thorough Literature Review and Feasibility**: The paper provides a well-rounded review of related work, demonstrating the authors' strong understanding of the field. The chosen approach appears feasible, with a clear plan for implementation and evaluation.\n\n## Weaknesses\n\n- **Small and Unclear Figures**: The figures in the paper are too small and lack clarity, making it difficult to interpret the visual information provided. Larger, more detailed figures would significantly improve the readability and comprehension of the model architecture and results.\n\n- **Writing Quality**: The writing could benefit from improved clarity and logical flow. More precise and sophisticated word choices would enhance the paper’s presentation.\n\n## Score\n\n- **Soundness**: 8/10\n- **Contribution**: 8/10\n- **Presentation**:7/10"},"confidence":{"value":3},"rating":{"value":8},"title":{"value":"Innovative Application of Mamba Architecture for Efficient Protein Sequence Prediction"}},"nonreaders":[],"tmdate":1731044036311,"tcdate":1730993070790,"writers":["tsinghua.edu.cn/THU/2024/Fall/AML","~Juncheng_Yu1"],"signatures":["~Juncheng_Yu1"],"forum":"I2FbCxyYVS","number":1,"license":"CC BY 4.0","cdate":1730993070790,"readers":["everyone"],"invitations":["tsinghua.edu.cn/THU/2024/Fall/AML/Submission52/-/Official_Review"],"mdate":1731044036311,"domain":"tsinghua.edu.cn/THU/2024/Fall/AML","replyto":"I2FbCxyYVS","id":"6lR2LzOofr","forumContent":{"TLDR":{"value":"This proposal explores implementing a Mamba architecture for efficient protein sequence generation, aiming to balance performance with reduced computational resource demands and making the architecture more suited to biological dynamics."},"venue":{"value":"THU 2024 Fall AML Submission"},"pdf":{"value":"/pdf/8d5ee7449a5695662b51c83289c76642879f2be4.pdf"},"keywords":{"value":["protein sequence generation","Mamba architecture","efficient modeling","protein language models","biological dynamics"]},"venueid":{"value":"tsinghua.edu.cn/THU/2024/Fall/AML/Submission"},"paperhash":{"value":"jiang|[proposalml]protein_sequence_generation_model"},"authorids":{"value":["~pan_jiang1","zeeshanzulfiqar79@outlook.com","ivanyazykov.mipt@gmail.com"]},"abstract":{"value":"This proposal outlines a novel approach for protein sequence generation, a key area in computational biology focused on generating sequences with specific functional and structural characteristics. Despite recent advances with deep learning models, challenges remain in handling long sequences and capturing biological dynamics. We propose implementing a Mamba architecture that leverages selective state-space updates to achieve efficient protein sequence generation. With linear computational complexity and effective handling of long-range dependencies, this architecture is designed to be more suited to biological dynamics while reducing resource requirements. A comprehensive evaluation on the UniRef50 dataset will demonstrate its potential to deliver competitive performance, providing a viable solution for research environments with limited computational resources."},"_bibtex":{"value":"@inproceedings{\njiang2024proposalmlprotein,\ntitle={[Proposal-{ML}]Protein Sequence Generation Model},\nauthor={pan jiang and Zeeshan Zulfiqar and Ivan Iazykov},\nbooktitle={Submitted to Tsinghua University Course: Advanced Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=I2FbCxyYVS},\nnote={under review}\n}"},"title":{"value":"[Proposal-ML]Protein Sequence Generation Model"},"authors":{"value":["pan jiang","Zeeshan Zulfiqar","Ivan Iazykov"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["linear RNNs","higher-order recurrence","block diagonal recurrence","normalization","synthetic tasks"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Linear recurrent networks (LRNNs) and linear state space models (SSMs) promise computational and memory efficiency on long-sequence modeling tasks, yet their diagonal state transitions limit expressivity. Dense and/or nonlinear architectures (e.g., LSTMs) on the other hand are provably more expressive, but computationally costly. \nHere, we explore how expressivity in LRNNs can be increased via richer state mixing across time and channels while maintaining competitive efficiency. Specifically, we introduce two structured LRNN architectures: (i) Higher-order Linear Recurrent Units (H-LRU), which generalize first-order recurrence to $m$-th order, mixing multiple past states, and (ii) Block-Diagonal LRUs (BD-LRU), which enable dense intra-block channel mixing. Per-channel (H-LRU) / per-row (BD-LRU) L1-normalization of selective gates stabilizes training and allows for scaling window/block sizes. In synthetic sequence-modeling benchmarks (compression, selective copying, associative recall), H-LRU is found to be the most parameter-efficient in compression, while the performance of BD-LRU matches or exceeds those of linear SSMs (Mamba), low-rank LRNNs (DeltaNet) and LSTM baselines. In permutation composition tasks ($S_3$-$S_5$), BD-LRU is found to efficiently solve these tasks at moderate block sizes, outperforming both linear and non-linear baselines. A parallel-scan implementation of the proposed architectures keeps the throughput competitive with diagonal LRNNs for moderate orders (H-LRU) and block sizes (BD-LRU), while preserving the efficiency that motivated LRNNs. \nThese results indicate that the structure of state mixing rather than width alone shapes expressivity of LRNNs, offering a practical route to closing the efficiency–expressivity gap in linear sequence models."},"_bibtex":{"value":"@misc{\ndubinin2026improved,\ntitle={Improved state mixing in higher-order and block diagonal linear recurrent networks},\nauthor={Igor Dubinin and Antonio Orvieto and Felix Effenberger},\nyear={2026},\nurl={https://openreview.net/forum?id=fF72TYOKjZ}\n}"},"title":{"value":"Improved state mixing in higher-order and block diagonal linear recurrent networks"},"pdf":{"value":"/pdf/7a4d7bd95df82a2db1bd82c38287ef7ad0d06fc5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"dubinin|improved_state_mixing_in_higherorder_and_block_diagonal_linear_recurrent_networks"},"authorids":{"value":["~Igor_Dubinin1","~Antonio_Orvieto3","~Felix_Effenberger1"]},"authors":{"value":["Igor Dubinin","Antonio Orvieto","Felix Effenberger"]}},"tmdate":1770805071859,"tcdate":1758277999110,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17598/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission17598/Authors"],"forum":"fF72TYOKjZ","license":"CC BY 4.0","number":17598,"cdate":1758277999110,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission17598/-/Full_Submission","ICLR.cc/2026/Conference/Submission17598/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770805071859,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"fF72TYOKjZ","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2501.17088v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"muñoz|mambashedder_posttransformer_compression_for_efficient_selective_structured_state_space_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:J._Pablo_Muñoz:","https://dblp.org/search/pid/api?q=author:Jinjie_Yuan:","~Nilesh_Jain1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2501.17088"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2501-17088,\n  publtype={informal},\n  author={J. Pablo Muñoz and Jinjie Yuan and Nilesh Jain},\n  title={Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models},\n  year={2025},\n  month={January},\n  cdate={1735689600000},\n  journal={CoRR},\n  volume={abs/2501.17088},\n  url={https://doi.org/10.48550/arXiv.2501.17088}\n}\n"},"abstract":{"value":"Large pre-trained models have achieved outstanding results in sequence modeling. The Transformer block and its attention mechanism have been the main drivers of the success of these models. Recently, alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address the inefficiencies of Transformers. This paper explores the compression of SSM-based models, particularly Mamba and its hybrids. We study the sensitivity of these models to the removal of selected components at different granularities to reduce the model size and computational overhead, thus improving their efficiency while maintaining accuracy. The proposed solutions, collectively referred to as Mamba-Shedder, achieve a speedup of up to 1.4x during inference, demonstrating that model efficiency can be improved by eliminating several redundancies with minimal impact on the overall model performance. The code is available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning."},"title":{"value":"Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models"},"authors":{"value":["J. Pablo Muñoz","Jinjie Yuan","Nilesh Jain"]}},"tmdate":1769363142616,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2501-17088"],"tcdate":1769363126374,"writers":["~"],"signatures":["~Nilesh_Jain1"],"forum":"OyxoEfRRsM","license":"CC BY-SA 4.0","number":800372,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769363142616,"domain":"DBLP.org","id":"OyxoEfRRsM","version":2},{"content":{"venue":{"value":"NAACL (Long Papers) 2025"},"pdf":{"value":"https://aclanthology.org/2025.naacl-long.195.pdf"},"venueid":{"value":"dblp.org/conf/NAACL/2025"},"paperhash":{"value":"muñoz|mambashedder_posttransformer_compression_for_efficient_selective_structured_state_space_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Juan_Pablo_Muñoz:","https://dblp.org/search/pid/api?q=author:Jinjie_Yuan:","~Nilesh_Jain1"]},"html":{"value":"https://doi.org/10.18653/v1/2025.naacl-long.195"},"_bibtex":{"value":"@inproceedings{DBLP:conf/naacl/MunozYJ25,\n  author={Juan Pablo Muñoz and Jinjie Yuan and Nilesh Jain},\n  title={Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models},\n  year={2025},\n  cdate={1735689600000},\n  pages={3851-3863},\n  url={https://doi.org/10.18653/v1/2025.naacl-long.195},\n  booktitle={NAACL (Long Papers)},\n  crossref={conf/naacl/2025-1}\n}\n"},"abstract":{"value":"Large pre-trained models have achieved outstanding results in sequence modeling. The Transformer block and its attention mechanism have been the main drivers of the success of these models. Recently, alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address the inefficiencies of Transformers. This paper explores the compression of SSM-based models, particularly Mamba and its hybrids. We study the sensitivity of these models to the removal of selected components at different granularities to reduce the model size and computational overhead, thus improving their efficiency while maintaining accuracy. The proposed solutions, collectively referred to as Mamba-Shedder, achieve a speedup of up to 1.4x during inference, demonstrating that model efficiency can be improved by eliminating several redundancies with minimal impact on the overall model performance. The code is available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning."},"title":{"value":"Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models"},"authors":{"value":["Juan Pablo Muñoz","Jinjie Yuan","Nilesh Jain"]}},"tmdate":1769363129579,"pdate":1767139200000,"externalIds":["dblp:conf/naacl/MunozYJ25"],"tcdate":1769363125972,"writers":["~"],"signatures":["~Nilesh_Jain1"],"forum":"yuGUuAySQG","license":"CC BY-SA 4.0","number":800356,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769363129579,"domain":"DBLP.org","id":"yuGUuAySQG","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2506.12606v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"lin|an_exploration_of_mamba_for_speech_selfsupervised_models"},"authorids":{"value":["~Tzu-Quan_Lin1","https://dblp.org/search/pid/api?q=author:Heng-Cheng_Kuo:","https://dblp.org/search/pid/api?q=author:Tzu-Chieh_Wei:","~Hsi-Chun_Cheng1","https://dblp.org/search/pid/api?q=author:Chun-Wei_Chen:","~Hsien-Fu_Hsiao1","https://dblp.org/search/pid/api?q=author:Yu_Tsao_0001:","https://dblp.org/search/pid/api?q=author:Hung-yi_Lee:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2506.12606"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2506-12606,\n  publtype={informal},\n  author={Tzu-Quan Lin and Heng-Cheng Kuo and Tzu-Chieh Wei and Hsi-Chun Cheng and Chun-Wei Chen and Hsien-Fu Hsiao and Yu Tsao and Hung-yi Lee},\n  title={An Exploration of Mamba for Speech Self-Supervised Models},\n  year={2025},\n  month={June},\n  cdate={1748736000000},\n  journal={CoRR},\n  volume={abs/2506.12606},\n  url={https://doi.org/10.48550/arXiv.2506.12606}\n}\n"},"abstract":{"value":"While Mamba has demonstrated strong performance in language modeling, its potential as a speech self-supervised (SSL) model remains underexplored, with prior studies limited to isolated tasks. To address this, we explore Mamba-based HuBERT models as alternatives to Transformer-based SSL architectures. Leveraging the linear-time Selective State Space, these models enable fine-tuning on long-context ASR with significantly lower compute. Moreover, they show superior performance when fine-tuned for streaming ASR. Beyond fine-tuning, these models show competitive performance on SUPERB probing benchmarks, particularly in causal settings. Our analysis shows that they yield higher-quality quantized representations and capture speaker-related features more distinctly than Transformer-based models. These findings highlight Mamba-based SSL as a promising and complementary direction for long-sequence modeling, real-time speech modeling, and speech unit extraction."},"title":{"value":"An Exploration of Mamba for Speech Self-Supervised Models"},"authors":{"value":["Tzu-Quan Lin","Heng-Cheng Kuo","Tzu-Chieh Wei","Hsi-Chun Cheng","Chun-Wei Chen","Hsien-Fu Hsiao","Yu Tsao","Hung-yi Lee"]}},"tmdate":1767779573075,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2506-12606"],"tcdate":1758790974458,"writers":["~"],"signatures":["~Hsi-Chun_Cheng1"],"forum":"POUblYsETp","license":"CC BY-SA 4.0","number":628609,"cdate":1748736000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1767779573075,"domain":"DBLP.org","id":"POUblYsETp","version":2},{"content":{"summary":{"value":"The paper introduces Mambular, an adaptation of the Mamba architecture specifically designed for tabular \bdata problems. The authors claim that their model leverages a sequential interpretation of tabular data, similar to its successful application in other domains such as text, vision, and time series. The proposed architecture incorporates pooling strategies and feature interaction mechanisms to enhance performance. The paper provides experimental results on various datasets, comparing Mambular with state-of-the-art models such as FT-Transformer, TabTransformer, XGBoost, and LightGBM. According to the authors, Mambular achieves competitive results, highlighting the feasibility of applying sequential models to tabular data."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"What is the rationale behind interpreting tabular data as sequences? Could the authors provide a theoretical explanation or empirical evidence showing that treating features as a sequence offers a distinct advantage?\n\nHow do the authors address the lack of consistency in performance improvements across the datasets? Were any specific types of datasets or characteristics where Mambular performed particularly well?\n\nCould the authors explain why comparisons were not made with a wider range of established tabular models, such as TabR[2], T2G-Former[3], and SAINT[4]?\n[2] TabR: Unlocking the Power of Retrieval-Augmented Tabular Deep Learning, 2023\n[3] T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interaction, 2023\n[4] SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training, 2021"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"The paper explores the application of the Mamba architecture to tabular data, leveraging lessons from its success in handling other data types like text and time series. While not entirely novel, this extension demonstrates an attempt to generalize Mamba to a new domain.\n\nThe authors present experimental results comparing Mambular with a limited selection of state-of-the-art models, such as FT-Transformer, TabTransformer, XGBoost, and LightGBM. While the comparisons are not exhaustive, these initial results provide some insight into the model’s potential.\n\nThe architecture incorporates feature interaction mechanisms and various pooling strategies, potentially providing flexibility to handle different types of tabular data characteristics."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Lack of novelty: One of the main weaknesses is the lack of significant novelty. The concept of adapting Mamba for tabular data is not substantially different from previously proposed models like MambaTab [1]. The minor modifications presented in Mambular mainly involve hyperparameter tuning and slight architectural changes, which do not justify the need for a new model.\n\n[1] MA Ahamed et al., MambaTab: A Plug-and-Play Model for Learning Tabular Data, MIPR, 2024.\n\n- Misalignment with tabular data characteristics: The paper fails to convincingly justify why treating tabular data as sequential offers any distinct advantage. Unlike sequential data such as text or time series, tabular data lacks a natural ordering of features. The authors did not provide clear evidence or theoretical grounding for why a sequence-based model should work better in this context.\n\n- Limited scope of experiments: The experimental results are based on only 12 datasets, and the comparison to state-of-the-art algorithms is not comprehensive. Given the varied nature of tabular data, a broader set of datasets and comparisons to more established tabular models would have strengthened the claims.\n\n- Inconsistent Performance: Despite the claims of Mambular’s superiority, the reported results do not consistently show a significant performance improvement over existing methods, such as XGBoost or LightGBM."}},"nonreaders":[],"tmdate":1731427504818,"tcdate":1730339014414,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1879/Reviewer_Na6p"],"signatures":["ICLR.cc/2025/Conference/Submission1879/Reviewer_Na6p"],"forum":"wElgE9qBb5","number":1,"license":"CC BY 4.0","cdate":1730339014414,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1879/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427504818,"domain":"ICLR.cc/2025/Conference","replyto":"wElgE9qBb5","id":"Xv6mFpXED5","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Tabular Deep Learning","Mamba","Sequential Models","SSM","Recurrent Neural Networks"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The analysis of tabular data has traditionally been dominated by gradient-boosted decision trees (GBDTs), known for their proficiency with mixed categorical and numerical features. However, recent deep learning innovations are challenging this dominance. We introduce Mambular, an adaptation of the Mamba architecture optimized for tabular data. We extensively benchmark Mambular against state-of-the-art models, including neural networks and tree-based methods, and demonstrate its competitive performance across diverse datasets.\nAdditionally, we explore various adaptations of Mambular to understand its effectiveness for tabular data. We investigate different pooling strategies, feature interaction mechanisms, and bi-directional processing. Our analysis shows that interpreting features as a sequence and passing them through Mamba layers results in surprisingly performant models.  The results highlight Mambular’s potential as a versatile and powerful architecture for tabular data analysis, expanding the scope of deep learning applications in this domain.\n The source code is available at: https://anonymous.4open.science/r/mamba-tabular-485F/"},"_bibtex":{"value":"@misc{\nthielmann2025mambular,\ntitle={Mambular: A Sequential Model for Tabular Deep Learning},\nauthor={Anton Frederik Thielmann and Manish Kumar and Christoph Weisser and Arik Reuter and Benjamin S{\\\"a}fken and Soheila Samiee},\nyear={2025},\nurl={https://openreview.net/forum?id=wElgE9qBb5}\n}"},"title":{"value":"Mambular: A Sequential Model for Tabular Deep Learning"},"pdf":{"value":"/pdf/4cfe83c12311b6ae99930d6ccfd4ada19aab102f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"thielmann|mambular_a_sequential_model_for_tabular_deep_learning"},"authorids":{"value":["~Anton_Frederik_Thielmann1","~Manish_Kumar4","~Christoph_Weisser1","~Arik_Reuter1","~Benjamin_Säfken1","~Soheila_Samiee1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Anton Frederik Thielmann","Manish Kumar","Christoph Weisser","Arik Reuter","Benjamin Säfken","Soheila Samiee"]}},"version":2},{"content":{"summary":{"value":"State Space Models (SSM), such as Mamba, demonstrate powerful representation capabilities in modeling long-range dependencies. However, their sequential nature necessitates multiple scans in different directions to compensate for the loss of spatial dependencies when flattening images into 1D sequences. This multi-directional scanning strategy significantly increases computational overhead. This paper proposes a new hierarchical Mamba network, Hi-Mamba, for image super-resolution. Hi-Mamba incorporates two key designs: (1) a Hierarchical Mamba Block (HMB) composed of Local SSMs (L-SSM) and Regional SSMs (R-SSM) with single-direction scanning, which aggregates multi-scale representations to enhance contextual modeling capabilities; (2) Direction-Alternating Hierarchical Mamba Groups (DA-HMG) that assign equivalent single-direction scans to cascaded HMBs, enriching spatial relationship modeling. The model is trained on two commonly used datasets: DIV2K and Flicker2K. It is evaluated on five standard super-resolution (SR) benchmark datasets: Set5, Set14, BSD100, Urban100, and Manga109, tested at scaling factors of ×2, ×3, and ×4. Ablation experiments are conducted for different scanning modes, fusion module effects, R-SSM channel numbers, and region sizes of n×n. Extensive experiments demonstrate that Hi-Mamba exhibits superior performance across the five benchmark datasets for efficient SR."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"My questions can refer to the content mentioned in the shortcomings, and the author can answer them one by one according to the shortcomings mentioned."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"Dual feature extraction capability: By combining the L-SSM and R-SSM modules, Hi-Mamba can extract local and regional features of the image at the same time, focusing on small-scale and large-scale contextual relationships respectively, thereby enhancing the expression ability of features.\n\nSuperior performance and computational efficiency: Experimental results show that Hi-Mamba performs well on multiple benchmark data sets. For example, in the ×2 SR task of the Urban100 data set, Hi-Mamba-S can significantly reduce the amount of calculation (294G FLOPs) while improving PSNR.\n\nHigher visual quality: In the ×4 SR task, Hi-Mamba can reduce blur artifacts and distortion more effectively than traditional CNN or Transformer models, retain more clear details and structural information, and improve the visual quality of the image. quality."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The proposed Hi-Mamba-T and Hi-Mamba-S variants do not show significant differences in PSNR and SSIM evaluation metrics compared to Hi-Mamba-T, aside from the differences in parameters and FLOPs. The advantages and roles of the two variants are not clearly stated.\n2. As the model variants Hi-Mamba-T, Hi-Mamba-S, and Hi-Mamba-L increase in depth and width, their parameters and FLOPs also increase. When using more complex models, the optimization performance in terms of computation may not be evident.\n3.Some newer super-resolution methods such as DAT and HAT are not compared in the article. Compared with these methods, the method proposed in the article does not have a great advantage.\n4.Although the improvement of the mamba variant structure used by the author has achieved certain results, it still looks relatively simple."}},"nonreaders":[],"tmdate":1731427888131,"tcdate":1730525792158,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3583/Reviewer_kVNh"],"signatures":["ICLR.cc/2025/Conference/Submission3583/Reviewer_kVNh"],"forum":"iPYwddLhXR","number":2,"license":"CC BY 4.0","cdate":1730525792158,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3583/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427888131,"domain":"ICLR.cc/2025/Conference","replyto":"iPYwddLhXR","id":"WqeDYoKn0d","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Image Super-Resolution","Mamba","Structured State Space Models"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSM), such as Mamba, have shown strong representation ability in modeling long-range dependency with linear complexity, achieving successful applications from high-level to low-level vision tasks. However, SSM's sequential nature necessitates multiple scans in different directions to compensate for the loss of spatial dependency when unfolding the image into a 1D sequence. This multi-direction scanning strategy significantly increases the computation overhead and is unbearable for high-resolution image processing. To address this problem, we propose a novel Hierarchical Mamba network, namely, Hi-Mamba, for image super-resolution (SR).  Hi-Mamba consists of two key designs: (1) The Hierarchical Mamba Block (HMB) assembled by a Local SSM (L-SSM) and a Region SSM (R-SSM) both with the single-direction scanning, aggregates multi-scale representations to enhance the context modeling ability. (2) The Direction Alternation  Hierarchical Mamba Group (DA-HMG) allocates the isomeric single-direction scanning into cascading HMBs to enrich the spatial relationship modeling.\nExtensive experiments demonstrate the superiority of Hi-Mamba across five benchmark datasets for efficient SR. For example, Hi-Mamba achieves a significant PSNR improvement of 0.29 dB on Manga109 for $\\times3$ SR, compared to the strong lightweight MambaIR."},"_bibtex":{"value":"@misc{\nqiao2024himamba,\ntitle={Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution},\nauthor={Junbo Qiao and Jincheng Liao and Wei Li and Yulun Zhang and Yong Guo and Yi Wen and Zhangxizi Qiu and Jiao Xie and Jie Hu and Shaohui Lin},\nyear={2024},\nurl={https://openreview.net/forum?id=iPYwddLhXR}\n}"},"title":{"value":"Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution"},"pdf":{"value":"/pdf/e75e168c76e93b40ac8e29dd9015c5e7d72edc8b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"qiao|himamba_hierarchical_mamba_for_efficient_image_superresolution"},"authorids":{"value":["~Junbo_Qiao1","~Jincheng_Liao1","~Wei_Li60","~Yulun_Zhang1","~Yong_Guo1","~Yi_Wen3","~Zhangxizi_Qiu2","~Jiao_Xie1","~Jie_Hu8","~Shaohui_Lin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Junbo Qiao","Jincheng Liao","Wei Li","Yulun Zhang","Yong Guo","Yi Wen","Zhangxizi Qiu","Jiao Xie","Jie Hu","Shaohui Lin"]}},"version":2},{"content":{"summary":{"value":"The paper investigates complex dynamical systems and aims to address limitations in modeling these systems using Koopman operators. It explores the application of large language model (LLM) technology to manage the nonlinear and time-varying characteristics of dynamic systems. The work introduces a framework combining Mamba with Koopman operators, leveraging Mamba’s matrix generation capabilities to predict the future states of time-varying systems. This approach preserves a linear state-space structure while enhancing the model’s predictive accuracy and adaptability, creating a linear invariant state-space model. The paper is well-structured and logically organized, presenting innovative ideas with strong experimental support and highlighting the potential impact of this approach on complex dynamical system modeling and control."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. The paper chooses the Mamba model as the basis for the overall approach but needs to provide quantifiable evidence supporting this choice. Could the authors provide relevant data to clarify the rationale for choosing the Mamba model?\n2. All validation experiments were conducted on a simulation platform, with no hardware experiments presented. Simulation results are not a substitute for results from real-world experiments. Could the authors conduct hardware experiments to verify the feasibility and effectiveness of the approach?\n3. The proposed framework aims to solve time-varying issues in time-varying systems but needs more discussion of delay errors and accumulated multi-step prediction errors, as well as the effects these errors may have on system stability and robustness. Did the authors consider these issues in the initial design phase? If not, could they now provide evidence to show the approach can address these issues?\n4. The paper indicates that MamKO relies on multi-step prediction to achieve scalability for complex systems. Could you provide a detailed comparison of the computational overhead between MamKO and other methods? For instance, do quantitative data on computation time or hardware resource consumption for equivalent tasks exist to demonstrate the efficiency of MamKO?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper provides comprehensive experimental steps, data, and ablation analyses to validate the superiority of the MamKO model.\n2. Through experiments on multiple benchmark systems, the paper demonstrates the advantages of MamKO in handling time-varying systems.\n3. The paper shows that MamKO modeling relies on multi-step prediction, enabling the model to scale to complex systems with multi-step dependencies effectively.\n4. Based on the MamKO model, the paper develops an MPC (Model Predictive Control) scheme, emphasizing its computationally efficient optimal control for nonlinear systems."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper does not provide quantitative metrics to justify why the Mamba model was chosen over other models, nor does it include comparative experiments with other LLM models.\n2. All experiments on time-varying systems were conducted in simulated environments, lacking consideration of real-world physical factors. Thus, there are questions regarding the actual performance of the MamKO model.\n3. Although the paper emphasizes solving time-varying problems in control systems, it lacks an analysis or discussion of the control scheme’s stability and robustness.\n4. The proposed approach uses multi-step historical data but needs to address the accumulation of errors that this may cause, which significantly affects the rigor of the model.\n5. While the paper aims to apply the MamKO framework to time-varying systems to address related issues, it does not explicitly discuss the delay errors present in time-varying systems or propose corresponding solutions."}},"nonreaders":[],"tmdate":1731428340591,"tcdate":1730550075498,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10895/Reviewer_41k8"],"signatures":["ICLR.cc/2025/Conference/Submission10895/Reviewer_41k8"],"forum":"hNjCVVm0EQ","number":2,"license":"CC BY 4.0","cdate":1730550075498,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10895/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428340591,"domain":"ICLR.cc/2025/Conference","replyto":"hNjCVVm0EQ","id":"L4gPOmLDJh","forumContent":{"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Koopman operator; model predictive control; nonlinear systems"]},"supplementary_material":{"value":"/attachment/0bf4c76c44568ae8539ede6166bb6a47f0d2be3a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The Koopman theory, which enables the transformation of nonlinear systems into linear representations, is a powerful and efficient tool to model and control nonlinear systems. However, the ability of the Koopman operator to model complex systems, particularly time-varying systems, is limited by the fixed linear state-space representation. To address the limitation, the large language model, Mamba, is considered a promising strategy for enhancing modeling capabilities while preserving the linear state-space structure.\nIn this paper, we propose a new framework, the Mamba-based Koopman operator (MamKO), which provides enhanced model prediction capability and adaptability, as compared to Koopman models with constant Koopman operators. Inspired by the Mamba structure, MamKO generates Koopman operators from online data; this enables the model to effectively capture the dynamic behaviors of the nonlinear system over time. A model predictive control system is then developed based on the proposed MamKO model. The modeling and control performance of the proposed method is evaluated through experiments on benchmark time-invariant and time-varying systems. The experimental results demonstrate the superiority of the proposed approach. Additionally, we perform ablation experiments to test the effectiveness of individual components of MamKO. This approach unlocks new possibilities for integrating large language models with control frameworks, and it achieves a good balance between advanced modeling capabilities and real-time control implementation efficiency."},"_bibtex":{"value":"@inproceedings{\nli2025mamko,\ntitle={Mam{KO}: Mamba-based Koopman operator for modeling and predictive control},\nauthor={ZHAOYANG LI and Minghao Han and Xunyuan Yin},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=hNjCVVm0EQ}\n}"},"title":{"value":"MamKO: Mamba-based Koopman operator for modeling and predictive control"},"pdf":{"value":"/pdf/49097a18837a65cc1415fac755f61c717559a104.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"li|mamko_mambabased_koopman_operator_for_modeling_and_predictive_control"},"authorids":{"value":["~ZHAOYANG_LI12","~Minghao_Han2","~Xunyuan_Yin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["ZHAOYANG LI","Minghao Han","Xunyuan Yin"]}},"version":2},{"content":{"TLDR":{"value":"Introducing MoE-Mamba, combining benefits of Mixture of Experts and SSMs."},"venue":{"value":"ME-FoMo 2024 Poster"},"pdf":{"value":"/pdf/0cbb2749296658df718da50399d4a2df7d479a66.pdf"},"keywords":{"value":["Mamba","LLM","Mixture of Experts","MoE","conditional computation","SSM"]},"venueid":{"value":"ICLR.cc/2024/Workshop/ME-FoMo"},"abstract":{"value":"State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms Mamba and matches the performance of Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in $2.35\\times$ *fewer training steps* while preserving the inference performance gains of Mamba against Transformer."},"_bibtex":{"value":"@inproceedings{\npi{\\'o}ro2024moemamba,\ntitle={MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts},\nauthor={Maciej Pi{\\'o}ro and Kamil Ciebiera and Krystian Kr{\\'o}l and Jan Ludziejewski and Micha{\\l} Krutul and Jakub Krajewski and Szymon Antoniak and Piotr Mi{\\l}o{\\'s} and Marek Cygan and Sebastian Jaszczur},\nbooktitle={ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models},\nyear={2024},\nurl={https://openreview.net/forum?id=LRp8rCaYH7}\n}"},"title":{"value":"MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts"}},"tmdate":1714893985985,"pdate":1709534845119,"tcdate":1706970564994,"writers":["ICLR.cc/2024/Workshop/ME-FoMo","ICLR.cc/2024/Workshop/ME-FoMo/Submission40/Authors"],"signatures":["ICLR.cc/2024/Workshop/ME-FoMo/Submission40/Authors"],"forum":"LRp8rCaYH7","license":"CC BY 4.0","number":40,"cdate":1706970564994,"readers":["everyone"],"invitations":["ICLR.cc/2024/Workshop/ME-FoMo/-/Submission","ICLR.cc/2024/Workshop/ME-FoMo/-/Post_Submission","ICLR.cc/2024/Workshop/ME-FoMo/-/Edit","ICLR.cc/2024/Workshop/ME-FoMo/Submission40/-/Revision"],"mdate":1714893985985,"odate":1712079580932,"domain":"ICLR.cc/2024/Workshop/ME-FoMo","id":"LRp8rCaYH7","version":2},{"content":{"summary":{"value":"Fore-Mamba3D is a Mamba-based 3D perception backbone for LiDAR-based object detection. It focuses on foreground voxels predicted by a learned scoring function. It propose a Regional-to-Global Sliding Window module which enhances long-range voxel interaction to prevent information loss from sparse foregrounds. It also introduce a Semantic-Assisted State Fusion module for fusing semantic and geometric features. It achieves efficient linear modeling and how state-of-the-art performance on nuScenes, KITTI, and Waymo."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Have the authors evaluated how alternative sequence ordering strategies, such as random shuffling or different spatial traversal patterns, affect the performance of the Regional-to-Global Sliding Window (RGSW)? If so, how sensitive is the model to the chosen voxel sequence order, and does it significantly influence spatial consistency or detection accuracy?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The paper presents an interesting and effective idea by selectively adapting foreground voxels for Mamba-based modeling. This approach substantially reduces computational overhead while maintaining high representational quality, showing a smart trade-off between efficiency and performance.\n\n2. The method is validated across multiple benchmark datasets (nuScenes, KITTI, Waymo) with consistent performance gains. The visual results Fig3 are clear and insightful, effectively illustrating how the proposed model can improve the contextual understanding of fused voxel features."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Although the proposed method introduces innovative foreground-focused encoding, the overall performance gain compared to prior Mamba-based or Transformer-based 3D detectors is relatively modest. The improvements, while consistent, may not fully justify the added architectural complexity or the additional design components.\n\n2. Unclear Spatial Consistency in the Regional-to-Global Sliding Window (RGSW): In the proposed RGSW strategy, the authors directly fuse the later half of the sequence with the previous half, which raises concerns about spatial coherence. Since the voxel sequence is constructed by selectively sampled voxels along a Hilbert curve, their ordering does not guarantee true spatial adjacency. This makes the fusion operation confusing, the merged regions may not correspond to physically neighboring spatial areas. As a result, the process may not fully respect local spatial correlation, and in extreme cases, it could behave similarly to random voxel concatenation, weakening the interpretability of the regional-to-global transition.\n\n3. The authors completely discard background voxel feature extraction, directly concatenating the processed foreground features. While this design choice further reduces computational cost, it introduces a potential issue, semantic strength inconsistency between foreground and background regions. In fact, background voxels play an important role in defining object boundaries and separating adjacent instances. Ignoring background information entirely may lead to inaccurate boundary localization, especially in cluttered or occluded scenes."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762921614401,"tcdate":1761626788276,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10258/Reviewer_jtLi"],"signatures":["ICLR.cc/2026/Conference/Submission10258/Reviewer_jtLi"],"forum":"e4t1775UJ1","number":2,"license":"CC BY 4.0","cdate":1761626788276,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10258/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762921614401,"domain":"ICLR.cc/2026/Conference","replyto":"e4t1775UJ1","id":"YEZLrJdAbu","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["3D Object Detection","Mamba","Foreground Representation"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Linear modeling methods like Mamba have been merged as the effective backbone for the 3D object detection task. However, previous Mamba-based methods utilize the bidirectional encoding for the whole non-empty voxel sequence, which contains abundant useless background information in the scenes. Though directly encoding foreground voxels appears to be a plausible solution, it tends to degrade detection performance. We attribute this to the response attenuation and restricted context representation in the linear modeling for fore-only sequences. To address this problem, we propose a novel backbone, termed Fore-Mamba3D, to focus on the foreground enhancement by modifying Mamba-based encoder. The foreground voxels are first sampled according to the predicted scores. Considering the response attenuation existing in the interaction of foreground voxels across different instances, we design a regional-to-global slide window (RGSW) to propagate the information from regional split to the entire sequence. Furthermore, a semantic-assisted and state spatial fusion module (SASFMamba) is proposed to enrich contextual representation by enhancing semantic and geometric awareness within the Mamba model. Our method emphasizes foreground-only encoding and alleviates the distance-based and causal dependencies in the linear autoregression model. The superior performance across various benchmarks demonstrates the effectiveness of Fore-Mamba3D in the 3D object detection task."},"_bibtex":{"value":"@inproceedings{\nning2026foremambad,\ntitle={Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection},\nauthor={Zhiwei Ning and Xuanang Gao and Jiaxi Cao and Runze Yang and Huiying Xu and Xinzhong Zhu and JIE YANG and Wei Liu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=e4t1775UJ1}\n}"},"title":{"value":"Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection"},"pdf":{"value":"/pdf/c88d1a4b5a3d89c3710599dac8b4ca8c7c9c1d6e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"ning|foremamba3d_mambabased_foregroundenhanced_encoding_for_3d_object_detection"},"authorids":{"value":["~Zhiwei_Ning2","~Xuanang_Gao1","~Jiaxi_Cao1","~Runze_Yang2","~Huiying_Xu1","~Xinzhong_Zhu1","~JIE_YANG18","~Wei_Liu38"]},"authors":{"value":["Zhiwei Ning","Xuanang Gao","Jiaxi Cao","Runze Yang","Huiying Xu","Xinzhong Zhu","JIE YANG","Wei Liu"]}},"version":2},{"content":{"summary":{"value":"This paper presents a hybrid approach that combines elements of Mamba and Transformer architectures, aiming to address two major challenges: the high computational complexity of Transformers in handling long contexts and the quality degradation issues encountered with Mamba. This approach aligns with prior research, including methods like Samba and Jamba.\n\nThe key contribution of the paper is its selective mechanism for token attention calculation. By incorporating a gating network, the model selectively skips attention computation for certain tokens, reducing inference costs. This optimization enhances the efficiency of the attention layer, achieving a notable speed-up without compromising performance and the paper demonstrates this with empirical results."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"see weaknesses"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper’s motivation is clear, focusing on an important and timely topic with practical significance.\n2. The writing is clear and well-organized, making it easy to understand.\n3. The selective attention concept is well-founded and adds a valuable perspective to the field"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The concept of selective attention is promising, as it aligns well with recent advances in efficient language models. However, similar approaches have been explored in prior work, including \"Power-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination\" and \"A Gated Self-attention Memory Network for Answer Selection.\" These studies also leverage selective focus on important tokens, prioritizing computation for tokens requiring additional context. Further distinction from these works, especially in terms of innovation and unique contributions, would enhance the impact of this paper.\n\nAmong previous research, \"Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling\" appears most comparable due to its hybrid structure and sliding window mechanism in attention. I would anticipate that Samba could achieve similar results in performance and latency to the model in this paper. A thorough empirical comparison with Samba would be beneficial to underscore the advantages and trade-offs of the proposed approach.\n\nIn Figure 1, perplexity seems to increase steadily from the start. Typically, one might expect an initial decrease in perplexity with context length before a rise as the length extends beyond a certain threshold such as the pre-training context length. Additionally, the claim that Taipan outperforms Mamba in terms of latency is unclear. Providing further clarification on latency measurements and factors contributing to Taipan’s efficiency would enhance the reader’s understanding.\n\nRegarding task performance, additional explanation is needed to clarify why Taipan outperforms Transfer on the tasks listed in Table 1, as many involve short-context scenarios. More supporting evidence to validate Taipan’s superiority on these tasks would strengthen the claims. Furthermore, including benchmarks on MMLU and GSM-8K, which require higher reasoning capabilities, would offer a more comprehensive assessment of the model's generalization and reasoning skills."}},"nonreaders":[],"tmdate":1731429282671,"tcdate":1730686630434,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9385/Reviewer_v4EA"],"signatures":["ICLR.cc/2025/Conference/Submission9385/Reviewer_v4EA"],"forum":"9DnKZbOr4r","number":3,"license":"CC BY 4.0","cdate":1730686630434,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9385/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429282671,"domain":"ICLR.cc/2025/Conference","replyto":"9DnKZbOr4r","id":"mCkSJuXbz6","forumContent":{"TLDR":{"value":"We introduce Taipan, a novel architecture that combines State Space Models with Selective Attention Layers, achieving superior performance in memory-intensive tasks while enabling efficient language modeling for sequences up to 1 million tokens."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Efficient Language Model","Model Architecture","Long-context Language Model","In-context Retrieval","Hybrid Architecture","Linear Complexity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long sequences due to quadratic computational complexity in training and linearly scaling memory costs during inference. Recent State Space Models (SSMs) such as Mamba offer alternatives with constant memory usage, but they underperform in tasks requiring extensive in-context retrieval. We introduce Taipan, a novel hybrid architecture that combines Mamba-2 with Selective Attention Layers (SALs). These SALs identify tokens requiring long-range interactions, remove less important features, and then augment their representations using the attention module. This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. By constraining the attention budget, Taipan extends accurate predictions to context lengths of up to 1 million tokens while preserving computational efficiency. Our experiments demonstrate Taipan's superior performance across various scales and tasks, offering a promising solution for efficient long-context language modeling."},"_bibtex":{"value":"@misc{\nnguyen2025taipan,\ntitle={Taipan: Efficient and Expressive State Space Language Models with Selective Attention},\nauthor={Chien Van Nguyen and Huy Huu Nguyen and Thang M. Pham and Ruiyi Zhang and Hanieh Deilamsalehy and Puneet Mathur and Ryan A. Rossi and Trung Bui and Viet Dac Lai and Franck Dernoncourt and Thien Huu Nguyen},\nyear={2025},\nurl={https://openreview.net/forum?id=9DnKZbOr4r}\n}"},"title":{"value":"Taipan: Efficient and Expressive State Space Language Models with Selective Attention"},"pdf":{"value":"/pdf/4c569111617737e5532042f859ac730be0ac18ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"nguyen|taipan_efficient_and_expressive_state_space_language_models_with_selective_attention"},"authorids":{"value":["~Chien_Van_Nguyen1","~Huy_Huu_Nguyen1","~Thang_M._Pham1","~Ruiyi_Zhang3","~Hanieh_Deilamsalehy1","~Puneet_Mathur2","~Ryan_A._Rossi2","~Trung_Bui1","~Viet_Dac_Lai1","~Franck_Dernoncourt1","~Thien_Huu_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chien Van Nguyen","Huy Huu Nguyen","Thang M. Pham","Ruiyi Zhang","Hanieh Deilamsalehy","Puneet Mathur","Ryan A. Rossi","Trung Bui","Viet Dac Lai","Franck Dernoncourt","Thien Huu Nguyen"]}},"version":2},{"content":{"summary":{"value":"The paper pioneers the utilization of Mamba in the anomaly detection field. Specifically, the core contribution of the proposed MambaAD is a Locality-Enhanced State Space module, which utilizes Mamba for global information and CNN for local information. Whereas the motivation and the core technical contribution of this paper are similar to other works like U-Mamba, VMamba, and MedMamba, the proposed MambaAD is the first to extensively explore Mamba in the anomaly detection field and achieve state-of-the-art multi-class anomaly detection performance with a low parameter count and computational demand, making this paper acceptable for me."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"See the weakness."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"This paper is the first to explore Mamba in the anomaly detection field. \n\nThe proposed MambaAD achieves state-of-the-art multi-class anomaly detection performance with lower computational demand and fewer parameters than other multi-class anomaly detection alternatives. \n\nThe experiments are extensive, featuring six distinct AD datasets with seven metrics."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The motivation for using Mamba for visual anomaly detection should be strengthened. e.g., what’s the core challenge for multi-class anomaly detection? why long-range modeling is important for multi-class anomaly detection?\n\n2. It would be better to report the performance with only global/local branches in Table 5, which can better show the influence of individual branches.\n\n3. While the proposed MambaAD achieves better detection performance in most metrics on the evaluated datasets, for some metrics, like AUPRO on VisA, the proposed MambaAD is not the best one. It would be better to offer some analysis on this issue.\n\n4. The current MambaAD framework appears to still rely on a pre-trained CNN encoder, utilizing the Mamba model only in the decoder. Why is this design chosen? What would be the implications if the entire structure were based on the Mamba model?\n\n5. It seems that only the Hilbert scanning method is ultimately used out of the five proposed hybrid scanning methods. How should we further interpret the ablation study in Table 3?"},"limitations":{"value":"See the weakness."}},"nonreaders":[],"tmdate":1730878757411,"tcdate":1720513808435,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2207/Reviewer_wfnx"],"signatures":["NeurIPS.cc/2024/Conference/Submission2207/Reviewer_wfnx"],"forum":"8VKxTlnejE","number":1,"license":"CC BY 4.0","cdate":1720513808435,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2207/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878757411,"domain":"NeurIPS.cc/2024/Conference","replyto":"8VKxTlnejE","id":"UfWhJet8DP","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"TLDR":{"value":"The first to utilize the Mamba model to address multi-class anomaly detection tasks."},"keywords":{"value":["Mamba","State Space Models","Anomaly Detection","Unsupervised Learning"]},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Recent advancements in anomaly detection have seen the efficacy of CNN- and transformer-based approaches. However, CNNs struggle with long-range dependencies, while transformers are burdened by quadratic computational complexity. Mamba-based models, with their superior long-range modeling and linear efficiency, have garnered substantial attention. This study pioneers the application of Mamba to multi-class unsupervised anomaly detection, presenting MambaAD, which consists of a pre-trained encoder and a Mamba decoder featuring (Locality-Enhanced State Space) LSS modules at multi-scales. The proposed LSS module, integrating parallel cascaded (Hybrid State Space) HSS blocks and multi-kernel convolutions operations, effectively captures both long-range and local information. The HSS block, utilizing (Hybrid Scanning) HS encoders, encodes feature maps into five scanning methods and eight directions, thereby strengthening global connections through the (State Space Model) SSM. The use of Hilbert scanning and eight directions significantly improves feature sequence modeling. Comprehensive experiments on six diverse anomaly detection datasets and seven metrics demonstrate state-of-the-art performance, substantiating the method's effectiveness. The code and models are available at https://lewandofskee.github.io/projects/MambaAD."},"_bibtex":{"value":"@inproceedings{\nhe2024mambaad,\ntitle={Mamba{AD}: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection},\nauthor={Haoyang He and Yuhu Bai and Jiangning Zhang and Qingdong He and Hongxu Chen and Zhenye Gan and Chengjie Wang and Xiangtai Li and Guanzhong Tian and Lei Xie},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=8VKxTlnejE}\n}"},"title":{"value":"MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection"},"pdf":{"value":"/pdf/b33454c402a9428ad357b8c1e894b852ce56a599.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"he|mambaad_exploring_state_space_models_for_multiclass_unsupervised_anomaly_detection"},"authorids":{"value":["~Haoyang_He1","~Yuhu_Bai1","~Jiangning_Zhang1","~Qingdong_He1","~Hongxu_Chen7","~Zhenye_Gan1","~Chengjie_Wang1","~Xiangtai_Li1","~Guanzhong_Tian1","~Lei_Xie6"]},"authors":{"value":["Haoyang He","Yuhu Bai","Jiangning Zhang","Qingdong He","Hongxu Chen","Zhenye Gan","Chengjie Wang","Xiangtai Li","Guanzhong Tian","Lei Xie"]}},"version":2},{"content":{"summary":{"value":"This paper addresses the issue of artifacts within the feature maps of Vision Mamba. Similar to vision transformers, high-norm tokens emerging in low-information background areas of images. To mitigate this, the authors introduce register tokens into Vision Mamba, resulting in a new architecture termed Mamba-R. This architecture includes two key modifications: 1) evenly inserting registers throughout the input token sequence, and 2) recycling registers for final decision predictions. Qualitative observations suggest that Mamba-R's feature maps are cleaner and more focused on semantically meaningful regions. Additional validation on downstream semantic segmentation tasks further supports Mamba-R's efficacy."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Please refer to weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper is well-written and presents its ideas in a clear and understandable manner.\n2. The exploration of alleviating artifacts in Vision Mamba is a valuable contribution to the field.\n3. The experiments provide evidence supporting the necessity of register tokens in Vision Mamba."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper primarily transfers observations and methods from Vision Transformers [1] to Vision Mamba without significant innovation.\n2. Despite the introduction of registration, the feature maps still exhibit noise compared to visualizations in prior work [1], why?\n3. Mamba-R introduces additional parameters and computations, as noted in Table 4, where a small improvement in mIOU (0.4) comes at the cost of 10M more parameters.\n4. The authors are encouraged to validate the effectiveness of registration in unsupervised object discovery, similar to the analysis in Table 3 of [1].\n5. The strategy of distributing register tokens evenly throughout the sequence appears less convincing, as indicated by marginal improvements (only +0.3 in Table 6) compared to puting registration tokens in the middle. This could be attributed to randomness rather than a systematic advantage.\n6. Generalization to other representative vision mambas, like VMamba [2] and MambaOut [3]?\n\n[1] Vision Transformers Need Registers\n[2] VMamba: Visual State Space Model\n[3] MambaOut: Do We Really Need Mamba for Vision?"}},"nonreaders":[],"tmdate":1731427468141,"tcdate":1730862644726,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1694/Reviewer_x3zz"],"signatures":["ICLR.cc/2025/Conference/Submission1694/Reviewer_x3zz"],"forum":"wxEASOHHdT","number":5,"license":"CC BY 4.0","cdate":1730862644726,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1694/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427468141,"domain":"ICLR.cc/2025/Conference","replyto":"wxEASOHHdT","id":"xkOaYCKBoG","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["State Space Models","Mamba","Representation Learning"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Similar to Vision Transformers, this paper identifies artifacts also present within the feature maps of Vision Mamba. These artifacts, corresponding to high-norm tokens emerging in low-information background areas of images, appear much more severe in Vision Mamba---they exist prevalently even with the tiny-sized model and activate extensively across background regions. To mitigate this issue, we follow the prior solution of introducing register tokens into Vision Mamba. To better cope with Mamba blocks' uni-directional inference paradigm, two key modifications are introduced: 1) evenly inserting registers throughout the input token sequence, and 2) recycling registers for final decision predictions. We term this new architecture MambaReg. Qualitative observations suggest, compared to vanilla Vision Mamba, MambaReg's feature maps appear cleaner and more focused on semantically meaningful regions.  Quantitatively, MambaReg attains stronger performance and scales better.  For example, on the ImageNet benchmark, our MambaReg-B attains 83.0% accuracy, significantly outperforming Vim-B's 81.8%; furthermore, we provide the first successful scaling to the large model size (i.e., with 341M parameters), attaining a competitive accuracy of 83.6% (84.5% if finetuned with 384x384 inputs). Additional validation on the downstream semantic segmentation task also supports MambaReg's efficacy."},"_bibtex":{"value":"@misc{\nwang2024mambareg,\ntitle={Mamba-Reg: Vision Mamba Also Needs Registers},\nauthor={Feng Wang and Jiahao Wang and Sucheng Ren and Guoyizhe Wei and Jieru Mei and Wei Shao and Yuyin Zhou and Alan Yuille and Cihang Xie},\nyear={2024},\nurl={https://openreview.net/forum?id=wxEASOHHdT}\n}"},"title":{"value":"Mamba-Reg: Vision Mamba Also Needs Registers"},"pdf":{"value":"/pdf/85e2e7cb78a7d2c589e0ea139ce0b63bae5fc9cd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"wang|mambareg_vision_mamba_also_needs_registers"},"authorids":{"value":["~Feng_Wang14","~Jiahao_Wang5","~Sucheng_Ren1","~Guoyizhe_Wei1","~Jieru_Mei2","~Wei_Shao8","~Yuyin_Zhou1","~Alan_Yuille1","~Cihang_Xie3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Feng Wang","Jiahao Wang","Sucheng Ren","Guoyizhe Wei","Jieru Mei","Wei Shao","Yuyin Zhou","Alan Yuille","Cihang Xie"]}},"version":2},{"content":{"summary":{"value":"The authors propose a clever method that exploits recent findings about the effectiveness of linear probing across diverse representation spaces in order to discover semantically-meaningful clusters (i.e. ones that correspond to human labeling) by seeing which linearly-separable clusters are preserved across multiple representation spaces. "},"soundness":{"value":"3 good"},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"- When producing reliable samples for SSL, what is the class balance? The usefulness is impacted significantly by the diversity of the samples HUME can produce. I see in Fig 3a) that samples are produced from each class, but is performance roughly the same across all classes?\n- Are all three of the large-pretrained models that you use pre-trained in unsupervised ways (e.g. none were ever actually trained on the ImageNet classes)?  \n- Unrelated to the review/scoring and purely out of curiosity, why did you name your method HUME? Is it an acronym for something?"},"rating":{"value":"8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":"4 excellent"},"contribution":{"value":"4 excellent"},"strengths":{"value":"- HUME is a very clever method that beautifully exploits recent findings about the effectiveness of linear probing across diverse representation spaces. \n- The paper is well-written and easy to follow. "},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- It's unclear whether performance gains are driven by HUME or by the strength of the second representation space. Additional baseline experiments with linear probing and clustering on top of the large pre-trained model representations (DINOv2, CLIP, ViT) would help clarify this. Another nice baseline evaluation that could help would be to simply concatenate the embeddings from the two representation spaces and repeat the linear probing and clustering evaluations on that. \n\n"},"limitations":{"value":"See weaknesses and questions sections. "}},"nonreaders":[],"tmdate":1702411088043,"tcdate":1688701512031,"writers":["NeurIPS.cc/2023/Conference","NeurIPS.cc/2023/Conference/Submission6997/Reviewer_uZTE"],"signatures":["NeurIPS.cc/2023/Conference/Submission6997/Reviewer_uZTE"],"forum":"3GpIeVYw8X","number":5,"license":"CC BY 4.0","cdate":1688701512031,"mdate":1702411088043,"readers":["everyone"],"invitations":["NeurIPS.cc/2023/Conference/Submission6997/-/Official_Review","NeurIPS.cc/2023/Conference/-/Edit"],"domain":"NeurIPS.cc/2023/Conference","replyto":"3GpIeVYw8X","id":"v9Lw9yOddg","forumContent":{"venue":{"value":"NeurIPS 2023 spotlight"},"keywords":{"value":["unsupervised learning","deep learning","generalization","self-supervised learning","clustering"]},"supplementary_material":{"value":"/attachment/92ae38550b5a67baf4e429155c54fcad75293dd0.pdf"},"_bibtex":{"value":"@inproceedings{\ngadetsky2023the,\ntitle={The Pursuit of Human Labeling: A New Perspective on Unsupervised Learning},\nauthor={Artyom Gadetsky and Maria Brbic},\nbooktitle={Thirty-seventh Conference on Neural Information Processing Systems},\nyear={2023},\nurl={https://openreview.net/forum?id=3GpIeVYw8X}\n}"},"title":{"value":"The Pursuit of Human Labeling: A New Perspective on Unsupervised Learning"},"paperhash":{"value":"gadetsky|the_pursuit_of_human_labeling_a_new_perspective_on_unsupervised_learning"},"TLDR":{"value":"We propose an unsupervised learning framework to find human labeling by searching for consistent labelings between different representation spaces."},"abstract":{"value":"We present HUME, a simple model-agnostic framework for inferring human labeling of a given dataset without any external supervision. The key insight behind our approach is that classes defined by many human labelings are linearly separable regardless of the representation space used to represent a dataset. HUME utilizes this insight to guide the search over all possible labelings of a dataset to discover an underlying human labeling. We show that the proposed optimization objective is strikingly well-correlated with the ground truth labeling of the dataset. In effect, we only train linear classifiers on top of pretrained representations that remain fixed during training, making our framework compatible with any large pretrained and self-supervised model. Despite its simplicity, HUME outperforms a supervised linear classifier on top of self-supervised representations on the STL-10 dataset by a large margin and achieves comparable performance on the CIFAR-10 dataset. Compared to the existing unsupervised baselines, HUME achieves state-of-the-art performance on four benchmark image classification datasets including the large-scale ImageNet-1000 dataset. Altogether, our work provides a fundamentally new view to tackle unsupervised learning by searching for consistent labelings between different representation spaces."},"pdf":{"value":"/pdf/0b0344e4b854907d561336645c1575259ab54721.pdf"},"venueid":{"value":"NeurIPS.cc/2023/Conference"},"authorids":{"value":["~Artyom_Gadetsky1","~Maria_Brbic1"]},"authors":{"value":["Artyom Gadetsky","Maria Brbic"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a unified theoretical framework to describe sequence models including Transformer and RNNs, and puts forward a series of design principles and experimental verifications to guide sequence modeling."},"soundness":{"value":3},"confidence":{"value":2},"questions":{"value":"1. Based on the design principles of the sequence modeling architecture proposed in the paper, what characteristics should the optimal model architecture for sequence modeling?\n2. Although these design principles can guide the design of the optimal sequence modeling architecture, for linear RNN architectures, the design of some components may not be conducive to hardware efficient parallel training of RNN. In this case, would a more general and expressive matrix gate be less useful than a simpler, more efficiently trained diagonal/scaler gate?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The design principles for sequence modeling architecture proposed in this paper are instructive.\n2. The analysis combining theory and experiment is convincing."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"A unified sequence modeling framework for transformer and RNN model architectures has been mentioned in various works, such as MetaLA [1], PaTH Attention [2], and log-linear-attention [3], which may diminish the contribution of this paper. Therefore, further comparison and discussion with similar related works will help highlight the contribution of this paper.\n\n[1] Yuhong Chou, et al. MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map. NeurIPS, 2024.\n[2] Songlin Yang, et al. PaTH Attention: Position Encoding via Accumulating Householder Transformations. NeurIPS, 2025.\n[3] Han Guo, et al. Log-Linear Attention. arXiv, 2025."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942929609,"tcdate":1761992425360,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission24085/Reviewer_gPgH"],"signatures":["ICLR.cc/2026/Conference/Submission24085/Reviewer_gPgH"],"forum":"9uChwgWHX9","number":3,"license":"CC BY 4.0","cdate":1761992425360,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission24085/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942929609,"domain":"ICLR.cc/2026/Conference","replyto":"9uChwgWHX9","id":"0Yz95wChzM","forumContent":{"TLDR":{"value":"Mixer blocks compute linear combinations of value vectors; we model their coefficients as outputs of autonomous linear dynamics with impulse inputs, unifying models from attention to SSMs and yielding design principles for sequence model design."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Sequence models","Design principles","Coefficient dynamics"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Deep sequence models, ranging from Transformers and State Space Models  (SSMs) to more recent approaches such as gated linear RNNs, fundamentally compute outputs as linear combinations of past value vectors. To draw insights and systematically compare such architectures, we develop a unified framework that makes this output operation explicit, by casting the linear combination coefficients as the outputs of autonomous linear dynamical systems driven by impulse inputs. This viewpoint, in spirit substantially different from approaches focusing on connecting linear RNNs with linear attention, reveals a common mathematical theme across diverse architectures and crucially captures softmax attention, on top of RNNs, SSMs, and related models. In contrast to new model proposals that are commonly evaluated on benchmarks, we derive design principles linking architectural choices to model properties. Thereby identifying tradeoffs between expressivity and efficient implementation, geometric constraints on input selectivity, and stability conditions for numerically stable training and information retention. By connecting several insights and observations from recent literature, the framework both explains empirical successes of recent designs and provides guiding principles for systematically designing new sequence model architectures."},"_bibtex":{"value":"@misc{\nsieber2026design,\ntitle={Design Principles for Sequence Models via Coefficient Dynamics},\nauthor={Jerome Sieber and Antonio Orvieto and Melanie Zeilinger and Carmen Amo Alonso},\nyear={2026},\nurl={https://openreview.net/forum?id=9uChwgWHX9}\n}"},"title":{"value":"Design Principles for Sequence Models via Coefficient Dynamics"},"pdf":{"value":"/pdf/0ba7218772bdf1386ec46825ed03f3b506371c5a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"sieber|design_principles_for_sequence_models_via_coefficient_dynamics"},"authorids":{"value":["~Jerome_Sieber1","~Antonio_Orvieto3","~Melanie_Zeilinger1","~Carmen_Amo_Alonso3"]},"authors":{"value":["Jerome Sieber","Antonio Orvieto","Melanie Zeilinger","Carmen Amo Alonso"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a new framework for inferring continuous cell cycle stages solely from label-free brightfield time-lapse images, using fluorescent Fucci signals as the ground truth. Specifically, the authors construct a large-scale dataset (over 1.3 million images) of RPE1 cells with nuclear (H2B) and Fucci signals, then systematically compare different deep sequence models: single-frame MLP, fixed temporal convolution (Causal CNN), causal state-space (Mamba, LSTM), and bidirectional transformer. They report that bidirectional and causal sequence models significantly outperform single-frame baselines, capturing subtle cell cycle transitions, such as G1/S, with around one-hour resolution. A key strength is demonstrating that morphological cues in brightfield can rival more explicit nuclear markers (like H2B) when equipped with powerful sequence encoders. The results highlight important implications for label-free monitoring of cell cycle states and open avenues for broader “smart” microscopy applications where fluorescent labeling is impractical or limited."},"justification_of_the_preliminary_rating":{"value":"This paper offers a valuable dataset and a thorough comparison of sequence models for continuous cell cycle prediction from brightfield images, a task with strong relevance in bioimaging and drug discovery. The key novelty lies in the temporal modeling of Fucci-like signals without needing actual fluorescent labeling, and the authors demonstrate significant gains over single-frame baselines. While certain aspects—such as testing on more cell lines, deeper interpretability, and partial-track complexities—could be expanded, the overall methodology is great, and results are compelling."},"strengths":{"value":"1. The authors provide a large dataset of labeled, time-lapse brightfield and fluorescent images (1.3 M images total) featuring multiple full cell cycles. \n2. They systematically compare a wide range of sequence models—from single-frame MLP to state-of-the-art transformers—using consistent architectural sizes and metrics (L1 error, DTW distance, checkpoint times). \n3. Predicting cell cycle phase transitions from standard brightfield images can be highly useful in labs lacking advanced or multiple fluorescence channels. \n4. The paper includes rigorous metrics (including biologically meaningful G1/S and S/G2 transitions) and also tests on out-of-distribution data (CDK4-6 inhibitor). These analyses strengthen the conclusions about sequence model robustness and generalizability.\n5. The manuscript is well-structured, with carefully presented figures, sufficient detail on model architectures, and a thoughtful discussion of biological checkpoints."},"weaknesses":{"value":"1. While the transformer and Mamba approaches are strong, the paper could benefit from a brief discussion of possible enhancements like hybrid multi-scale convolution-transformer backbones or variational latent models for capturing subtle morphological transitions.\n2. The paper emphasizes predictive performance but offers relatively little about how the sequence models learn to capture morphological signals of the cell cycle."},"confidence":{"value":4},"detailed_comments":{"value":"1. The authors use L1 loss with a time-warped DTW metric for validation. Clarifying how hyperparameters in DTW (e.g., warping penalties) were selected would be beneficial.\n2. The G1/S and S/G2 transitions are inferred from Fucci intensities. Including a short discussion on the biological variability in these landmarks may help interpret the model’s timing offsets.\n3. Sequence models like Transformers can be GPU-intensive for long time series. The authors might note memory usage or speed trade-offs, guiding future large-scale experiments.\n4. Since the dataset is publicly available, releasing checkpoints of the best performing models (or code) would accelerate community benchmarking.\n5. The model’s improved generalization for the transformer is promising. More insights into which morphological features shift under Palbociclib would be a valuable extension."},"recommendation":{"value":["Oral","Poster"]},"questions_to_address_in_the_rebuttal":{"value":"1. Could the authors provide more analysis of which morphological features the Transformer or Mamba exploit, for instance via attention maps or saliency approaches?\n2. How feasible is it to run the best-performing transformer in real-time or near real-time? Are there any latency concerns for live-cell experiments?\n3. What do the authors propose to mitigate the diminished improvements in partial-tracking scenarios? Could a hierarchical approach or prior knowledge about morphological transitions help?"},"preliminary_rating":{"value":4}},"nonreaders":[],"tmdate":1747661876991,"tcdate":1740202296349,"writers":["MIDL.io/2025/Conference","MIDL.io/2025/Conference/Submission241/Reviewer_T5Cg"],"signatures":["MIDL.io/2025/Conference/Submission241/Reviewer_T5Cg"],"forum":"npYk02a6Qj","number":2,"license":"CC BY 4.0","cdate":1740202296349,"readers":["everyone"],"invitations":["MIDL.io/2025/Conference/Submission241/-/Official_Review","MIDL.io/2025/Conference/-/Edit"],"mdate":1747661876991,"domain":"MIDL.io/2025/Conference","replyto":"npYk02a6Qj","id":"WUA5Mj2jrA","forumContent":{"TLDR":{"value":"Sequence models for continuous cell cycle stage prediction from brightfield images"},"venue":{"value":"MIDL 2025 Oral"},"midl_latex_submission_checklist":{"value":["The paper compiles correctly using the pdflatex compiler.","Replace NNN with your OpenReview submission ID.","The LaTeX file includes the correct header commands before \\title.","Hyperref is preloaded; do not modify it.","If included, must contact program chairs immediately.","Format names correctly and avoid Unicode characters.","Title page, citations, repositories, data sources, and acknowledgements must reflect correct authorship.","Did not use \\begin{thebibliography} directly to insert references.","Avoid using \\scalebox; prefer \\resizebox{\\textwidth}{!}{...}.","Zip archive should only contain needed files.","Clean final version without visual artifacts.","Ensure proper LaTeX encoding of characters.","No separate supplementary PDF uploads.","Check page limit and content structure carefully."]},"keywords":{"value":["Cell cycle prediction","label-free microscopy","sequence-models","Mamba"]},"abstract":{"value":"Understanding cell cycle dynamics is crucial for studying biological processes such as growth, development and disease progression. While fluorescent protein reporters like the Fucci system allow live monitoring of cell cycle phases, they require genetic engineering and occupy additional fluorescence channels, limiting broader applicability in complex experiments. In this study, we conduct a comprehensive evaluation of deep learning methods for predicting continuous Fucci signals using non-fluorescence brightfield imaging, a widely available label-free modality. To that end, we generated a large dataset of 1.3 M images of dividing RPE1 cells with full cell cycle trajectories to quantitatively compare the predictive performance of distinct model categories including single time-frame models, causal state space models and bidirectional transformer models. We show that both causal and transformer-based models significantly outperform single- and fixed frame approaches, enabling the prediction of visually imperceptible transitions like G1/S within 1h resolution. Our findings underscore the importance of sequence models for accurate predictions of cell cycle dynamics and highlight their potential for label-free imaging."},"_bibtex":{"value":"@inproceedings{\nleger2025sequence,\ntitle={Sequence models for continuous cell cycle stage prediction from brightfield images},\nauthor={Louis-Alexandre Leger and Maxine Leonardi and Andrea Salati and Felix Naef and Martin Weigert},\nbooktitle={Medical Imaging with Deep Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=npYk02a6Qj}\n}"},"title":{"value":"Sequence models for continuous cell cycle stage prediction from brightfield images"},"latex_code":{"value":"/attachment/2dddafd38a15281d03217f1d402cd9855e759040.zip"},"paper_type":{"value":"Methodological Development"},"secondary_subject_area":{"value":"Image Acquisition and Reconstruction"},"pdf":{"value":"/pdf/62eccc68f1c819f0b6954a49d75c5089eb3a5314.pdf"},"copyright_form":{"value":"/attachment/f94482f4e62f40914f3ec6883af966704ae30ef0.pdf"},"venueid":{"value":"MIDL.io/2025/Conference"},"paperhash":{"value":"leger|sequence_models_for_continuous_cell_cycle_stage_prediction_from_brightfield_images"},"primary_subject_area":{"value":"Unsupervised Learning and Representation Learning"},"authorids":{"value":["~Louis-Alexandre_Leger1","~Maxine_Leonardi1","~Andrea_Salati1","~Felix_Naef1","~Martin_Weigert2"]},"registration":{"value":"Yes"},"authors":{"value":["Louis-Alexandre Leger","Maxine Leonardi","Andrea Salati","Felix Naef","Martin Weigert"]}},"version":2},{"content":{"research_area_keywords":{"value":"Linear sequence modeling, mixture of experts, sequence parallelism, hybrid models"},"languages_studied":{"value":"English,Chinese"},"venue":{"value":"ACL ARR 2025 February Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025linearmoe,\ntitle={Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - February 2025},\nyear={2025},\nurl={https://openreview.net/forum?id=m6TZ6nB0MD},\nnote={under review}\n}"},"title":{"value":"Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts"},"contribution_types":{"value":["Model analysis & interpretability","NLP engineering experiment","Approaches low compute settings-efficiency","Publicly available software and/or pre-trained models"]},"abstract":{"value":"Linear Sequence Modeling (LSM) like linear attention, state space modeling and linear RNNs, and Mixture-of-Experts (MoEs) have recently emerged as significant architectural improvements. In this paper, we introduce Linear-MoE, a production-level system for modeling and training large-scale models that integrate LSM with MoEs. Linear-MoE leverages the advantages of both LSM modules for linear-complexity sequence modeling and MoE layers for sparsely activation, aiming to offer high performance with efficient training. The Linear-MoE system comprises: 1) Modeling subsystem, which provides a unified framework supporting all instances of LSM. and 2) Training subsystem, which facilitates efficient training by incorporating advanced parallelism, particularly Sequence Parallelism designed for Linear-MoE models. Additionally, we explore hybrid models that combine Linear-MoE layers with standard Transformer-MoE layers with its Sequence Parallelism to further enhance model flexibility and performance. Evaluations on two model series, A0.3B-2B and A1B-7B, demonstrate Linear-MoE achieves efficiency gains while maintaining competitive performance on various benchmarks, showcasing its potential as a next-generation foundational model architecture. Code: \\url{https://anonymous.4open.science/r/Linear-MoE-AD77}"},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/5d19dae77bf8dc956fc4921ac4ff38b2ec7ad958.pdf"},"research_area":{"value":"Efficient/Low-Resource Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/February/Submission"}},"tmdate":1782961992785,"tcdate":1739618476963,"writers":["aclweb.org/ACL/ARR/2025/February","aclweb.org/ACL/ARR/2025/February/Submission3264/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/February/Submission3264/Authors"],"forum":"m6TZ6nB0MD","license":"CC BY 4.0","number":3264,"cdate":1739618476963,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/February/-/Submission","aclweb.org/ACL/ARR/2025/February/-/Edit","aclweb.org/ACL/ARR/2025/February/-/Post_Submission","aclweb.org/ACL/ARR/2025/February/-/Desk_Reject_Verification_Flag","aclweb.org/ACL/ARR/2025/February/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2025/February/-/Preprint_Post_Submission"],"mdate":1782961992785,"odate":1746821355472,"domain":"aclweb.org/ACL/ARR/2025/February","id":"m6TZ6nB0MD","version":2},{"content":{"summary":{"value":"The paper introduces Q-Mamba, an offline learning framework for Meta-Black-Box Optimization (MetaBBO) designed to improve both the optimization performance and training efficiency of existing online MetaBBO methods. Q-Mamba employs a Decomposed Q-function Representation (DQR) to address the challenge of controlling a low-level optimizer’s multiple configurable components, breaking down the complex joint-action space into sequential Q-functions for each component. The framework uses a Mamba-based neural network Q-Learner to model these sequential Q-functions, enhancing long-sequence modeling for improved policy learning. By leveraging offline trajectory data from both exploration and exploitation, Q-Mamba aims to circumvent issues associated with online learning, such as high training costs and inefficiency. Additionally, Q-Mamba integrates Conservative Q-Learning (CQL) regularization to address distributional shifts during offline training. Experimental results indicate that Q-Mamba performs competitively, often surpassing both online and offline baselines in efficiency and effectiveness on a range of black-box optimization tasks."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"__[Questions and Comments]__\n\n* Typo: in Line 143 on page 3, shouldn't \"improving\" be used instead of \"reducing\" in the given context?\n\n* Notation for Q-functions: why do the authors denote Q-functions as if they are probability distributions? For instance, in Equation 1, the notation used is $Q(a_{1:K}^t|s^t)$. Wouldn’t it be clearer to write this as $Q(s^t, a_{1:K}^t)$? If there is a specific reason for adopting the conditional notation, an explanation would be helpful.\n\n* Reward signal based on relative improvement: as noted in the weaknesses, why did the authors choose to use relative improvement in objective values, rather than, e.g., an absolute value, as the reward signal? This approach assumes access to an optimization oracle. Have the authors tried alternative reward types? Do they anticipate that the performance of Q-Mamba might degrade if optimal objective values are not incorporated?\n\n* Impact of action dimension ordering: also noted in the weaknesses, the paper would benefit from a discussion on the impact of action dimension ordering on performance. How sensitive is Q-Mamba to this ordering, and do the authors have any insights on how different orderings might affect outcomes?\n\n* CQL regularization: in CQL, the values of OOD actions are minimized rather than regularized to zero. Is there a reason why the authors adopt a zero-regularization scheme in Equation 8? Additionally, if my understanding is correct, the relevant discussion in lines 358-362 should be updated to show how this regularization term differs from what's in CQL.\n\n* Exploration capability: why might Q-Mamba exhibit weaker exploration capabilities than other online baselines? Given that Q-Mamba underperforms its offline variant, it would be interesting to understand what limitations might be affecting its exploration during online training.\n\n* In Table 1, the confidence intervals of RLPSO and Q-Mamba largely overlap. However, the authors state that \"Q-Mamba significantly outperforms the online baselines RLPSO, LDP, and GLEET, which control...\". This statement could be misleading given the overlapping confidence intervals and would benefit from being rephrased.\n\n* Details of experiments in Section 5.3: can the authors provide a detailed description of the experiments conducted in Section 5.3 and consider adding this information to the appendix?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The paper offers a novel synthesis of several established techniques: it combines the decomposition of action dimensions in the Q-function to increase efficiency, employs the Mamba architecture for improved handling of long sequences, models meta-black-box optimization as a sequential decision-making problem addressed through RL, and utilizes offline reinforcement learning to further enhance efficiency.\n\n- The OOD generalization experiments in Section 5.3 demonstrate remarkable performance by Q-Mamba. Notably, the model, meta-trained on synthetic problems, effectively generalizes to more complex MuJoCo continuous control tasks.\n\n- The paper's presentation is clear and well-motivated, with empirical evaluations that convincingly support the authors’ claims."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The concrete problem setup and formal definition of MetaBBO appear only in Section 4.1, making it challenging for readers to fully comprehend the framework in earlier sections. Introducing the MetaBBO problem definition earlier would facilitate a quicker understanding of the framework's main contributions.\n\n- Q-Mamba employs a decomposed Q-function, where each action dimension’s value is estimated auto-regressively. While this approach is effective for handling large joint-action spaces, the paper does not analyze how the ordering of action dimensions might affect the performance. A systematic evaluation of the impact of action dimension ordering would add valuable insights to the empirical results.\n\n- Q-Mamba uses the relative improvement in objective values as its reward, which assumes knowledge of the optimal objective value --- a condition that may not always be practical in real-world applications. The paper would benefit from discussing alternative reward shaping methods and their potential impact on learning outcomes.\n\n- In Section 5.2, the online variant of Q-Mamba underperforms the offline version, which the authors attribute to the advantage of a diverse offline dataset collected from other online methods. However, this result suggests that Q-Mamba may lack effective exploration capabilities during online training, an aspect that could be further analyzed or addressed."}},"nonreaders":[],"tmdate":1731427533233,"tcdate":1730686622286,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2058/Reviewer_VHiT"],"signatures":["ICLR.cc/2025/Conference/Submission2058/Reviewer_VHiT"],"forum":"cSgEW7EZ9h","number":3,"license":"CC BY 4.0","cdate":1730686622286,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2058/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427533233,"domain":"ICLR.cc/2025/Conference","replyto":"cSgEW7EZ9h","id":"dvCNgRx55Z","forumContent":{"TLDR":{"value":"A novel MetaBBO framework which achieves both effectiveness and efficiency for auto-configuring black-box optimizer."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Black-Box Optimization","Dynamic Algorithm Configuration","Learning to Optimize","Offline Reinforcement Learning","Mamba"]},"primary_area":{"value":"optimization"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent progress in Meta-Black-Box-Optimization (MetaBBO) has demonstrated that meta-training a neural network based meta-level control policy over an optimization task distribution could significantly enhance the optimization performance of the low-level black-box optimizers. However, achieving such performance enhancement requires effective policy optimization/search method to locate optimal control policy within a massive joint-action space. The online learning fashion of existing works further makes the efficiency of MetaBBO problematic. To address these technical challenges, we propose an offline learning framework in this paper, termed Q-Mamba. Concretely, our method uses a Mamba neural network architecture to meta-learn decomposed Q-functions for each configurable component in the low-level optimizer. By decomposing the Q-function of the configuration decisions of all components in an optimizer, we can apply effective sequence modelling to avoid searching the control policy in the massive joint-action space. Furthermore, by leveraging the long-sequence modelling advantage of Mamba and moderate offline trajectory samples, Q-Mamba can be efficiently trained through a synergy of offline Temporal-Difference update and Conservative Q-Learning regularization to achieve competitive performance against the online learning paradigms. Through extensive benchmarking, we observe that Q-Mamba achieves competitive or even superior optimization performance to prior online/offline learning baselines, while significantly improving the training efficiency of existing online learning baselines. Additional ablation studies show that each of the proposed key designs contributes to this good performance."},"_bibtex":{"value":"@misc{\nma2025metablackboxoptimization,\ntitle={Meta-Black-Box-Optimization through Offline Q-function Learning with Mamba Architecture},\nauthor={Zeyuan Ma and Hongshu Guo and Zhou Jiang and Zhiguang Cao and Yue-Jiao Gong},\nyear={2025},\nurl={https://openreview.net/forum?id=cSgEW7EZ9h}\n}"},"title":{"value":"Meta-Black-Box-Optimization through Offline Q-function Learning with Mamba Architecture"},"pdf":{"value":"/pdf/2f9052c209e77207cb87544eb40cb37b2a701261.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"ma|metablackboxoptimization_through_offline_qfunction_learning_with_mamba_architecture"},"authorids":{"value":["~Zeyuan_Ma1","~Hongshu_Guo1","~Zhou_Jiang2","~Zhiguang_Cao1","~Yue-Jiao_Gong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zeyuan Ma","Hongshu Guo","Zhou Jiang","Zhiguang Cao","Yue-Jiao Gong"]}},"version":2},{"content":{"summary":{"value":"This paper introduces sequence modeling layers called Test-Time Training (TTT) layers. The key contributions are:\n1. Introduction of a machine learning model (e.g., linear model or MLP) that is trained through self-supervised learning during inference/test time. This allows the model to learn and adapt to patterns in the input sequence on the fly.\n2. Two practical instantiations: TTT-Linear and TTT-MLP, which use a linear model and a two-layer MLP respectively as their hidden states. \n3. Empirical validation showing that:\n    * Both TTT variants match or exceed performance of Transformers and Mamba models across different model sizes (125M to 1.3B parameters)\n    * TTT-Linear achieves faster inference than Transformers at 8k context and matches Mamba's speed\nThe authors also introduce practical optimizations like mini-batch TTT and a dual form implementation that significantly improve wall-clock time performance, making TTT-Linear immediately practical for real applications."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"Please see weakness"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"The paper is tackles an interesting problem but lacks novelty and experimental rigour."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The major weakness of the paper is the missing novelty, as well as comparison to appropriate baselines, in my opinion.  \n\n* Novelty: Linear self-attention and its variants are clearly motivated already by learning at inference. This is discussed in e.g.  Schlag et al., 2021 (Linear Transformers Are Secretly Fast Weight Programmers) and citations within. Additionally, and building on Schlag, are at least 3 missing citations i.e. Yang et al. 2024 (Parallelizing Linear Transformers with the Delta Rule over Sequence Length) as well as von Oswald et al. 2022, (Transformers Learn In-Context by Gradient Descent) and von Oswald et al. 2023 (Uncovering mesa-optimization algorithms in Transformers) which also build on this framework. \n\t\n* Please provide a comprehensive analysis how TTT-linear differs from these layers. \n\n* Yang et al. here is therefore also one obvious baseline, which is missing. First, the parallelized delta net implementation has better throughput as well as perplexity scores than Mamba and is therefore a crucial and missing baseline here. Additionally, and probably  even closer baseline are results of gated linear self-attention layers e.g. Yang et al. 2023 (Gated Linear Attention Transformers with Hardware-Efficient Training), which seem to be an extension of TTT-linear, because of the gates. \n\t\n* These missing references also highlight that the following statements might be untrue:\n\t1) The “novel” class of layers is in my opinion a false statement and needs correction. \n\t2) Related literature, see references, is able to parallelize across time (gated) linear self-attention and the delta rule, showing that they are able, to an extent, to leverage matmul units on GPUs (and TPUs). What is the relationship to Mini-Batch TTT, is this identical? \n \n\t\n\n* Further things: \n\t\n\t- Figure 3: 1) Perplexity seems very low here on the Pile. 2) Perplexity should be lower for 8k than for 2k, especially for Transformers. \n\t- Missing evals: Please Incorporate evals such as Table 1 and 2 in  Yang et al. 2024 (Parallelizing Linear Transformers with the Delta Rule over Sequence Length)"}},"nonreaders":[],"tmdate":1733219677830,"tcdate":1730935289804,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5310/Reviewer_o5Ft"],"signatures":["ICLR.cc/2025/Conference/Submission5310/Reviewer_o5Ft"],"forum":"eifW0W0xgt","number":4,"license":"CC BY 4.0","cdate":1730935289804,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5310/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733219677830,"domain":"ICLR.cc/2025/Conference","replyto":"eifW0W0xgt","id":"QaCuin28Yk","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["test-time training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state. Inspired by prior work, we present a practical framework to instantiate sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Both TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer, they can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. With preliminary systems optimization, TTT-Linear is already faster than Transformer at 8k context and matches Mamba in wall-clock time. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research."},"_bibtex":{"value":"@misc{\nsun2025learning,\ntitle={Learning to (Learn at Test Time): {RNN}s with Expressive Hidden States},\nauthor={Yu Sun and Xinhao Li and Karan Dalal and Jiarui Xu and Arjun Vikram and Genghan Zhang and Yann Dubois and Xinlei Chen and Xiaolong Wang and Sanmi Koyejo and Tatsunori Hashimoto and Carlos Guestrin},\nyear={2025},\nurl={https://openreview.net/forum?id=eifW0W0xgt}\n}"},"title":{"value":"Learning to (Learn at Test Time): RNNs with Expressive Hidden States"},"pdf":{"value":"/pdf/be1b3f1cf37ec5812fb041e8bbdd1569d92a9992.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sun|learning_to_learn_at_test_time_rnns_with_expressive_hidden_states"},"authorids":{"value":["~Yu_Sun1","~Xinhao_Li3","~Karan_Dalal1","~Jiarui_Xu1","~Arjun_Vikram1","~Genghan_Zhang1","~Yann_Dubois1","~Xinlei_Chen1","~Xiaolong_Wang3","~Sanmi_Koyejo1","~Tatsunori_Hashimoto1","~Carlos_Guestrin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yu Sun","Xinhao Li","Karan Dalal","Jiarui Xu","Arjun Vikram","Genghan Zhang","Yann Dubois","Xinlei Chen","Xiaolong Wang","Sanmi Koyejo","Tatsunori Hashimoto","Carlos Guestrin"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"This paper questions the necessity of self-attention in long-term sequence forecasting and introduces MambaTS, which models global dependencies across time and variables by leveraging causal relationships through a single linear scan."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forcasting; State Space Model"]},"supplementary_material":{"value":"/attachment/dc191874f3a708459705ff4c9016637ffa6cb3d7.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"In recent years, Transformers have become the de-facto architecture for long-term sequence forecasting (LTSF), yet they face challenges associated with the self-attention mechanism, including quadratic complexity and permutation invariant bias. This raises an important question: \\emph{do we truly need the self-attention mechanism to establish long-range dependencies in LTSF?} Recognizing the significance of causal relationships in multivariate LTSF, we propose MambaTS, which leverages causal relationships to model global dependencies across time and variables through a single linear scan. However, causal graphs are often unknown. To address this, we introduce variable-aware scan along time (VAST), which dynamically discovers variable relationships during training and decodes the optimal variable scan order by solving the shortest path visiting all nodes problem during inference. MambaTS employs the latest Mamba model as its backbone. We suggest that the causal convolution in Mamba is unnecessary due to the presence of independent variables, leading to the development of the Temporal Mamba Block (TMB). To mitigate model overfitting, we further incorporate a dropout mechanism for selective parameters in TMB. Extensive experiments conducted on eight public datasets demonstrate that MambaTS achieves new state-of-the-art performance."},"_bibtex":{"value":"@misc{\ncai2025mambats,\ntitle={Mamba{TS}: Improved Selective State Space Models for Long-term Time Series Forecasting},\nauthor={Xiuding Cai and Yaoyao Zhu and Xueyao Wang and Yu Yao},\nyear={2025},\nurl={https://openreview.net/forum?id=vEtDApqkNR}\n}"},"title":{"value":"MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting"},"pdf":{"value":"/pdf/d5d8697d5fd93b05c6b17f60b0f8962287f5c5f2.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"cai|mambats_improved_selective_state_space_models_for_longterm_time_series_forecasting"},"authorids":{"value":["~Xiuding_Cai1","~Yaoyao_Zhu2","~Xueyao_Wang1","~Yu_Yao4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xiuding Cai","Yaoyao Zhu","Xueyao Wang","Yu Yao"]}},"tmdate":1738735811713,"tcdate":1727442843900,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10330/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10330/Authors"],"forum":"vEtDApqkNR","license":"CC BY 4.0","number":10330,"cdate":1727442843900,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission10330/-/Full_Submission","ICLR.cc/2025/Conference/Submission10330/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735811713,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"vEtDApqkNR","version":2},{"content":{"summary":{"value":"This paper proposes MambaSIC, a novel stereo image compression framework that leverages the Mamba-based Stereo Visual State Space Block (Stereo VSSB) to efficiently capture long-range inter-view correlations with linear complexity. It further introduces a bi-directional multi-reference entropy model that employs a checkerboard partition and inter-view priors for accurate probability estimation and fast entropy coding. Experiments on InStereo2K and Cityscapes show that MambaSIC achieves state-of-the-art rate–distortion performance."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"See weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The MambaSIC achieves both high compression efficiency and coding speed.\n2. The paper is well-organized with clear figures."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The disparity between the left and right views can be significantly large. Within the proposed Stereo Visual State Space Block (Stereo VSSB), the straightforward concatenation of features from both views may introduce substantial irrelevant or noisy information. This unaligned fusion could contaminate the original feature representations and degrade their discriminative capacity, ultimately reducing the overall effectiveness of inter-view redundancy modeling.\n2. The integration of the Mamba mechanism into stereo feature interaction is implemented in a rather direct and naive manner. The authors simply apply Mamba-based state-space modeling across two views without introducing any geometry-aware or adaptive fusion strategy. As a result, the conceptual novelty of the Stereo VSSB is limited, since it mainly reuses Mamba’s existing selective scan structure without substantial modification or theoretical adaptation to stereo correspondence.\n3. Can the author visualize the feature map or attention-map in the Stereo VSSB to make it clear that Mamba does capture meaningful inter-view dependencies?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916359155,"tcdate":1761486836304,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2750/Reviewer_V835"],"signatures":["ICLR.cc/2026/Conference/Submission2750/Reviewer_V835"],"forum":"0dHrYUd17W","number":1,"license":"CC BY 4.0","cdate":1761486836304,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2750/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916359155,"domain":"ICLR.cc/2026/Conference","replyto":"0dHrYUd17W","id":"u3erBJGIme","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Stereo image compression; visual state space model"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Stereo image compression (SIC) has become increasingly vital with its applications surging in fields such as 3D reconstruction and autonomous navigation. Previous methods leverage cross-attention to model inter-view redundancy and employ autoregressive entropy models to predict probability distributions, achieving impressive rate-distortion performance. However, they suffer from slow coding speed due to the quadratic complexity of cross-attention mechanisms and the spatial autoregressive iterations of the entropy models. To address these limitations, we propose MambaSIC, which introduces two key innovations.  First, we propose a Mamba-based stereo visual state space block (stereo VSSB) that leverages its linear complexity and long-range modeling capabilities to more rapidly and efficiently capture redundancy information between the two views. Second, to accelerate the compression process and enhance the accuracy of probability distribution estimation, we introduce a bi-directional multi-reference entropy model that utilizes a checkerboard partitioning strategy and the stereo VSSB to get rich inter-view priors. Experimental results demonstrate that our MambaSIC outperforms the state-of-the-art methods in both rate-distortion performance and coding efficiency. Moreover, it achieves the smallest inter-view PSNR discrepancy, resulting in more balanced reconstruction quality."},"_bibtex":{"value":"@misc{\nqin2025mambasic,\ntitle={Mamba{SIC}: Mamba-based Stereo Image Compression with Bi-directional Multi-reference Entropy Model},\nauthor={Shiyu Qin and Xinjie Zhang and Zhening Liu and Jinpeng Wang and Bin Chen and Jiawei Li and Yifan Ren and Shu-Tao Xia and Jun Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=0dHrYUd17W}\n}"},"title":{"value":"MambaSIC: Mamba-based Stereo Image Compression with Bi-directional Multi-reference Entropy Model"},"pdf":{"value":"/pdf/9496a712bd6ced6d32edf6793dca83982cb63fa4.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"qin|mambasic_mambabased_stereo_image_compression_with_bidirectional_multireference_entropy_model"},"authorids":{"value":["~Shiyu_Qin1","~Xinjie_Zhang2","~Zhening_Liu1","~Jinpeng_Wang3","~Bin_Chen4","~Jiawei_Li4","~Yifan_Ren4","~Shu-Tao_Xia1","~Jun_Zhang25"]},"authors":{"value":["Shiyu Qin","Xinjie Zhang","Zhening Liu","Jinpeng Wang","Bin Chen","Jiawei Li","Yifan Ren","Shu-Tao Xia","Jun Zhang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2411.15469v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"cheng|mambacl_optimizing_selective_state_space_model_in_null_space_for_continual_learning"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:De_Cheng:","https://dblp.org/search/pid/api?q=author:Yue_Lu:","https://dblp.org/search/pid/api?q=author:Lingfeng_He:","~Shizhou_Zhang3","https://dblp.org/search/pid/api?q=author:Xi_Yang_0011:","https://dblp.org/search/pid/api?q=author:Nannan_Wang_0001:","https://dblp.org/search/pid/api?q=author:Xinbo_Gao_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2411.15469"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2411-15469,\n  publtype={informal},\n  author={De Cheng and Yue Lu and Lingfeng He and Shizhou Zhang and Xi Yang and Nannan Wang and Xinbo Gao},\n  title={Mamba-CL: Optimizing Selective State Space Model in Null Space for Continual Learning},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2411.15469},\n  url={https://doi.org/10.48550/arXiv.2411.15469}\n}\n"},"abstract":{"value":"Continual Learning (CL) aims to equip AI models with the ability to learn a sequence of tasks over time, without forgetting previously learned knowledge. Recently, State Space Models (SSMs), particularly the Mamba model, have achieved notable success in computer vision. Building on the strengths of SSMs, this study explores leveraging the Mamba model for CL. Therefore, we introduce Mamba-CL, a framework that continuously fine-tunes the core SSMs of the large-scale Mamba foundation model by updating parameters orthogonal to the feature subspace of previous tasks. This approach theoretically guarantees the consistency objective aiming to preserves consistent output for each SSM module across both previous and current tasks, so as to overcome catastrophic forgetting issue. Specifically, we achieve this goal by deducing the overall consistency constraints on four key time-invariant parameters in the Mamba model, streamlining its recurrent state-space structure and non-linear discretization process in SSM. In practice, we apply the null-space projection to efficiently implement the orthogonality within Mamba model. Extensive experiments on four class-incremental benchmarks demonstrate the effectiveness of Mamba-CL for anti-forgetting, achieving superior performances to state-of-the-art methods. Code is available in the supplementary materials."},"title":{"value":"Mamba-CL: Optimizing Selective State Space Model in Null Space for Continual Learning"},"authors":{"value":["De Cheng","Yue Lu","Lingfeng He","Shizhou Zhang","Xi Yang","Nannan Wang","Xinbo Gao"]}},"tmdate":1737786956206,"pdate":1704067200000,"tcdate":1737786952011,"writers":["~"],"signatures":["~Shizhou_Zhang3"],"forum":"HTRUsl7Pld","license":"CC BY-SA 4.0","number":272107,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1737786956206,"domain":"DBLP.org","id":"HTRUsl7Pld","version":2},{"content":{"summary":{"value":"This paper proposes a new generic vision backbone called Vision Filter (ViF) , built upon a novel Fourier Neural Filter (FNF) module. The authors argue that while the Fourier Neural Operator (FNO) is promising for its global modeling capabilities and quasi-linear complexity , it suffers from an \"over-smoothing\" effect and a \"bandwidth bottleneck,\" making it difficult to capture local, high-frequency patterns. To address this, the FNF module introduces two key components: 1) Adaptive Modulation (AM) in the frequency domain to enhance sensitivity to high-frequency components , and 2) Selective Activation (SA) to balance local time-domain and global frequency-domain information flow. This design creates an input-dependent integral kernel. Extensive experiments on image classification (ImageNet-1K) , object detection (COCO) , and semantic segmentation (ADE20K) demonstrate that ViF outperforms existing SOTA Transformer- and Mamba-based backbones."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"In addition to the above weaknesses, I also have several questions and suggestions for the authors:\n1. Could ViF be integrated into existing architectures (e.g., ConvNeXt or BEiT) as a modular replacement, and what challenges might arise?\n2. Does the proposed FNF introduce any implicit bias toward certain frequency bands, and how is this mitigated?\n3. Are there plans to extend FNF to video or multimodal vision tasks?\n4. Could the authors provide additional visualization results illustrating spectral activations or learned frequency responses?"},"rating":{"value":6},"details_of_ethics_concerns":{"value":"No ethics review needed."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The authors introduce an adaptive kernel that effectively couples time- and frequency-domain information, providing theoretical novelty.\n2. The paper demonstrates strong empirical performance across multiple visual benchmarks and model scales.\n3. The authors conduct comprehensive ablation studies that clearly quantify the contribution of each component (AM, SA, LC).\n4. The proposed ViF achieves a favorable balance between model complexity and performance, offering quasi-linear computational efficiency."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper lacks experiments exploring the scalability of ViF on larger datasets such as ImageNet-22K or LAION.\n2. The authors do not evaluate ViF under self-supervised or foundation model pre-training settings, which limits its generalization claims. I believe this is crucial for vision backbone models.\n3. The paper lacks detailed frequency-domain visualizations (e.g., spectral activation maps) to empirically support its frequency modeling claims.\n4. The improvements over strong Mamba variants on dense prediction tasks are relatively marginal compared to the architectural complexity.\n5. The authors have not released any implementation code at submission time, reducing reproducibility."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918428016,"tcdate":1761981936459,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6047/Reviewer_kGtH"],"signatures":["ICLR.cc/2026/Conference/Submission6047/Reviewer_kGtH"],"forum":"Q5SSA6IonA","number":4,"license":"CC BY 4.0","cdate":1761981936459,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6047/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918428016,"domain":"ICLR.cc/2026/Conference","replyto":"Q5SSA6IonA","id":"ixqhBSQtxs","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["representation learning"]},"supplementary_material":{"value":"/attachment/46b601e6cab658387a955095350002cd51924698.pdf"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Effective information extraction has long been a central challenge in Computer Vision (CV). Transformer- and Mamba-based backbones have significantly advanced this field by providing powerful long-range modeling capability, even though they are initially developed for Natural Language Processing (NLP). Recent progress has highlighted the potential of Fourier Neural Operator (FNO), which, with its favorable quasi-linear complexity and strong global modeling capacity, offers a promising alternative for visual representation learning. However, FNO exhibits a fundamental limitation in capturing local high-frequency patterns due to the over-smoothing effect and bandwidth bottleneck. To address this limitation, we propose Vision Filter (ViF), as a generic backbone for CV, consisting of two complementary components: adaptive modulation for enhancing sensitivity to high-frequency component in the frequency domain, and selective activation for balancing local time-domain and global frequency-domain information flow. Extensive experiments reveal that ViF consistently outperforms prominent variants of Transformer- and Mamba-based backbones across diverse visual tasks, including image classification, object detection, and semantic segmentation. ViF demonstrates lower computational complexity than Transformer-based models and better structural modeling than Mamba-based models, which suffer from spatial disruption due to their directional scanning mechanism. The joint time- and frequency-domain mechanism introduced in ViF may establish a promising paradigm for designing effective visual representation learning, bridging local high-frequency information with global low-frequency information."},"_bibtex":{"value":"@misc{\nxu2026fourier,\ntitle={Fourier Neural Filter as Generic Vision Backbone},\nauthor={Chenheng Xu and Yuanhao Ban and Dan Wu and Cho-Jui Hsieh and Ying Nian Wu and Yixin Zhu},\nyear={2026},\nurl={https://openreview.net/forum?id=Q5SSA6IonA}\n}"},"title":{"value":"Fourier Neural Filter as Generic Vision Backbone"},"pdf":{"value":"/pdf/357d80af090cafb82d30a7f6570ca1a0576217e3.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xu|fourier_neural_filter_as_generic_vision_backbone"},"authorids":{"value":["~Chenheng_Xu1","~Yuanhao_Ban1","~Dan_Wu7","~Cho-Jui_Hsieh1","~Ying_Nian_Wu1","~Yixin_Zhu1"]},"authors":{"value":["Chenheng Xu","Yuanhao Ban","Dan Wu","Cho-Jui Hsieh","Ying Nian Wu","Yixin Zhu"]}},"version":2},{"content":{"summary":{"value":"This paper introduces UIT-Pred, a unified architecture for trajectory prediction in autonomous driving that robustly handles variable-length input histories and missing data. The method combines a time-aware input representation, a Bidirectional Time Decay Mamba (BTD-Mamba) module (extending state space models with bidirectional and decay-aware processing), and a multimodal prediction module using integrated state encoding and cross-attention fusion. The approach is validated empirically on the Argoverse 1 and Argoverse 2 benchmarks, demonstrating state-of-the-art performance under several observation regimes and scenarios. Extensive ablations and qualitative results further support the claims."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. How robust is UIT-Pred to adversarial missing patterns? Any chance for synthetic stress-testing to be added or analyzed?\n2. In the fusion module, are positional encodings used within the cross-attention layers (cf. Equations 14-20)? If so, how are these adapted for variable-length or irregularly timed data? \n3. Why did the authors choose to weight all loss terms equally in $L_\\text{total}$? (cf. equation 23)"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"+ The paper tackles an important and under-addressed real-world challenge: predicting agent trajectories under intermittent (i.e., variable-length and missing) observation sequences, directly motivated by realistic issues such as sensor occlusion and communication drops in autonomous driving.\n+ The method brings together several principled ideas—temporal normalization, bidirectional SSMs with time decay, and unified fusion—to handle variable and incomplete data without reliance on input masking, explicit validity masks, or duplicative augmentation. The proposed Bidirectional Time Decay Mamba module is theoretically motivated and technically well-integrated, extending recent progress in SSMs in sequence modeling to the irregular sampling domain."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. While the decay process in BTD-Mamba is heuristically motivated (see Equations 4–5), the formulation and parameterization of the decay via an MLP, as well as the use of a negative exponential, seem ad-hoc. There is no theoretical analysis on how this decay mechanism affects information retention or gradient propagation through highly irregular gaps. A more principled or formal treatment would be valuable.\n2. Ambiguity in Loss Weighting and Training Objective: The main loss function (Equation 23 and detailed appendix losses) weights all terms equally without analysis or justification. No tuning or sensitivity study is reported regarding these hyperparameters. In multi-loss systems, uniform weighting is rarely optimal. This could materially affect empirical results (see Equations 24–33, Table 8 in the appendix for auxiliary losses – but no broader tuning study).\n3. Certain key steps gloss over important mathematical details: the cross-attention decoders used extensively for graph, lane, and multimodal fusion are described abstractly ($\\mathcal{D}(·)$ in Equations 14–20) but lack explicit architectural detail (are positional encodings used? What are the dimensions of queries/keys/values in multi-modal context?).\n4.  There are a few spots where notational clarity is lacking, particularly in not distinguishing between predicted, reconstructed, and observed trajectories within tensors in the main methodology (Equations 15–17). Indexing switches between $j$, $t$, and $l$ within equations can be confusing."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926859528,"tcdate":1761961030182,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16838/Reviewer_UY3o"],"signatures":["ICLR.cc/2026/Conference/Submission16838/Reviewer_UY3o"],"forum":"ThZeZdRxna","number":3,"license":"CC BY 4.0","cdate":1761961030182,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16838/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926859528,"domain":"ICLR.cc/2026/Conference","replyto":"ThZeZdRxna","id":"dH9uuBARUZ","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Trajectory prediction","Autonomous Driving","Attention","Mamba","Intermittent Data"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Trajectory prediction is a fundamental component of autonomous driving, requiring models that can handle intermittent observation patterns such as variable-length histories and missing data. Existing state-of-the-art methods, however, often assume fixed-length trajectories and complete input, which challenges their applicability in real-world scenarios where sensor occlusions, communication delays, and temporal sparsity are common. Moreover, conventional approaches typically address tasks such as trajectory prediction, variable-length modeling, or missing data handling in isolation, making them less effective in multi-task settings that naturally arise in practice. To address these challenges, we propose Universal Intermittent Trajectory Predictor (UIT-Pred) that processes inputs with the time index features, which capture temporal variations to effectively adapt to diverse input patterns within the domain. Particularly, We extend recent State Space Models (SSMs) by introducing the Bidirectional Time Decay Mamba (BTD-Mamba), designed to capture dependencies both forward and backward along the sequence. By integrating a decay process, BTD-Mamba effectively analyzes trajectories while maintaining relationships under intermittent observation. Furthermore, the proposed prediction module employs state encoding to capture the underlying motion patterns in the input data and models a multimodal trajectory distribution to account for uncertainty in future predictions. These components are fused through a unified fusion module, enabling the model to jointly reason over observed dynamics and potential future behaviors. Extensive experiments on Argoverse 1 and Argoverse 2 datasets validate the effectiveness of the proposed model. By simultaneously handling prediction, variable-length observations, and missing inputs within a universal architecture, the framework proposes to meet the challenges of real-world autonomous driving systems."},"_bibtex":{"value":"@misc{\nbharilya2026uitpred,\ntitle={{UIT}-Pred: Universal Intermittent Trajectory Predictor for Autonomous Driving},\nauthor={Vibha Bharilya and Neetesh Kumar},\nyear={2026},\nurl={https://openreview.net/forum?id=ThZeZdRxna}\n}"},"title":{"value":"UIT-Pred: Universal Intermittent Trajectory Predictor for Autonomous Driving"},"pdf":{"value":"/pdf/09ddf9106e39f3ead7441bc862c70e71d213fcad.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"bharilya|uitpred_universal_intermittent_trajectory_predictor_for_autonomous_driving"},"authorids":{"value":["~Vibha_Bharilya1","~Neetesh_Kumar1"]},"authors":{"value":["Vibha Bharilya","Neetesh Kumar"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2503.05447v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"sun|linearmoe_linear_sequence_modeling_meets_mixtureofexperts"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Weigao_Sun:","~Disen_Lan1","https://dblp.org/search/pid/api?q=author:Tong_Zhu_0002:","https://dblp.org/search/pid/api?q=author:Xiaoye_Qu:","https://dblp.org/search/pid/api?q=author:Yu_Cheng_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2503.05447"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-05447,\n  publtype={informal},\n  author={Weigao Sun and Disen Lan and Tong Zhu and Xiaoye Qu and Yu Cheng},\n  title={Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts},\n  year={2025},\n  month={March},\n  cdate={1740787200000},\n  journal={CoRR},\n  volume={abs/2503.05447},\n  url={https://doi.org/10.48550/arXiv.2503.05447}\n}\n"},"abstract":{"value":"Linear Sequence Modeling (LSM) like linear attention, state space models and linear RNNs, and Mixture-of-Experts (MoE) have recently emerged as significant architectural improvements. In this paper, we introduce Linear-MoE, a production-level system for modeling and training large-scale models that integrate LSM with MoE. Linear-MoE leverages the advantages of both LSM modules for linear-complexity sequence modeling and MoE layers for sparsely activation, aiming to offer high performance with efficient training. The Linear-MoE system comprises: 1) Modeling subsystem, which provides a unified framework supporting all instances of LSM. and 2) Training subsystem, which facilitates efficient training by incorporating various advanced parallelism technologies, particularly Sequence Parallelism designed for Linear-MoE models. Additionally, we explore hybrid models that combine Linear-MoE layers with standard Transformer-MoE layers with its Sequence Parallelism to further enhance model flexibility and performance. Evaluations on two model series, A0.3B-2B and A1B-7B, demonstrate Linear-MoE achieves efficiency gains while maintaining competitive performance on various benchmarks, showcasing its potential as a next-generation foundational model architecture. Code: https://github.com/OpenSparseLLMs/Linear-MoE."},"title":{"value":"Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts"},"authors":{"value":["Weigao Sun","Disen Lan","Tong Zhu","Xiaoye Qu","Yu Cheng"]}},"tmdate":1747644447516,"pdate":1735689600000,"tcdate":1747644441845,"writers":["~"],"signatures":["~Disen_Lan1"],"forum":"a0O9L7KM39","license":"CC BY-SA 4.0","number":525557,"cdate":1740787200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747644447516,"domain":"DBLP.org","id":"a0O9L7KM39","version":2},{"content":{"summary":{"value":"The paper proposes MambaExtend a computationally efficient way to improve Mamba's ability handle longer contexts. \nThe method is based on the observation that when presented with longer sequences than during training, the time discretization parameter of Mamba assumes values which the model cannot handle. \nAs a result, the authors propose to use a similar approach to position interpolation in transformers and scale the discretization parameter per layer independently to a range which Mamba has seen during training. \nThe authors propose two methods to estimate the scaling factor for the discretization parameter of each layer a gradient-based and a zero-order optimization method using finite-differences to estimate the scaling factor and ensure that the scaling factor can only be positive.\nThe authors demonstrate improved performance on pass-key retrieval tasks and show significantly lower perplexity at high context lengths."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- Please see the main weakness. In a sense, I think it would be important to understand whether the performance degradation you observe for Mamba is an artifact of the training using a transformer pipeline where the data is shuffled into non-consecutive chunks compared to an RNN training pipeline where the data isn't chunked and the last hidden state of the previous batch is the first hidden state of the current batch."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- The proposed method allows Mamba to be used with much higher context lengths than seen during training. Given that long-context handling is the computational advantage of Mamba models, it's great that the proposed method allows to alleviate performance degradation.\n- The results on pass-key retrieval are convincing. \n- The method is well-motivated and easy to understand.\n- The authors propose two methods: a more accurate but also more memory heavy gradient based method and a less accurate but less memory intensive zero-order method which makes sense to me as they are applicable depending on the model size and hardware constraints."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- While the proposed method is effective at making an existing checkpoint of Mamba more capable of handling long contexts, I feel like it treats more a symptom rather than a cause. As shown in Wang et al. (2024) (Figure 4) Mamba can handle long context effectively if trained using truncated backpropagation through time on consecutive chunks of the data (rather than shuffling). Crucially, the initial state of Mamba is changed from zero to the last hidden state in the sequence. As a result Mamba doesn't need to start from scratch at token position i every time. \nThis is challenging to implement in Mamba 1 as the initial state can't be set but is an exposed option for Mamba 2. I think it would be interesting to understand whether the observed differences in the discretization parameter for longer context still occur in models trained in this way or whether it makes the proposed method less effective. This could be checked experimentally using an existing small checkpoint (340m) and then fine-tuning on a small dataset of around 1B token using TBTT.\n- The code for reproducibility was not provided.\n\n\n\n\nReferences:\n- Wang, S. (2024). LongSSM: On the Length Extension of State-space Models in Language Modelling. arXiv preprint arXiv:2406.02080."}},"nonreaders":[],"tmdate":1732531417045,"tcdate":1730196047557,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9212/Reviewer_UZhN"],"signatures":["ICLR.cc/2025/Conference/Submission9212/Reviewer_UZhN"],"forum":"LgzRo1RpLS","number":1,"license":"CC BY 4.0","cdate":1730196047557,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9212/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732531417045,"domain":"ICLR.cc/2025/Conference","replyto":"LgzRo1RpLS","id":"0psmGzQogN","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Long Context Generalization","Discretization Step","SSM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The inherent quadratic complexity of the attention mechanism in transformer models has driven the research community to explore alternative architectures with sub-quadratic complexity, such as state-space models. Mamba has established itself as a leading model within this emerging paradigm, achieving state-of-the-art results in various language modeling benchmarks. However, despite its impressive performance, Mamba's effectiveness is limited by its pre-training context length, resulting in a pronounced degradation when the model is tasked with handling longer contexts. Our investigation reveals that Mamba's inability to generalize effectively to long contexts is primarily due to the out-of-distribution (OOD) discretization steps. To address this critical limitation, we introduce _**MambaExtend**_, a novel framework designed to significantly enhance the context extension capabilities of Mamba. Specifically, MambaExtend leverages a _**training-free**_ approach to calibrate _only_ the scaling factors of discretization modules for different layers. We demonstrate both gradient-based and gradient-free zeroth-order optimization to learn the optimal scaling factors for each Mamba layer, requiring orders of magnitude fewer updates as opposed to the parameter fine-tuning-based alternatives. \nUsing this approach, we achieve a training-free context extension of up to 32x, expanding the context from 2k to 64k tokens with minimal increases in perplexity. In contrast to existing fine-tuning methods, MambaExtend selectively calibrates the scaling factors, requiring up to $\\mathbf{5.42 * 10^6} \\times$ fewer parameter updates and incurring up to $\\mathbf{3.87} \\times$ lower peak memory usage, while delivering comparable or superior long-context performance across multiple tasks. Codes and checkpoints are available here$^1$."},"_bibtex":{"value":"@inproceedings{\nazizi2025mambaextend,\ntitle={MambaExtend: A Training-Free Approach to Improve Long Context Extension of Mamba},\nauthor={Seyedarmin Azizi and Souvik Kundu and Mohammad Erfan Sadeghi and Massoud Pedram},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=LgzRo1RpLS}\n}"},"title":{"value":"MambaExtend: A Training-Free Approach to Improve Long Context Extension of Mamba"},"pdf":{"value":"/pdf/f1f70d718d90155c62d9c0fae5c8af6c4eb7400a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"azizi|mambaextend_a_trainingfree_approach_to_improve_long_context_extension_of_mamba"},"authorids":{"value":["~Seyedarmin_Azizi1","~Souvik_Kundu2","~Mohammad_Erfan_Sadeghi1","~Massoud_Pedram1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seyedarmin Azizi","Souvik Kundu","Mohammad Erfan Sadeghi","Massoud Pedram"]}},"version":2},{"content":{"venue":{"value":"J. Big Data 2024"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1186/s40537-024-01001-9.pdf"},"venueid":{"value":"dblp.org/journals/JBD/2024"},"paperhash":{"value":"li|cmmamba_channel_mixing_mamba_for_time_series_forecasting"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Qiang_Li:","~Jiwei_Qin1","https://dblp.org/search/pid/api?q=author:Daishun_Cui:","https://dblp.org/search/pid/api?q=author:Dezhi_Sun:","https://dblp.org/search/pid/api?q=author:Dacheng_Wang:"]},"html":{"value":"https://doi.org/10.1186/s40537-024-01001-9"},"_bibtex":{"value":"@article{DBLP:journals/jbd/LiQCSW24,\n  author={Qiang Li and Jiwei Qin and Daishun Cui and Dezhi Sun and Dacheng Wang},\n  title={CMMamba: channel mixing Mamba for time series forecasting},\n  year={2024},\n  month={December},\n  cdate={1733011200000},\n  journal={J. Big Data},\n  volume={11},\n  number={1},\n  pages={153},\n  url={https://doi.org/10.1186/s40537-024-01001-9}\n}\n"},"abstract":{"value":"Transformer-based methods have achieved excellent results in the field of time series forecasting due to their powerful ability to model sequences and capture their long-term dependencies. However, Transformer-based methods are still computationally inefficient and have high costs when modeling long sequences. In addition, previous work ignored the destruction of the original in-channel features by too many mixed features when performing channel mixing. To address the above issues, we introduce Mamba into the field of time series forecasting. The selective state-space model Mamba models long sequences more efficiently by expanding linearly according to the sequence length. Based on this, we propose Channel Mixing Mamba (CMMamba), which uses bidirectional Mamba to model sequences and uses a channel mixing mechanism to select appropriate mixed features to enhance the feature representation within the original channel. By introducing Mamba, we achieve efficient modeling of sequences. We also propose Deep Convolutional Structure (DCS) to learn cross-channel dependencies and temporal order information. Extensive experimental results on six real-world public datasets demonstrate the effectiveness of CMMamba."},"title":{"value":"CMMamba: channel mixing Mamba for time series forecasting"},"authors":{"value":["Qiang Li","Jiwei Qin","Daishun Cui","Dezhi Sun","Dacheng Wang"]}},"tmdate":1768482848907,"pdate":1704067200000,"externalIds":["dblp:journals/jbd/LiQCSW24"],"tcdate":1768482830986,"writers":["~"],"signatures":["~Qin_JiWei1"],"forum":"xKm78pz2JX","license":"CC BY-SA 4.0","number":764104,"cdate":1733011200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1768482848907,"domain":"DBLP.org","id":"xKm78pz2JX","version":2},{"content":{"summary":{"value":"This paper studies the in-context learning abilities of the Mamba (S6) architecture, as a cheaper alternative to transformers (specifically, their quadratic attention cost in the sequence length). To study this problem, the authors consider finite-state k-order Markov Chains, and evaluate Mamba and Transformers in their ability to output the transition probabilities of the optimal Laplacian smoothing estimator in-context. Empirically, the findings include Mamba fitting the Laplacian smoothing estimator with only 1 layer, and convolution being the main ingredient in this ability. Furthermore, the authors show that a simplified Mamba variant can theoretically encode all the bits needed for storing the Laplacian smoothing operator with specific choices of its parameters."},"soundness":{"value":3},"confidence":{"value":2},"questions":{"value":"- Did the authors check that 2 or more layers Mamba achieves the same result as 1-layer Mamba in the considered experiments? Or in other words, what is the impact on performance if the model is over-parameterized with respect to the bits needed to encode the Laplacian smoothing estimator?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":4},"strengths":{"value":"- The paper is clearly written and well motivated.\n- The methodology considered in this study is sound and made explicit: Building on the empirical observation that the convolution is the key ingredient in the ICL abilities of Mamba, the authors construct a minimal architecture that allows a theoretical analysis all while keeping key similarities with the original architecture. This allows the derivation of sound theoretical results characterizing Mamba's ability to learn in-context, as function of the choices in parameters and the hidden dimension.\n- Given equivalences between language modeling, and k-order Markov Chains (e.g. n-gram models which previously had been the standard in language modeling), I believe that the insights included in this study are an important step towards building more efficient architectures in general, and in Language modelling in particular."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Theorem 1 is insightful in proving the in-context ability of Mamba. However, in my understanding it only tackles the model misspecification problem, in the sense that it proves that the optimal solution can be represented by the model (zero approximation error). I think it would be interesting to discuss the intersections between the theoretically chosen parameters that achieve the result in Theorem 1, and those actually obtained by optimizing Eq 3 on a given training set (with or without distribution set) starting from standard random initializations of these parameters. This is partially done, with the observation that $a_t \\approx 1$ at convergence, but can also be extended to other parameters (for instance, the count-independent term in Eq 5 that need to sum up to $\\beta \\mathbf{1}$).\n- The natural language modeling experiments only validate the key role of the convolution in Mamba. I think that even the theoretical results (especially Th 2) can also be validated through experiments. For instance, by generating language from an n-gram model, one can show that a given hidden dimension size of Mamba is strictly needed in order to be able to perform in-context learning tasks in the generated language corpus. Of course this comes with challenges related to the exponential dependence on the Markov Chain order, but having a discussion around this would be a nice addition to the paper."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927221392,"tcdate":1762100534732,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17277/Reviewer_5agm"],"signatures":["ICLR.cc/2026/Conference/Submission17277/Reviewer_5agm"],"forum":"kmK3WSCOCT","number":3,"license":"CC BY 4.0","cdate":1762100534732,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17277/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927221392,"domain":"ICLR.cc/2026/Conference","replyto":"kmK3WSCOCT","id":"1L3Uw2vGeg","forumContent":{"TLDR":{"value":"We uncover an interesting phenomenon where a single-layer Mamba represents the Bayes optimal Laplacian smoothing estimator when trained on Markov chains and we demonstrate it theoretically and empirically."},"venue":{"value":"ICLR 2026 Oral"},"keywords":{"value":["State-space models","Markov chains","In-context learning","Laplacian smoothing"]},"supplementary_material":{"value":"/attachment/7481132bbc16e969531b0db2b1a4e3d8dbafeefc.zip"},"primary_area":{"value":"interpretability and explainable AI"},"abstract":{"value":"While transformer-based language models have driven the AI revolution thus far, their computational complexity has spurred growing interest in viable alternatives, such as structured state space sequence models (SSMs) and Selective SSMs. Among these, Mamba (S6) and its variant Mamba-2 have shown remarkable inference speed-ups over transformers while achieving comparable or superior performance on complex language modeling tasks. However, despite these architectural innovations and empirical successes, the fundamental learning capabilities of Mamba remain poorly understood. In this paper, we address this gap by studying in-context learning (ICL) on Markov chains and uncovering an interesting phenomenon: even a single-layer Mamba efficiently learns the in-context Laplacian smoothing estimator, which is both Bayes and minimax optimal. To explain this, we theoretically characterize the representation capacity of Mamba and reveal the fundamental role of convolution in enabling it to represent the optimal Laplacian smoothing. These theoretical insights align strongly with empirical results and, to the best of our knowledge, represent the first formal connection between Mamba and optimal statistical estimators. Finally, we outline promising research directions inspired by these findings."},"_bibtex":{"value":"@inproceedings{\nbondaschi2026from,\ntitle={From Markov to Laplace: How Mamba In-Context Learns Markov Chains},\nauthor={Marco Bondaschi and Nived Rajaraman and Xiuying Wei and Razvan Pascanu and Caglar Gulcehre and Michael Gastpar and Ashok Vardhan Makkuva},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=kmK3WSCOCT}\n}"},"title":{"value":"From Markov to Laplace: How Mamba In-Context Learns Markov Chains"},"pdf":{"value":"/pdf/7c5974f3688d2ba6f4817747b376897b745b3197.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"bondaschi|from_markov_to_laplace_how_mamba_incontext_learns_markov_chains"},"authorids":{"value":["~Marco_Bondaschi1","~Nived_Rajaraman1","~Xiuying_Wei1","~Razvan_Pascanu1","~Caglar_Gulcehre1","~Michael_Gastpar1","~Ashok_Vardhan_Makkuva1"]},"authors":{"value":["Marco Bondaschi","Nived Rajaraman","Xiuying Wei","Razvan Pascanu","Caglar Gulcehre","Michael Gastpar","Ashok Vardhan Makkuva"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a Mamba-based framework called RoboMamba that utilizes multimodal state space model for robotic reasoning and manipulation. It addresses the limitations of existing MLLMs in complex reasoning and high computational costs. RoboMamba integrates a vision encoder with the Mamba model to align visual data with language embedding, enabling robust visual reasoning and action capabilities. The model uses a simple lightweight policy head for action pose prediction, requiring minimal fine-tuning. Experiments show RoboMamba excels in reasoning and pose prediction tasks, achieving faster inference speeds and state-of-the-art performance in both simulated and real-world environments."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please refer to the weaknesses of the paper."},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Using architecture with linear computational complexity like Mamba for improving efficiency fits quite well with the robotic reasoning and manipulation. \n2. The design of training strategy is simple and high-efficient, verifying the transferability of the proposed framework.\n3. Experimental results on public benchmarks as well as simulated and real-world environments demonstrate the effectiveness and efficiency of the proposed method."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Besides Mamba-like architecture, there exists many network architectures with linear computational complexity, like XCiT[1], xLSTM[2], hence my first concern is can these architectures work for the proposed framework? \n[1] Ali, Alaaeldin, et al. \"Xcit: Cross-covariance image transformers.\" Advances in neural information processing systems 34 (2021): 20014-20027.\n[2] Beck, Maximilian, et al. \"xLSTM: Extended Long Short-Term Memory.\" arXiv preprint arXiv:2405.04517 (2024).\n\n2. Following the first question, does Mamba has some unique attributes and advantages specially fit the robotic field? This work lack of related comparison and analysis.\n\n3. The Fig.1 seems a bit of rough, for example, the formats and colors of arrows are inconsistent, some borders have jagged edges, and the overall resolution is not high enough."},"limitations":{"value":"Please refer to the weaknesses of the paper."}},"nonreaders":[],"tmdate":1730879041133,"tcdate":1720668209675,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission5916/Reviewer_vV5n"],"signatures":["NeurIPS.cc/2024/Conference/Submission5916/Reviewer_vV5n"],"forum":"JxOQeg1NkH","number":2,"license":"CC BY 4.0","cdate":1720668209675,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission5916/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879041133,"domain":"NeurIPS.cc/2024/Conference","replyto":"JxOQeg1NkH","id":"sX6IIYYSrK","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Robot Manipulation","Multimodal Large Language Model","State Space Model"]},"supplementary_material":{"value":"/attachment/a8ec83e43dde9f6210ec7009438418ab8d320803.zip"},"primary_area":{"value":"robotics"},"abstract":{"value":"A fundamental objective in robot manipulation is to enable models to comprehend visual scenes and execute actions. Although existing Vision-Language-Action (VLA) models for robots can handle a range of basic tasks, they still face challenges in two areas: (1) insufficient reasoning ability to tackle complex tasks, and (2) high computational costs for VLA model fine-tuning and inference. The recently proposed state space model (SSM) known as Mamba demonstrates promising capabilities in non-trivial sequence modeling with linear inference complexity. Inspired by this, we introduce RoboMamba, an end-to-end robotic VLA model that leverages Mamba to deliver both robotic reasoning and action capabilities, while maintaining efficient fine-tuning and inference. Specifically, we first integrate the vision encoder with Mamba, aligning visual tokens with language embedding through co-training, empowering our model with visual common sense and robotic-related reasoning. To further equip RoboMamba with SE(3) pose prediction abilities, we explore an efficient fine-tuning strategy with a simple policy head. We find that once RoboMamba possesses sufficient reasoning capability, it can acquire manipulation skills with minimal fine-tuning parameters (0.1\\% of the model) and time. In experiments, RoboMamba demonstrates outstanding reasoning capabilities on general and robotic evaluation benchmarks. Meanwhile, our model showcases impressive pose prediction results in both simulation and real-world experiments, achieving inference speeds 3 times faster than existing VLA models."},"_bibtex":{"value":"@inproceedings{\nliu2024robomamba,\ntitle={RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation},\nauthor={Jiaming Liu and Mengzhen Liu and Zhenyu Wang and Pengju An and Xiaoqi Li and Kaichen Zhou and Senqiao Yang and Renrui Zhang and Yandong Guo and Shanghang Zhang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=JxOQeg1NkH}\n}"},"title":{"value":"RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation"},"pdf":{"value":"/pdf/d121b003663e64b010981e0a4b9e271adebd7c2d.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"liu|robomamba_efficient_visionlanguageaction_model_for_robotic_reasoning_and_manipulation"},"authorids":{"value":["~Jiaming_Liu2","~Mengzhen_Liu1","~Zhenyu_Wang11","~Pengju_An1","~Xiaoqi_Li3","~Kaichen_Zhou1","~Senqiao_Yang1","~Renrui_Zhang1","~Yandong_Guo2","~Shanghang_Zhang4"]},"authors":{"value":["Jiaming Liu","Mengzhen Liu","Zhenyu Wang","Pengju An","Xiaoqi Li","Kaichen Zhou","Senqiao Yang","Renrui Zhang","Yandong Guo","Shanghang Zhang"]}},"version":2},{"content":{"comment":{"value":"**Other sub-quadratic algorithms for sequence processing**\n\nAnother promising line of work focuses on applying deep State Space Models (SSMs) [6-9] and Linear RNNs [10-12] to long-range sequence and language modeling. Fundamentally, these architectures model interactions in sequence dimension by a linear recurrence:\n\n$\n\\begin{align*}\n&x_t = \\mathbf{\\overline{A}}x_{t-1}  + \\mathbf{\\overline{B}}u_t, \\\\\n& y_t = \\mathbf{\\overline{C}}x_t + \\mathbf{\\overline{D}}u_t,\n\\end{align*}\n$\n\nwhere recurrence matrix $\\mathbf{\\overline{A}}$ and other parameters are data-independent matrices which form and initialization are defining properties for a particular SSM/ RNN architecture. \nThe linear recurrence is advantageous during inference as it runs in $O(N)$ time. For training, it also can be unrolled into a convolutional kernel $\\mathbf{K} = \\begin{bmatrix} \\mathbf{\\overline{C}  \\overline{B}}, & \\mathbf{\\overline{C} \\overline{A} \\overline{B}}, & \\ldots, & \\mathbf{\\overline{C} \\overline{A}}^{N-1}\\mathbf{\\overline{B}} \\end{bmatrix} $ to compute $y = \\mathbf{K} * u$ via Fast Fourier Transform (FFT) in $O (N \\log N)$ time. Here, we set $\\mathbf{D}=0$ for ease of exposition, but in practice it's usually set to identity to act as a skip-connection ubiquitous in modern deep NN architectures.\n\nAmong other novel algorithms which rely on FFT or its generalizations such as Monarch matrices [13], are Long Convolutions [14], Hyena [15], and Monarch Mixer [16], with the latter using sub-quadratic primitives both for computations along the sequence length and the model dimension. \n\nWhile being sub-quadratic, these algorithms are still slower than linear time as in DenseAttention. However, recently [9] introduced data-dependent gating for SSM parameters and low-level, hardware efficient CUDA implementation for parallel-scan operation which allows for fast linear-time processing both during training and inference. And [17] adopt a similar gating mechanism for causal linear attention which allows to drop the denominator in the linear attention formula but also admits no parallel training without resorting to low-level implementations.\n\n---\n\nReferences: \n\n[1] Touvron et al., \"LLaMA: Open and Efficient Foundation Language Models.\" arXiv preprint arXiv:2302.13971, 2023.\n\n[2] Xiong et al., \"On Layer Normalization in the Transformer Architecture.\" ICML 2020\n\n[3] Katharopoulos et al., \"Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention.\" ICML 2020\n\n[4] Shen et al., \"Efficient Attention: Attention with Linear Complexities.\" WACV 2021\n\n[5] Koohpayegani and Pirsiavash, \"SimA: Simple Softmax-free Attention for Vision Transformers.\" WACV 2024\n\n[6] Gu et al., \"Efficiently Modeling Long Sequences with Structured State Spaces.\" ICLR 2022\n\n[7] Gupta et al., \"Diagonal State Spaces are as Effective as Structured State Spaces.\" NeurIPS 2022\n\n[8] Ma et al., \"Mega: Moving Average Equipped Gated Attention.\" ICLR 2023\n\n[9] Gu and Dao, \"Mamba: Linear-Time Sequence Modeling with Selective State Spaces.\" CoLM 2024\n\n[10] Beck et al., \"xLSTM: Extended Long Short-Term Memory.\" NeurIPS 2024\n\n[11] Orvieto et al., \"Resurrecting Recurrent Neural Networks for Long Sequences.\" ICML 2023\n\n[12] Peng et al., \"RWKV: Reinventing RNNs for the Transformer Era.\" Findings of EMNLP 2023\n\n[13] Dao et al., \"Monarch: Expressive Structured Matrices for Efficient and Accurate Training.\" ICML 2022 \n\n[14] Fu et al., \"Simple Hardware-Efficient Long Convolutions for Sequence Modeling.\" ICML 2023\n\n[15] Poli et al., \"Hyena Hierarchy: Towards Larger Convolutional Language Models.\" ICML 2023\n\n[16] Fu et al., \"Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture.\" NeurIPS 2023\n\n[17] Yang et al., \"Gated Linear Attention Transformers with Hardware-Efficient Training.\" ICML 2024\n\n--- \nWe thank you for the opportunity to improve our paper and will include the above exposition in the next revision of the manuscript (please note that the present revision contains an abbreviated un-proofread version which will be updated)."},"title":{"value":"Response to Reviewer oGkz. Part 2"}},"tmdate":1732904566730,"tcdate":1732904566730,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13608/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13608/Authors"],"forum":"2bIQBDSfRk","number":23,"license":"CC BY 4.0","cdate":1732904566730,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13608/-/Official_Comment"],"mdate":1732904566730,"domain":"ICLR.cc/2025/Conference","replyto":"MmwZFCIsZQ","id":"XJEzmaRy8k","forumContent":{"TLDR":{"value":"We propose a novel DenseAttention architecture which achieves 1) favorable computational efficiency;  2) linear time & space complexity by simplification and reduction of standard Transformer architecture."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["self-attention","deep learning","transformer architecture","nlp","efficient transformers","DenseAttention","long context","Long Range Arena"]},"supplementary_material":{"value":"/attachment/78e4b43b1d574d7e27aaf0271c25bf639f64491b.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The ubiquitous Transformer architecture suffers from two main bottlenecks: 1) low computational and memory efficiency, leading to suboptimal hardware utilization, and 2) quadratic time complexity with respect to sequence length $N$, making it slow and costly for large data contexts. We propose a novel DenseAttention Network architecture, a straightforward simplification of the standard Transformer block that addresses these issues and serves as a drop-in replacement for language modeling tasks. We eliminate memory-bound components in DenseAttention, including Softmax, masking, one skip connection, and both LayerNorms, as well as key, value, and output projection matrices, as they become redundant. Despite these removals, it maintains exact $N \\times N$ pairwise interactions between tokens. By exploiting the associativity of matrix multiplications, DenseAttention can be computed with $O(N^2d)$ or $O(Nd^2)$ time and space complexity, depending on the context. To handle the absence of Softmax and prevent numerical instability, we introduce MaxNormActivation at both ends of the Transformer block. We also devise Cosine Relative Positional Embeddings as a computationally efficient replacement for RoPE, and simple LocalAttention variations of the block to help the model focus on details in extremely long contexts. \n\nDenseAttention competes with FlashAttention in speed on small sequences and outperforms it by orders of magnitude on large contexts. We pre-train encoder language models on sequences up to 16K in length, which perform similarly or better than baseline BERT-large, while significantly improving speed and efficiency.  Finally, we achieve state-of-the-art on the LRA benchmark among the Transformer-based architectures."},"_bibtex":{"value":"@misc{\nargatkiny2025denseattention,\ntitle={DenseAttention: No-Compromise Exact All \\$N {\\textbackslash}times N\\$  Interactions Algorithm with \\$O(N)\\$ Space and Time Complexity},\nauthor={Andrew Argatkiny},\nyear={2025},\nurl={https://openreview.net/forum?id=2bIQBDSfRk}\n}"},"title":{"value":"DenseAttention: No-Compromise Exact All $N \\times N$  Interactions Algorithm with $O(N)$ Space and Time Complexity"},"pdf":{"value":"/pdf/efeb6787f0c4f6394dce7d2e3d58e8e6dae7e3ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"argatkiny|denseattention_nocompromise_exact_all_n_\\times_n_interactions_algorithm_with_on_space_and_time_complexity"},"authorids":{"value":["~Andrew_Argatkiny1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Andrew Argatkiny"]}},"version":2},{"content":{"comment":{"value":"> \"Why choose mamba over other linear transformers (deltanet, GLA..) and state space models?\"\n\n**Match to data and objective**. Our sequences are long, sparse, and effectively streaming, and the pretraining task is causal k-NN forecasting along outward-moving trajectories. This is exactly the regime where selective SSMs like Mamba are strong: they provide causal, linear-time recurrence and an inductive bias closer to signal propagation in a medium than to the bidirectional, full-history behavior typical of many linear-attention Transformers. That alignment between dynamics and physics motivated our choice.\n\n**Empirical comparison to linear Transformers**. To directly test whether a linear-attention Transformer can serve as an equally good backbone in our setting, we performed additional experiments with **Linformer** [1] and **Longformer** [2] as representative linear Transformers. At comparable backbone sizes (5.2M and 5.27M parameters, respectively) and with largely identical downstream adapter sizes and training setups (200 epochs each), both models perform significantly worse than the Mamba2 counterpart on the track reconstruction task: Linformer and Longformer achieve ARI scores of 0.59 and 0.55, respectively, compared to 0.91 ARI for Mamba2 at similar parameter count. This large gap suggests that, for our long, sparse, causal geometry regime and k-NN forecasting objective, a selective SSM is substantially more effective than these representative linear-attention alternatives.\n\nOur goal here is not to exhaustively rank every linear-attention variant, but to answer a simpler question: can a linear-attention Transformer, as a class, match a selective SSM backbone on our NPP tasks under comparable capacity and training budget? For that purpose we deliberately chose Linformer and Longformer because they are (i) widely used and well-supported, (ii) cover complementary inductive biases in the linear-attention family (low-rank global projections vs. local sliding-window + global tokens), and (iii) can be plugged into our architecture with minimal changes, allowing a controlled, apples-to-apples comparison. Given that both of these reasonably strong linear-attention baselines underperform Mamba2 by a large margin at matched parameter count and training time, we view it as unlikely that swapping in yet another linear-attention variant (DeltaNet/GLA) would overturn the qualitative conclusion; a broader study of that family is therefore left for follow-up work.\n\n[1] Wang, Sinong, et al., *Linformer: Self-attention with linear complexity.* arXiv:2006.04768 (2020).\n\n[2] Beltagy, Iz, et al., *Longformer: The long-document transformer.* arXiv:2004.05150 (2020).\n\n> \"How long are the sequence lengths for the input? State space models struggle with extremely long context.\"\n\n**Our sequences are not “extremely long” in the sense in which SSMs are known to struggle**. In our setup, each event is serialized into a 1D sequence whose length equals the number of reconstructed TPC spacepoints in that event. As described in Sec. 3 and Appendix A, the number of spacepoints per event ranges from a few hundred to tens of thousands, with a mean of 856 (Fig. 7a). Thus, the typical sequence length is $O(10³)$, and even the busiest p+p events in this dataset remain at $O(10⁴)$ tokens, far below the extreme-context regimes ($\\geq$ 10⁵ tokens) where memory or optimization issues in SSMs tend to appear [1].\nWe will clarify in the manuscript that (i) sequence length equals the number of spacepoints per event, with the distribution reported in Fig. 7, and (ii) our use of Mamba targets the $O(10⁴)$ token regime, rather than the extreme long-context settings where SSMs are known to be more problematic.\n\n[1] A. Gu et al., *Efficiently Modeling Long Sequences with Structured State Spaces*, ICLR 2022."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764831471267,"tcdate":1764745993299,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7782/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7782/Authors"],"forum":"qaI3cLFsiX","number":6,"license":"CC BY 4.0","cdate":1764745993299,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7782/-/Official_Comment"],"mdate":1764831471267,"domain":"ICLR.cc/2026/Conference","replyto":"QFwhIazxzv","id":"fasE2tOqQY","forumContent":{"TLDR":{"value":"We present a scalable foundation model for sparse nuclear and particle physics detector data, capable of adaptation to diverse downstream tasks after pretraining."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Foundation Model","State Space Model","Neural Scaling","Particle Tracking","Nuclear Physics","Particle Physics"]},"supplementary_material":{"value":"/attachment/87460ebbfd31e692941657bb7ccc084990769618.zip"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"abstract":{"value":"Large language models have revolutionized artificial intelligence by enabling large, generalizable models trained through self-supervision. This paradigm has inspired the development of scientific foundation models (FMs). However, applying this capability to experimental particle physics is challenging due to the sparse, spatially distributed nature of detector data, which differs dramatically from natural language. This work addresses if an FM for particle physics can scale and generalize across diverse tasks. We introduce a new dataset with more than 11 million particle collision events and a suite of downstream tasks and labeled data for evaluation. We propose a novel self-supervised training method for detector data and demonstrate its neural scalability with models that feature up to 188 million parameters. With frozen weights and task-specific adapters, this FM consistently outperforms baseline models across all downstream tasks. The performance also exhibits robust data-efficient adaptation. Further analysis reveals that the representations extracted by the FM are task-agnostic but can be specialized via a single linear mapping for different downstream tasks."},"_bibtex":{"value":"@inproceedings{\npark2026fmnpp,\ntitle={{FM}4{NPP}: A Scaling Foundation Model for Nuclear and Particle Physics},\nauthor={David Keetae Park and Shuhang Li and Yi Huang and Xihaier Luo and Haiwang Yu and Yeonju Go and Christopher Pinkenburg and Yuewei Lin and Shinjae Yoo and Joseph D. Osborn and Jin Huang and Yihui Ren},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=qaI3cLFsiX}\n}"},"title":{"value":"FM4NPP: A Scaling Foundation Model for Nuclear and Particle Physics"},"pdf":{"value":"/pdf/a8447a796e51fc50328ede271c7cf61e3c313155.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"park|fm4npp_a_scaling_foundation_model_for_nuclear_and_particle_physics"},"authorids":{"value":["~David_Keetae_Park2","~Shuhang_Li2","~Yi_Huang1","~Xihaier_Luo1","~Haiwang_Yu1","~Yeonju_Go1","~Christopher_Pinkenburg1","~Yuewei_Lin1","~Shinjae_Yoo1","~Joseph_D._Osborn1","~Jin_Huang9","~Yihui_Ren1"]},"authors":{"value":["David Keetae Park","Shuhang Li","Yi Huang","Xihaier Luo","Haiwang Yu","Yeonju Go","Christopher Pinkenburg","Yuewei Lin","Shinjae Yoo","Joseph D. Osborn","Jin Huang","Yihui Ren"]}},"version":2},{"content":{"summary":{"value":"The paper presents an approach for designing neural network policies, in the context of deep reinforcement learning (DRL) with continuous state and action spaces, that are more amenable to verification compared to standard networks. A standard approach to verifying continuous DRL systems is to abstract the system after learning the policy and to then verify the abstracted system. In contrast, the proposed approach first abstracts the state space (such that the state space is discretized) and then learns a linear policy for each abstract state. The coefficients of the linear policy are learnt using a coefficient network that maps each abstract state to its resulting policy coefficients. This mapping itself need not be linear and can be a complex neural network. However, since the state space is discretized and the policy for each abstract state is fixed and linear, tight and efficient verification of the resulting DRL system becomes feasible. Moreover, the state abstraction computation is itself encoded as the first layer of the coefficient network, so that the resulting policy network (referred to as piece-wise linear decision neural networks or PLDNNs) has a standard input (system state) and an output (actions), and can be directly trained using off-the-shelf DRL algorithms. The empirical evaluation shows that the performance of PLDNNs, measured in terms of cumulative reward and robustness of the policy, is comparable to standard policy networks. At the same time, DRL systems with PLDNN policies are much more amenable to verification."},"presentation":{"value":"3 good"},"contribution":{"value":"4 excellent"},"soundness":{"value":"4 excellent"},"strengths":{"value":"Even when the system dynamics are known, formal verification of safety and liveness properties of DRL systems is challenging to scale, specially for continuous state and action spaces. The paper proposes a very interesting idea for architecting policy networks that are easier to verify. I believe that the proposed notion of abstracting the state space before learning the policy will be a fruitful research direction. I also find the observation that a small set of linear policies can perform as well as a complicated non-linear policy to be surprising and interesting in its own right. I should add the disclaimer that I am not an expert on the topic of reinforcement learning and do not have a good sense of the broader literature on the topic."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"My main concern is about the \"scalability of verification vs cumulative reward\" tradeoff for DRL systems with high-dimensional state spaces (for instance, when the inputs to the policy network are images). As the state space becomes more complex, I suspect that a finer-grained partition becomes necessary to learn a good policy, directly impacting the scalability of verification. The presented evaluation considers system states with up to 12 dimensions, so it is hard to make an assessment about PLDNNs in settings where inputs are images required hundreds to thousands of dimensions.\nI am also curious about how the proposed approach would be used when the action space is discrete. Would each abstract state be mapped to a fixed action? Some discussion about this setting could be interesting and helpful. Finally, I have minor concerns about the empirical evaluation (described in the Questions section below)."},"confidence":{"value":"3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked."},"questions":{"value":"I have some clarification questions about the evaluation.\n\n1. What were the model architectures used for the results reported in Section 5.1? Are these the same architectures as in Section 5.2?\n\n2. Are the architectures for PLDNN and DNN the same (except for the first layer) for the results in Section 5.1 and Section 5.2? If not, how were the DNN architectures chosen and how do we ensure that it is an apples-to-apples comparison?\n\n3. Is the time reported in Table 1 in seconds? And is it the total training time over all episodes?\n\n4. How is the reward for the robustness experiments calculated, i.e., what is the horizon length? More generally, what is the reward function? The appendix gives some details; it says that for the 6 regular benchmarks \"we set a negative reward when the agent is not in the goal region. Once the agent reaches the goal region, it will be awarded a positive reward\". How and what is the positive reward assigned? Why is the final cumulative reward so different across the 6 regular benchmarks (based on Fig 5 and 7) if their reward functions are so similar?\n\n5. Are the models used for the robustness experiment the same as the ones used for the cumulative reward experiment? If yes, then why do you say that 10 different policies are trained for the former while you conduct five trials for the latter? My understanding is that the models used for the robustness experiment are trained on non-perturbed data but the evaluation is conducted with perturbed data. If this is so, then the same models can and should be used for both sets of experiments.\n\n6. What are precisely the properties of the DRL systems being verified? Based on Table 3 in the appendix, my guess is that for B1-B5, Tora, and QUAD the verified property is a liveness property whereas for CartPole it is a safety property? Can you please confirm? Is the time horizon for properties bounded or unbounded? It would be helpful to include this detail in the paper as well.\n\nSome more general questions:\n\n1. Can you comment on the applicability of this approach in settings where the inputs are high-dimensional images?\n\n2. Can the approach be adapted to settings where the action space is discrete?"},"rating":{"value":"7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations."},"code_of_conduct":{"value":"Yes"},"limitations":{"value":"The paper does not necessarily discuss the limitations and might benefit from a small discussion about the same."}},"nonreaders":[],"tmdate":1702411321575,"tcdate":1688577332765,"writers":["NeurIPS.cc/2023/Conference","NeurIPS.cc/2023/Conference/Submission11232/Reviewer_63JN"],"signatures":["NeurIPS.cc/2023/Conference/Submission11232/Reviewer_63JN"],"forum":"hpYb4eUinX","number":2,"license":"CC BY 4.0","cdate":1688577332765,"mdate":1702411321575,"readers":["everyone"],"invitations":["NeurIPS.cc/2023/Conference/Submission11232/-/Official_Review","NeurIPS.cc/2023/Conference/-/Edit"],"domain":"NeurIPS.cc/2023/Conference","replyto":"hpYb4eUinX","id":"4hfkQO1Lii","forumContent":{"venue":{"value":"NeurIPS 2023 poster"},"keywords":{"value":["Deep reinforcement learning","Reachability analysis","Hybrid system","State abstraction"]},"supplementary_material":{"value":"/attachment/1f9c93099db3a0b9dbac289f4722fd659502611e.zip"},"_bibtex":{"value":"@inproceedings{\ntian2023boosting,\ntitle={Boosting Verification of Deep Reinforcement Learning via Piece-Wise Linear Decision Neural Networks},\nauthor={Jiaxu Tian and Dapeng Zhi and Si Liu and Peixin Wang and Cheng Chen and Min Zhang},\nbooktitle={Thirty-seventh Conference on Neural Information Processing Systems},\nyear={2023},\nurl={https://openreview.net/forum?id=hpYb4eUinX}\n}"},"title":{"value":"Boosting Verification of Deep Reinforcement Learning via Piece-Wise Linear Decision Neural Networks"},"paperhash":{"value":"tian|boosting_verification_of_deep_reinforcement_learning_via_piecewise_linear_decision_neural_networks"},"TLDR":{"value":"We propose a novel DNN-based piece-wise linear decision model to facilitate the verification of DRL systems while achieving the comparable performance as ordinary DNN."},"abstract":{"value":"Formally verifying deep reinforcement learning (DRL) systems suffers from both inaccurate verification results and limited scalability. The major obstacle lies in the large overestimation introduced inherently during training and then transforming the inexplicable decision-making models, i.e., deep neural networks (DNNs), into easy-to-verify models. In this paper, we propose an inverse transform-then-train approach, which first encodes a DNN into an equivalent set of efficiently and tightly verifiable linear control policies and then optimizes them via reinforcement learning.  We accompany our inverse approach with a novel neural network model called piece-wise linear decision neural networks (PLDNNs), which are compatible with most existing DRL training algorithms with comparable performance against conventional DNNs. Our extensive experiments show that, compared to DNN-based  DRL systems, PLDNN-based systems can be more efficiently and tightly verified with up to $438$ times speedup and a significant reduction in overestimation.  In particular, even a complex $12$-dimensional DRL system is efficiently verified with up to 7 times deeper computation steps."},"pdf":{"value":"/pdf/99bfdabf2307b194ff8587b72ad510e61a0a0d55.pdf"},"venueid":{"value":"NeurIPS.cc/2023/Conference"},"authorids":{"value":["~Jiaxu_Tian1","~Dapeng_Zhi1","~Si_Liu6","~Peixin_Wang1","~Cheng_Chen9","~Min_Zhang6"]},"authors":{"value":["Jiaxu Tian","Dapeng Zhi","Si Liu","Peixin Wang","Cheng Chen","Min Zhang"]}},"version":2},{"content":{"venue":{"value":"IJCAI 2025"},"pdf":{"value":"https://www.ijcai.org/proceedings/2025/0824.pdf"},"venueid":{"value":"dblp.org/conf/IJCAI/2025"},"paperhash":{"value":"hou|rrgmamba_efficient_radiology_report_generation_with_state_space_model"},"authorids":{"value":["~Xiaodi_Hou3","~Xiaobo_Li5","https://dblp.org/search/pid/api?q=author:Mingyu_Lu:","https://dblp.org/search/pid/api?q=author:Simiao_Wang:","~Yijia_Zhang4"]},"html":{"value":"https://doi.org/10.24963/ijcai.2025/824"},"_bibtex":{"value":"@inproceedings{DBLP:conf/ijcai/Hou0LW025,\n  author={Xiaodi Hou and Xiaobo Li and Mingyu Lu and Simiao Wang and Yijia Zhang},\n  title={RRG-Mamba: Efficient Radiology Report Generation with State Space Model},\n  year={2025},\n  cdate={1735689600000},\n  pages={7410-7418},\n  url={https://doi.org/10.24963/ijcai.2025/824},\n  booktitle={IJCAI},\n  crossref={conf/ijcai/2025}\n}\n"},"abstract":{"value":"Recent advancements in radiology report generation have utilized deep neural networks such as CNNs and Transformers, achieving notable improvements in generating accurate and detailed reports. However, their practical adoption is hindered by the challenge of balancing global dependency modeling with computational efficiency. The state space model, particularly its enhanced variant Mamba, offers promising linear-complexity solutions for long-range dependency modeling. Despite its strengths, Mamba’s fixed positional encoding limits its ability to effectively capture complex spatial dependencies. To address this gap, we propose RRG-Mamba, an advanced framework for efficient radiology report generation. Within the RRGMamba, we enhance the vanilla Mamba by integrating rotary position encoding (RoPE), enabling dynamic modeling of relative positional information in visual feature sequences. Furthermore, we design a global dependency learning module to optimize long-range visual feature sequence modeling. Extensive experiments on publicly available datasets, including IU X-Ray and MIMIC-CXR, demonstrate that RRG-Mamba achieves a 3.7% improvement in BLEU-4 score over existing models, along with significant gains in computational and memory efficiency. Our code is available at https://github.com/Eleanorhxd/RRG-Mamba."},"title":{"value":"RRG-Mamba: Efficient Radiology Report Generation with State Space Model"},"authors":{"value":["Xiaodi Hou","Xiaobo Li","Mingyu Lu","Simiao Wang","Yijia Zhang"]}},"tmdate":1769047799840,"pdate":1735689600000,"externalIds":["dblp:conf/ijcai/Hou0LW025"],"tcdate":1768457400235,"writers":["~"],"signatures":["~Yijia_Zhang4"],"forum":"y68fPhFPQe","license":"CC BY-SA 4.0","number":758974,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769047799840,"domain":"DBLP.org","id":"y68fPhFPQe","version":2},{"content":{"comment":{"value":"We believe our work would be beneficial to the 3D molecular community and would genuinely appreciate your recognition.\n\nWe sincerely thank you for your time! Hope we have addressed your concerns through practical efforts and shown the contributions and significance of our work. We look forward to your reply and further discussions, thanks!\n\nSincerely,\n\nAuthors\n\n[1] New Basis Set Exchange: An Open, Up-to-Date Resource for the Molecular Sciences Community, 2019\n\n[2] PotentialNet for molecular property prediction, 2018\n\n[3] International Journal of Molecular Sciences\n\n[4] WIREs Computational Molecular Science\n\n[5] Molecular Sciences (ISSN: 2998-8977)\n\n[6] Retrieval-based controllable molecule generation, 2022\n\n[7] C5t5: Controllable generation of organic molecules with transformers, 2021\n\n[8] CProMG: controllable protein-oriented molecule generation with desired binding affinity and drug-like properties, 2023\n\n[9] Equivariant diffusion for molecule generation in 3d\n\n[10] Geometric Latent Diffusion Models for 3D Molecule Generation\n\n[11] Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies, 2024\n\n[12] Scaling Laws for Neural Language Models, 2020\n\n[13] Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2023"},"title":{"value":"Response to Reviewer J6yi (Part IV)"}},"tmdate":1732136864133,"tcdate":1732136864133,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5194/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5194/Authors"],"forum":"HbZrxBXzks","number":5,"license":"CC BY 4.0","cdate":1732136864133,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5194/-/Official_Comment"],"mdate":1732136864133,"domain":"ICLR.cc/2025/Conference","replyto":"uu0tW9mIpp","id":"1n3aLg9R2E","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Generative models","language models","tokenization","3d molecule generation","SE(3)-equivariance"]},"supplementary_material":{"value":"/attachment/d817704200f2ad4a725e3233fb9a358564155235.pdf"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We consider molecule generation in 3D space using language models (LMs), which requires discrete tokenization of 3D molecular geometries. Although tokenization of molecular graphs exists, that for 3D geometries is largely unexplored. Here, we attempt to bridge this gap by proposing the Geo2Seq, which converts molecular geometries into SE(3)-invariant 1D discrete sequences. Geo2Seq consists of canonical labeling and invariant spherical representation steps, which together maintain geometric and atomic fidelity in a format conducive to LMs. Our experiments show that, when coupled with Geo2Seq, various LMs excel in molecular geometry generation, especially in controlled generation tasks."},"_bibtex":{"value":"@misc{\nli2025geometry,\ntitle={Geometry Informed Tokenization of Molecules for Language Model Generation},\nauthor={Xiner Li and Limei Wang and Youzhi Luo and Carl Edwards and Shurui Gui and Yuchao Lin and Heng Ji and Shuiwang Ji},\nyear={2025},\nurl={https://openreview.net/forum?id=HbZrxBXzks}\n}"},"title":{"value":"Geometry Informed Tokenization of Molecules for Language Model Generation"},"pdf":{"value":"/pdf/c2332ad9a60ef3e3d8eea3ae837c46cadfb48aef.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"li|geometry_informed_tokenization_of_molecules_for_language_model_generation"},"authorids":{"value":["~Xiner_Li1","~Limei_Wang1","~Youzhi_Luo1","~Carl_Edwards1","~Shurui_Gui1","~Yuchao_Lin1","~Heng_Ji3","~Shuiwang_Ji1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xiner Li","Limei Wang","Youzhi Luo","Carl Edwards","Shurui Gui","Yuchao Lin","Heng Ji","Shuiwang Ji"]}},"version":2},{"content":{"TLDR":{"value":"A new sequence model for long sequence modeling employed on various tasks."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["state space models","neural network architectures","deep learning architectures","sequence modeling","event-based vision","event cameras","neural odes"]},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"A central challenge in sequence modeling is efficiently handling tasks with extended contexts. While recent state-space models (SSMs) have made significant progress in this area, they often lack input-dependent filtering or require substantial increases in model complexity to handle input variability. We address this gap by introducing S7, a simplified yet powerful SSM that can handle input dependence while incorporating stable reparameterization and specific design choices to dynamically adjust state transitions based on input content, maintaining efficiency and performance. We prove that this reparameterization ensures stability in long-sequence modeling by keeping state transitions well-behaved over time. Additionally, it controls the gradient norm, enabling efficient training and preventing issues like exploding or vanishing gradients. S7 significantly outperforms baselines across various sequence modeling tasks, including neuromorphic event-based datasets, Long Range Arena benchmarks, and various physical and biological time series. Overall, S7 offers a more straightforward approach to sequence modeling without relying on complex, domain-specific inductive biases, achieving significant improvements across key benchmarks."},"_bibtex":{"value":"@misc{\nsoydan2024s,\ntitle={S7: Selective and Simplified State Space Layers for Sequence Modeling},\nauthor={Taylan Soydan and Nikola Zubic and Nico Messikommer and Siddhartha Mishra and Davide Scaramuzza},\nyear={2024},\nurl={https://openreview.net/forum?id=4wtcXV0kbi}\n}"},"title":{"value":"S7: Selective and Simplified State Space Layers for Sequence Modeling"},"pdf":{"value":"/pdf/1fffab6aa0f02c9b5326306d49f2faa5435961a9.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"soydan|s7_selective_and_simplified_state_space_layers_for_sequence_modeling"},"authorids":{"value":["~Taylan_Soydan1","~Nikola_Zubic1","~Nico_Messikommer1","~Siddhartha_Mishra1","~Davide_Scaramuzza1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Taylan Soydan","Nikola Zubic","Nico Messikommer","Siddhartha Mishra","Davide Scaramuzza"]}},"tmdate":1731588607203,"tcdate":1726837100753,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2167/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission2167/Authors"],"forum":"4wtcXV0kbi","license":"CC BY 4.0","number":2167,"cdate":1726837100753,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission2167/-/Full_Submission","ICLR.cc/2025/Conference/-/Withdrawn_Submission"],"mdate":1731588607203,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"4wtcXV0kbi","version":2},{"content":{"summary":{"value":"This paper proposes a Mamba-based framework, namely MambaLLIE, for low-light image enhancement. Specifically, the authors claim that they have two technical contributions:\n\n(i) A global-then-local state space block that integrates a local-enhanced state space module and an implicit Retinex-aware selective kernel module to capture intricate global and local dependencies.\n\n(ii) An implicit Retinex-aware selective kernel mechanism to guide deeper neural representations and segregate them into independent positive and negative illumination components before integrating them."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"I am curious about the training time comparison between the proposed MambaLLIE and other methods."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"(i) The idea is novel and interesting. Mamba is a scorching research topic in computer vision now and has shown promising performance in many high-level vision tasks like image recognition, object detection, image segmentation, etc. But until now, there have been fewer efforts dedicated to the low-light image enhancement task, a small low-level topic. Thus, it is very exciting to see this wonderful work.\n\n(ii) The presentation is well-dressed. I like the style of the figures in this paper. For example, the teaser figure can clearly show the advantages in larger receptive fields of the proposed MambaLLIE than previous Transformer-based methods and Mamba-based image restoration methods. For another example, the pipeline in Figure 2 can also clearly show the workflow and the details of each submodule of the whole framework.\n\n(iii)  The performance is good and solid. The proposed MambaLLIE significantly outperforms the state-of-the-art methods on six benchmarks including LOL-v2-syn, LOL-v2-real, SMID, SDSD-indoor, and SDSD-outdoor, as reported in Table 1. The visual comparisons in Figure 3 also show the effectiveness of the proposed method. Looks good."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"(i) The motivation is not clear. To be specific, why use Mamba for low-light image enhancement? The authors claim that the Mamba can help capture long-range dependences. However, Transformer architecture can also model the long-range dependences. Why use Mamba instead of Transformer? Also, why design the space state module like this? The insight of the proposed Mamba should be analyzed in more detail.\n\n(ii) A technique should be explained, i.e., what is the scanning operation in the state space module? The authors mention the scan in Mamba but I could not find any mathematical formula or text description in the paper to explain the computation process of this scan.\n\n(iii) In table 1, although the performance of the proposed MambaLLIE is better than the state-of-the-art Transformer-based method Retinexformer, its computational complexity and memory cost are larger than those of Retinexformer. For example, the FLOPS and the Params are 33% and 38% higher. In addition, what about the comparison of performance on the MIT-Adobe-FiveK dataset (sRGB output mode)?\n\n(iv) In Figure 4, it seems that Retinexformer achieves more visually pleasant results. For example, in the third line of Figure 6, Retinexformer reconstructs the complete edge of the redwood in the left zoomed-in patch but the proposed MambaLLIE fails. Thus, it is better to compare MambaLLIE and the state-of-the-art method on unpaired datasets such as the LIME, NPE, MEF, DICM, and VV datasets.\n\n(v) The source code and pre-trained models have not been submitted. The reproducibility cannot be checked."},"limitations":{"value":"Yes, the authors have analyzed the limitations in the main paper."}},"nonreaders":[],"tmdate":1730879264200,"tcdate":1719122177168,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission8743/Reviewer_1rUY"],"signatures":["NeurIPS.cc/2024/Conference/Submission8743/Reviewer_1rUY"],"forum":"l6xVqzm72i","number":1,"license":"CC BY 4.0","cdate":1719122177168,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission8743/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879264200,"domain":"NeurIPS.cc/2024/Conference","replyto":"l6xVqzm72i","id":"8xsXb3pJRo","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Low Light Enhancement","State Space Models","Feature Control"]},"supplementary_material":{"value":"/attachment/7614fe2adeff99c6083bb41f73ca3e8b76498b64.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Recent advances in low light image enhancement have been dominated by Retinex-based learning framework, leveraging convolutional neural networks (CNNs) and Transformers. However, the vanilla Retinex theory primarily addresses global illumination degradation and neglects local issues such as noise and blur in dark conditions. Moreover, CNNs and Transformers struggle to capture global degradation due to their limited receptive fields. While state space models (SSMs) have shown promise in the long-sequence modeling, they face challenges in combining local invariants and global context in visual data. In this paper, we introduce MambaLLIE, an implicit Retinex-aware low light enhancer featuring a global-then-local state space design. We first propose a Local-Enhanced State Space Module (LESSM) that incorporates an augmented local bias within a 2D selective scan mechanism, enhancing the original SSMs by preserving local 2D dependency. Additionally, an Implicit Retinex-aware Selective Kernel module (IRSK) dynamically selects features using spatially-varying operations, adapting to varying inputs through an adaptive kernel selection process. Our Global-then-Local State Space Block (GLSSB) integrates LESSM and IRSK with layer normalization (LN) as its core. This design enables MambaLLIE to achieve comprehensive global long-range modeling and flexible local feature aggregation. Extensive experiments demonstrate that MambaLLIE significantly outperforms state-of-the-art CNN and Transformer-based methods. Our code is available at https://github.com/wengjiangwei/MambaLLIE."},"_bibtex":{"value":"@inproceedings{\nweng2024mamballie,\ntitle={Mamba{LLIE}: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Space},\nauthor={Jiangwei Weng and Zhiqiang Yan and Ying Tai and Jianjun Qian and Jian Yang and Jun Li},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=l6xVqzm72i}\n}"},"title":{"value":"MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Space"},"pdf":{"value":"/pdf/90cb332aa165948530e0ea9a1b5949966821bef1.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"weng|mamballie_implicit_retinexaware_low_light_enhancement_with_globalthenlocal_state_space"},"authorids":{"value":["~Jiangwei_Weng1","~Zhiqiang_Yan1","~Ying_Tai1","~Jianjun_Qian2","~Jian_Yang1","~Jun_Li16"]},"authors":{"value":["Jiangwei Weng","Zhiqiang Yan","Ying Tai","Jianjun Qian","Jian Yang","Jun Li"]}},"version":2},{"content":{"summary":{"value":"This paper investigates the poor length generalization properties of recurrent models, in particular Mamba-2, and finds that a mechanism the authors term State Collapse (SC) is responsible. Based on this insight, the authors propose three mechanisms to mitigate state collapse in Mamba models, and find that it improves the length generalization capabilities of Mamba-2 models."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- How general are these methods for recurrent models? Do they transfer well beyond Mamba-2?\n- Of the proposed methods, which are more or less effective, and in which contexts?\n- Is the tradeoff at short sequences inherent?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- The work tackles an important problem from a new direction, informing long-context training methods from mechanistic understanding of recurrent models.\n- The newlines prompt is a compelling synthetic (if models can’t do this, something is wrong!) and provides focus and grounding to the work.\n- The provided methods all appear to improve the length-generalization capabilities of Mamba-2 models. The authors provide analysis on both synthetics and a range of language modeling tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- While the three proposed methods show promise, additional comparisons between them would enhance the paper's practical value. Figure 9 provides some insights, but more detailed ablation studies comparing their relative strengths and limitations would be valuable for practitioners.\n- The generalizability of these methods beyond Mamba-2 presents an opportunity for future exploration. Given that RWKV and GLA are mentioned in the introduction, including experiments with these architectures could help demonstrate the broader applicability of the proposed approach.\n- The methods currently show some performance trade-offs at shorter sequence lengths. Additional analysis or ablations of this phenomenon through the paper's theoretical framework could provide valuable insights and potentially suggest ways to maintain performance across all sequence lengths.\n- The paper would benefit from engaging more with relevant prior work on recurrent neural network length generalization, including recent contributions such as Jelassi, Samy, et al. \"Repeat after me: Transformers are better than state space models at copying.\" arXiv preprint arXiv:2402.01032 (2024); Arora, Simran, et al. \"Zoology: Measuring and improving recall in efficient language models.\" arXiv preprint arXiv:2312.04927 (2023); and Ben-Kish, Assaf, et al. \"DeciMamba: Exploring the Length Extrapolation Potential of Mamba.\" arXiv preprint arXiv:2406.14528 (2024). While this work offers a novel perspective, situating it within the broader literature would strengthen its contribution.\n- The visualization of results, particularly in figures 2, 6, and 12, could be more accessible to readers. For instance, figure 2 would benefit from clearer labeling and explanation of metrics like \"answer depth,\" and the color scheme implications could be more explicitly defined in the caption or text."}},"nonreaders":[],"tmdate":1731428501176,"tcdate":1730095104221,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12121/Reviewer_hmho"],"signatures":["ICLR.cc/2025/Conference/Submission12121/Reviewer_hmho"],"forum":"cu2CT2VAvs","number":3,"license":"CC BY 4.0","cdate":1730095104221,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12121/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428501176,"domain":"ICLR.cc/2025/Conference","replyto":"cu2CT2VAvs","id":"7LSoLaeqJS","forumContent":{"TLDR":{"value":"Systematic study of linear diagonal RNNs' length generalization failure and memory capacity in language modeling and contextual information retrieval."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["RNN","foundation models","long-context"]},"supplementary_material":{"value":"/attachment/d71dbd3e7ce14131c3720da88c049fa27def3e94.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"One essential advantage of recurrent neural networks (RNNs) over transformer-based language models is their linear computational complexity concerning the sequence length, which makes them much faster in handling long sequences during inference. However, most publicly available RNNs (e.g., Mamba and RWKV) are trained on sequences with less than 10K tokens, and their effectiveness in longer contexts remains largely unsatisfying so far. In this paper, we study the cause of the inability to process long context for RNNs and suggest critical mitigations. First, we investigate *state explosion* (SE) in Mamba-2 when processing long sequences, a phenomenon where some channels of the state exhibit exploding values that cause severe performance degradation. With controlled experiments, we discover that the model fails to forget the earlier tokens when there is more information than it can remember. We attribute this to overfitting due to the recurrent state being overparameterized for the training length, thereby establishing a relationship between SE and the capacity of the state. To support this hypothesis, we make an important empirical observation: for any given state size, there exists a training length threshold such that SE is exhibited if and only if the training length is greater than this threshold. Empirically searching for this threshold for different state sizes reveals that it is a linear function of the state size. We also search for the maximum context length at which the model can recall contextual information and find that this context length scales exponentially to the state size. Based on this, we empirically train a Mamba-2 370M with near-perfect passkey retrieval accuracy on 256K context length. This suggests a promising future for RNN-based long-context modeling. Code and model checkpoints will be publicly released."},"_bibtex":{"value":"@misc{\nchen2025stuffed,\ntitle={Stuffed Mamba: State Collapse and State Capacity of {RNN}-Based Long-Context Modeling},\nauthor={Yingfa Chen and Xinrong Zhang and Shengding Hu and Xu Han and Zhiyuan Liu and Maosong Sun},\nyear={2025},\nurl={https://openreview.net/forum?id=cu2CT2VAvs}\n}"},"title":{"value":"Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling"},"pdf":{"value":"/pdf/81b99123fa4ffe7fe2c3aa7b6d9993ea1a3889f6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chen|stuffed_mamba_state_collapse_and_state_capacity_of_rnnbased_longcontext_modeling"},"authorids":{"value":["~Yingfa_Chen1","~Xinrong_Zhang1","~Shengding_Hu2","~Xu_Han2","~Zhiyuan_Liu1","~Maosong_Sun1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yingfa Chen","Xinrong Zhang","Shengding Hu","Xu Han","Zhiyuan Liu","Maosong Sun"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Human trajectory prediction","Selective state-space models"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Human trajectory forecasting is crucial for safe navigation in crowded environments, requiring models that balance accuracy with computational efficiency. Efficiently modeling social interactions is key to performance in dense crowds. Yet, most recent methods rely on attention mechanisms, which are effective at capturing complex dependencies, but incur quadratic computational costs that scale poorly with the growing number of neighbors. Recently, Selective State-Space Models have provided a linear-time alternative; however, their inherently sequential design is misaligned with the unstructured and dynamic nature of social interactions. To address this challenge, we propose Social-Mamba, a forecasting architecture that reformulates social interactions as structured sequential processes. At its core is the Cycle Mamba block, a novel module that enables continuous bidirectional information flow. Social-Mamba organizes agents on a semantically ordered egocentric grid and introduces social triplet factorization, which decomposes interactions into temporal, egocentric, and goal-centric scans. These are dynamically integrated through a learnable social gate and global scan to generate accurate and efficient trajectory predictions. Extensive experiments on five trajectory forecasting benchmarks show that Social-Mamba achieves state-of-the-art accuracy while offering superior parameter efficiency and computational scalability. Furthermore, embedding Social-Mamba into a flow-matching framework further enhances both accuracy and efficiency, establishing it as a flexible and robust foundation for future trajectory forecasting research."},"_bibtex":{"value":"@misc{\nluan2026socialmamba,\ntitle={Social-Mamba: Efficient Human Trajectory Forecasting with State-Space Models},\nauthor={Po-Chien Luan and Wuyang Li and Yang Gao and Alexandre Alahi},\nyear={2026},\nurl={https://openreview.net/forum?id=9XCnaA2CQm}\n}"},"title":{"value":"Social-Mamba: Efficient Human Trajectory Forecasting with State-Space Models"},"pdf":{"value":"/pdf/e8274978dcc35c7a13afdaa601d4b91dd99b4728.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"luan|socialmamba_efficient_human_trajectory_forecasting_with_statespace_models"},"authorids":{"value":["~Po-Chien_Luan1","~Wuyang_Li1","~Yang_Gao15","~Alexandre_Alahi3"]},"authors":{"value":["Po-Chien Luan","Wuyang Li","Yang Gao","Alexandre Alahi"]}},"tmdate":1770804836434,"tcdate":1757420322865,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission3412/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission3412/Authors"],"forum":"9XCnaA2CQm","license":"CC BY 4.0","number":3412,"cdate":1757420322865,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission3412/-/Full_Submission","ICLR.cc/2026/Conference/Submission3412/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804836434,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"9XCnaA2CQm","version":2},{"content":{"summary":{"value":"This paper investigates the application of structured state space sequence models (SSMs), specifically MAMBA, in the molecular modeling of SMILES string formats. The model is pre-trained on a large dataset of 91 million SMILES strings from PubChem, resulting in 4 billion molecular tokens, Through pretraining and finetuning, the experimental results demonstrate that the proposed methodology achieves competitive performance on prediction and generation tasks with improvements in inference efficiency."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"* \"In Section 2.2, the authors propose using a language decoder alongside MAMBA's inherent decoding capabilities. This design choice appears to contradict the paper's efficiency claims, as self-attention mechanisms typically incur higher computational costs compared to MAMBA modules. Could the authors justify this architectural decision?\"\n\n* \"Figure 1's architectural representation requires clarification. The diagram suggests the SSM module is integrated as a standalone component following the molecular encoder's embeddings. The authors should specify whether the term 'MAMBA-based encoder' (line 124) indicates MAMBA modules are:\na) supplementary components to an existing encoder (e.g., transformers, GNNs), or\nb) fundamental building blocks comprising the encoder's internal layers.\"\n\n* \"Section 4.3 would benefit from a detailed description of the generation task implementation, specifically addressing the input format and the utilization of pretrained model components.\"\n\n* \"While MAMBA is renowned for circumventing the cubic complexity associated with sequence length in self-attention mechanisms, the incorporation of a language decoder during pretraining (Section 4.3) raises questions about computational efficiency. How do the authors reconcile these seemingly contradictory design choices?\"\n\n* \"Regarding Section 4.2, the superior generalization performance of MAMBA-based representations warrants further analysis. The results suggest this advantage stems from the MAMBA architecture itself rather than large-scale pretraining, given that transformer-based pretrained baselines demonstrate comparatively lower generalization capabilities.\"\n\n* \"The authors' description in Section 2.3 of the two-phase strategy, particularly regarding dataset partitioning, is described as counter-intuitive. This warrants a more comprehensive analysis and justification of the approach.\""},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":1},"strengths":{"value":"1.   The motivation is clearly articulated. the application of Mamba to the biological domain, specifically to long-sequence data like SMILES strings, appears promising due to its ability to efficiently capture complex patterns and dependencies within these sequences.The authors effectively establish the need for an alternative to Transformer-based architectures in handling long-sequence data in chemistry. \n\n2.   Experimental results on tasks such as property prediction, reaction yield prediction, and molecular generation show positive outcomes."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.   The contribution is overlapped with the work from [1], lots of accliam and desription are same from the work. which make the contribution of this work is marginal and unclear.\n2.   There is a disconnect between the stated motivation(deal with long-sequence) and the experimental design. Specifically, the experiments are primarily conducted on molecules with lengths of 49 ± 45, which is considerably shorter than the maximum sequence length supported by transformer-based models (e.g., 512 for ChemBERTa [1]). Although inference speed improvements are reported, experiments on longer sequences, such as proteins, would provide stronger evidence for the model’s applicability to long-sequence tasks.\n3.   The manuscript lacks sufficient details regarding the experimental implementation. Several key aspects require clarification (please see questions below).\n\n[1] Eduardo Soares, Victor Shirasuna, Emilio Vital Brazil, Renato Cerqueira, Dmitry Zubarev, and\nKristin Schmidt. A large encoder-decoder family of foundation models for chemical language.\narXiv preprint arXiv:2407.20267, 2024.\n[2] Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: large-scale selfsupervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.09885, 2020."}},"nonreaders":[],"tmdate":1731429570783,"tcdate":1730521909071,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11844/Reviewer_CqKH"],"signatures":["ICLR.cc/2025/Conference/Submission11844/Reviewer_CqKH"],"forum":"7dsC1w4yzP","number":3,"license":"CC BY 4.0","cdate":1730521909071,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11844/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429570783,"domain":"ICLR.cc/2025/Conference","replyto":"7dsC1w4yzP","id":"Zzzct1QlmF","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","foundation model","molecular property prediction","classification","molecular reconstruction","synthesis yield prediction"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We present a novel approach to chemical foundation models, leveraging structured state space sequence models (SSMs) to overcome the limitations of traditional Transformer-based architectures. While Transformers have achieved state-of-the-art results in chemical tasks such as property prediction and molecule generation, their self-attention mechanism is constrained by its inability to model data outside of a finite context window and its quadratic scaling with respect to window length. In contrast, SSMs offer a promising alternative for sequence modeling, enabling the capture of complex patterns and dependencies in molecular structures. Our Mamba architecture, a simplified end-to-end SSM-based neural network, eliminates the need for attention and MLP blocks, allowing for faster inference. We pre-train Mamba on a large, curated dataset of 91 million SMILES samples (equivalent to 4 billion molecular tokens) sourced from PubChem, and evaluate its performance on various benchmark datasets. Our experiments demonstrate the SSM's capacity to provide state-of-the-art results while maintaining fast inference, supporting complex tasks such as molecular property prediction, classification, molecular reconstruction, and synthesis yield prediction. This work advances the state-of-the-art in AI methodology in chemical sciences, offering a promising direction for future research in molecular modeling and discovery."},"_bibtex":{"value":"@misc{\nsoares2024mambabased,\ntitle={Mamba-based Chemical Foundational Model for Fast Inference},\nauthor={Eduardo Soares and Emilio Vital Brazil and Victor Yukio Shirasuna and Dmitry Zubarev and Renato Cerqueira and Kristin Schmidt},\nyear={2024},\nurl={https://openreview.net/forum?id=7dsC1w4yzP}\n}"},"title":{"value":"Mamba-based Chemical Foundational Model for Fast Inference"},"pdf":{"value":"/pdf/2afc255dfea68dbe51a16069cf43bcf4bd16c18e.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"soares|mambabased_chemical_foundational_model_for_fast_inference"},"authorids":{"value":["~Eduardo_Soares1","~Emilio_Vital_Brazil1","~Victor_Yukio_Shirasuna1","~Dmitry_Zubarev1","~Renato_Cerqueira1","~Kristin_Schmidt1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Eduardo Soares","Emilio Vital Brazil","Victor Yukio Shirasuna","Dmitry Zubarev","Renato Cerqueira","Kristin Schmidt"]}},"version":2},{"content":{"venue":{"value":"IEEE Transactions on Geoscience and Remote Sensing"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/36/10354519/10703171.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"he|3dssmamba_3dspectralspatial_mamba_for_hyperspectral_image_classification"},"html":{"value":"https://doi.org/10.1109/TGRS.2024.3472091"},"abstract":{"value":"Hyperspectral image (HSI) classification constitutes the fundamental research in remote sensing fields. Convolutional neural networks (CNNs) and Transformers have demonstrated impressive capability in capturing spectral-spatial contextual dependencies. However, these architectures suffer from limited receptive fields and quadratic computational complexity, respectively. Fortunately, recent Mamba architectures built upon the state space models (SSMs) integrate the advantages of long-range sequence modeling and linear computational efficiency, exhibiting substantial potential in low-dimensional scenarios. Motivated by this, we propose a novel 3D-spectral-spatial mamba (3DSS-Mamba) framework for HSI classification, allowing for global spectral-spatial relationship modeling with greater computational efficiency. Technically, a spectral-spatial token generation (SSTG) module is designed to convert the HSI cube into a set of 3-D spectral-spatial tokens. To overcome the limitations of traditional Mamba, which is confined to modeling causal sequences and inadaptable to high-dimensional scenarios, a 3D-spectral-spatial selective scanning (3DSS) mechanism is introduced, which performs pixel-wise selective scanning on 3-D hyperspectral tokens along the spectral and spatial dimensions. Five scanning routes are constructed to investigate the impact of dimension prioritization. The 3DSS scanning mechanism combined with conventional mapping operations forms the 3D-spectral-spatial mamba block (3DMB), enabling the extraction of global spectral-spatial semantic representations. Experimental results and analysis demonstrate that the proposed method outperforms the state-of-the-art methods on HSI classification benchmarks. The code is available at https://github.com/IIP-Team/3DSS-Mamba."},"title":{"value":"3DSS-Mamba: 3D-Spectral-Spatial Mamba for Hyperspectral Image Classification"},"authors":{"value":[{"fullname":"Yan He"},{"fullname":"Bing Tu","username":"~Bing_Tu1"},{"fullname":"Bo Liu"},{"fullname":"Jun Li"},{"fullname":"Antonio Plaza"}]}},"tmdate":1789092963268,"pdate":1704067200000,"externalIds":["doi:10.1109/tgrs.2024.3472091"],"tcdate":1762941874428,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Bing_Tu1"],"forum":"60IIKsRtyi","license":"CC BY-SA 4.0","number":14805,"cdate":1760462998041,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/-/Edit"],"mdate":1789092963268,"domain":"OpenReview.net/Public_Article","id":"60IIKsRtyi","version":2},{"content":{"comment":{"value":"**References**\n\n\\[1\\]: Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret. arXiv:2006.16236, 2020.\n\n\\[2\\]: Retentive Network: A Successor to Transformer for Large Language Models. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei. arXiv:2307.08621, 2023.\n\n\\[3\\]: Random Feature Attention. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, Lingpeng Kong. arXiv:2103.02143, 2021.\n\n\\[4\\]: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. Tri Dao, Albert Gu. arXiv:2405.21060, 2024.\n\n\\[5\\]: Rethinking Attention with Performers. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller. arXiv:2009.14794, 2022.\n\n\\[6\\]: Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Albert Gu, Tri Dao. arXiv:2312.00752, 2024.\n\\[7\\]: Efficiently Modeling Long Sequences with Structured State Spaces. Albert Gu, Karan Goel, Christopher Ré. arXiv:2111.00396, 2022.\n\n\\[8\\]: Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers. Sukjun Hwang, Aakash Lahoti, Tri Dao, Albert Gu. arXiv:2407.09941, 2024.\n\n\\[9\\]: Liquid Structural State-Space Models. Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, Daniela Rus. arXiv:2209.12951, 2022.\n\n\\[10\\]: Deep Contextualized Word Representations. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer. arXiv:1802.05365, 2018.\n\n\\[11\\]: xLSTM: Extended Long Short-Term Memory. Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter\n\n[12]: State Space Models as Foundation Models: A Control Theoretic Overview. Carmen Amo Alonso, Jerome Sieber, and Melanie N. Zeilinge\n\n[13]: Raymond A DeCarlo. Linear systems: A state variable approach with numerical implementation. Prentice-Hall, Inc., 1989.\n\n[14]: HiPPO: Recurrent Memory with Optimal Polynomial Projections: Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Re\n\n[15]: Parallelizing Linear Transformers with the Delta Rule over Sequence Length: Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, Yoon Kim\n\n---\n\nWe hope that our clarifications have provided a clearer understanding of our contributions and that we have successfully addressed the reviewer's concerns. In case the reviewer has any remaining concerns, we are more than willing to address them"},"title":{"value":"Authors Rebuttal to Reviewer sped (4/4)"}},"tmdate":1731945531094,"tcdate":1731881127574,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13771/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13771/Authors"],"forum":"VxJjw52dZu","number":10,"license":"CC BY 4.0","cdate":1731881127574,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13771/-/Official_Comment"],"mdate":1731945531094,"domain":"ICLR.cc/2025/Conference","replyto":"wsfYqaAFzJ","id":"X0byEYzMEe","forumContent":{"TLDR":{"value":"We provide a bidirectional selective recurrent form of full kernelized attention with learnable mask allowing scalability in context length and superior inference efficiency"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Self-attention","transformer","bidirectionality","mamba","ssm","rnn","inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We introduce LION, a novel sequence-to-sequence framework that unifies the bidirectionality and parallelized training of Transformers with the fast inference of recurrent neural networks. LION is built upon a mathematical formulation where full kernelized attention with a learnable mask is efficiently computed using a bidirectional selective recurrent model, matching the effectiveness of softmax-based attention with constant-time inference. Our framework naturally accounts for spatial and temporal relationships within input sequences, reducing reliance on heuristic positional embeddings and facilitating straightforward scalability in context length and resolution. Using our framework and inspired by the recent state-space models, we propose three main running examples LIOn-LIT, LION-RETNET, and LION-S, a transformer with selective mask and recurrent inference. Numerical evaluations on tasks such as language modeling, the Long-Range Arena, and image classification show that LION framework achieves performance on par with state-of-the-art models while delivering fast training and inference efficiency."},"_bibtex":{"value":"@misc{\nafzal2025lion,\ntitle={{LION}: A bidirectional framework that trains like a Transformer and infers like an {RNN}},\nauthor={Arshia Afzal and Elias Abad Rocamora and Leyla Naz Candogan and Pol Puigdemont and Francesco Tonin and Yongtao Wu and Mahsa Shoaran and Volkan Cevher},\nyear={2025},\nurl={https://openreview.net/forum?id=VxJjw52dZu}\n}"},"title":{"value":"LION: A bidirectional framework that trains like a Transformer and infers like an RNN"},"pdf":{"value":"/pdf/6e8b8585752d6f27da24ffe876859459a5b56efe.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"afzal|lion_a_bidirectional_framework_that_trains_like_a_transformer_and_infers_like_an_rnn"},"authorids":{"value":["~Arshia_Afzal1","~Elias_Abad_Rocamora1","~Leyla_Naz_Candogan1","~Pol_Puigdemont1","~Francesco_Tonin1","~Yongtao_Wu1","~Mahsa_Shoaran1","~Volkan_Cevher1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Arshia Afzal","Elias Abad Rocamora","Leyla Naz Candogan","Pol Puigdemont","Francesco Tonin","Yongtao Wu","Mahsa Shoaran","Volkan Cevher"]}},"version":2},{"content":{"summary":{"value":"This paper addresses the challenges of long-term time series forecasting (LTSF) by tackling the Lookback Window Limitation (LWL) encountered in current Transformer-based models. The authors argue that while longer lookback windows can theoretically provide richer historical insights, they often lead to redundancy and overfitting to temporal noise, particularly given the quadratic complexity of Transformers.\n\nTo mitigate these issues, the paper introduces the Information Bottleneck Filter (IBF), which employs information bottleneck theory to extract crucial subsequences from input data, thereby reducing redundancy. Additionally, the authors propose the Hybrid-Transformer-Mamba (HTM), a novel architecture that combines the linear complexity and long-range modeling capabilities of Mamba with the short-sequence strengths of Transformers.\n\nThrough integration of these model-agnostic components into existing methods, the authors conduct experiments across seven datasets, demonstrating significant improvements in performance. Notably, the introduction of the PIH (Patch-IBF-HTM) strategy enables the extension of the lookback window to 1024, achieving state-of-the-art results and underscoring the value of utilizing longer historical sequences for forecasting tasks."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"I find it puzzling that while the authors identify the Lookback Window Limitation (LWL) and propose corresponding solutions, they also state in the conclusion that \"longer lookback windows are not always beneficial for all datasets.\" This suggests that the core issue may not be fully addressed. Could the authors provide a more detailed explanation of this apparent contradiction?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper introduces the Information Bottleneck Filter (IBF) and Hybrid-Transformer-Mamba (HTM) as novel approaches to address the Lookback Window Limitation (LWL) in long-term time series forecasting. The integration of these concepts demonstrates an attempt to enhance model performance and reduce redundancy in input data. The authors conduct experiments across seven datasets, providing empirical validation of their methods, which achieve competitive results in the context of existing literature. The paper is organized clearly, making it accessible to readers."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Concerns on SOTA Comparisons**: As a submission for 2025, it is surprising that the authors only compare with the 2023 PatchTST and the 2024 S-Mamba. Additionally, the comparisons with FEDformer and Autoformer (2022 and 2021, respectively) do not adequately represent the current state of the art. Please annotate each method with its publication year to provide context. While including S-Mamba is valuable, it is still an unpeer-reviewed work. A more comprehensive comparison with recent methods would enhance the submission's relevance and credibility.\n\n2. **Fairness of Window Size Comparisons**: The parameter settings for the baselines in Table 1 are not clearly defined. Results for FEDformer, Autoformer, and Informer seem to replicate those from PatchTST, but their lookback window $ L $ does not uniformly match the PIH setting of 1024. This raises fairness concerns and contradicts the authors' claim of not intentionally disadvantaging these models. I recommend creating two tables: one where all models are evaluated at $ L = 1024 $ and another using each model’s optimal $ L $. Presenting both together would effectively demonstrate the significance of PIH's larger lookback window.\n\n3. **Experimental Rigor**: The authors state that PIH outperforms other models under shorter windows (e.g., 336 and 512), but this claim lacks rigor since only DLinear and NLinear are compared. According to Table 1, PatchTST appears to show superior performance. Additionally, since both PatchTST and DLinear/NLinear assume channel independence, it is crucial to compare with channel-dependent models in this context.\n\n4. **Minor Corrections**:\n   - In Figure 6, the left panel legend incorrectly states \"PHI\" instead of \"PIH.\"\n   - There is a duplicate paragraph on page 2: \"Moreover, it does not mitigate the quadratic complexity inherent in Transformers.\" This should be corrected."}},"nonreaders":[],"tmdate":1731428336006,"tcdate":1730204826462,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6771/Reviewer_gzbk"],"signatures":["ICLR.cc/2025/Conference/Submission6771/Reviewer_gzbk"],"forum":"hVpAjJPfgZ","number":2,"license":"CC BY 4.0","cdate":1730204826462,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6771/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428336006,"domain":"ICLR.cc/2025/Conference","replyto":"hVpAjJPfgZ","id":"gv5Mp7g5Nz","forumContent":{"TLDR":{"value":"We adopts a Transformer-Mamba hybrid architecture and information bottleneck for extreme long lookback windows in LSTF."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["long-term time series forcasting","Mamba","Information Bottleneck"]},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Long-term time series forecasting (LTSF) aims to predict future trends based on historical data. While longer lookback windows theoretically provide more comprehensive insights, current Transformer-based models face the Lookback Window Limitation (LWL). On one hand, longer windows introduce redundant information, which can hinder model learning. On the other hand, Transformers tend to overfit temporal noise rather than extract meaningful temporal information when dealing with longer sequences, compounded by their quadratic complexity. In this paper, we aim to overcome LWL, enabling models to leverage more historical information for improved performance. Specifically, to mitigate information redundancy, we introduce the Information Bottleneck Filter (IBF), which applies information bottleneck theory to extract essential subsequences from the input. Additionally, to address the limitations of the Transformer architecture in handling long sequences, we propose the Hybrid-Transformer-Mamba (HTM), which combines the linear complexity and long-range modeling capabilities of Mamba with the Transformer's strength in modeling short sequences. We integrate these two model-agnostic modules into various existing methods and conduct experiments on seven datasets. The results demonstrate that incorporating these modules effectively overcomes the lookback window limitations. Notably, by combining them with the Patch strategy, we design the PIH (\\textbf{P}atch-\\textbf{I}BF-\\textbf{H}TM), successfully extending the window length to 1024—a significantly larger window than previously achieved—and achieving state-of-the-art results, highlighting the potential of exploring even longer windows."},"_bibtex":{"value":"@misc{\nzhang2024overcoming,\ntitle={Overcoming Lookback Window Limitations: Exploring Longer Windows in Long-Term Time Series Forecasting},\nauthor={Jiahui Zhang and Wenjie Du and Zhengyang Zhou and Qihe Huang and Xuqiang Li and Yang Wang},\nyear={2024},\nurl={https://openreview.net/forum?id=hVpAjJPfgZ}\n}"},"title":{"value":"Overcoming Lookback Window Limitations: Exploring Longer Windows in Long-Term Time Series Forecasting"},"pdf":{"value":"/pdf/d81ea206a0e4871a9de5d22719240efde5117469.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhang|overcoming_lookback_window_limitations_exploring_longer_windows_in_longterm_time_series_forecasting"},"authorids":{"value":["~Jiahui_Zhang7","~Wenjie_Du2","~Zhengyang_Zhou1","~Qihe_Huang2","~Xuqiang_Li1","~Yang_Wang32"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jiahui Zhang","Wenjie Du","Zhengyang Zhou","Qihe Huang","Xuqiang Li","Yang Wang"]}},"version":2},{"content":{"summary":{"value":"The paper investigates simplified versions of traditional recurrent neural networks (RNNs), specifically LSTMs and GRUs, adapting them for efficient training by removing their hidden state dependencies, enabling parallel training. The proposed **minLSTM** and **minGRU** models retain the functional structure of their predecessors but omit time dependencies in their gates, reducing parameter count and improving computational efficiency. Empirical tests indicate these models achieve comparable performance to contemporary state-of-the-art sequence models, suggesting that streamlined versions of older RNN architectures may offer viable alternatives in sequence modeling tasks."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. **Layer Depth in Computational Comparisons:** (Please refer to Weaknesses) Were the computational benchmarks (Figure 1) conducted with models having comparable quantitative performances? The paper could benefit from a comparison on computational complexity normalized by model performances or depth. \n\n2. **Parameter Matching for Competitors:** Related to previous question. When taking results from other papers (as reported in the Appendix), were the competitor models matched in terms of parameters and model size? Variability in model configurations could impact the validity of performance comparisons.\n\n3. **Choice of Datasets:** The model is mainly tested on very simple datasets (mostly synthetic) that are not very representative for the current literature (see [3]). The authors should comment on why this choice taken.  If the model cannot scale to large datasets due to its inner working and memory requirements (as reported in the Limitation section), this is something that hinder the paper contributions.  For instance, the Long Range Arena benchmark has became a standardize framework to test sequence models performances, that could help also in identifying the model ability to preserve long term dependencies -- that is something extremely requested by current literature. The removal of time dependecy questions the model ability to perform in such benchmarks, which I believe should be better investigated. Thus,  at least a discussion on this is required. \n\n4. **Theoretical Comparison with xLSTM [1]:** Could the authors elaborate on the theoretical distinctions and advantages of minLSTM over xLSTM, especially in terms of efficiency and sequence modeling?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The approach effectively repurposes older RNNs by leveraging simplifications that enable parallel training, providing an interesting contrast to complex, modern architectures.\n\n- The proposed models significantly reduce training time, achieving speed improvements of up to 175x for sequence lengths of 512, which is a notable practical advantage.\n\n- This work challenges the abandonment of RNNs in favor of more recent architectures and suggests potential for simpler, more interpretable models in long-sequence processing."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"*Insufficient Comparison Context:* From the current text, it is not clear whether the computational comparisons (Figure 1) were carried on considering the fact that the proposed models potentially require more layer to achieve competitor quantitative performances on tasks (Table 1). This fact potentially skews the results if competitors require fewer layers for comparable performance.\n\n*Limited Dataset Representativeness:* The model's evaluation primarily relies on synthetic, simplified datasets, which are not representative of those used in current literature benchmarks (see Section 7/Table 4 in  [3]). This limits insights into the model’s scalability and ability to handle real-world data complexities (and it is not clear whether this was caused by memory requirements drawbacks). Moreover, the removal of hidden state dependencies from gates raises concerns regarding the model ability to preserve long range dependendencies - something that should be investigated. \n\n*Literature Comparisons and references:* The paper references but does not thoroughly compare against xLSTM, a similar recent work aimed at improving LSTM performance. A more detailed theoretical (at least) comparison with xLSTM would strengthen the paper's argument for minLSTM's contributions. Moreover, the paper is inspired and puts emphasis (even in the title) on the recent **resurgence** of RNNs - a concept that was highligthed by recent surveys [2,3] that could help the reader better contextualize the work.  In the aforementioned surveys, other very related works [4,5] are described that should be at least mentioned \n\n[1] Beck, Maximilian, et al. \"xLSTM: Extended Long Short-Term Memory.\" arXiv preprint arXiv:2405.04517 (2024).\n\n[2] Tiezzi, Matteo, et al. \"On the resurgence of recurrent models for long sequences: Survey and research opportunities in the transformer era.\" arXiv preprint arXiv:2402.08132 (2024).\n\n[3] Tiezzi, Matteo, et al. \"State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era.\" arXiv preprint arXiv:2406.09062 (2024).\n\n[4] Bradbury, James, et al. \"Quasi-recurrent neural networks.\" arXiv preprint arXiv:1611.01576 (2016).\n\n[5] Martin, Eric, and Chris Cundy. \"Parallelizing linear recurrent neural nets over sequence length.\" arXiv preprint arXiv:1709.04057 (2017)."}},"nonreaders":[],"tmdate":1732578817288,"tcdate":1730403619745,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8241/Reviewer_aGUN"],"signatures":["ICLR.cc/2025/Conference/Submission8241/Reviewer_aGUN"],"forum":"GrmFFxGnOR","number":3,"license":"CC BY 4.0","cdate":1730403619745,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8241/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732578817288,"domain":"ICLR.cc/2025/Conference","replyto":"GrmFFxGnOR","id":"AVNRzxZT49","forumContent":{"TLDR":{"value":"Revisiting decade-old RNNs (LSTMs and GRUs), we introduce minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters, (2) are parallelizable during training, and (3) match the performance of recent sequence models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Recurrent Neural Networks","RNNs","Sequence Modelling","Efficiency","LSTMs","GRUs","Parallel Scan"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers—particularly with respect to sequence length—have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively.\nIn this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers."},"_bibtex":{"value":"@misc{\nfeng2025were,\ntitle={Were {RNN}s All We Needed?},\nauthor={Leo Feng and Frederick Tung and Mohamed Osama Ahmed and Yoshua Bengio and Hossein Hajimirsadeghi},\nyear={2025},\nurl={https://openreview.net/forum?id=GrmFFxGnOR}\n}"},"title":{"value":"Were RNNs All We Needed?"},"pdf":{"value":"/pdf/a56137c0e88e5c5b4003b438b48e61ee149bd102.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"feng|were_rnns_all_we_needed"},"authorids":{"value":["~Leo_Feng1","~Frederick_Tung1","~Mohamed_Osama_Ahmed2","~Yoshua_Bengio1","~Hossein_Hajimirsadeghi1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Leo Feng","Frederick Tung","Mohamed Osama Ahmed","Yoshua Bengio","Hossein Hajimirsadeghi"]}},"version":2},{"content":{"venue":{"value":"ICDM (Workshops) 2025"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/11415623/11415713/11415909.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"hong|an_efficient_selfsupervised_framework_for_longsequence_eeg_modeling"},"html":{"value":"https://doi.org/10.1109/ICDMW69685.2025.00039"},"_bibtex":{"value":"@inproceedings{DBLP:conf/icdm/HongMG25,\n  author={Jiazhen Hong and Geoffrey Mackellar and Soheila Ghane},\n  title={An Efficient Self-Supervised Framework for Long-Sequence EEG Modeling},\n  year={2025},\n  cdate={1735689600000},\n  pages={285-292},\n  url={https://doi.org/10.1109/ICDMW69685.2025.00039},\n  booktitle={ICDM (Workshops)},\n  crossref={conf/icdm/2025w}\n}\n"},"abstract":{"value":"Electroencephalogram (EEG) signals suffer from low signal-to-noise ratio (SNR) and high inter-subject variability, making generalization challenging. While Transformer-based self-supervised methods have advanced EEG representation learning, their quadratic complexity limits efficiency in long-range modeling. We propose EEGM2, a self-supervised framework with a U-shaped encoder-decoder integrated with Mamba-2, achieving linear complexity, reduced memory usage, and faster inference. Moreover, the selective information propagation mechanism of Mamba-2 enables the model to effectively capture long-range dependencies, and a joint L1-spectral loss preserves both temporal dynamics and spectral characteristics. Experiments show that EEGM2 achieves state-of-the-art performance in short- and long-sequence modeling, with strong cross-subject generalization and cross-domain transferability. EEGM2 thus provides an efficient, scalable solution suitable for resource-constrained BCI devices. Models and code are available at: https://github.com/Emotiv/EEGM2-ICDM2025"},"title":{"value":"An Efficient Self-Supervised Framework for Long-Sequence EEG Modeling"},"authors":{"value":[{"fullname":"Jiazhen Hong","username":""},{"fullname":"Geoffrey Mackellar","username":""},{"fullname":"Soheila Ghane","username":""}]}},"tmdate":1778193775901,"pdate":1767139200000,"externalIds":["dblp:conf/icdm/HongMG25"],"tcdate":1778193597819,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jiazhen_Hong3"],"forum":"GoFlM3oiyF","license":"CC BY-SA 4.0","number":14257,"cdate":1735689600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Author_Removal"],"mdate":1778193775901,"domain":"OpenReview.net/Public_Article","id":"GoFlM3oiyF","version":2},{"content":{"venue":{"value":"2025 IEEE International Conference on Data Mining Workshops (ICDMW)"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"hong|an_efficient_selfsupervised_framework_for_longsequence_eeg_modeling"},"authorids":{"value":["~Jiazhen_Hong3","~Geoffrey_Mackellar1","~Soheila_Ghane1"]},"html":{"value":"https://ieeexplore.ieee.org/document/11415909"},"abstract":{"value":"Electroencephalogram (EEG) signals suffer from low signal-to-noise ratio (SNR) and high inter-subject variability, making generalization challenging. While Transformer-based self-supervised methods have advanced EEG representation learning, their quadratic complexity limits efficiency in long-range modeling. We propose EEGM2, a self-supervised framework with a U-shaped encoder-decoder integrated with Mamba-2, achieving linear complexity, reduced memory usage, and faster inference. Moreover, the selective information propagation mechanism of Mamba-2 enables the model to effectively capture long-range dependencies, and a joint L1-spectral loss preserves both temporal dynamics and spectral characteristics. Experiments show that EEGM2 achieves state-of-the-art performance in short- and long-sequence modeling, with strong cross-subject generalization and cross-domain transferability. EEGM2 thus provides an efficient, scalable solution suitable for resource-constrained BCI devices. Models and code are available at: https://github.com/Emotiv/EEGM2-ICDM2025"},"title":{"value":"An Efficient Self-Supervised Framework for Long-Sequence EEG Modeling"},"authors":{"value":["Jiazhen Hong","Geoffrey Mackellar","Soheila Ghane"]}},"tmdate":1778191302553,"pdate":1773115200000,"tcdate":1778191302553,"writers":["~Jiazhen_Hong3","~Geoffrey_Mackellar1","~Soheila_Ghane1"],"signatures":["~Jiazhen_Hong1"],"forum":"BEN8RzZdXb","license":"CC BY-NC-ND 4.0","number":49812,"cdate":1778191302553,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1778191302553,"domain":"OpenReview.net/Archive","id":"BEN8RzZdXb","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2407.06136v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"li|mambafscil_dynamic_adaptation_with_selective_state_space_model_for_fewshot_classincremental_learning"},"authorids":{"value":["~Xiaojie_Li3","~Yibo_Yang2","~Jianlong_Wu1","~Bernard_Ghanem1","","~Min_Zhang9"]},"html":{"value":"https://doi.org/10.48550/arXiv.2407.06136"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2407-06136,\n  publtype={informal},\n  author={Xiaojie Li and Yibo Yang and Jianlong Wu and Bernard Ghanem and Liqiang Nie and Min Zhang},\n  title={Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2407.06136},\n  url={https://doi.org/10.48550/arXiv.2407.06136}\n}\n"},"abstract":{"value":"Few-shot class-incremental learning (FSCIL) confronts the challenge of integrating new classes into a model with minimal training samples while preserving the knowledge of previously learned classes. Traditional methods widely adopt static adaptation relying on a fixed parameter space to learn from data that arrive sequentially, prone to overfitting to the current session. Existing dynamic strategies require the expansion of the parameter space continually, leading to increased complexity. In this study, we explore the potential of Selective State Space Models (SSMs) for FSCIL, leveraging its dynamic weights and strong ability in sequence modeling to address these challenges. Concretely, we propose a dual selective SSM projector that dynamically adjusts the projection parameters based on the intermediate features for dynamic adaptation. The dual design enables the model to maintain the robust features of base classes, while adaptively learning distinctive feature shifts for novel classes. Additionally, we develop a class-sensitive selective scan mechanism to guide dynamic adaptation. It minimizes the disruption to base-class representations caused by training on novel data, and meanwhile, forces the selective scan to perform in distinct patterns between base and novel classes. Experiments on miniImageNet, CUB-200, and CIFAR-100 demonstrate that our framework outperforms the existing state-of-the-art methods. The code is available at \\url{https://github.com/xiaojieli0903/Mamba-FSCIL}."},"title":{"value":"Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning"},"authors":{"value":["Xiaojie Li","Yibo Yang","Jianlong Wu","Bernard Ghanem","Liqiang Nie","Min Zhang"]}},"tmdate":1774412141826,"pdate":1704067200000,"tcdate":1731007036301,"writers":["~"],"signatures":["~Yibo_Yang2"],"forum":"1cg5ibSBBO","license":"CC BY-SA 4.0","number":174084,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1774412141826,"domain":"DBLP.org","id":"1cg5ibSBBO","version":2},{"content":{"TLDR":{"value":"Inspired by Ebbinghaus' forgetting curve, we propose a continuous state space model for dynamic graph modeling and achieve state-of-the-art performance."},"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["Temporal Graph","Link Prediction","Long-term Sequence Modeling"]},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Dynamic graph modeling aims to uncover evolutionary patterns in real-world systems, enabling accurate social recommendation and early detection of cancer cells. Inspired by the success of recent state space models in efficiently capturing long-term dependencies, we propose DyG-Mamba by translating dynamic graph modeling into a long-term sequence modeling problem. Specifically, inspired by Ebbinghaus' forgetting curve, we treat the irregular timespans between events as control signals, allowing DyG-Mamba to dynamically adjust the forgetting of historical information. This mechanism ensures effective usage of irregular timespans, thereby improving both model effectiveness and inductive capability. In addition, inspired by Ebbinghaus' review cycle, we redefine core parameters to ensure that DyG-Mamba selectively reviews historical information and filters out noisy inputs, further enhancing the model’s robustness. Through exhaustive experiments on 12 datasets covering dynamic link prediction and node classification tasks, we show that DyG-Mamba achieves state-of-the-art performance on most datasets, while demonstrating significantly improved computational and memory efficiency. Our code is available at https://github.com/Clearloveyuan/DyG-Mamba."},"_bibtex":{"value":"@inproceedings{\nli2025dygmamba,\ntitle={DyG-Mamba: Continuous State Space Modeling on Dynamic Graphs},\nauthor={Dongyuan Li and Shiyin Tan and Ying Zhang and Ming Jin and Shirui Pan and Manabu Okumura and Renhe Jiang},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=ja2wA4UncJ}\n}"},"title":{"value":"DyG-Mamba: Continuous State Space Modeling on Dynamic Graphs"},"pdf":{"value":"/pdf/9140326d07557c09bd27ce8451624ceec0bfd37c.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"li|dygmamba_continuous_state_space_modeling_on_dynamic_graphs"},"authorids":{"value":["~Dongyuan_Li1","~Shiyin_Tan1","~Ying_Zhang16","~Ming_Jin3","~Shirui_Pan1","~Manabu_Okumura2","~Renhe_Jiang1"]},"authors":{"value":["Dongyuan Li","Shiyin Tan","Ying Zhang","Ming Jin","Shirui Pan","Manabu Okumura","Renhe Jiang"]}},"tmdate":1783627389262,"pdate":1758216896963,"tcdate":1746782000248,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission12036/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission12036/Authors"],"forum":"ja2wA4UncJ","license":"CC BY 4.0","number":12036,"cdate":1746782000248,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission12036/-/Full_Submission","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission12036/-/Camera_Ready_Revision"],"mdate":1783627389262,"odate":1761704868377,"domain":"NeurIPS.cc/2025/Conference","id":"ja2wA4UncJ","version":2},{"content":{"summary":{"value":"This paper introduces SparseSSM, a training-free pruning framework that extends the Optimal Brain Surgeon (OBS) methodology to Mamba-based state-space models (SSMs). The authors derive an approximate Hessian for the time-shared, discretized state-transition matrix 𝐴 log , propose a power-law temporal aggregation to merge per-step importance scores, and add a component-wise sensitivity analysis to guide FFN pruning. Experiments on multiple Mamba variants (130 M – 2.8 B params) show ≈ 50 % sparsity with minimal degradation on language-modeling and zero-shot tasks, with limited but measurable speedups."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. What theoretical or empirical reasoning supports the power-law aggregation form? Have exponential or learned schemes been tested?\n\n2. Why does performance degrade for large (2.8 B +) models—fundamental to one-shot OBS or fixable with design changes? \n\n3.  To the best of my understanding, \"Efficient Unstructured Pruning of Mamba SSMs,(arXiv 2505.08299) showed better results on WIKItext for the 50% case? Would it be possible to at least pick wikitext and one more dataset from that to show how your work stands with this paper? This is an accepted paper from EMNLP 25. \n\n4. Have you examined eigenvalue distributions pre-/post-pruning to confirm stability of the selective SSM dynamics?\n\n.5. Do you plan to include Long Range Arena or other long-context benchmarks to highlight SSM advantages? It is not mendatory but I would appreciate.  \n\n6. What value of the exponent p is used, and how sensitive are the results to its tuning?\n\nIf my questions/ concerns/limitations are properly addressed, I am happy to increase my score. I appreciate the authors for their wonderful work!"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Novel technical adaptation – Thought OBS is known; however, this work extends second-order (OBS) pruning to time-shared, discretized SSM parameters, a non-trivial advance beyond Transformer-focused methods.\n\n2. Mathematical rigor – Theorem 1 and Appendix A give a clear Hessian-trace derivation consistent with the Gauss–Newton assumption.\n\n3. Comprehensive experiments – Multiple model sizes, datasets, and sparsity regimes, including unstructured/structured variants.\n\n4. Practical value – Training-free, minimal compute overhead, and modest speed/memory savings useful for deployment.\n\n5. Clear exposition – Strong alignment between problem motivation and architectural properties of Mamba."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Temporal aggregation lacks theoretical grounding – The power-law choice is ad hoc; no comparison with exponential or learned weighting.\n\n2. Scalability and robustness – Performance drops at larger model scales; no analysis of failure modes or eigenvalue stability.\n\n3. Limited comparative context – No direct cost-benefit discussion versus iterative training-based pruning, achieving higher compression.\n\n4. Modest hardware validation – Reported 1.05–1.09 × parallel speedup lacks wall-clock analysis on target devices.\n\n5. Missing benchmarks – No Long Range Arena or robustness tests; would strengthen claims for long-sequence modeling.\n."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915756505,"tcdate":1761602160012,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1381/Reviewer_ef7V"],"signatures":["ICLR.cc/2026/Conference/Submission1381/Reviewer_ef7V"],"forum":"bzb9OJVfCP","number":2,"license":"CC BY 4.0","cdate":1761602160012,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1381/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915756505,"domain":"ICLR.cc/2026/Conference","replyto":"bzb9OJVfCP","id":"oIuPDpwFfa","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We introduce an OBS–based pruning methodology tailored to SSM-based architectures such as Mamba, effectively reducing redundancy while preserving model performance."},"keywords":{"value":["Neural network pruning","Mamba","Hessian-based pruning"]},"supplementary_material":{"value":"/attachment/6b5f630754dd0367304e126456137bc23a85f5bf.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"State-space language models such as Mamba match Transformer quality while permitting linear complexity inference, yet still comprise billions of parameters that hinder deployment. Existing one-shot pruning methods are tailored to attention blocks and fail to account for the time-shared and discretized state-transition matrix at the heart of the selective state-space module (SSM). In this paper, we introduce SparseSSM, the first training-free pruning framework that extends the classic optimal brain surgeon (OBS) framework to state space architectures. Our layer-wise algorithm (i) derives an approximate second-order saliency score that aggregates Hessian-trace information across time steps, (ii) incorporates a component sensitivity analysis to guide feed-forward network (FFN) pruning, which also sheds light on where redundancy resides in mamba architecture, (iii) can be easily extended to semi-structured and structured sparsity. Empirically, we prune 50% of SSM weights without fine-tuning and observe no zero-shot accuracy loss, achieving the current state-of-the-art pruning algorithm for Mamba-based LLMs."},"_bibtex":{"value":"@misc{\ntuo2026sparsessm,\ntitle={Sparse{SSM}: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot},\nauthor={Kaiwen Tuo and Huan Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=bzb9OJVfCP}\n}"},"title":{"value":"SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot"},"pdf":{"value":"/pdf/8e219971d001aa872a5a0e48ae02be58f8abb56b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"tuo|sparsessm_efficient_selective_structured_state_space_models_can_be_pruned_in_oneshot"},"authorids":{"value":["~Kaiwen_Tuo1","~Huan_Wang3"]},"authors":{"value":["Kaiwen Tuo","Huan Wang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2404.11778v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"deng|cumamba_selective_state_space_models_with_channel_learning_for_image_restoration"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Rui_Deng:","~Tianpei_Gu1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2404.11778"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2404-11778,\n  publtype={informal},\n  author={Rui Deng and Tianpei Gu},\n  title={CU-Mamba: Selective State Space Models with Channel Learning for Image Restoration},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2404.11778},\n  url={https://doi.org/10.48550/arXiv.2404.11778}\n}\n"},"abstract":{"value":"Reconstructing degraded images is a critical task in image processing. Although CNN and Transformer-based models are prevalent in this field, they exhibit inherent limitations, such as inadequate long-range dependency modeling and high computational costs. To overcome these issues, we introduce the Channel-Aware U-Shaped Mamba (CU-Mamba) model, which incorporates a dual State Space Model (SSM) framework into the U-Net architecture. CU-Mamba employs a Spatial SSM module for global context encoding and a Channel SSM component to preserve channel correlation features, both in linear computational complexity relative to the feature map size. Extensive experimental results validate CU-Mamba's superiority over existing state-of-the-art methods, underscoring the importance of integrating both spatial and channel contexts in image restoration."},"title":{"value":"CU-Mamba: Selective State Space Models with Channel Learning for Image Restoration"},"authors":{"value":["Rui Deng","Tianpei Gu"]}},"tmdate":1717613642645,"pdate":1704067200000,"tcdate":1717613634130,"writers":["~"],"signatures":["~Tianpei_Gu1"],"forum":"6DTYeUx5bn","license":"CC BY-SA 4.0","number":24273,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1717613642645,"domain":"DBLP.org","id":"6DTYeUx5bn","version":2},{"content":{"summary":{"value":"The paper proposes BioSASANet, a self-interpretable approach for genomic sequence modeling that extends the Shapley Additive Self-Attribution (SASANet) framework by incorporating Mamba/MambaDNA modules to capture long-range nucleotide interactions and reverse-complementary structure, achieving competitive predictive accuracy and more biologically faithful, nucleotide-level attributions on histone-mark and plant promoter datasets."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- Can the authors test BioSASANet on additional genomics tasks or use in silico mutagenesis to validate attribution causality beyond MP@k?\n\n- Can the authors provide ablations for $\\lambda_s$ and $\\lambda_v$, or provide discussion on this?\n\n- Can the authors add runtime and/or memory comparisons to support efficiency claims?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- Well-grounded theoretical foundation:\nBuilds directly on the Shapley Additive Self-Attribution (SASA) framework, preserving provable convergence to true Shapley values while adapting it to biological sequence data.While SASANet (Sun et al., 2025) introduced the general Shapley Additive Self-Attribution framework, BioSASANet extends it into a biologically specialized regime. It integrates reverse complementarity (RC)-aware modeling, Mamba/MambaDNA-based long-range sequence encoding, and nucleotide-level Shapley convergence within DNA sequences. \n\n- Biologically informed architecture:\nIncorporates the NMCSM and MambaDNA modules, enabling modeling of long-range dependencies and bidirectional RC equivariance specific to DNA. This directly addresses the limitations of SASANet’s generic sequential encoder and grounds the interpretability mechanism in actual biological constraints.\n\n- Faithful and interpretable results:\nConsistently surpasses post-hoc baselines (DeepLIFT, GradientSHAP, LIME) on MP@k metrics and visually aligns high-attribution positions with true promoter or histone-mark regions.\n\n- Architectural flexibility:\nDemonstrates compatibility with multiple backbone DNA LMs, showing potential generality across future genomics models.\n\n- Clarity and presentation quality:\nThe paper maintains a clear narrative flow from theoretical background to biological application, supported by equations, figures, pseudocode, parameter tables.\n\n- Significance and potential impact:\nThe work aims to connect model interpretability with biological understanding."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Limited in evaluation:\nCurrent experiments focus only on histone-mark and promoter tasks. Extended evaluation to additional genomic contexts/tasks )(e.g., enhancer prediction, variant effect modeling, cross-species generalization etc.) would demonstrate robustness and transferability.\n\n- Limitation in the attribution validation approach:\nMP@k captures overlap with known motifs but does not directly ensure causality (biological correctness). One possible way to  to test attribution stability would be to conduct in silico mutagenesis. This is quite easy to conduct with even with limited comptational budget.\n\n- Ablation missing:\nWhile loss balancing hyperparameteres $\\lambda_s$ and $\\lambda_v$ are mentioned, their individual effects are not clear. I would suggest authors add ablations isolating each component to clarify their contributions\n\n- Efficiency and scalability claims not empirically verified:\nThe paper cites Mamba’s linear complexity, however actual runtime and memory analysis (especially in genomic context) is missing.\n\n- Figure + reference missing (line 151)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762920795462,"tcdate":1761984645700,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission9091/Reviewer_cZLE"],"signatures":["ICLR.cc/2026/Conference/Submission9091/Reviewer_cZLE"],"forum":"jT8djAd7T4","number":3,"license":"CC BY 4.0","cdate":1761984645700,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission9091/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762920795462,"domain":"ICLR.cc/2026/Conference","replyto":"jT8djAd7T4","id":"vDLGIn8gkB","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Interpretable Machine Learning","Biological Sequence Analysis","Applications in Biology"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"abstract":{"value":"Deep learning models have significantly advanced genomics sequence modeling by uncovering complex patterns, but their black-box nature limits the biological insights that can be derived from their predictions. While recent efforts have employed post-hoc methods to identify important nucleotides or motifs, these approaches are decoupled from the prediction process and often suffer from accuracy limitations. In this paper, we introduce the Shapley Additive Self-Attribution (SASA) framework to genomics sequence modeling and propose BioSASANet. BioSASANet integrates a marginal contribution-based sequential module that captures long-range nucleotide interactions, and a positional Shapley value module that explicitly models the reverse complementarity (RC) property of genomic sequences, enabling position-aware, biologically grounded self-attribution. Experiments on two genomics tasks show that BioSASANet achieves accurate predictive performance while providing more faithful nucleotide-level attributions. Additionally, its flexible design allows integration with state-of-the-art DNA language model backbones, enabling advanced models with Shapley value-based self-interpretation."},"_bibtex":{"value":"@misc{\ncui2026biosasanet,\ntitle={Bio{SASAN}et: Self-Interpretable Shapley Attribution for Deep Genomics Sequence Modeling},\nauthor={Shuting Cui and Ying Sun and Yuting Zhang and Hengshu Zhu},\nyear={2026},\nurl={https://openreview.net/forum?id=jT8djAd7T4}\n}"},"title":{"value":"BioSASANet: Self-Interpretable Shapley Attribution for Deep Genomics Sequence Modeling"},"pdf":{"value":"/pdf/3f6a731b512cae4f4242793705a026451cc99e33.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cui|biosasanet_selfinterpretable_shapley_attribution_for_deep_genomics_sequence_modeling"},"authorids":{"value":["~Shuting_Cui1","~Ying_Sun4","~Yuting_Zhang5","~Hengshu_Zhu1"]},"authors":{"value":["Shuting Cui","Ying Sun","Yuting Zhang","Hengshu Zhu"]}},"version":2},{"content":{"summary":{"value":"This paper focuses on addressing the core tradeoff between parallelizable training and efficient sequential inference in neural sequence models. It formalizes the Sequential-Parallel Duality (SPD), a property where models support near-constant-depth parallel training and linear-time/constant-space sequential inference, and introduces a generalized model class called Prefix-Scannable Models (PSMs) to unify and extend existing efficient architectures."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. The paper uses the Blelloch scan’s fixed binary-tree parenthesisation for non-associative aggregators (e.g., softmax attention). Is there a theoretical or empirical justification for this specific parenthesisation over alternative groupings (e.g., left-to-right sequential, random binary trees)? Could different parenthesisations impact task performance or complexity?\n\n2. For the Multi Query Associative Recall (MQAR) task, Transformer-PSM with chunk size 64 achieves perfect accuracy, but chunk size 32 degrades performance on long sequences (512 tokens). What causes this sensitivity to chunk size, and is there a principled way to select chunk size for different tasks without grid search?\n\n3. The paper compares Transformer-PSM to GPT-2, Mamba, and Sliding Window Transformer (SWT). Why were other efficient models (e.g., RetNet, GLA, xLSTM’s mLSTM) not included in the comparisons, especially for language modeling or associative recall? How would Transformer-PSM perform against these models on the same benchmarks?\n\n4. The paper claims O(log n) inference memory for PSMs, but chunk size (c) interacts with this scaling (Appendix C). For large chunk sizes (e.g., c=256), does the memory scaling still hold, or does the chunk-local KV cache in the Inf module dominate memory usage?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper demonstrates originality through three key contributions. First, it introduces a unified theoretical framework (Prefix-Scannable Models, PSMs) that formalizes sequential-parallel duality (SPD), a core property of modern efficient sequence models (e.g., Mamba, GLA, RetNet). This unification resolves the fragmentation of prior work by showing that diverse architectures are special cases of PSMs with associative aggregation operators. Second, it generalizes PSMs to non-associative aggregation functions (e.g., softmax attention), expanding the model design space beyond existing affine/linear constraints while retaining efficient inference. Third, the instantiation of Transformer-PSM is a creative combination: it merges Transformer-style self-attention (for expressive power) with PSM’s linear-time inference (for efficiency), addressing a long-standing tradeoff between performance and scalability.\n\n2. It provides strict formal definitions (e.g., SPD, state kernel, PSM) and rigorous proofs (e.g., Proposition 3.2, Theorem 3.5) for complexity guarantees (O(n) training compute, O(log n) inference memory). Appendixes supplement detailed derivations (e.g., affine recurrence proofs, computational complexity analysis) to support core claims. Empirically, the experiment design is comprehensive and controlled: it targets three canonical tasks (state tracking S₅, multi-query associative recall, WikiText-103 language modeling) and compares against strong baselines (GPT-2, Mamba, Sliding Window Transformer). Results are interpretable (e.g., Fig. 3 shows superior length generalization; Fig. 6 validates constant inference latency) and reproducible (detailed training protocols, public code plan).\n\n3. Theoretically, it deepens the connection between parallel prefix-scan algorithms and neural sequence modeling, providing a general framework for analyzing and designing efficient models—this will guide future research on SPD-compatible architectures (e.g., log-linear attention, as noted in the discussion). Practically, it addresses critical limitations of existing models: Transformer-PSM matches Transformer’s expressive power (e.g., perfect associative recall) while achieving Mamba-like linear inference efficiency (e.g., constant per-token latency for long sequences). This balance is transformative for real-world applications (e.g., long-document processing, streaming NLP) where both training parallelism and inference scalability are essential. Additionally, the paper’s length generalization results (e.g., S₅ task scaling to 160+ tokens) open new avenues for modeling algorithmic tasks that require precise state tracking."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper’s core contribution of generalizing PSMs to non-associative aggregation operators is underdeveloped, as it only instantiates `Aggθ` with Transformer-style softmax attention (Transformer-PSM). The design space of non-associative operators (e.g., additive attention variants, kernelized attention, or task-specific non-associative functions) remains unexplored. This raises critical questions: Are PSM’s advantages (e.g., length generalization on `S5` tasks) inherent to the prefix-scan framework, or specific to softmax attention? Without testing alternative non-associative `Aggθ` implementations, the paper fails to validate that PSMs truly expand the efficient sequence model design space beyond softmax-based variants.\n\n2. The paper demonstrates that chunk size (`c`) significantly impacts performance (e.g., Transformer-PSM with `c=64` achieves perfect accuracy on MQAR, while `c=32` degrades on long sequences; Fig. 4) and perplexity (Fig. 5). The paper does not quantify the tradeoff between chunk size, computation cost (e.g., `O(c²)` for chunk-local self-attention), and memory usage (e.g., `O(c log(n/c))`). How does `c` scale with sequence length (e.g., optimal `c` for 1k vs. 10k token sequences)?\n\n\n3. Missing Ablation Studies on Core PSM Components. \nThe PSM framework is defined by three modules: `Enc` (chunk encoder), `Aggθ` (aggregator), and `Infφ` (inference head). However, the paper does not perform component-wise ablation studies to isolate which modules drive PSM’s performance advantages. Does the simple embedding `Enc` (used in Transformer-PSM) limit performance, or is a more complex encoder (e.g., convolutional, hierarchical) unnecessary?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924584470,"tcdate":1761958397637,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14117/Reviewer_LR1S"],"signatures":["ICLR.cc/2026/Conference/Submission14117/Reviewer_LR1S"],"forum":"tuLF84azND","number":3,"license":"CC BY 4.0","cdate":1761958397637,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14117/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924584470,"domain":"ICLR.cc/2026/Conference","replyto":"tuLF84azND","id":"rLiQZl6UBx","forumContent":{"TLDR":{"value":"Propose new architecture Transformer-PSM that generalizes state space models to use softmax attention"},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["state space models","linear RNNs","linear transformers","sequence modeling"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Modern neural sequence models are designed to meet the dual mandate of parallelizable training and fast sequential inference. Recent developments have given rise to various models, such as Gated Linear Attention (GLA) and Mamba, that achieve such ``sequential-parallel duality.'' This raises a natural question: can we characterize the full class of neural sequence models that support near-constant-time parallel evaluation and linear-time, constant-space sequential inference? We begin by describing a broad class of such models -- state space models -- as those whose state updates can be computed using the classic parallel prefix scan algorithm with a custom associative aggregation operator. We then define a more general class, Prefix-Scannable Models (PSMs), by relaxing the state aggregation operator to allow arbitrary (potentially non-associative) functions such as softmax attention. This generalization unifies many existing architectures, including element-wise RNNs (e.g., Mamba) and linear transformers (e.g., GLA, Mamba2, mLSTM), while also introducing new models with softmax-like operators that achieve O(1) amortized compute per token and log(N) memory for sequence length N. We empirically evaluate such models on illustrative small-scale language modeling and canonical synthetic tasks, including state tracking and associative recall. Empirically, we find that PSMs retain the expressivity of transformer-based architectures while matching the inference efficiency of state space models -- in some cases exhibiting better length generalization than either."},"_bibtex":{"value":"@inproceedings{\nyau2026sequential,\ntitle={Sequential Parallel Duality in Prefix Scannable Models},\nauthor={Morris Yau and Sharut Gupta and Valerie Engelmayer and Kazuki Irie and Stefanie Jegelka and Jacob Andreas},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=tuLF84azND}\n}"},"title":{"value":"Sequential Parallel Duality in Prefix Scannable Models"},"pdf":{"value":"/pdf/f5eb77bbc97daf92bd3d76db45182ed7855ed21b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"yau|sequential_parallel_duality_in_prefix_scannable_models"},"authorids":{"value":["~Morris_Yau3","~Sharut_Gupta1","~Valerie_Engelmayer1","~Kazuki_Irie1","~Stefanie_Jegelka3","~Jacob_Andreas1"]},"authors":{"value":["Morris Yau","Sharut Gupta","Valerie Engelmayer","Kazuki Irie","Stefanie Jegelka","Jacob Andreas"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a novel continuous-time dynamic graphs (CTDGs) representation model named DyGMamba to address the challenges of learning long-term temporal dependencies. DyGMemba consists of a node-level state space model (SSM) and edge-level SSM, which aims to learn the node-level and edge-level representations, respectively. The outputs of node-level SSM and edge-level SSM are combined for future link prediction. The authors conduct extensive experiments to evaluate DyGMamba."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"See the weakness."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"1. The experiments are extensive."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The novelty of this work is low. Some important modules are highly similar to existing works. In specific: \n   - In Section 3.1,  The \"Encode Neighbor Features\" and \" Patching Neighbors\" sections are highly similar to DyGFormer [1]. \n   - The main network architecture is taken from original Mamba [2], without incorporating structure information of dynamic graph (see Eq. 5, 6 and 7).  \n\n2. The authors claim that applying Mamba for dynamic graph learning is to address the challenge of computation complexity in capturing very long-term temporal dependencies. This raises several concerns: \n\n   - Why should we learn from very long history? Intuitively, very long history should be discarded since it has minor impact on current event. In addition, in Fig 3 (a), the performance of DyGFormer and variant A is not increasing as sequence length increases. This supports that the model performance does not necessarily increase as sequence length increases.\n\n   - In addition, I think DyGFormer has address the problem of learning long-term dependencies by its patching technique. It can learn very long history if its patch size is large enough. \n\n3. What is \"critical temporal information\" mentioned in Line 54? How to define it? Is there evidence that DyGMamba can indeed capture this?  \n\n4. The performance improvement of DyGMamba is marginal (Table 1 and 2). Most improvements are within 0.5\\%.\n \n5. In Fig.3, why the experiments are only conducted on Enron? More datasets should be included.  \n\n[1] Yu L, Sun L, Du B, et al. Towards better dynamic graph learning: New architecture and unified library[J]. Advances in Neural Information Processing Systems, 2023, 36: 67686-67700. \n\n[2] Gu A, Dao T. Mamba: Linear-time sequence modeling with selective state spaces[J]. arXiv preprint arXiv:2312.00752, 2023."}},"nonreaders":[],"tmdate":1731428326066,"tcdate":1730709764071,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5146/Reviewer_TJxG"],"signatures":["ICLR.cc/2025/Conference/Submission5146/Reviewer_TJxG"],"forum":"f3gCs2a4ZD","number":4,"license":"CC BY 4.0","cdate":1730709764071,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5146/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428326066,"domain":"ICLR.cc/2025/Conference","replyto":"f3gCs2a4ZD","id":"9Tw9Ir69zN","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["dynamic graph","state space model"]},"supplementary_material":{"value":"/attachment/2dc1a3502a1d53b635f3c846cad8277b987ac60b.zip"},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Learning useful representations for continuous-time dynamic graphs (CTDGs) is challenging, due to the concurrent need to span long node interaction histories and grasp nuanced temporal details. In particular, two problems emerge: (1) Encoding longer histories requires more computational resources, making it crucial for CTDG models to maintain low computational complexity to ensure efficiency; (2) Meanwhile, more powerful models are needed to identify and select the most critical temporal information within the extended context provided by longer histories. To address these problems, we propose a CTDG representation learning model named DyGMamba, originating from the popular Mamba state space model (SSM). DyGMamba first leverages a node-level SSM to encode the sequence of historical node interactions. Another time-level SSM is then employed to exploit the temporal patterns hidden in the historical graph, where its output is used to dynamically select the critical information from the interaction history. We validate DyGMamba experimentally on the dynamic link prediction task. The results show that our model achieves state-of-the-art in most cases. DyGMamba also maintains high efficiency in terms of computational resources, making it possible to capture long temporal dependencies with a limited computation budget."},"_bibtex":{"value":"@misc{\nding2025dygmamba,\ntitle={Dy{GM}amba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models},\nauthor={Zifeng Ding and Yifeng Li and Yuan He and Antonio Norelli and Jingcheng Wu and Volker Tresp and Yunpu Ma and Michael M. Bronstein},\nyear={2025},\nurl={https://openreview.net/forum?id=f3gCs2a4ZD}\n}"},"title":{"value":"DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models"},"pdf":{"value":"/pdf/0921c754d017d250e0d2f4b59a779766e47b30a6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ding|dygmamba_efficiently_modeling_longterm_temporal_dependency_on_continuoustime_dynamic_graphs_with_state_space_models"},"authorids":{"value":["~Zifeng_Ding1","~Yifeng_Li4","~Yuan_He5","~Antonio_Norelli2","~Jingcheng_Wu3","~Volker_Tresp1","~Yunpu_Ma1","~Michael_M._Bronstein1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zifeng Ding","Yifeng Li","Yuan He","Antonio Norelli","Jingcheng Wu","Volker Tresp","Yunpu Ma","Michael M. Bronstein"]}},"version":2},{"content":{"summary":{"value":"### ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction\n\n#### Main Research Problem\n\n* **Embodied Cognition (EC)**: The theory posits that intelligence arises from continuous sensorimotor interaction with the world.\n* **VLM Limitation**: Modern Vision-Language Models (VLMs) are trained **disembodiedly** on vast static datasets.\n* **Core Question**: **To what extent does embodied cognition emerge** in these VLMs despite their non-interactive training, specifically their ability to understand the **consequences** of actions and the dynamics of the environment?\n\n#### Research Methodology (The ENACT Benchmark)\n\n* **Framework**: Introduced the **ENACT** benchmark, grounded in a Partially Observable Markov Decision Process (POMDP) framework, to evaluate EC through **egocentric** (first-person) interaction world modeling.\n* **Task Types**: Two complementary **sequence reordering** tasks test a model's understanding of how the world evolves with its actions:\n    1.  **Forward World Modeling**: Given a sequence of abstract actions, the model must predict and correctly order the sequence of **future states**.\n    2.  **Inverse World Modeling**: Given a sequence of state changes (observations), the model must infer and correctly order the corresponding **actions**.\n* **Data**: A scalable dataset of 8,972 QA pairs derived from diverse, long-horizon household activities in the BEHAVIOR simulator.\n\n#### Key Experimental Findings\n\n1.  **Significant Gap**: State-of-the-art VLMs (including GPT-5) show a **substantial performance gap** compared to human performance.\n2.  **Horizon Failure**: Model accuracy **drops sharply** as the interaction time horizon (number of steps) increases, indicating a lack of robust, **long-term** understanding of action consequences.\n3.  **Embodied Biases**: VLMs exhibit distinct embodied biases, such as a preference for recognizing **right-side actions** and a significant performance drop when camera views deviate from typical human perspectives.\n4.  **Error Analysis**: The primary error modes for top models are **Omission** (failing to recognize a state change that occurred) and **Hallucination** (recognizing a state change that did not occur).\n\n#### Conclusion\n\nThe ENACT benchmark demonstrates that while powerful, current VLMs lack the necessary robust, long-term understanding required for embodied world modeling, indicating that **key elements of embodied intelligence have not fully emerged** from their static training paradigm."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. I still have some doubts about the selection of keyframes. The author emphasized during the construction of the dataset to avoid having repetitive frames and overly similar frames as adjacent keyframes, and proposed a series of methods for this purpose. However, what readers are concerned about is whether it is no longer similar to the previous frame. This is particularly crucial throughout the entire trajectory. In the readers' understanding, a key frame contains sufficient event information. Can the author propose a series of criteria to measure this?\n\n2. In my opinion, this is actually a piece of work that assesses the model's ability to understand the embodied world. Since the author attempts to propose a benchmark or dataset, it seems that more measurement criteria are necessary in the eyes of the readers. The current embodied tasks do not seem to be satisfied with merely evaluating the \"ordering tasks\" of forward and backward modeling. And if such a benchmark only includes these two evaluation tasks, then a series of video generation models such as Sora would be excluded (we also observed this in the experimental part). However, we know that many of the current world models are actually constructed based on image/video generation models. This would significantly limit the widespread use of benchmarks."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"### Main Strengths of the ENACT Paper\n\n* **1. Novelty and Comprehensiveness in Evaluation (Consequence-Aware World Modeling)**\n    * **Summary**: ENACT elevates the evaluation of embodied AI beyond simple perception or isolated interactions to **consequence-aware world modeling** over **extended time horizons**.\n    * **Detail**: It introduces complementary **Forward** (Action $\\rightarrow$ State) and **Inverse** (State $\\rightarrow$ Action) sequence reordering tasks, providing a holistic test of a VLM's ability to reason about the **causal effects** of its actions, unlike prior static or limited benchmarks.\n\n* **2. Focus on Egocentric Interaction and Fidelity (Robust POMDP Framework)**\n    * **Summary**: The benchmark is explicitly designed around **egocentric (first-person)** interaction trajectories within a **Partially Observable Markov Decision Process (POMDP)** framework.\n    * **Detail**: By utilizing data from the **high-fidelity BEHAVIOR simulator**, ENACT ensures the evaluation tasks are realistic and relevant to the challenges faced by embodied agents operating in complex, dynamic household environments.\n\n* **3. Effective Diagnosis of SOTA VLM Failures (Long-Term Reasoning and Biases)**\n    * **Summary**: ENACT provides a crucial diagnostic tool by successfully revealing fundamental weaknesses in the **state-of-the-art VLMs** (e.g., GPT-5) concerning long-term embodied reasoning.\n    * **Detail**: Experimental findings pinpoint specific failure modes—such as a **sharp performance drop** with increasing horizon, and prevalent **Omission** and **Hallucination** errors—which are essential for directing future research efforts in embodied AI development."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"### Main Limitations of the ENACT Benchmark (Weaknesses)\n\n* **1. Reliance on Simulation and the Sim-to-Real Gap**\n    * **Summary**: The entire ENACT dataset and evaluation are restricted to high-fidelity trajectories within the **BEHAVIOR simulator**.\n    * **Detail**: This introduces the inherent **sim-to-real gap** limitation. The model's strong performance (or failures) in the clean, deterministic physics and visuals of the simulation may not directly translate to the stochastic, noisy, and complex dynamics of a real-world robotic agent.\n\n* **2. Abstraction of States and Actions (Missing Low-Level Embodiment)**\n    * **Summary**: The benchmark evaluates understanding based on high-level **symbolic scene graphs** and **abstract action primitives** (e.g., `Open(object)`), rather than raw sensorimotor data.\n    * **Detail**: By removing the necessity to process raw pixels, continuous motor commands, and solve low-level control problems, ENACT fails to test critical aspects of true embodied cognition related to continuous planning and fine-grained sensor processing.\n\n* **3. Limitation to Sequence Reordering (Absence of Generative Modeling)**\n    * **Summary**: The core evaluation is a **sequence reordering task**, which tests recognition and temporal alignment against a limited set of choices.\n    * **Detail**: This task is less demanding than **free-form generative world modeling**, where the VLM would have to *generate* novel, unconstrained future states or action sequences from scratch, which is the ultimate goal of a robust world model.\n\n* **4. Focus on Functional Dynamics (Neglects Fine-Grained Physics)**\n    * **Summary**: The benchmark primarily assesses object-level, functional state changes (e.g., object is open/closed, object is grasped/placed).\n    * **Detail**: It may insufficiently test complex, fine-grained **physical reasoning** such as geometric constraints, stability, momentum, or soft-body dynamics, which are integral to sophisticated manipulation in unstructured environments."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764359526450,"tcdate":1761861574466,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission445/Reviewer_NXiX"],"signatures":["ICLR.cc/2026/Conference/Submission445/Reviewer_NXiX"],"forum":"Patx6MRipw","number":3,"license":"CC BY 4.0","cdate":1761861574466,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission445/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764359526450,"domain":"ICLR.cc/2026/Conference","replyto":"Patx6MRipw","id":"RW62O45Ft3","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce ENACT, which probes embodied cognition in VLMs via egocentric world modeling. Two permutation tasks (forward/inverse) built from scalable simulator data with online verification."},"keywords":{"value":["Embodied Agents","Vision Language Models","Benchmarking","World Modeling"]},"supplementary_material":{"value":"/attachment/5c329bb06f444a57d906394178066d9fa4d5fec3.zip"},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"Embodied cognition argues that intelligence arises from continuous sensorimotor interaction with the world. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? To investigate this, we introduce ENACT, a benchmark that probes this question through world modeling from egocentric interaction. Grounded in a partially observable Markov decision process (POMDP) framework, ENACT comprises two complementary sequence reordering tasks: forward world modeling (predicting an ordered sequence of future states from actions) and inverse world modeling (inferring an ordered sequence of actions from state changes). Correctly solving these tasks indicates that the model has a solid understanding of how the environment will evolve given one's actions. Our scalable dataset contains 8,972 QA pairs derived from diverse, long-horizon household activities in the BEHAVIOR simulator. Experiments reveal a significant performance gap between state-of-the-art VLMs and humans, which widens dramatically as interaction horizons lengthen. We find that models consistently solve the inverse problem better than the forward one and exhibit strong embodied biases, showing a preference for right-handed actions and performance degradation with camera perspectives that deviate from those of human vision."},"_bibtex":{"value":"@inproceedings{\nwang2026enact,\ntitle={{ENACT}: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction},\nauthor={Qineng Wang and Wenlong Huang and Yu Zhou and Hang Yin and Tianwei Bao and Jianwen Lyu and Weiyu Liu and Ruohan Zhang and Jiajun Wu and Li Fei-Fei and Manling Li},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=Patx6MRipw}\n}"},"title":{"value":"ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction"},"pdf":{"value":"/pdf/5f1bea99dde54c2004fdcec05663ca0cb7c57702.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"wang|enact_evaluating_embodied_cognition_with_world_modeling_of_egocentric_interaction"},"authorids":{"value":["~Qineng_Wang1","~Wenlong_Huang1","~Yu_Zhou20","~Hang_Yin11","~Tianwei_Bao1","~Jianwen_Lyu1","~Weiyu_Liu1","~Ruohan_Zhang1","~Jiajun_Wu1","~Li_Fei-Fei1","~Manling_Li1"]},"authors":{"value":["Qineng Wang","Wenlong Huang","Yu Zhou","Hang Yin","Tianwei Bao","Jianwen Lyu","Weiyu Liu","Ruohan Zhang","Jiajun Wu","Li Fei-Fei","Manling Li"]}},"version":2},{"content":{"venue":{"value":"AAAI 2026"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/38649/42611"},"venueid":{"value":"dblp.org/conf/AAAI/2026"},"paperhash":{"value":"yao|dimts_bridge_the_gap_between_selective_state_space_models_and_time_series_for_generative_modeling"},"authorids":{"value":["~Zihao_Yao1","~Jiankai_Zuo1",""]},"html":{"value":"https://doi.org/10.1609/aaai.v40i19.38649"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/YaoZZ26,\n  author={Zihao Yao and Jiankai Zuo and Yaying Zhang},\n  title={DiM-TS: Bridge the Gap Between Selective State Space Models and Time Series for Generative Modeling},\n  year={2026},\n  cdate={1767225600000},\n  pages={16137-16144},\n  url={https://doi.org/10.1609/aaai.v40i19.38649},\n  booktitle={AAAI},\n  crossref={conf/aaai/2026}\n}\n"},"abstract":{"value":"Time series data plays a pivotal role in a wide variety of fields but faces challenges related to privacy concerns. Recently, synthesizing data via diffusion models is viewed as a promising solution. However, existing methods still struggle to capture long-range temporal dependencies and complex channel interrelations. In this research, we aim to utilize the sequence modeling capability of a State Space Model called Mamba to extend its applicability to time series data generation. We firstly analyze the core limitations in State Space Model, namely the lack of consideration for correlated temporal lag and channel permutation. Building upon the insight, we propose Lag Fusion Mamba and Permutation Scanning Mamba, which enhance the model's ability to discern significant patterns during the denoising process. Theoretical analysis reveals that both variants exhibit a unified matrix multiplication framework with the original Mamba, offering a deeper understanding of our method. Finally, we integrate two variants and introduce Diffusion Mamba for Time Series (DiM-TS), a high-quality time series generation model that better preserves the temporal periodicity and inter-channel correlations. Comprehensive experiments on public datasets demonstrate the superiority of DiM-TS in generating realistic time series while preserving diverse properties of data."},"title":{"value":"DiM-TS: Bridge the Gap Between Selective State Space Models and Time Series for Generative Modeling"},"authors":{"value":["Zihao Yao","Jiankai Zuo","Yaying Zhang"]}},"tmdate":1777454901509,"pdate":1798675200000,"tcdate":1775098702028,"writers":["~"],"signatures":["~Zihao_Yao1"],"forum":"b03H6rbqFl","license":"CC BY-SA 4.0","number":862862,"cdate":1767225600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1777454901509,"domain":"DBLP.org","id":"b03H6rbqFl","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.18312v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"yao|dimts_bridge_the_gap_between_selective_state_space_models_and_time_series_for_generative_modeling"},"authorids":{"value":["~Zihao_Yao1","","https://dblp.org/search/pid/api?q=author:Yaying_Zhang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2511.18312"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-18312,\n  publtype={informal},\n  author={Zihao Yao and Jiankai Zuo and Yaying Zhang},\n  title={DiM-TS: Bridge the Gap between Selective State Space Models and Time Series for Generative Modeling},\n  year={2025},\n  month={November},\n  cdate={1761955200000},\n  journal={CoRR},\n  volume={abs/2511.18312},\n  url={https://doi.org/10.48550/arXiv.2511.18312}\n}\n"},"abstract":{"value":"Time series data plays a pivotal role in a wide variety of fields but faces challenges related to privacy concerns. Recently, synthesizing data via diffusion models is viewed as a promising solution. However, existing methods still struggle to capture long-range temporal dependencies and complex channel interrelations. In this research, we aim to utilize the sequence modeling capability of a State Space Model called Mamba to extend its applicability to time series data generation. We firstly analyze the core limitations in State Space Model, namely the lack of consideration for correlated temporal lag and channel permutation. Building upon the insight, we propose Lag Fusion Mamba and Permutation Scanning Mamba, which enhance the model's ability to discern significant patterns during the denoising process. Theoretical analysis reveals that both variants exhibit a unified matrix multiplication framework with the original Mamba, offering a deeper understanding of our method. Finally, we integrate two variants and introduce Diffusion Mamba for Time Series (DiM-TS), a high-quality time series generation model that better preserves the temporal periodicity and inter-channel correlations. Comprehensive experiments on public datasets demonstrate the superiority of DiM-TS in generating realistic time series while preserving diverse properties of data."},"title":{"value":"DiM-TS: Bridge the Gap between Selective State Space Models and Time Series for Generative Modeling"},"authors":{"value":["Zihao Yao","Jiankai Zuo","Yaying Zhang"]}},"tmdate":1777454048392,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2511-18312"],"tcdate":1768906177839,"writers":["~"],"signatures":["~Jiankai_Zuo1"],"forum":"Kyu8NZBgXW","license":"CC BY-SA 4.0","number":771719,"cdate":1761955200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1777454048392,"domain":"DBLP.org","id":"Kyu8NZBgXW","version":2},{"content":{"venue":{"value":"MM2024 Poster"},"supplementary_material":{"value":"/attachment/e6771c423b45e04e75a97a0afcf72c23724da4f0.zip"},"abstract":{"value":"Image deblurring aims to restore a high-quality image from its corresponding blurred. The emergence of CNNs and Transformers has enabled significant progress. However, these methods often face the dilemma between eliminating long-range degradation perturbations and maintaining computational efficiency. While the selective state space model (SSM) shows promise in modeling long-range dependencies with linear complexity, it also encounters challenges such as local pixel forgetting and channel redundancy. To address this issue, we propose an efficient image deblurring network that leverages selective state spaces model to aggregate enriched and accurate features. Specifically, we introduce an aggregate local and global information block (ALGBlock) designed to effectively capture and integrate both local invariant properties and non-local information. The ALGBlock comprises two primary modules: a module for capturing local and global features (CLGF), and  a feature aggregation module (FA).  The CLGF module is composed of two branches: the global branch captures long-range dependency features via a selective state spaces model, while the local branch employs simplified channel attention to model local connectivity, thereby reducing local pixel forgetting and channel redundancy. In addition, we design a FA module to accentuate the local part by recalibrating the weight during the aggregation of the two branches for restoration. Experimental results demonstrate that the proposed method outperforms state-of-the-art approaches on widely used benchmarks."},"relevance_to_conference":{"value":"In this paper, we propose an efficiently aggregates local and global features network (ALGNet) that leverages selective state spaces model for image deblurring. Extensive experiments demonstrate that our ALGNet achieves favorably performance against state-of-the-art methods."},"_bibtex":{"value":"@inproceedings{\ngao2024learning,\ntitle={Learning Enriched Features via Selective State Spaces Model for Efficient Image Deblurring},\nauthor={Hu Gao and Bowen Ma and Ying Zhang and Jingfan Yang and Jing Yang and Depeng Dang},\nbooktitle={ACM Multimedia 2024},\nyear={2024},\nurl={https://openreview.net/forum?id=yoofksFKk9}\n}"},"title":{"value":"Learning Enriched Features via Selective State Spaces Model for Efficient Image Deblurring"},"secondary_subject_area":{"value":["[Experience] Multimedia Applications"]},"pdf":{"value":"/pdf/edef71b7f4e8b881c4803643a7655e9bd3b3fda3.pdf"},"venueid":{"value":"acmmm.org/ACMMM/2024/Conference"},"paperhash":{"value":"gao|learning_enriched_features_via_selective_state_spaces_model_for_efficient_image_deblurring"},"primary_subject_area":{"value":"[Content] Vision and Language"},"authorids":{"value":["~Hu_Gao1","~Bowen_Ma3","~Ying_Zhang27","~Jingfan_Yang1","~Jing_Yang18","~Depeng_Dang1"]},"authors":{"value":["Hu Gao","Bowen Ma","Ying Zhang","Jingfan Yang","Jing Yang","Depeng Dang"]}},"tmdate":1721528143061,"pdate":1721513487319,"tcdate":1711711891698,"writers":["acmmm.org/ACMMM/2024/Conference","acmmm.org/ACMMM/2024/Conference/Submission1277/Authors"],"signatures":["acmmm.org/ACMMM/2024/Conference/Submission1277/Authors"],"forum":"yoofksFKk9","license":"CC BY 4.0","number":1277,"cdate":1711711891698,"readers":["everyone"],"invitations":["acmmm.org/ACMMM/2024/Conference/-/Submission","acmmm.org/ACMMM/2024/Conference/-/Post_Submission","acmmm.org/ACMMM/2024/Conference/Submission1277/-/Revision","acmmm.org/ACMMM/2024/Conference/-/Edit"],"mdate":1721528143061,"odate":1721513487319,"domain":"acmmm.org/ACMMM/2024/Conference","id":"yoofksFKk9","version":2},{"content":{"summary":{"value":"The paper proposes GenKSR, a framework that learns a conditional generative model instantiated with a Mamba state-space architecture to model measurement distributions arising in Krylov Quantum Diagonalization and Sample-based KQD. Conditioned on Hamiltonian parameters and the Krylov time step, the CGM generates synthetic bitstrings that are fed into classical KQD/SKQD post-processing, enabling ground-state energy estimation for unseen Hamiltonians and extrapolation to larger Krylov dimensions without additional quantum executions. The paper validates GenKSR with noiseless simulations on Heisenberg chains up to 15 qubits and hardware SKQD on a 20-qubit subset of IBM fez, showing that the CGM reproduces the noisy measurement distribution and supports energy estimation at higher Krylov dimensions than seen in training. It also argues for Mamba’s near-linear sequence complexity versus Transformer baselines."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. In Sec. 3.2 you state: “The discrete evolution index $t_l$ is mapped into a dense vector via a learnable embedding layer.” This sounds like a standard learned lookup over the set of evolution indices present in training ( $t_l$ in your setup). However, your experiments later extrapolate to larger Krylov dimensions (D=15). Can you clarify how the embedding rows for $l > 5$ are initialized and whether they receive any gradient signals. I am confused how untrained rows can extrapolate effectively.\n2. Following the previous question, in ML we know “length / depth / horizon” generalization is usually brittle (sequence length in LLMs, longer rollouts in RL, deeper diffusion steps, etc.). How do you interpret the successful generalization to extrapolated Krylov dimensions suppose it actually works? \n3. Do you need to train a separate model for each Hamiltonian family? In this case, it still requires substantial quantum resources and undermines the advantages in resource saving.\n4. Does Transformer achieve better RMSE in larger qubits with less shots in Table 1? Is it possible to extrapolate to qubit number larger than 15 to see if Transformer outperforms in resource-restricted settings?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Clear and timely problem framing. The paper targets a bottleneck in NISQ-era Krylov methods: every new Hamiltonian and every higher Krylov dimension costs new quantum shots. Turning this into an offline-learn-once, infer-many pipeline is a sensible strategy.\n2. Experiments on real hardware. Simulated 15-qubit Heisenberg and a real 20-qubit IBM experiment provide a fairly convincing story that the approach is not simulator-only. The fact that they can learn the noise and regenerate it for test cases is a nice touch.\n3. Generalization to unseen Hamiltonians and larger Krylov dimensions.  Training on $D\\le 5$ and evaluating on up to D=15 is a nontrivial test. The paper shows that both Transformer and Mamba follow the correct shape of the energy curve, demonstrating effectiveness in generalization."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The Hamiltonian families are closely related (1D Heisenberg/XXZ). It remains unclear how well GenKSR transfers across lattice geometries, boundary conditions, anisotropies, and initial states. An explicit OOD Hamiltonian evaluation would strengthen the generalization claim.\n2. Lack of ablations on conditioning. It is informative to disentangle contributions from (i) Hamiltonian encoder, (ii) time-step embedding forms, and (iii) Mamba depth/width.\n3. The complexity motivation for replacing Transformers by Mamba is valid for naive quadratic attention, but the paper does not compare against modern linear/sparse/structured-attention Transformers, which seems a bit outdated from ML's perspective."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764359088760,"tcdate":1761970030777,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23036/Reviewer_c4Tv"],"signatures":["ICLR.cc/2026/Conference/Submission23036/Reviewer_c4Tv"],"forum":"a1MteiJJ6g","number":4,"license":"CC BY 4.0","cdate":1761970030777,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23036/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764359088760,"domain":"ICLR.cc/2026/Conference","replyto":"a1MteiJJ6g","id":"axpLrOTgCc","forumContent":{"TLDR":{"value":"We propose GenKSR, a generative Krylov subspace representation based on Mamba that reduces quantum experiments for scalable eigensolvers."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Quantum computing","generative models","Krylov subspace"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"abstract":{"value":"Predicting ground state energies of quantum many-body systems is one of the central computational challenges in quantum chemistry, physics, and materials science. Krylov subspace methods, such as Krylov Quantum Diagonalization (KQD) and Sample-based Krylov Quantum Diagonalization (SKQD), are promising approaches for this task on near-term quantum computers. However, both require repeated quantum circuit executions for each Krylov subspace and for every new Hamiltonian, posing a major bottleneck under noisy hardware constraints. We introduce Generative Krylov Subspace Representations (GenKSR), a framework that learns a classical generative representation of the entire Krylov diagonalization process. To enable effective modeling of quantum systems, GenKSR leverages a conditional generative model (CGM) framework. We investigate two representative backbone architectures, the standard Transformer and the Mamba state-space model. While Transformers provide a strong baseline, Mamba offers a more computationally efficient alternative due to its linear-time complexity. By learning the distribution of measurement outcomes conditioned on Hamiltonian parameters and evolution time, GenKSR generates Krylov subspace samples for unseen Hamiltonians and for larger subspace dimensions than those used in training. This enables full energy reconstruction purely from the classical model, without additional quantum experiments. We validate our approach through simulations of 15-qubit 1D and 16-qubit 2D Heisenberg models, as well as a hardware experiment on a 20-qubit XXZ chain executed on an IBM quantum processor. Our model successfully learns the distribution from experimental data and generates a high-fidelity representation of the quantum process. This representation enables classical reproduction of experimental outcomes, supports reliable energy estimates for unseen Hamiltonians, and significantly reduces the need for further quantum computation."},"_bibtex":{"value":"@misc{\nlee2026generative,\ntitle={Generative Krylov Subspace Representations for Scalable Quantum Eigensolvers},\nauthor={Changwon Lee and Daniel K. Park},\nyear={2026},\nurl={https://openreview.net/forum?id=a1MteiJJ6g}\n}"},"title":{"value":"Generative Krylov Subspace Representations for Scalable Quantum Eigensolvers"},"pdf":{"value":"/pdf/2c44d7d9faeb0274acc4960f4fba235e0c57381d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"lee|generative_krylov_subspace_representations_for_scalable_quantum_eigensolvers"},"authorids":{"value":["~Changwon_Lee1","~Daniel_K._Park1"]},"authors":{"value":["Changwon Lee","Daniel K. Park"]}},"version":2},{"content":{"summary":{"value":"This work proposes a hierarchical Mamba network for efficient image super-resolution. The fusion of Local SSM and Region SSM enables improved contextual modeling. Additionally, the authors proposed a DA-HMG block to further enrich the spatial relationship modeling capabilities. The extensive experiments validate the effectiveness of the proposed model."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. The paper introduces region SSM and local SSM. However, from a theoretical perspective, how do these designs enhance contextual information modeling to achieve performance gains? It would be better to first address this problem in section 3.3.1, rather than directly introducing the structure of the block.\n2. Are there any visual comparison results other than the Urban100 benchmark?\n3. Minor: it seems that Figure 6(c) is mistakenly placed."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":1},"strengths":{"value":"1. The paper is well-structured and easy to follow. The figures provide clear structure information about different key components of the proposed method. Moreover, the figures are clean and well-organized, with appropriate color schemes that enhance clarity and readability.\n\n2. The visual comparisons on the Urban100 benchmark demonstrate the effectiveness of the proposed method. Hi-Mamba restores better local textures than competitive methods, while maintaining higher LAM attributions and area of contributions."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The motivation for this paper largely draws from previous works, such as MambaIR [1], LocalMamba [2], Vmamba [3], and MSVMamba [4], with limited innovation. First, the introduction section does not adequately elaborate on the paper’s motivation. For instance, the reasons why Local SSM and Region SSM enhance modeling capabilities are not sufficiently explained. Additionally, using Local SSM and multi-scale features is not novel within Mamba-related studies (e.g., LocalMamba and MSVMamba).\n\n2. The directional alternation mentioned in the paper is also not a new concept; similar alternating scanning is discussed in MambaCSR [5]. While this scanning approach can reduce computation to some extent, it also introduces certain performance trade-offs which can be further discussed.\n\n[1] VMamba: Visual State Space Model, Yue Liu et al. CoRR abs/2401.10166.\n\n[2] Guo H, Li J, Dai T, et al. Mambair: A simple baseline for image restoration with state-space model[C]//European Conference on Computer Vision. Springer, Cham, 2025: 222-241.\n\n[3] Huang T, Pei X, You S, et al. Localmamba: Visual state space model with windowed selective scan[J]. arXiv preprint arXiv:2403.09338, 2024.\n\n[4] Shi Y, Dong M, Xu C. Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model[J]. arXiv preprint arXiv:2405.14174, 2024.\n\n[5] Ren Y, Li X, Guo M, et al. MambaCSR: Dual-Interleaved Scanning for Compressed Image Super-Resolution With SSMs[J]. arXiv preprint arXiv:2408.11758, 2024."}},"nonreaders":[],"tmdate":1731427887785,"tcdate":1730621857729,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3583/Reviewer_6h8G"],"signatures":["ICLR.cc/2025/Conference/Submission3583/Reviewer_6h8G"],"forum":"iPYwddLhXR","number":3,"license":"CC BY 4.0","cdate":1730621857729,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3583/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427887785,"domain":"ICLR.cc/2025/Conference","replyto":"iPYwddLhXR","id":"NDgoXl6sKE","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Image Super-Resolution","Mamba","Structured State Space Models"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSM), such as Mamba, have shown strong representation ability in modeling long-range dependency with linear complexity, achieving successful applications from high-level to low-level vision tasks. However, SSM's sequential nature necessitates multiple scans in different directions to compensate for the loss of spatial dependency when unfolding the image into a 1D sequence. This multi-direction scanning strategy significantly increases the computation overhead and is unbearable for high-resolution image processing. To address this problem, we propose a novel Hierarchical Mamba network, namely, Hi-Mamba, for image super-resolution (SR).  Hi-Mamba consists of two key designs: (1) The Hierarchical Mamba Block (HMB) assembled by a Local SSM (L-SSM) and a Region SSM (R-SSM) both with the single-direction scanning, aggregates multi-scale representations to enhance the context modeling ability. (2) The Direction Alternation  Hierarchical Mamba Group (DA-HMG) allocates the isomeric single-direction scanning into cascading HMBs to enrich the spatial relationship modeling.\nExtensive experiments demonstrate the superiority of Hi-Mamba across five benchmark datasets for efficient SR. For example, Hi-Mamba achieves a significant PSNR improvement of 0.29 dB on Manga109 for $\\times3$ SR, compared to the strong lightweight MambaIR."},"_bibtex":{"value":"@misc{\nqiao2024himamba,\ntitle={Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution},\nauthor={Junbo Qiao and Jincheng Liao and Wei Li and Yulun Zhang and Yong Guo and Yi Wen and Zhangxizi Qiu and Jiao Xie and Jie Hu and Shaohui Lin},\nyear={2024},\nurl={https://openreview.net/forum?id=iPYwddLhXR}\n}"},"title":{"value":"Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution"},"pdf":{"value":"/pdf/e75e168c76e93b40ac8e29dd9015c5e7d72edc8b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"qiao|himamba_hierarchical_mamba_for_efficient_image_superresolution"},"authorids":{"value":["~Junbo_Qiao1","~Jincheng_Liao1","~Wei_Li60","~Yulun_Zhang1","~Yong_Guo1","~Yi_Wen3","~Zhangxizi_Qiu2","~Jiao_Xie1","~Jie_Hu8","~Shaohui_Lin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Junbo Qiao","Jincheng Liao","Wei Li","Yulun Zhang","Yong Guo","Yi Wen","Zhangxizi Qiu","Jiao Xie","Jie Hu","Shaohui Lin"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a contrastive learning training framework for Mamba-based models in multivariate time series forecasting (MTSF) tasks. The authors introduce a repeated data augmentation and contrastive learning approach for both inter- and intra-sequence levels. Experiments demonstrate that replacing partial blocks of Mamba models with the proposed pre-trained blocks can enhance their performance."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1, Could you clarify whether all MambaPB blocks share the same parameters, or are they trained independently during the self-training phase? If all MambaPB blocks share the same parameters, this would make the approach more interesting.\n\n2. In what way does the proposed augmentation strategy improve upon conventional contrastive learning approaches?\n\n3. Is there an ablation study that examines how varying levels of Gaussian noise impact performance?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This is one of the first works exploring contrastive learning in Mamba models for time series.\n2. The proposed method demonstrates performance gains on recent Mamba-based models for time series forecasting.\n3. Based on the visualizations in Figures 3 and 4, I believe the pretraining is effective."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The repetitive augmentation does not seem novel, as it appears to simply create negative pairs and concatenate positive and negative pairs together. This approach likely increases the input size and, consequently, the computational cost. Traditional contrastive learning can also generate negative pairs with different strategies, and I suspect that with a smaller input size, the model might be even more efficient. (Please correct me if I am mistaken.)\n\n2. The paper does not include comparisons with non-Mamba methods. This is crucial, as improvements over weak baselines alone may not provide strong evidence of the proposed method’s contribution.\n\n3. The experimental evaluation may be insufficient for ICLR standards. Recent works often include either a larger variety of datasets or more comprehensive experimental setups.\n\nMinors: There are some typos and repeated sentences, and the overall presentation could be improved."}},"nonreaders":[],"tmdate":1732812412009,"tcdate":1730399357442,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9847/Reviewer_3whs"],"signatures":["ICLR.cc/2025/Conference/Submission9847/Reviewer_3whs"],"forum":"tIURLNBTPx","number":1,"license":"CC BY 4.0","cdate":1730399357442,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9847/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732812412009,"domain":"ICLR.cc/2025/Conference","replyto":"tIURLNBTPx","id":"sUIFO7XGwz","forumContent":{"TLDR":{"value":"We propose a Repetitive Contrastive Learning method to enhance the sequence selection capability of Mamba SSM, which can bring improvements to multiple Mamba-based models in time series prediction problems."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Self-supervised learning; Time Series Prediction"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The prediction of long sequences has always been a challenge in time series forecasting tasks. Due to Mamba's sequence selection capability, many Mamba-based models have been proposed, achieving state-of-the-art results in long sequence prediction problems. However, much research has focused on integrating mamba-ssm into specific model structures for better performance, while the core of mamba-ssm, its sequence selection capability, has not been deeply explored. We believe there is significant potential in Mamba's sequence selection capability and propose a Repetitive Contrastive Learning (RCL) method to enhance it. Specifically, we utilize Repeating Sequence Augmentation to expand the sequence while introducing Gaussian noise, thereby enhancing the Mamba block's sequence selection capability through both inter-sequence and intra-sequence contrastive techniques. Then our methods transfer parameters directly from a single pretrained Mamba block to a variety of Mamba-based models. This approach provides superior initialization for forecasting tasks. Our experiments consistently demonstrate that this technique improves the forecasting performance of many Mamba-based models, without imposing additional memory requirements."},"_bibtex":{"value":"@misc{\nyan2025repetitive,\ntitle={Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction},\nauthor={WenBo Yan and Hanzhong Cao and Ying Tan},\nyear={2025},\nurl={https://openreview.net/forum?id=tIURLNBTPx}\n}"},"title":{"value":"Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction"},"pdf":{"value":"/pdf/25ba192abf3d52ee17ddd05bad92e932a9f5e0c0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"yan|repetitive_contrastive_learning_enhances_mambas_selectivity_in_time_series_prediction"},"authorids":{"value":["~WenBo_Yan1","~Hanzhong_Cao1","~Ying_Tan1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["WenBo Yan","Hanzhong Cao","Ying Tan"]}},"version":2},{"content":{"comment":{"value":"Dear Reviewer 76n7,\n\nThank you for your prompt reply and for raising the score. We are pleased to hear that you found our rebuttal satisfactory and appreciate your recognition of the practical value of our work. Below, we address the raised question.\n\n- *Do you think the current framework can become scalable with a more smaller model size?*:\n\n    We firmly believe that the core contributions of our work—the FILM encoder, temporally adaptable spatio-temporal down/upsampling blocks, and flow regularization—are not closely tied to model size and should therefore enhance the performance of smaller video VAEs as well. As discussed in **Section 5** and **Appendix A.6** of the revised manuscript, the primary computational bottleneck in our model lies in the spatio-temporal attention blocks, which also play a notable role in enhancing network performance, as demonstrated in Table 4d. As mentioned in **Section 5**, a promising direction for future work would be to explore more computationally efficient spatio-temporal attention mechanisms, such as Mamba [1], for video VAE architectures. This could alleviate current computational challenges and enhance the scalability of our framework. We will elaborate on this further in the final version of our paper.\n\n**References**\n\n[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces"}},"tmdate":1732534352620,"tcdate":1732534352620,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2418/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission2418/Authors"],"forum":"aRD1NqcXTC","number":17,"license":"CC BY 4.0","cdate":1732534352620,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2418/-/Official_Comment"],"mdate":1732534352620,"domain":"ICLR.cc/2025/Conference","replyto":"KSZtx5DdFT","id":"VbYa5z9Elk","forumContent":{"TLDR":{"value":"A causal video VAE for joint image and video tokenization"},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Autoencoding","Generative Modelling","Causal Video VAE","FILM","Video Tokenization"]},"supplementary_material":{"value":"/attachment/64a77b89dfe3e76ce46a81375afe1fd4c32272aa.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Generative modeling has seen significant advancements in image and video synthesis. However, the curse of dimensionality remains a significant obstacle, especially for video generation, given its inherently complex and high-dimensional nature. Many existing works rely on low-dimensional latent spaces from pretrained image autoencoders. However, this approach overlooks temporal redundancy in videos and often leads to temporally incoherent decoding. To address this issue, we propose a video compression network that reduces the dimensionality of visual data both spatially and temporally. Our model, based on a variational autoencoder, employs causal 3D convolution to handle images and videos jointly. The key contributions of our work include a scale-agnostic encoder for preserving video fidelity, a novel spatio-temporal down/upsampling block for robust long-sequence modeling, and a flow regularization loss for accurate motion decoding. \nOur approach outperforms competitors in video quality and compression rates across various datasets. Experimental analyses also highlight its potential as a robust autoencoder for video generation training."},"_bibtex":{"value":"@inproceedings{\nargaw2025highquality,\ntitle={High-Quality Joint Image and Video Tokenization with Causal {VAE}},\nauthor={Dawit Mureja Argaw and Xian Liu and Qinsheng Zhang and Joon Son Chung and Ming-Yu Liu and Fitsum Reda},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=aRD1NqcXTC}\n}"},"title":{"value":"High-Quality Joint Image and Video Tokenization with Causal VAE"},"pdf":{"value":"/pdf/55e58cfb0679e477e52cfc55ece2e1183259bcb5.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"argaw|highquality_joint_image_and_video_tokenization_with_causal_vae"},"authorids":{"value":["~Dawit_Mureja_Argaw1","~Xian_Liu1","~Qinsheng_Zhang1","~Joon_Son_Chung1","~Ming-Yu_Liu1","~Fitsum_Reda1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Dawit Mureja Argaw","Xian Liu","Qinsheng Zhang","Joon Son Chung","Ming-Yu Liu","Fitsum Reda"]}},"version":2},{"content":{"summary":{"value":"The authors consider the convergence of single time scale actor critic algorithm in continuous state and action spaces with infinite horizon average reward as the performance metric. The state and action spaces considered are assumed to be compact and the value functions and the policies are parametrized through deep neural networks. The average reward, the relative value function and the policy are all updated on a similar time scale. Finite time convergence guarantees to a stationary point are provided under the overparametrized neural network regime. They resort to a NTK style analysis where by the virtue of overparametrization, the weights do not change much in magnitude by the end of the training period."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Equation 1, what does the measure $d(a\\times s')$ represent?\n2. Theorem 4.9 $\\alpha = \\frac{c}{\\sqrt{T}}$, what are the typical values of $c$ in this learning rate?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. Most prior literature in RL (especially average reward RL) considers finite state and action spaces. Hence the setting of continuous state and action spaces is relatively understudied given its significance. \n2. Neural networks (NN) have consistently demonstrated strong capabilities as function approximators, with prior results showing that they can approximate continuous functions to any desired accuracy. However, their theoretical foundations remain relatively underexplored. This paper addresses a crucial problem, particularly given the extensive practical applications of neural networks.\n3. They characterize tight bounds in terms of dependence on the number of iterations and number of parameters of the NN."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Since this paper is motivated by the promise of neural networks in policy optimization in the realm of RL, simulations demonstrating the same would increase the strength of the paper. As of now, the paper analyzes the standard actor critic algorithm and provides finite time bounds, but relating these to whats observed in practice would vastly help with understanding (especially in terms of determining the various step sizes required for updating different quantities of interest).\n2. The paper provides convergence to a stationary point which can be a local minima. However, some of the recent works have proposed global convergence guarantees for average reward problems. See ref below. Although these are for finite state and action spaces, the optimization landscape might be the same independent of the finiteness of the problem parameters.\n3. The assumptions require the nonlinear activations to be differentiable and smooth, this precludes the use of ReLU. However the authors suggest other nonlinear activations which indeed satisfy these assumptions. Once again, a simulation to demonstrate their practicality would help.\n4. Regarding Assumption 4.3: Does this assumption require the original value functions to also be smooth with respect to the underlying parameter $\\theta$. Since the parameters are initialized from a normal distribution, it is not clear as to how assumption 4.3 can be satisfied in practice (the uniform boundedness of the optimal parameters corresponding to the value function estimate). In case of linear function approximations, this assumption is trivial since typically $\\|\\phi\\| \\leq 1$, where $\\phi$ is the feature vector. Its not clear whether this assumption can be satisfied in a straight forward manner when using neural networks. \n5. Regarding Assumption 4.4: Since this assumption holds for every state, when dealing with uncountably infinite states, it is unclear as to how $\\lambda_1$ can be uniformly bounded from below across all states.\n\n\n\n\n[1] Improved Sample Complexity for Global Convergence of Actor-Critic Algorithms\nN Kumar, P Agrawal, G Ramponi, KY Levy, S Mannor 2024\n[2] On the global convergence of policy gradient in average reward markov decision processes\nN Kumar, Y Murthy, I Shufaro, KY Levy, R Srikant, S Mannor 2024"}},"nonreaders":[],"tmdate":1731428309933,"tcdate":1730689472795,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6608/Reviewer_7fyF"],"signatures":["ICLR.cc/2025/Conference/Submission6608/Reviewer_7fyF"],"forum":"iC4hBE9xYe","number":3,"license":"CC BY 4.0","cdate":1730689472795,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6608/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428309933,"domain":"ICLR.cc/2025/Conference","replyto":"iC4hBE9xYe","id":"SLPTIXxHqv","forumContent":{"TLDR":{"value":"We establish the finite-time convergence of single-timescale neural actor-critic in continuous state-action space"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Single-Timescale Actor-Critic","Continuous State-Action Space","Deep Neural Networks"]},"primary_area":{"value":"learning theory"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Actor-critic (AC) algorithms have been the powerhouse behind many successful yet challenging applications. However, the theoretical understanding of finite-time convergence in AC's most practical form remains elusive. Existing research often oversimplifies the algorithm and only considers simple finite state and action spaces. We analyze the more practical single-timescale AC on continuous state and action spaces and use deep neural network approximations for both critic and actor. \nOur analysis reveals that the iterates of the more practical framework we consider converge towards the stationary point at rate $\\widetilde{\\mathcal{O}}(T^{-1/2})+\\widetilde{\\mathcal{O}}(m^{-1/2})$, where $T$ is the total number of iterations and $m$ is the width of the deep neural network.  To our knowledge, this is the first finite-time analysis of single-timescale AC in continuous state and action spaces, which further narrows the gap between theory and practice."},"_bibtex":{"value":"@misc{\nchen2025provable,\ntitle={Provable Convergence of Single-Timescale Neural Actor-Critic in Continuous Spaces},\nauthor={Xuyang Chen and Fengzhuo Zhang and Guojian Wang and Lin Zhao},\nyear={2025},\nurl={https://openreview.net/forum?id=iC4hBE9xYe}\n}"},"title":{"value":"Provable Convergence of Single-Timescale Neural Actor-Critic in Continuous Spaces"},"pdf":{"value":"/pdf/93c6a2bb8f6afbde13c985bb9fa2cafa4044e3de.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chen|provable_convergence_of_singletimescale_neural_actorcritic_in_continuous_spaces"},"authorids":{"value":["~Xuyang_Chen1","~Fengzhuo_Zhang1","~Guojian_Wang2","~Lin_Zhao3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xuyang Chen","Fengzhuo Zhang","Guojian Wang","Lin Zhao"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for the response. We want to highlight that all the recent baselines asked by reviewers are included in the N-body experiment. The N-body experiment was moved to the Appendix because (i) our work tackles large-scale geometric systems, and standard 5-particle N-body experiment is not a representative task for modeling large-scale geometric systems; (ii) to accommodate the new results and reviewers' suggestion within 10 pages limit.\n\nAlso, we want to highlight that associative recall is a widely recognized experiment, routinely used to evaluate the contextual learning capabilities of sequence models. For instance, it has been used to benchmark models such as Hyena [1] and even serves as the primary motivation for models like Mamba (Figure 2 in [2]). Furthermore, recent studies demonstrate that mechanistic interpretability tasks are indicative of a model's scaling behavior [3], where improved performance on tasks such as associative recall directly correlates with better generalization to real-life downstream tasks. Given its broad acceptance in the literature, we consider associative recall a serious experiment to evaluate models' context learning abilities.\n\n[1] Poli et al. Hyena Hierarchy: Towards Larger Convolutional Language Models. 2023. \\\n[2] Gu and Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023 \\\n[3] Poli et al. Mechanistic Design and Scaling of Hybrid Architectures. 2024.\n\nWe also kindly ask the reviewer to consider **all dimensions of our work, including theoretical and methodological contributions**, alongside the experimental part, which is important but not the only component of the paper."},"title":{"value":"Authors response"}},"tmdate":1733151311342,"tcdate":1733151311342,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3951/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission3951/Authors"],"forum":"evzUQDjXKV","number":18,"license":"CC BY 4.0","cdate":1733151311342,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3951/-/Official_Comment"],"mdate":1733151311342,"domain":"ICLR.cc/2025/Conference","replyto":"eHu9MNy4S6","id":"7unjWdb8Hk","forumContent":{"TLDR":{"value":"We introduce the SE(3)-Hyena, an equivariant long-convolutional model that efficiently captures global geometric context at sub-quadratic complexity"},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["architecrture","equivariance","global context","long convolution","scalability","mechanistic interpretability"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Modeling global geometric context while maintaining equivariance is crucial for accurate predictions in many fields such as biology, chemistry, or when modeling physical systems. Yet, this is challenging due to the computational demands of processing high-dimensional data at scale. Existing approaches such as equivariant self-attention or distance-based message passing, suffer from quadratic complexity with respect to sequence length, while localized methods sacrifice global information. Inspired by the recent success of state-space and long-convolutional models, in this work, we introduce the SE(3)-Hyena operator, the first equivariant network that adopts a long-convolutional framework for geometric systems. SE(3)-Hyena captures global geometric context at sub-quadratic complexity while maintaining equivariance to rotations and translations. Evaluated on the task of all-atom property prediction of large RNA molecules, SE(3)-Hyena matches or outperforms equivariant self-attention while requiring significantly less memory and compute for long geometric sequences. Additionally, we propose equivariant associative recall as a new mechanistic interpretability task for studying the contextual learning capabilities of equivariant models. Notably, our model processes the geometric context of $30k$ tokens $20 \\times$ faster than the equivariant transformer and allows $72 \\times$ longer context within the same memory budget. The code will be released upon the acceptance."},"_bibtex":{"value":"@misc{\nmoskalev2024sehyena,\ntitle={{SE}(3)-Hyena Operator for Scalable Equivariant Learning},\nauthor={Artem Moskalev and Mangal Prakash and Junjie Xu and Rui Liao and Tommaso Mansi},\nyear={2024},\nurl={https://openreview.net/forum?id=evzUQDjXKV}\n}"},"title":{"value":"SE(3)-Hyena Operator for Scalable Equivariant Learning"},"pdf":{"value":"/pdf/19aec26deb3541f0c80480cce1c6eb1699c9af93.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"moskalev|se3hyena_operator_for_scalable_equivariant_learning"},"authorids":{"value":["~Artem_Moskalev1","~Mangal_Prakash1","~Junjie_Xu1","~Rui_Liao2","~Tommaso_Mansi5"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Artem Moskalev","Mangal Prakash","Junjie Xu","Rui Liao","Tommaso Mansi"]}},"version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Scalable probabilistic models","large language models","Kalman Filtering","Uncertainty"]},"supplementary_material":{"value":"/attachment/63bcd8968c51b9e1414e27341962fa62b924f9b3.zip"},"_bibtex":{"value":"@inproceedings{\nshaj2026kalman,\ntitle={Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modeling and State Tracking},\nauthor={Vaisakh Shaj and Cameron Barker and Aidan Scannell and Andras Szecsenyi and Elliot J. Crowley and Amos Storkey},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=9h7sSJe4jN}\n}"},"title":{"value":"Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modeling and State Tracking"},"paperhash":{"value":"shaj|kalman_linear_attention_parallel_bayesian_filtering_for_efficient_language_modeling_and_state_tracking"},"originally_submitted_PDF":{"value":"/pdf/ed3eb17adf2a0e7d3ebd5d1cd91b435b80f388b2.pdf"},"TLDR":{"value":"Parallelized Kalman filter updates as a sequence mixer/block for efficient language modelling and state tracking."},"primary_area":{"value":"probabilistic_methods->bayesian_models_and_methods"},"abstract":{"value":"State-space language models such as Mamba and gated linear attention (GLA) offer linear-complexity, parallelisable alternatives to transformers, but often lack the expressivity and robust state-tracking needed for complex reasoning. We address this by reframing sequence modeling through a probabilistic lens, with Bayesian filters as the core primitive. Classical Kalman filters provide principled state and uncertainty estimation but are typically viewed as inherently sequential. We show that reparameterising the Kalman filter in information form casts its updates as an associative scan, enabling efficient parallel training. The resulting Kalman Linear Attention (KLA) layer performs time-parallel probabilistic inference while maintaining explicit belief-state uncertainty, offering strictly more expressive nonlinear updates and gating than GLA variants while retaining their computational advantages. On language modeling tasks, KLA matches or outperforms modern SSMs and GLAs on discrete token-manipulation, state-tracking, and zero-shot commonsense-reasoning benchmarks, and is among the first stacked Bayesian-filtering primitives pretrained at the billion-token scale."},"link_to_code":{"value":"https://github.com/vaisakh-shaj/kalman-linear-attention"},"pdf":{"value":"/pdf/77bbedacbe5fde4f5de50a01b8e04abc888bd9dc.pdf"},"lay_summary":{"value":"The AI systems behind today's chatbots, called transformers, are powerful but grow slow and resource-hungry as the text they handle gets longer. Researchers have built leaner, faster alternatives, but these often struggle to keep careful track of information across long passages - a skill that genuine reasoning needs.\n\nWe turned to an unlikely source: the Kalman filter, a 60-year-old algorithm that estimates the hidden state of a system from noisy measurements while keeping track of its own uncertainty. It famously helped guide the Apollo missions to the Moon and remains a gold standard in engineering and neuroscience. The catch was that it seemed to work only one step at a time - far too slowly to learn from the vast text collections modern AI needs. We found that rewriting the filter in a different mathematical form reveals hidden structure that lets all its steps be computed at once, in parallel, and built this into a neural network layer we call Kalman Linear Attention.\n\nBecause the model continually reasons about its own uncertainty, useful behaviours that other systems bolt on by hand emerge on their own. It matches or improves on leading efficient models for language, and is among the first such probabilistic methods trained at large scale - a step toward AI that has a built-in sense of what it doesn't know."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Vaisakh_Shaj1","~Cameron_Barker1","~Aidan_Scannell1","~Andras_Szecsenyi1","~Elliot_J._Crowley1","~Amos_Storkey1"]},"authors":{"value":["Vaisakh Shaj","Cameron Barker","Aidan Scannell","Andras Szecsenyi","Elliot J. Crowley","Amos Storkey"]}},"tmdate":1790067413301,"pdate":1777576733268,"tcdate":1769183465456,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission25965/Authors"],"signatures":["ICML.cc/2026/Conference/Submission25965/Authors"],"forum":"9h7sSJe4jN","license":"CC BY 4.0","number":25965,"cdate":1769183465456,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission25965/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission25965/-/Camera_Ready_Revision"],"mdate":1790067413301,"odate":1782341999037,"domain":"ICML.cc/2026/Conference","id":"9h7sSJe4jN","version":2},{"content":{"summary":{"value":"This paper presents a new neural network architecture for multi-variate time series forecasting. The main idea is to iteratively extract linear and non-linear patterns from the time series which are easier to learn and forecast. The linear patterns are extracted using 1D convolutions (part of the Li Block) where as the non linear features (part of the No Block) are extracted using linear projections on the frequency (FFT) and time domain (temporal variation features). The inter time-series dependencies are modeled using a weighted mean (akin to a pooling function) where the weights are computed using a softmax function.\n\nThe LiNo blocks are iteratively applied to the residuals to build predictors for each residual. All the predictors are finally added to produce the final prediction. The authors have presented extensive experimentation showing the effectiveness of the approach yielding state of the art results on several benchmark datasets."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- What is the advantage of using frequency domain operations? How do they benefit the predictions? Is it only useful if there are cyclical seasonal patterns? Will it be useful in cases where there is no seasonality or the frequency of the seasonal pattern can shift?\n- Line 239 to 243: Can you elaborate on how the softmax is applied and the weighted mean is computed? It would be helpful to include a figure showing this operation in detail.\n- How is the FFT and IFFT calculated? In particular, is this operation differentiable is is back-propagated through?\n- Where does H_i sit in figure 2?\n- Figure 2 No Block: Why is the MLP applied to the final prediction (right before adding the prediction from the Li Block) but not when computing the residual for the next block?\n- Figure 2: What does the backbone consist of?\n- What is the purpose of dropout in Equation 2? Does it help with spreading out the coefficients across the whole input window rather than have it concentrated on a few time steps?\n\nSome suggested clarifications in the text:\n- Line 078: \"Another problem is the design of current nonlinear models, which mainly focus on\none or two types of nonlinear patterns.\"\nWhich types of non-linear patterns are being talked about here.\n- Line 090: What is \"traditional RRD\"?\n- Line 143: Typo \"closely close\"\n- Eq 3: did you mean T<D?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"Strengths:\n- The author presented a novel time series forecasting method based on the idea of iteratively extracting linear and non-linear features from the residuals. While the idea has been introduced earlier in the literature (RRD), this paper extends the idea using neural networks.\n- The experiments performed in this paper are extensive and the baselines used are state of the art. Comparison with several state of the art baselines on benchmark performance shows improved prediction accuracy.\n- Additionally, the analysis presented in the paper including the ablation studies are extensive furthering strengthening the proposed method."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Weaknesses\n- Several parts of the paper are unclear. In particular the design of the model (see questions below)\n- The paper builds on the ideas of residual modeling and non-linear feature extraction using FFT, both which have been explored earlier in the literature [1, 2, 3] ([1] has not been cited), weakening the novelty of the approach slightly.\n\n[1] Oreshkin, Boris N., et al. \"N-BEATS: Neural basis expansion analysis for interpretable time series forecasting.\" arXiv preprint arXiv:1905.10437 (2019).\n\n[2] Challu, Cristian, et al. \"Nhits: Neural hierarchical interpolation for time series forecasting.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 6. 2023.\n\n[3] Xu, Zhijian, Ailing Zeng, and Qiang Xu. \"FITS: Modeling time series with $10 k $ parameters.\" arXiv preprint arXiv:2307.03756 (2023)."}},"nonreaders":[],"tmdate":1731427452451,"tcdate":1731000563069,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1614/Reviewer_236E"],"signatures":["ICLR.cc/2025/Conference/Submission1614/Reviewer_236E"],"forum":"JeZS4jQF77","number":4,"license":"CC BY 4.0","cdate":1731000563069,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1614/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427452451,"domain":"ICLR.cc/2025/Conference","replyto":"JeZS4jQF77","id":"LYKD0KeoFD","forumContent":{"TLDR":{"value":"We propose LiNo, a framework offering better linear and nonlinear patterns decomposition in time series data to enhance forecasting accuracy and robustness."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forecasting","Deep learning."]},"supplementary_material":{"value":"/attachment/f36dc2f069b68d2e524f94233db2602455035352.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Forecasting models are pivotal in a data-driven world with vast volumes of time series data that appear as a compound of vast $\\textbf{Li}$near and $\\textbf{No}$nlinear patterns. \nRecent deep time series forecasting models struggle to utilize seasonal and trend decomposition to separate the entangled components. Such a strategy only explicitly extracts simple linear patterns like trends, leaving the other linear modes and vast unexplored nonlinear patterns to the residual. Their flawed linear and nonlinear feature extraction models and shallow-level decomposition limit their adaptation to the diverse patterns present in real-world scenarios.\nGiven this, we innovate Recursive Residual Decomposition by introducing explicit extraction of both linear and nonlinear patterns. This deeper-level decomposition framework, which is named $\\textbf{LiNo}$, captures linear patterns using a Li block which can be a moving average kernel, and models nonlinear patterns using a No block which can be a Transformer encoder. The extraction of these two patterns is performed alternatively and recursively. To achieve the full potential of LiNo, we develop the current simple linear pattern extractor to a general learnable autoregressive model, and design a novel No block that can handle all essential nonlinear patterns.\nRemarkably, the proposed LiNo achieves state-of-the-art on thirteen real-world benchmarks under univariate and multivariate forecasting scenarios. Experiments show that current forecasting models can deliver more robust and precise results through this advanced Recursive Residual Decomposition. We hope this work could offer insight into designing more effective forecasting models. Code is available at this anonymous repository: https://anonymous.4open.science/r/LiNo-8225/."},"_bibtex":{"value":"@misc{\nyu2025lino,\ntitle={LiNo: Advancing Recursive Residual Decomposition of Linear and Nonlinear Patterns for Robust Time Series Forecasting},\nauthor={Guoqi Yu and Yaoming Li and Xiaoyu Guo and Shujun Wang and Zirui Liu and Dayu Wang and Tong Yang},\nyear={2025},\nurl={https://openreview.net/forum?id=JeZS4jQF77}\n}"},"title":{"value":"LiNo: Advancing Recursive Residual Decomposition of Linear and Nonlinear Patterns for Robust Time Series Forecasting"},"pdf":{"value":"/pdf/fa10c828099387707e229fc9c4a29cb300051efb.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"yu|lino_advancing_recursive_residual_decomposition_of_linear_and_nonlinear_patterns_for_robust_time_series_forecasting"},"authorids":{"value":["~Guoqi_Yu1","~Yaoming_Li1","~Xiaoyu_Guo6","~Shujun_Wang1","~Zirui_Liu2","~Dayu_Wang1","~Tong_Yang6"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Guoqi Yu","Yaoming Li","Xiaoyu Guo","Shujun Wang","Zirui Liu","Dayu Wang","Tong Yang"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a Decomposed Q-function Representation (DQR), enabling sequence modeling-based representation for each component’s Q-function to simplify the learning of an optimal control policy within the entire configuration space of a black-box optimizer. Specifically, using DQR and the Mamba architecture, the paper introduces Q-Leaner, which models each component configuration in the optimizer as a separate time step, auto-regressively predicting the corresponding decomposed Q-function conditioned on the current optimization status and previously selected component configurations, with CQL regularization. Experimental results demonstrate that Q-Mamba achieves competitive optimization performance compared to prior online/offline learning baselines, while requiring at most half the training budget of the online baselines."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. What motivated the choice of Mamba as the sequence modeling architecture? In Table 1, the inference time of Decision Transformer (DT) is comparable to that of Q-Mamba. What primarily drives the performance differences between DT and Q-Mamba—is it due to the additional Q-module? Additionally, how would Q-Mamba compare to QT [1] or QDT [2], which also integrate Q-modules into DT to enhance performance?\n2. How does Q-Mamba perform in comparison with other methods in the D4RL environment, a widely used testbed for offline RL methods?\n3. Could the authors clarify the problem definition and the necessity of using an offline RL approach in a more accessible way? As I understand, the primary focus of this research is to identify components of the BBO optimization algorithm that enhance generalization.\n4. The total training objective is directly defined by the Bellman backup update in Equation 8, differing from the supervised learning objective typically used in sequence modeling for offline RL. How, then, is DT applied in this context? What is the training objective for DT in this setting?\n\n[1] Shengchao Hu, et al. Q-value Regularized Transformer for Offline Reinforcement Learning. 2024 ICML.\n\n[2] Taku Yamagata, et al. Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL. 2023 ICML."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The application of sequence modeling for MetaBBO is intriguing.\n2. The paper is well-written."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Including the algorithms for Q-Mamba would enhance the clarity of the optimization pipeline, detailing the input and output to facilitate reader comprehension."}},"nonreaders":[],"tmdate":1731427533435,"tcdate":1730266344382,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2058/Reviewer_uKGW"],"signatures":["ICLR.cc/2025/Conference/Submission2058/Reviewer_uKGW"],"forum":"cSgEW7EZ9h","number":1,"license":"CC BY 4.0","cdate":1730266344382,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2058/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427533435,"domain":"ICLR.cc/2025/Conference","replyto":"cSgEW7EZ9h","id":"49Toti2ntS","forumContent":{"TLDR":{"value":"A novel MetaBBO framework which achieves both effectiveness and efficiency for auto-configuring black-box optimizer."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Black-Box Optimization","Dynamic Algorithm Configuration","Learning to Optimize","Offline Reinforcement Learning","Mamba"]},"primary_area":{"value":"optimization"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent progress in Meta-Black-Box-Optimization (MetaBBO) has demonstrated that meta-training a neural network based meta-level control policy over an optimization task distribution could significantly enhance the optimization performance of the low-level black-box optimizers. However, achieving such performance enhancement requires effective policy optimization/search method to locate optimal control policy within a massive joint-action space. The online learning fashion of existing works further makes the efficiency of MetaBBO problematic. To address these technical challenges, we propose an offline learning framework in this paper, termed Q-Mamba. Concretely, our method uses a Mamba neural network architecture to meta-learn decomposed Q-functions for each configurable component in the low-level optimizer. By decomposing the Q-function of the configuration decisions of all components in an optimizer, we can apply effective sequence modelling to avoid searching the control policy in the massive joint-action space. Furthermore, by leveraging the long-sequence modelling advantage of Mamba and moderate offline trajectory samples, Q-Mamba can be efficiently trained through a synergy of offline Temporal-Difference update and Conservative Q-Learning regularization to achieve competitive performance against the online learning paradigms. Through extensive benchmarking, we observe that Q-Mamba achieves competitive or even superior optimization performance to prior online/offline learning baselines, while significantly improving the training efficiency of existing online learning baselines. Additional ablation studies show that each of the proposed key designs contributes to this good performance."},"_bibtex":{"value":"@misc{\nma2025metablackboxoptimization,\ntitle={Meta-Black-Box-Optimization through Offline Q-function Learning with Mamba Architecture},\nauthor={Zeyuan Ma and Hongshu Guo and Zhou Jiang and Zhiguang Cao and Yue-Jiao Gong},\nyear={2025},\nurl={https://openreview.net/forum?id=cSgEW7EZ9h}\n}"},"title":{"value":"Meta-Black-Box-Optimization through Offline Q-function Learning with Mamba Architecture"},"pdf":{"value":"/pdf/2f9052c209e77207cb87544eb40cb37b2a701261.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"ma|metablackboxoptimization_through_offline_qfunction_learning_with_mamba_architecture"},"authorids":{"value":["~Zeyuan_Ma1","~Hongshu_Guo1","~Zhou_Jiang2","~Zhiguang_Cao1","~Yue-Jiao_Gong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zeyuan Ma","Hongshu Guo","Zhou Jiang","Zhiguang Cao","Yue-Jiao Gong"]}},"version":2},{"content":{"comment":{"value":"We thank the Reviewer for the careful reading and detailed comments and questions. \n\nWe respond point-by-point below for their concerns.\n\n---\n\n### Weakness 1 — Positioning / accessibility (“what is FSSM and who is it for?”)\n\n> “It is advertised for using SSMs to enhance spectral representation learning, but it is barely accessible to either community…”\n\nFSSM is **explicitly designed as a *learnable DFT-style spectral front-end***, i.e., a drop-in replacement for FFT / learnable DFT / learnable filterbanks in radar and audio pipelines [1–3].\n\n- At initialization, the modes are **placed on the DFT grid**, with unit magnitude and neutral bias correction, so the module behaves like a standard DFT/STFT [1].  \n- During training, these modes **learn center frequencies, bandwidths (via damping), and gains**, i.e., they become *task-adaptive DFT bins* in the spirit of learnable filterbanks such as SincNet and LEAF [2, 3].\n\nWe use an SSM parameterization **only as an implementation procedure** to realize these learnable DFT bins as stable, efficient recurrences (for streaming and linear-time computation) [4, 5]; FSSM is **not proposed as a new generic “SSM block” competing with Mamba/S4**, but as a **spectral front-end**.\n\n**Changes in the paper.** At the start of Section 3, we will make this role explicit, e.g.:\n\n> “FSSM is a *learnable DFT front-end*: each mode is a damped complex sinusoid initialized to match a DFT bin, then trained to adapt its center frequency and bandwidth. We implement these modes as stable state-space recurrences for efficiency, but conceptually they are learnable DFT filters.”\n\nWe will also add a short intuitive paragraph showing a single FSSM mode as “a complex narrowband filter that can be initialized as a DFT basis vector and is learnable.\n\n**References:**\n\n[1] A. V. Oppenheim and R. W. Schafer, *Discrete-Time Signal Processing*, 3rd ed., Prentice Hall, 2009.  \n[2] M. Ravanelli and Y. Bengio, “Speaker Recognition from Raw Waveform with SincNet,” *IEEE SLT*, 2018.  \n[3] J. Zeghidour et al., “LEAF: A Learnable Frontend for Audio Classification,” *ICLR*, 2021.  \n[4] A. Gu et al., “Efficiently Modeling Long Sequences with Structured State Spaces,” *ICLR*, 2022.    \n[5] A. Gu and T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” arXiv:2312.00752, 2023."},"title":{"value":"Authors Response (1/n)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764553028297,"tcdate":1764553028297,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23270/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission23270/Authors"],"forum":"SyTuymSbBJ","number":14,"license":"CC BY 4.0","cdate":1764553028297,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23270/-/Official_Comment"],"mdate":1764553028297,"domain":"ICLR.cc/2026/Conference","replyto":"Vm1WS03S8H","id":"zDCEAr2Yhj","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We propose a state-space model for learning spectral representations of time series data inspired by discrete Fourier transforms."},"keywords":{"value":["frequency spectrum","state space models","spectral operators","band-selective kernels","radar object detection","audio keyword recognition","sequence models"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"We introduce the first state space model (FSSM) with frequency selective spectral operators, parameterizing a family of stable, causal, band-selective kernels whose spectral weights are conditioned on the end task. This yields a representation that adapts its characteristics per task domain while retaining linear-time inference and memory. The key novelty is the trainable spectral front-end through which the model can adapt frequency weighting and inter-bin window size. We show the effectiveness of our learned spectral representations on two independent domains: radar object detection and speech keyword recognition, outperforming state of the art frequency based methods in both domains while maintaining competitive throughput and computational overhead. We further show the robustness of our approach under input perturbations, demonstrating the value of stabilized sequential operators in spectral representation learning."},"_bibtex":{"value":"@misc{\nsen2026fssm,\ntitle={{FSSM}: Frequency-Selective State Space Models for Spectral Representation Learning},\nauthor={Anuvab Sen and Mir Sayeed Mohammad and Saibal Mukhopadhyay},\nyear={2026},\nurl={https://openreview.net/forum?id=SyTuymSbBJ}\n}"},"title":{"value":"FSSM: Frequency-Selective State Space Models for Spectral Representation Learning"},"pdf":{"value":"/pdf/c79f7107ce97de50e6f86b3e654576f0e1200e0c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"sen|fssm_frequencyselective_state_space_models_for_spectral_representation_learning"},"authorids":{"value":["~Anuvab_Sen1","~Mir_Sayeed_Mohammad1","~Saibal_Mukhopadhyay2"]},"authors":{"value":["Anuvab Sen","Mir Sayeed Mohammad","Saibal Mukhopadhyay"]}},"version":2},{"content":{"summary":{"value":"In this paper, the authors propose MamKo, combining Koopman operator theory (projecting nonlinear systems into a high-dimensional linear space) with Mamba (a LLM based on state-space models) for accurate and efficient modeling of time-varying systems. Furthermore, based on the derived model, a model-based control approach is proposed. Simulations with different, time-varying systems demonstrate that MamKo can be superior to other SOTA methods. \n\nThe main contribution is combining these two methods (with some minor modifications), creating a powerful framework for time-varying systems."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"See weaknesses"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"- The paper is well written, the ideas are clearly formulated, and the results are well presented.\n- To the best of my knowledge, it is the first time that these two methods have been combined and applied to model time-varying systems. Thus, it's a novel and original approach.\n- The simulation results are quite convincing that the proposed method can significantly improve the modeling and control of time-varying systems."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The simulation is limited to learning method but ignores \"classic\" system identification. There is a rich literature on identifying time-varying parameters in dynamical systems. In particular, for the cart pendulum example, a comparison to classic system identification would strengthen the paper.\n- Maybe I missed the part but how is the optimal dimension of the linear space, i.e. the dimension of A_k in (5) determined?\n- In the introduction, the computational time of other koopman based approaches is discussed. However, there is no proper discussion/evaluation/theory on the training/inference time with the proposed framework."}},"nonreaders":[],"tmdate":1731428340719,"tcdate":1730672789655,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10895/Reviewer_9kPH"],"signatures":["ICLR.cc/2025/Conference/Submission10895/Reviewer_9kPH"],"forum":"hNjCVVm0EQ","number":4,"license":"CC BY 4.0","cdate":1730672789655,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10895/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428340719,"domain":"ICLR.cc/2025/Conference","replyto":"hNjCVVm0EQ","id":"458Xmn7FeE","forumContent":{"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Koopman operator; model predictive control; nonlinear systems"]},"supplementary_material":{"value":"/attachment/0bf4c76c44568ae8539ede6166bb6a47f0d2be3a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The Koopman theory, which enables the transformation of nonlinear systems into linear representations, is a powerful and efficient tool to model and control nonlinear systems. However, the ability of the Koopman operator to model complex systems, particularly time-varying systems, is limited by the fixed linear state-space representation. To address the limitation, the large language model, Mamba, is considered a promising strategy for enhancing modeling capabilities while preserving the linear state-space structure.\nIn this paper, we propose a new framework, the Mamba-based Koopman operator (MamKO), which provides enhanced model prediction capability and adaptability, as compared to Koopman models with constant Koopman operators. Inspired by the Mamba structure, MamKO generates Koopman operators from online data; this enables the model to effectively capture the dynamic behaviors of the nonlinear system over time. A model predictive control system is then developed based on the proposed MamKO model. The modeling and control performance of the proposed method is evaluated through experiments on benchmark time-invariant and time-varying systems. The experimental results demonstrate the superiority of the proposed approach. Additionally, we perform ablation experiments to test the effectiveness of individual components of MamKO. This approach unlocks new possibilities for integrating large language models with control frameworks, and it achieves a good balance between advanced modeling capabilities and real-time control implementation efficiency."},"_bibtex":{"value":"@inproceedings{\nli2025mamko,\ntitle={Mam{KO}: Mamba-based Koopman operator for modeling and predictive control},\nauthor={ZHAOYANG LI and Minghao Han and Xunyuan Yin},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=hNjCVVm0EQ}\n}"},"title":{"value":"MamKO: Mamba-based Koopman operator for modeling and predictive control"},"pdf":{"value":"/pdf/49097a18837a65cc1415fac755f61c717559a104.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"li|mamko_mambabased_koopman_operator_for_modeling_and_predictive_control"},"authorids":{"value":["~ZHAOYANG_LI12","~Minghao_Han2","~Xunyuan_Yin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["ZHAOYANG LI","Minghao Han","Xunyuan Yin"]}},"version":2},{"content":{"summary":{"value":"This paper proposes TS-Mamba, a state-of-the-art method for online video super-resolution (VSR), leveraging trajectory-aware shifted state space models (SSMs) for efficient spatio-temporal information aggregation. The model addresses challenges in long-term temporal modeling for real-time applications while keeping computational complexity low. The approach combines token-level spatio-temporal aggregation with a novel trajectory-aware shifted Mamba aggregation (TSMA) module. TS-Mamba constructs trajectories within video frames to select similar tokens from previous frames and aggregates them using shifted SSM blocks. This enables improved video frame restoration with reduced computational overhead. The proposed model is evaluated on several benchmark datasets (REDS, Vimeo-90K-T, Vid4) and outperforms five state-of-the-art online VSR methods in terms of PSNR/SSIM while reducing complexity by over 22.7%."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See the weakness."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Performance: The model achieves superior performance in terms of PSNR/SSIM and visual quality across multiple benchmark datasets (REDS, Vid4, Vimeo-90K-T) and degradation types (BI and BD), demonstrating its robustness in real-world video restoration scenarios.\n\n2. Computational Efficiency: TS-Mamba successfully reduces complexity by 22.7% in terms of MACs compared to existing methods, making it a strong candidate for real-time online VSR applications. The model is also one of the fastest among the tested methods.\n\n3. Comprehensive Ablation Study: The authors provide an extensive ablation study validating the importance of trajectory-aware components and the shifted SSM blocks, as well as the impact of different design choices (e.g., token number, shift operations). This strengthens the validity of their claims and helps illustrate the contributions of each part of the model."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Lack of Comparison. The proposed scheme fails to discuss or compare with several recent restoration schemes that leverage state-space models (SSMs, e.g. Mamba) for superior performance. For instance, MambaIR and MambaIRv2 introduced a residual Mamba-based backbone (with convolution and channel attention) to capture global dependencies in image super-resolution and denoising, outperforming a SwinIR Transformer baseline. The more recent TAMambaIR improves efficiency by modulating the state-space transition for complex textures and using multi-directional scanning, achieving state-of-the-art results across image restoration tasks (e.g. super-resolution, deraining, low-light enhancement). In the video domain, VSRM proposes dual Spatial-to-Temporal and Temporal-to-Spatial Mamba blocks for long-range spatio-temporal feature extraction and a deformable cross-Mamba alignment module for flexible frame alignment, yielding new state-of-the-art performance on VSR benchmarks. Likewise, The authors should incorporate and evaluate these methods to ensure a comprehensive comparison with the current state-of-the-art in video super-resolution. （I believe that the VSR method, when transformed into the online VSR setting, can be compared.）\n\n2. Insufficient Theoretical Justification: While the paper presents an innovative approach, the theoretical justification for the trajectory-aware shifted state space model is somewhat lacking. Specifically, a formal analysis of the long-term spatio-temporal aggregation process and a comparison with existing models using similar principles (e.g., in flow-guided deformable attention models) would provide more clarity. Additionally, the authors mention the use of shifted operations to enhance spatial continuity but could further explore the theoretical implications of this operation on model behavior.\n\n3. Insufficient Validation. A notable weakness is the absence of experiments on real-world VSR datasets. The method’s results are only reported on standard synthetic benchmarks (REDS4, Vid4, Vimeo-90K) with bicubic or simulated blur degradation, but no evaluations on real degraded videos were provided . As real applications involve unknown and complex degradations (compression artifacts, sensor noise, motion blur, etc.), the paper should have validated the approach on established real-world video SR benchmarks. For example, testing on datasets like RealVSR (ICCV 2021), which is built with a dual-camera system on the iPhone 11 Pro Max, would demonstrate the model’s robustness to in-the-wild conditions\n\n4. Potential on Certain Datasets: The model performs exceptionally well on the benchmarks but lacks a more nuanced discussion of failure cases or scenarios where the method might struggle. A brief mention of potential limitations (e.g., when frames are highly dynamic or occlusions are prevalent) would enhance the robustness of the paper's claims."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919222114,"tcdate":1761894628265,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7012/Reviewer_2xrU"],"signatures":["ICLR.cc/2026/Conference/Submission7012/Reviewer_2xrU"],"forum":"RygnSGcV49","number":4,"license":"CC BY 4.0","cdate":1761894628265,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7012/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919222114,"domain":"ICLR.cc/2026/Conference","replyto":"RygnSGcV49","id":"YoywqqaWkw","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Video Super-resolution","Online","Mamba","Trajectory"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Online video super-resolution (VSR) is an important technique for many real-world video processing applications, which aims to restore the current high-resolution video frame based on temporally previous frames. Most of the existing online VSR methods solely employ one neighboring previous frame to achieve temporal alignment, which limits long-range temporal modeling of videos. Recently, state space models (SSMs) have been proposed with linear computational complexity and a global receptive field, which significantly improve computational efficiency and performance. In this context, this paper presents a novel online VSR method based on Trajectory-aware Shifted SSMs (TS-Mamba), leveraging both long-term trajectory modeling and low-complexity Mamba to achieve efficient spatio-temporal information aggregation. Specifically, TS-Mamba first constructs the trajectories within a video to select the most similar tokens from the previous frames. Then, a Trajectory-aware Shifted Mamba Aggregation (TSMA) module consisting of proposed shifted SSMs blocks is employed to aggregate the selected tokens. The shifted SSMs blocks are designed based on Hilbert scannings and corresponding shift operations to compensate for scanning losses and strengthen the spatial continuity of Mamba. Additionally, we propose a trajectory-aware loss function to supervise the trajectory generation, ensuring the accuracy of token selection when training our model. Extensive experiments on three widely used VSR test datasets demonstrate that compared with six online VSR benchmark models, our TS-Mamba achieves state-of-the-art performance in most cases and over 22.7% complexity reduction (in MACs). The source code for TS-Mamba is available at https://github.com/QZ1-boy/TS-Mamba."},"_bibtex":{"value":"@inproceedings{\nzhu2026trajectoryaware,\ntitle={Trajectory-aware Shifted State Space Models for Online Video Super-Resolution},\nauthor={Qiang Zhu and Xiandong MENG and Yuxuan Jiang and Fan Zhang and David Bull and Shuyuan Zhu and Bing Zeng and Ronggang Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=RygnSGcV49}\n}"},"title":{"value":"Trajectory-aware Shifted State Space Models for Online Video Super-Resolution"},"pdf":{"value":"/pdf/63207ba1dfa93f1f2594d2cd3d80db2064dc0722.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhu|trajectoryaware_shifted_state_space_models_for_online_video_superresolution"},"authorids":{"value":["~Qiang_Zhu6","~Xiandong_MENG1","~Yuxuan_Jiang8","~Fan_Zhang6","~David_Bull1","~Shuyuan_Zhu1","~Bing_Zeng1","~Ronggang_Wang1"]},"authors":{"value":["Qiang Zhu","Xiandong MENG","Yuxuan Jiang","Fan Zhang","David Bull","Shuyuan Zhu","Bing Zeng","Ronggang Wang"]}},"version":2},{"content":{"summary":{"value":"The paper introduces an LPR framework for Mamba. The method breaks the Mamba architecture into three parts (SiLU activation, selective SSM, and multiplicative gating) and analyzes the layers using relevance scores. The evaluations on languages and images show that the proposed method is more precise and faithful than other explanation methods. They also show several use cases including gender bias, long-range capabilities, and the needle-in-a-haystack test."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. Tables 10 and 11 show the runtime comparisons with and without using fast CUDA kernels. When using fast CUDA kernels, other methods including gradient × Input, SmoothGrad, and Integrated Gradients improve the speed a lot (22-25x faster), but the proposed MambaLRP has only 1.3x speed-up. What is the reason?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The paper is easy to follow. \n- The introduced LRP framework is faster and provides more faithful explanations than other explanation methods. \n- The experiments show interesting interpretations of Mamba specifically for gender bias, long-context capabilities, and needle-in-a-haystack test in Mamba."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The current paper is limited to the explainability of Mamba only. I believe LRP can be applied to Transformers. It would be more interesting to compare the behavior of different methods instead of showing the behavior of one method.\n\n- This point is related to the first point. The use case experiments are interesting, but it would be more useful if this could be compared with different methods. For instance, in the long context capability of Mamba, the paper mainly shows that Mamba can use earlier context. Instead, the paper can show that one method has better long context capability than the other in terms of accuracy, and it is verified by LRP. Also, the behavior of different methods can be compared such as 1) if the Transformer-based method is not as good as Mamba in terms of long-range modeling, what does the Transformer model focus on 2) how does the behavior change with tasks (generation vs QA) 3) how does it change with the training datasets, etc. \n\nOverall, the paper lacks a more interesting and useful analysis of the methods using LRP as explained above. This limits the contribution of the paper."},"limitations":{"value":"The paper briefly explained the limitation in Section 7."}},"nonreaders":[],"tmdate":1730879312741,"tcdate":1721588911347,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission9423/Reviewer_4ExW"],"signatures":["NeurIPS.cc/2024/Conference/Submission9423/Reviewer_4ExW"],"forum":"2n1Ysn1EDl","number":3,"license":"CC BY 4.0","cdate":1721588911347,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission9423/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879312741,"domain":"NeurIPS.cc/2024/Conference","replyto":"2n1Ysn1EDl","id":"coQeSdb3ga","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Explainable AI","explainability","interpretability","state space models","Mamba","long-range dependency"]},"primary_area":{"value":"interpretability_and_explainability"},"abstract":{"value":"Recent sequence modeling approaches using selective state space sequence models, referred to as Mamba models, have seen a surge of interest. These models allow efficient processing of long sequences in linear time and are rapidly being adopted in a wide range of applications such as language modeling, demonstrating promising performance. To foster their reliable use in real-world scenarios, it is crucial to augment their transparency. Our work bridges this critical gap by bringing explainability, particularly Layer-wise Relevance Propagation (LRP), to the Mamba architecture. Guided by the axiom of relevance conservation, we identify specific components in the Mamba architecture, which cause unfaithful explanations. To remedy this issue, we propose MambaLRP, a novel algorithm within the LRP framework, which ensures a more stable and reliable relevance propagation through these components. Our proposed method is theoretically sound and excels in achieving state-of-the-art explanation performance across a diverse range of models and datasets. Moreover, MambaLRP facilitates a deeper inspection of Mamba architectures, uncovering various biases and evaluating their significance. It also enables the analysis of previous speculations regarding the long-range capabilities of Mamba models."},"_bibtex":{"value":"@inproceedings{\njafari2024mambalrp,\ntitle={Mamba{LRP}: Explaining Selective State Space Sequence Models},\nauthor={Farnoush Rezaei Jafari and Gr{\\'e}goire Montavon and Klaus Robert Muller and Oliver Eberle},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=2n1Ysn1EDl}\n}"},"title":{"value":"MambaLRP: Explaining Selective State Space Sequence Models"},"pdf":{"value":"/pdf/d253dfe5b75f15bb96b1033361699ed2984da98a.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"jafari|mambalrp_explaining_selective_state_space_sequence_models"},"authorids":{"value":["~Farnoush_Rezaei_Jafari1","~Grégoire_Montavon1","~Klaus_Robert_Muller1","~Oliver_Eberle1"]},"authors":{"value":["Farnoush Rezaei Jafari","Grégoire Montavon","Klaus Robert Muller","Oliver Eberle"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2412.00776v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"zhao|learning_mamba_as_a_continual_learner"},"authorids":{"value":["~Chongyang_Zhao1","https://dblp.org/search/pid/api?q=author:Dong_Gong:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2412.00776"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-00776,\n  publtype={informal},\n  author={Chongyang Zhao and Dong Gong},\n  title={Learning Mamba as a Continual Learner},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2412.00776},\n  url={https://doi.org/10.48550/arXiv.2412.00776}\n}\n"},"abstract":{"value":"Continual learning (CL) aims to efficiently learn and accumulate knowledge from a data stream with different distributions. By formulating CL as a sequence prediction task, meta-continual learning (MCL) enables to meta-learn an efficient continual learner based on the recent advanced sequence models, e.g., Transformers. Although attention-free models (e.g., Linear Transformers) can ideally match CL's essential objective and efficiency requirements, they usually perform not well in MCL. Considering that the attention-free Mamba achieves excellent performances matching Transformers' on general sequence modeling tasks, in this paper, we aim to answer a question -- Can attention-free Mamba perform well on MCL? By formulating Mamba with a selective state space model (SSM) for MCL tasks, we propose to meta-learn Mamba as a continual learner, referred to as MambaCL. By incorporating a selectivity regularization, we can effectively train MambaCL. Through comprehensive experiments across various CL tasks, we also explore how Mamba and other models perform in different MCL scenarios. Our experiments and analyses highlight the promising performance and generalization capabilities of Mamba in MCL."},"title":{"value":"Learning Mamba as a Continual Learner"},"authors":{"value":["Chongyang Zhao","Dong Gong"]}},"tmdate":1741146974389,"pdate":1704067200000,"tcdate":1737160249420,"writers":["~"],"signatures":["~Chongyang_Zhao1"],"forum":"T2f1kjgrmf","license":"CC BY-SA 4.0","number":267901,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1741146974389,"domain":"DBLP.org","id":"T2f1kjgrmf","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Continual Learning","Sequence Modelling"]},"primary_area":{"value":"transfer learning, meta learning, and lifelong learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Continual learning (CL) aims to efficiently learn and accumulate knowledge from a data stream with different distributions. By formulating CL as a sequence prediction task, meta-continual learning (MCL) enables to meta-learn an efficient continual learner based on the recent advanced sequence models, e.g., Transformers. Although attention-free models (e.g., Linear Transformers) can ideally match CL's essential objective and efficiency requirements, they usually perform not well in MCL. Considering that the attention-free Mamba achieves excellent performances matching Transformers' on general sequence modeling tasks, in this paper, we aim to answer a question -- Can attention-free Mamba perform well on MCL? By formulating Mamba with a selective state space model (SSM) for MCL tasks, we propose to meta-learn Mamba as a continual learner, referred to as MambaCL. By incorporating a selectivity regularization, we can effectively train MambaCL. Through comprehensive experiments across various CL tasks, we also explore how Mamba and other models perform in different MCL scenarios. Our experiments and analyses highlight the promising performance and generalization capabilities of Mamba in MCL."},"_bibtex":{"value":"@misc{\nzhao2025learning,\ntitle={Learning Mamba as a Continual Learner},\nauthor={Chongyang Zhao and Dong Gong},\nyear={2025},\nurl={https://openreview.net/forum?id=1TXDtnDIsV}\n}"},"title":{"value":"Learning Mamba as a Continual Learner"},"pdf":{"value":"/pdf/65bb972e176ce4ef1a4d56629ad29b0e78847ac1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhao|learning_mamba_as_a_continual_learner"},"authorids":{"value":["~Chongyang_Zhao1","~Dong_Gong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chongyang Zhao","Dong Gong"]}},"tmdate":1738735610872,"tcdate":1726186115934,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission9/Authors"],"forum":"1TXDtnDIsV","license":"CC BY 4.0","number":9,"cdate":1726186115934,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission9/-/Full_Submission","ICLR.cc/2025/Conference/Submission9/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735610872,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"1TXDtnDIsV","version":2},{"content":{"summary":{"value":"This paper explores enhancing EHR data processing models by assessing the impact of context length and model architecture on clinical prediction tasks. Four architectures—GPT, Llama, Hyena, and Mamba—were tested with various context lengths, with Mamba (16k token context) achieving state-of-the-art results in 9 of 14 EHRSHOT benchmark tasks. The study identifies three EHR-specific factors influencing model performance: copy-forwarding (repeated diagnoses), irregular time intervals, and disease progression. Findings show longer contexts generally improve performance, with robustness in EHR data handling, though results vary across architectures, as Hyena showed performance drops beyond 4k tokens. This research advances medical data prediction and offers insights on processing long-sequence data, despite limitations like transformer model computational demands and single-institution data."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Since the author's testing only showed good results with small Mamba models, it's uncertain whether larger Mamba models would perform better than traditional GPT."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Introduce non-Transformer architectures (such as Mamba) to process medical data.\n\n2. The paper demonstrates the potential of the Mamba model’s long-context capabilities for future clinical applications.\n\n3. The advantage of this paper is that Mamba can perform well with linear complexity."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Both GPT and LLaMA have a maximum context length of only 4096 tokens, so it's not appropriate for the authors to conduct tests with 16K length as a benchmark.\n\n2.The authors should provide additional details about what the 14 tasks in EHRSHOT include to facilitate better comparison.\n\n3. The paper mentions using the standard deviation of time intervals to divide patients into four groups (Q1-Q4). Regarding the groups with the most regular time intervals and the most irregular time intervals, the standards vary across different diseases, and testing of time intervals should be conducted according to specific diseases.\n\n4. The paper mentions achieving an AUROC of 0.807, but it's confusing that the specific content of the 14 tasks is not listed."}},"nonreaders":[],"tmdate":1732576185942,"tcdate":1731084232512,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13179/Reviewer_ACwG"],"signatures":["ICLR.cc/2025/Conference/Submission13179/Reviewer_ACwG"],"forum":"zg3ec1TdAP","number":5,"license":"CC BY 4.0","cdate":1731084232512,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13179/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732576185942,"domain":"ICLR.cc/2025/Conference","replyto":"zg3ec1TdAP","id":"z0pnflZ41R","forumContent":{"TLDR":{"value":"An analysis of the impact of context length on foundation models trained on structured electronic health record (EHR) data for clinical prediction tasks."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["ehr","foundation model","long context","clinical prediction making","healthcare"]},"supplementary_material":{"value":"/attachment/b6ee9ac9232dd3da6857ced0d9ddb2aa470cc775.pdf"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Foundation Models (FMs) trained on Electronic Health Records (EHRs) have achieved state-of-the-art results on numerous clinical prediction tasks. However, prior EHR FMs typically have context windows of $<$1k tokens, which prevents them from modeling full patient EHRs which can exceed 10k's of events. For making clinical predictions, both model performance and robustness to the unique properties of EHR data are crucial. Recent advancements in subquadratic long-context architectures (e.g. Mamba) offer a promising solution. However, their application to EHR data has not been well-studied. We address this gap by presenting the first systematic evaluation of the effect of context length on modeling EHR data. We find that longer context models improve predictive performance -- our Mamba-based model surpasses the prior state-of-the-art on 9/14 tasks on the EHRSHOT prediction benchmark. Additionally, we measure robustness to three unique, previously underexplored properties of EHR data: (1) the prevalence of ``copy-forwarded\" diagnoses which create artificial token repetition in EHR sequences; (2) the irregular time intervals between EHR events which can lead to a wide range of timespans within a context window; and (3) the natural increase in disease complexity over time which makes later tokens in the EHR harder to predict than earlier ones. Stratifying our EHRSHOT results, we find that higher levels of each property correlate negatively with model performance (e.g., a 14% higher Brier loss between the least and most irregular patients), but that longer context models are more robust to more extreme levels of these properties. Our work highlights the potential for using long-context architectures to model EHR data, and offers a case study on how to identify and quantify new challenges in modeling sequential data motivated by domains outside of natural language. We release all of our model checkpoints and code."},"_bibtex":{"value":"@inproceedings{\nwornow2025context,\ntitle={Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on {EHR} Data},\nauthor={Michael Wornow and Suhana Bedi and Miguel Angel Fuentes Hernandez and Ethan Steinberg and Jason Alan Fries and Christopher Re and Sanmi Koyejo and Nigam Shah},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=zg3ec1TdAP}\n}"},"title":{"value":"Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data"},"pdf":{"value":"/pdf/b21eff6a613bf00fac9c545c0a6c7f2df1c9e5a0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wornow|context_clues_evaluating_long_context_models_for_clinical_prediction_tasks_on_ehr_data"},"authorids":{"value":["~Michael_Wornow1","~Suhana_Bedi1","~Miguel_Angel_Fuentes_Hernandez1","~Ethan_Steinberg1","~Jason_Alan_Fries1","~Christopher_Re1","~Sanmi_Koyejo1","~Nigam_Shah1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Michael Wornow","Suhana Bedi","Miguel Angel Fuentes Hernandez","Ethan Steinberg","Jason Alan Fries","Christopher Re","Sanmi Koyejo","Nigam Shah"]}},"version":2},{"content":{"venue":{"value":"NeurIPS 2025 poster"},"TLDR":{"value":"LrcSSM is a nonlinear RNN with parallel prefix-scan updates, formal gradient stability, and compute-optimal scaling. It outperforms LRU, S5, and Mamba on long-range tasks - matching SSM speed without attention or FFT overhead."},"keywords":{"value":["state space model","non-linear RNN","liquid neural network","parallelization","sequence modeling"]},"supplementary_material":{"value":"/attachment/72a1758b1e8d1aeb5a379ade6be10e8f7ece6b00.zip"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"We present LrcSSM, a $\\textit{non-linear}$ recurrent model that processes long sequences as fast as today's linear state-space layers. By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel, giving $\\mathcal{O}(TD)$ computational work and memory and only $\\mathcal{O}(\\log T)$ sequential depth, for input-sequence length $T$ and a state dimension $D$. Moreover, LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 and Mamba do not provide. Importantly, the diagonal Jacobian structure of our model results in no performance loss compared to the original model with dense Jacobian, and the approach can be generalized to other non-linear recurrent models, demonstrating broader applicability. On a suite of long-range forecasting tasks, we demonstrate that LrcSSM outperforms Transformers, LRU, S5, and Mamba."},"_bibtex":{"value":"@inproceedings{\nfarsang2025parallelization,\ntitle={Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling},\nauthor={M{\\'o}nika Farsang and Radu Grosu},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=ENYvdnyhLl}\n}"},"title":{"value":"Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling"},"pdf":{"value":"/pdf/6102c945cda7f097a0fe360fe046def01669279b.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"farsang|parallelization_of_nonlinear_statespace_models_scaling_up_liquidresistance_liquidcapacitance_networks_for_efficient_sequence_modeling"},"authorids":{"value":["~Mónika_Farsang1","~Radu_Grosu1"]},"authors":{"value":["Mónika Farsang","Radu Grosu"]}},"tmdate":1783627344893,"pdate":1758217145678,"tcdate":1746902095574,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission17571/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission17571/Authors"],"forum":"ENYvdnyhLl","license":"CC BY 4.0","number":17571,"cdate":1746902095574,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission17571/-/Full_Submission","NeurIPS.cc/2025/Conference/Submission17571/-/Supplementary_Material","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission17571/-/Camera_Ready_Revision"],"mdate":1783627344893,"odate":1761704952145,"domain":"NeurIPS.cc/2025/Conference","id":"ENYvdnyhLl","version":2},{"content":{"comment":{"value":"###  **Q1.  Hyperparameter lambda and the effect of regularization**\n\n\nAs the reviewer mentioned **Table I.1** illustrates that the inclusion of the regularization loss significantly improves the performance **regardless of the value of lambda**.\n\n\nThis phenomenon is reasonable, as shown in **Figure 9** and **Figure G.1**, where the regularization loss converges very quickly early in the training process for our proposed methods. This rapid convergence suggests that the regularization effectively aligns the two reversed-order vectors with ease, yielding consistent performance improvements regardless of the value of lambda.\n\n&nbsp;\n\n\n---\n###  **Q2.  Motivation of (1) single unidirectional Mamba and (2) removal of 1D-conv.**\n\n\n> (1) **Motivation of single unidirectional Mamba**\n\n\nPrevious research used two separate Mambas to capture channel dependencies from both directions [A, B] in an effort to address the channel order bias. However, we found that using bidirectional Mamba does not always yield the best performance (**Table 1**) and can be inefficient (**L51--52**). By using a single Mamba with regularization loss, we effectively address this bias, achieving superior performance (**Table 2,4,5**)  and improved efficiency (**Table 13**) compared to using two Mambas.\"\n\n&nbsp;\n\n> **(2) Motivation of removal of 1D-conv**\n\n\nThe 1D-conv in Mamba [C] was originally designed to capture local information in sequential data (**L66--68**), making it less suitable for non-sequential data (i.e., channels in TS). However, previous works have overlooked the need to remove the 1D-conv when using Mamba to capture channel dependencies (CD) [D,E,F], where channels do not have sequential order. This has motivated us to remove it from the Mamba block. Further details on the removal of the 1D-conv are provided in **Appendix D**.\n\n\n&nbsp;\n\n\n[A] Wang, Zihan, et al. \"Is mamba effective for time series forecasting?.\" arXiv (2024)\n\n\n[B] Liang, Aobo, et al. \"Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting.\" arXiv (2024) \n\n\n[C] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv (2024)\n\n\n[D] Weng, Zixuan, et al. \"Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting.\" arXiv (2024)\n\n\n[E] Cai, Xiuding, et al. \"MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting.\" arXiv (2024)\n\n\n[F] Ma, Shusen, et al. \"Fmamba: Mamba based on fast-attention for multivariate time-series forecasting.\" arXiv (2024)\n\n&nbsp;\n\n\n---\n###  **Q3. Lack of details regarding CCM**\n\n\nWe apologize for the lack of detail regarding the proposed pretraining task, CCM. To address this, we have included pseudocode for the task in **Appendix J** and provided additional guidance for readers in **L244**."}},"tmdate":1731649227105,"tcdate":1731647408212,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission69/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission69/Authors"],"forum":"fyl82vAale","number":9,"license":"CC BY 4.0","cdate":1731647408212,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission69/-/Official_Comment"],"mdate":1731649227105,"domain":"ICLR.cc/2025/Conference","replyto":"Jc94nxQCsE","id":"jt2HiO1f6w","forumContent":{"TLDR":{"value":"We propose SOR-Mamba, a time series forecasting method that uses Mamba with regularization to address the sequential order bias in capturing channel dependencies."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","State-space Model","Time Series"]},"supplementary_material":{"value":"/attachment/a3e6bf94e3ea1b0642f98af3f08c4590c0c5d03e.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data.\nHowever, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.\nExtensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios."},"_bibtex":{"value":"@misc{\nlee2025sequential,\ntitle={Sequential Order-Robust Mamba for Time Series Forecasting},\nauthor={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\nyear={2025},\nurl={https://openreview.net/forum?id=fyl82vAale}\n}"},"title":{"value":"Sequential Order-Robust Mamba for Time Series Forecasting"},"pdf":{"value":"/pdf/2172851f730b05caa6144b837024da19c84263c0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"lee|sequential_orderrobust_mamba_for_time_series_forecasting"},"authorids":{"value":["~Seunghan_Lee1","~Juri_Hong1","~Kibok_Lee1","~Taeyoung_Park1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seunghan Lee","Juri Hong","Kibok Lee","Taeyoung Park"]}},"version":2},{"content":{"summary":{"value":"## **Summary**\nThis paper proposes **White-Basilisk**, a hybrid architecture for code vulnerability detection that integrates Mamba (state-space modeling), linear self-attention, and a Mixture of Experts (MoE) framework. The model aims to achieve efficient long-context processing while maintaining strong representational power. It is evaluated on five public vulnerability datasets (e.g., BigVul, PRIMEVUL, VulDeepecker) and reports state-of-the-art results with only 200M parameters. The paper also claims improved scalability and performance on imbalanced data distributions, positioning White-Basilisk as a compact yet competitive alternative to larger transformer-based models."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"1. **Theoretical Justification**  \n   The main contribution seems to be an engineering combination of Mamba, linear self-attention, and Mixture of Experts.  \n   - Could you clarify the theoretical motivation behind this hybrid design?  \n   - Why should these mechanisms be complementary rather than redundant?  \n   - Is there any analytical or empirical evidence that this combination improves generalization or stability beyond computational efficiency?\n\n2. **Motivation and Problem Framing**  \n   The introduction presents a general “bigger is not always better” argument but lacks domain-specific reasoning.  \n   - What *conceptual failures* of existing models does your method aim to solve in code vulnerability detection?  \n   - How does the proposed hybrid architecture leverage structural characteristics of code, such as control or data flow?\n\n3. **Ablation and Component Analysis**  \n   Multiple components are introduced, yet no ablation studies are provided.  \n   - Could you present quantitative results for removing or replacing **Mamba**, **MoE**, or **Basilisk Self-Attention**?  \n   - Which component contributes most to the reported performance improvements and why?\n\n4. **Fine-Tuning and Evaluation Details**  \n   - Were duplicate or overlapping samples across datasets (e.g., *BigVul*, *VulDeepecker*) removed before fine-tuning?  \n   - How were early stopping, validation splits, and hyperparameter selections determined?  \n   - Please clarify whether class balancing or weighting was applied.\n\n5. **Reproducibility and Statistical Reliability**  \n   - Could you provide standard deviations or significance tests for key metrics?  \n   - If possible, please release the evaluation scripts or processed subsets to facilitate reproducibility."},"rating":{"value":4},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"## **Strengths**\n\n1. **Experiment Quality**  \n   The experimental section is extensive in dataset coverage, spanning multiple well-known benchmarks. The reported results suggest strong empirical performance and scalability, demonstrating the potential of hybrid sequence modeling beyond traditional transformers.\n\n2. **Clarity**  \n   The paper is generally well-organized, with detailed architectural diagrams and formulas. The presentation of computational complexity and efficiency claims is clear.\n\n4. **Significance**  \n   The work tackles a high-impact, real-world problem — automated software vulnerability detection — where efficiency and scalability are crucial. If the reported gains are validated, the proposed hybrid approach could have meaningful implications for applying large-sequence models in security-critical domains."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"## **1. Theoretical and Motivation Issues**\n\n### **1) Substantive Theoretical Contribution is weak**\nThe paper’s central contribution — combining Mamba, linear self-attention, and Mixture of Experts — is essentially an engineering-level hybridization of three existing techniques, each already well-studied in prior work. There is no mathematical derivation or new theoretical insight to show why this combination yields better generalization or stability. For example, the “Basilisk Self-Attention” is described mainly as an incremental variant of Infini-Attention, without any theoretical justification for its claimed improvements.\n\n### **2) Weak and overly generic motivation**\nThis paper doesn't articulate why existing models fail conceptually or how the proposed hybrid mechanism directly addresses those failures beyond computational efficiency. The “bigger is not always better” argument lacks concrete analytical grounding and does not build a clear hypothesis linking the hybrid architecture to the domain-specific characteristics of code vulnerability detection.\n\n## **2. Experimental Weaknesses**\n\n### **1) Missing Ablation Studies**\nAlthough the paper lists multiple architectural components, it does not isolate their individual contributions. The “Ablation Study” section mentioned in the appendix index isn't clearly presented in the main results. There are no results showing the effect of removing Mamba, MoE, or Basilisk attention individually, making it impossible to attribute the reported gains to some specific design parts.\n\n### **2) Insufficient Details in Fine-Tuning and Evaluation**\nFine-tuning hyperparameters, validation-split criteria, and early stopping strategies are only listed generically (in an appendix table) without explanations or empirical rationale. Moreover, it remains unclear whether data deduplication, class balancing, or data leakage prevention were applied during fine-tuning—critical issues in vulnerability datasets such as BigVul and VulDeepecker.\n\n## **3. Writing Issues**\n\nThe introduction focuses excessively on industrial motivation while missing conceptual gaps in existing architectures. The theoretical and experimental sections are densely filled with formulae and data but lack an interpretive statement."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933673072,"tcdate":1761825859602,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20162/Reviewer_ShmY"],"signatures":["ICLR.cc/2026/Conference/Submission20162/Reviewer_ShmY"],"forum":"Qq3efdQp3f","number":4,"license":"CC BY 4.0","cdate":1761825859602,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20162/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933673072,"domain":"ICLR.cc/2026/Conference","replyto":"Qq3efdQp3f","id":"o6qgLGgbmD","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["vulnerability detection","llm","linear attention","mamba","white-basilisk","nlp","moe","efficient","source code","c","c++"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"The proliferation of software vulnerabilities presents a significant challenge to cybersecurity, necessitating more effective detection methodologies. We introduce White-Basilisk, a hybrid approach to vulnerability detection that demonstrates strong performance with efficient architectural design. Utilizing an architecture that integrates Mamba layers, linear self-attention, and a Mixture of Experts framework, White-Basilisk achieves state-of-the-art results in vulnerability detection tasks with a parameter count of only 200M. The model's capacity to process extended sequences enables comprehensive analysis of large codebases in a single pass, addressing context limitations that affect current approaches. White-Basilisk exhibits robust performance on imbalanced, real-world datasets, while maintaining computational efficiency that facilitates deployment across diverse organizational scales. This research establishes new benchmarks in code security and provides empirical evidence that compact, efficiently designed models can achieve competitive performance on specialized tasks, contributing to our understanding of architectural efficiency in domain-specific applications."},"_bibtex":{"value":"@misc{\nlamprou2026whitebasilisk,\ntitle={White-Basilisk: A Hybrid Model for Code Vulnerability Detection},\nauthor={Ioannis Lamprou and Alexander Shevtsov and Ioannis Arapakis and Sotiris Ioannidis},\nyear={2026},\nurl={https://openreview.net/forum?id=Qq3efdQp3f}\n}"},"title":{"value":"White-Basilisk: A Hybrid Model for Code Vulnerability Detection"},"pdf":{"value":"/pdf/16cc726186b8b13cc45dcfb8f674abd5d1058896.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"lamprou|whitebasilisk_a_hybrid_model_for_code_vulnerability_detection"},"authorids":{"value":["~Ioannis_Lamprou1","~Alexander_Shevtsov1","~Ioannis_Arapakis1","~Sotiris_Ioannidis1"]},"authors":{"value":["Ioannis Lamprou","Alexander Shevtsov","Ioannis Arapakis","Sotiris Ioannidis"]}},"version":2},{"content":{"comment":{"value":"---\n\n### Weakness 4: Analysis of design choices (phase alignment, damping, parameterization)\n\n> “There are multiple components proposed in section 3, and many of them are not totally intuitive. Is it possible to have ablation for the phase alignment, learnable damping, and the proposed parameterizations?”\n\nWe agree that these ablations are important. We have run internal variants and will report the results. In summary:\n\n- **No learnable damping (fix \\(r_k = 1\\)).**  \n\n  This removes explicit bandwidth control and effectively reduces each mode to an undamped sinusoid. Performance drops more clearly, especially in noisy low-SNR settings: up to about 2.5% loss on Speech Commands accuracy and a 3.8% mIoU/F1 drop on RADIal, illustrating that being able to tune the effective window length (and thus leakage/side-lobe behavior) is important [1, 2].\n\n- **Unconstrained parameterization (no stability constraints).**  \n\n  This occasionally yields unstable modes and erratic training curves; gradients can explode for certain sequences. The constrained parameterization we adopt (ensuring stable poles for the underlying recurrences [3, 4]) eliminates these issues and yields smoother, more reliable optimization.\n\n---\n\n**References**\n\n[1] A. V. Oppenheim and R. W. Schafer, *Discrete-Time Signal Processing*, 3rd ed., Prentice Hall, 2009.  \n[2] F. J. Harris, “On the Use of Windows for Harmonic Analysis with the Discrete Fourier Transform,” *Proc. IEEE*, vol. 66, no. 1, pp. 51–83, 1978.    \n[3] A. Gu et al., “Efficiently Modeling Long Sequences with Structured State Spaces,” *ICLR*, 2022.  \n[4] A. Gu and T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” arXiv:2312.00752, 2023."},"title":{"value":"Authors Response (4/n)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764840730115,"tcdate":1764557466403,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23270/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission23270/Authors"],"forum":"SyTuymSbBJ","number":17,"license":"CC BY 4.0","cdate":1764557466403,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23270/-/Official_Comment"],"mdate":1764840730115,"domain":"ICLR.cc/2026/Conference","replyto":"Vm1WS03S8H","id":"99ZcfhBNtx","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We propose a state-space model for learning spectral representations of time series data inspired by discrete Fourier transforms."},"keywords":{"value":["frequency spectrum","state space models","spectral operators","band-selective kernels","radar object detection","audio keyword recognition","sequence models"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"We introduce the first state space model (FSSM) with frequency selective spectral operators, parameterizing a family of stable, causal, band-selective kernels whose spectral weights are conditioned on the end task. This yields a representation that adapts its characteristics per task domain while retaining linear-time inference and memory. The key novelty is the trainable spectral front-end through which the model can adapt frequency weighting and inter-bin window size. We show the effectiveness of our learned spectral representations on two independent domains: radar object detection and speech keyword recognition, outperforming state of the art frequency based methods in both domains while maintaining competitive throughput and computational overhead. We further show the robustness of our approach under input perturbations, demonstrating the value of stabilized sequential operators in spectral representation learning."},"_bibtex":{"value":"@misc{\nsen2026fssm,\ntitle={{FSSM}: Frequency-Selective State Space Models for Spectral Representation Learning},\nauthor={Anuvab Sen and Mir Sayeed Mohammad and Saibal Mukhopadhyay},\nyear={2026},\nurl={https://openreview.net/forum?id=SyTuymSbBJ}\n}"},"title":{"value":"FSSM: Frequency-Selective State Space Models for Spectral Representation Learning"},"pdf":{"value":"/pdf/c79f7107ce97de50e6f86b3e654576f0e1200e0c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"sen|fssm_frequencyselective_state_space_models_for_spectral_representation_learning"},"authorids":{"value":["~Anuvab_Sen1","~Mir_Sayeed_Mohammad1","~Saibal_Mukhopadhyay2"]},"authors":{"value":["Anuvab Sen","Mir Sayeed Mohammad","Saibal Mukhopadhyay"]}},"version":2},{"content":{"summary":{"value":"This paper investigate video snapshot compression imaging reconstruction task by Quad-Bayer CFA pattern into color video SCI. They design a Residual-Mamba-Block consisting of ST-Mamba, Edge-Detail-Reconstruction module and Channel-wise attention module to enhance reconstruction quality and edge details. Experimental results on many datasets validate its effectiveness."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Why we need Mamba instead of self-attention in this task?\n\nWhy quad-Bayer pattern is better than a single Bayer-patterned measurement?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"It is interesting to introduce Mamba into a new task.\n\nThey investigate long sequence modeling to further validate the effectiveness of ST-Mamba.\n\nThey proposed a novel framework for video SCI reconstruction task, which outperform existing methods not only in effectiveness but also in efficiency."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The contribution (ii) and (iii) have limited novelty. It seems that the author simply adapt ST-Mamba to video SCI reconstruction task without elaborated designs. The proposed Residual-Mamba-Block is an incremental combination of ST-Mamba from Vivim, DWConv and Channel-wise Attention. The author should further clarify the novelty.\n\nMore description of EDR module should be given. Why EDR module can provide more edge details?\n\nObserved from Table 4, the improvements on large-scale simulation color video are not consistent across datasets. More analysis on such phenomenon should be given. While the author claim the results on Messi and Hummingbird is strained by parameters and FLOPs, specific quantitative comparison should be provided."},"limitations":{"value":"The author is suggested to compare the efficiency of the proposed method and existing methods on long videos to validate the effectiveness on long sequence."}},"nonreaders":[],"tmdate":1730879651460,"tcdate":1720453433125,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission13667/Reviewer_xnVL"],"signatures":["NeurIPS.cc/2024/Conference/Submission13667/Reviewer_xnVL"],"forum":"U4WeoyRHPd","number":2,"license":"CC BY 4.0","cdate":1720453433125,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission13667/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879651460,"domain":"NeurIPS.cc/2024/Conference","replyto":"U4WeoyRHPd","id":"7LxDJuwGly","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Snapshot Compressive Imaging; Compressive Imaging; Deep Learning; Attention; Coded Aperture Compressive Temporal Imaging (CACTI); State Space Models"]},"supplementary_material":{"value":"/attachment/5497c5a57122f835d37bd551c00f9d42d8905317.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Color video snapshot compressive imaging (SCI) employs computational imaging techniques to capture multiple sequential video frames in a single Bayer-patterned measurement. With the increasing popularity of quad-Bayer pattern in mainstream smartphone cameras for capturing high-resolution videos, mobile photography has become more accessible to a wider audience. However, existing color video SCI reconstruction algorithms are designed based on the traditional Bayer pattern. When applied to videos captured by quad-Bayer cameras, these algorithms often result in color distortion and ineffective demosaicing, rendering them impractical for primary equipment. To address this challenge, we propose the MambaSCI method, which leverages the Mamba and UNet architectures for efficient reconstruction of quad-Bayer patterned color video SCI. To the best of our knowledge, our work presents the first algorithm for quad-Bayer patterned SCI reconstruction, and also the initial application of the Mamba model to this task. Specifically, we customize Residual-Mamba-Blocks, which residually connect the Spatial-Temporal Mamba (STMamba), Edge-Detail-Reconstruction (EDR) module, and Channel Attention (CA) module. Respectively, STMamba is used to model long-range spatial-temporal dependencies with linear complexity, EDR is for better edge-detail reconstruction, and CA is used to compensate for the missing channel information interaction in Mamba model. Experiments demonstrate that MambaSCI surpasses state-of-the-art methods with lower computational and memory costs. PyTorch style pseudo-code for the core modules is provided in the supplementary materials. Code is at https://github.com/PAN083/MambaSCI."},"_bibtex":{"value":"@inproceedings{\npan2024mambasci,\ntitle={Mamba{SCI}: Efficient Mamba-{UN}et for Quad-Bayer Patterned Video Snapshot Compressive Imaging},\nauthor={Zhenghao Pan and Haijin Zeng and Jiezhang Cao and Yongyong Chen and Kai Zhang and Yong Xu},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=U4WeoyRHPd}\n}"},"title":{"value":"MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging"},"pdf":{"value":"/pdf/72c3ea7ea0eeba8e9718d04e2061a016d54bfee0.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"pan|mambasci_efficient_mambaunet_for_quadbayer_patterned_video_snapshot_compressive_imaging"},"authorids":{"value":["~Zhenghao_Pan1","~Haijin_Zeng1","~Jiezhang_Cao2","~Yongyong_Chen1","~Kai_Zhang8","~Yong_Xu9"]},"authors":{"value":["Zhenghao Pan","Haijin Zeng","Jiezhang Cao","Yongyong Chen","Kai Zhang","Yong Xu"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2409.07163v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"cao|mamba_policy_towards_efficient_3d_diffusion_policy_with_hybrid_selective_state_models"},"authorids":{"value":["~Jiahang_Cao1","~Qiang_Zhang10","https://dblp.org/search/pid/api?q=author:Jingkai_Sun:","https://dblp.org/search/pid/api?q=author:Jiaxu_Wang:","~Hao_Cheng15","https://dblp.org/search/pid/api?q=author:Yulin_Li:","~Jun_Ma7","https://dblp.org/search/pid/api?q=author:Yecheng_Shao:","https://dblp.org/search/pid/api?q=author:Wen_Zhao:","https://dblp.org/search/pid/api?q=author:Gang_Han:","https://dblp.org/search/pid/api?q=author:Yijie_Guo:","https://dblp.org/search/pid/api?q=author:Renjing_Xu:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2409.07163"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2409-07163,\n  publtype={informal},\n  author={Jiahang Cao and Qiang Zhang and Jingkai Sun and Jiaxu Wang and Hao Cheng and Yulin Li and Jun Ma and Yecheng Shao and Wen Zhao and Gang Han and Yijie Guo and Renjing Xu},\n  title={Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2409.07163},\n  url={https://doi.org/10.48550/arXiv.2409.07163}\n}\n"},"abstract":{"value":"Diffusion models have been widely employed in the field of 3D manipulation due to their efficient capability to learn distributions, allowing for precise prediction of action trajectories. However, diffusion models typically rely on large parameter UNet backbones as policy networks, which can be challenging to deploy on resource-constrained devices. Recently, the Mamba model has emerged as a promising solution for efficient modeling, offering low computational complexity and strong performance in sequence modeling. In this work, we propose the Mamba Policy, a lighter but stronger policy that reduces the parameter count by over 80% compared to the original policy network while achieving superior performance. Specifically, we introduce the XMamba Block, which effectively integrates input information with conditional features and leverages a combination of Mamba and Attention mechanisms for deep feature extraction. Extensive experiments demonstrate that the Mamba Policy excels on the Adroit, Dexart, and MetaWorld datasets, requiring significantly fewer computational resources. Additionally, we highlight the Mamba Policy's enhanced robustness in long-horizon scenarios compared to baseline methods and explore the performance of various Mamba variants within the Mamba Policy framework. Our project page is in https://andycao1125.github.io/mamba_policy/."},"title":{"value":"Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Models"},"authors":{"value":["Jiahang Cao","Qiang Zhang","Jingkai Sun","Jiaxu Wang","Hao Cheng","Yulin Li","Jun Ma","Yecheng Shao","Wen Zhao","Gang Han","Yijie Guo","Renjing Xu"]}},"tmdate":1784541562712,"pdate":1704067200000,"tcdate":1731476930064,"writers":["~"],"signatures":["~Hao_Cheng15"],"forum":"bvyZQUcNWH","license":"CC BY-SA 4.0","number":203396,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784541562712,"domain":"DBLP.org","id":"bvyZQUcNWH","version":2},{"content":{"summary":{"value":"This paper proposes MBMamba, a structure-aware image deblurring network that augments the Mamba state-space model with a memory buffering mechanism (MemVSSM) and an Ising-inspired regularization loss.\nThe memory buffer preserves historical context to mitigate pixel forgetting and channel redundancy inherent in Mamba’s flatten-and-scan strategy, while the Ising loss enforces spatial coherence by mimicking energy minimization between neighboring pixels.\nThe model achieves state-of-the-art PSNR/SSIM on both synthetic (GoPro, HIDE) and real-world (RealBlur) datasets, with notably reduced computational cost."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"The FIFO memory and cross-attention introduce potential non-stationarity in gradient propagation. Does the memory cause gradient explosion or saturation in deeper decoders? How is the memory reset between sub-decoders? These engineering choices critically affect convergence yet are not reported.\n\nProvide a mathematical derivation or theoretical interpretation of how memory fusion modifies the Mamba update rule. Include sensitivity studies for memory depth K, chunk size N, and Ising loss weight.\n\nCompare Ising loss with TV or Laplacian smoothness regularizers to establish distinct advantages."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1.\tElegant structural innovation.\nThe proposed memory buffer mechanism cleverly leverages temporal-like dependency modeling to retain local context within the Mamba framework, without modifying the core architecture. This is a nontrivial conceptual advancement, positioning MBMamba as a “plug-in” enhancement rather than a structural overhaul.\n2.\tIsing-inspired loss bridging physics and perception.\nThe introduction of the Ising regularizer is an intriguing idea that connects statistical physics with image restoration. Its simplicity and interpretability make it both effective and theoretically meaningful for enforcing spatial smoothness."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tLack of theoretical or mathematical grounding of the memory buffer mechanism\nThe design is described algorithmically but not theoretically. How does the memory depth K influence the state propagation in Mamba’s continuous SSM formulation? Is the memory fusion equivalent to modifying the recurrent transition kernel? Does this break the linearity and stability guarantees of Mamba’s state evolution? A deeper analysis (e.g., spectrum or stability study) would elevate the contribution from empirical to theoretical.\n2.\tPotential redundancy between Mamba’s state memory and the added buffer\nMamba inherently encodes long-range dependencies through selective state updates. The proposed buffer essentially adds a short-term memory atop it. The authors should clarify whether the memory buffer complements or duplicates the SSM’s internal recurrence. Ablations varying K or removing cross-attention could clarify this synergy."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764358205640,"tcdate":1762001958944,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7244/Reviewer_NRXa"],"signatures":["ICLR.cc/2026/Conference/Submission7244/Reviewer_NRXa"],"forum":"uAmbHy4R8f","number":3,"license":"CC BY 4.0","cdate":1762001958944,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7244/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764358205640,"domain":"ICLR.cc/2026/Conference","replyto":"uAmbHy4R8f","id":"F5GuhJY2HS","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Image restoration","Image deblurring","Mamba"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"The Mamba architecture has emerged as a promising alternative to CNNs and Transformers for image deblurring. However, its flatten-and-scan strategy often results in local pixel forgetting and channel redundancy, limiting its ability to effectively aggregate 2D spatial information. Although existing methods mitigate this by modifying the scan strategy or incorporating local feature modules, it increase computational complexity and hinder real-time performance. In this paper, we propose a structure-aware image deblurring network  without changing the original Mamba architecture. Specifically, we design a memory buffer mechanism to preserve historical information for later fusion, enabling reliable modeling of relevance between adjacent features. Additionally, we introduce an Ising-inspired regularization loss that simulates the energy minimization of the physical system's \"mutual attraction\" between pixels, helping to maintain image structure and coherence. Building on this, we develop MBMamba. Experimental results show that our method outperforms state-of-the-art approaches."},"_bibtex":{"value":"@misc{\ngao2026mbmamba,\ntitle={{MBM}amba: When Memory Buffer Meets Mamba for Structure-Aware Image Deblurring},\nauthor={Hu Gao and XiaoNing Lei and Xichen Xu and Depeng Dang and Lizhuang Ma},\nyear={2026},\nurl={https://openreview.net/forum?id=uAmbHy4R8f}\n}"},"title":{"value":"MBMamba: When Memory Buffer Meets Mamba for Structure-Aware Image Deblurring"},"pdf":{"value":"/pdf/aea2419687a8eab97916ee02e0d79b5e35b65c47.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"gao|mbmamba_when_memory_buffer_meets_mamba_for_structureaware_image_deblurring"},"authorids":{"value":["~Hu_Gao1","~XiaoNing_Lei1","~Xichen_Xu1","~Depeng_Dang1","~Lizhuang_Ma1"]},"authors":{"value":["Hu Gao","XiaoNing Lei","Xichen Xu","Depeng Dang","Lizhuang Ma"]}},"version":2},{"content":{"summary":{"value":"This paper presents VMamba, a novel vision backbone model inspired by the famous Mamba state-space sequence model. The main contribution of VMamba is its ability to achieve efficient visual representation learning with linear computational complexity. The core of VMamba is the VSS block, which incorporates the 2D-Selective-Scan module (SS2D), thereby extending the Mamba model that is a 1D selective scan good for NLP tasks. With SS2D, we can work nicely with inductive biases associated with 2D image space. \n\nVMamba's architecture consists of multiple stages with hierarchical representations (similar to ViT). The authors introduce three model sizes: Tiny, Small, and Base. The VSS blocks replace the S6 module from Mamba with the SS2D module, and further enhancements are made by eliminating unnecessary components and optimizing the architecture for better computational efficiency - using the Triton language.\n\nExtensive experiments demonstrate VMamba's promising performance across various visual perception tasks, including image classification on ImageNet-1K, object detection, instance segmentation on MSCOCO, and semantic segmentation on ADE20K. VMamba consistently achieves superior accuracy and throughput compared to existing benchmark models, showcasing its scalability and adaptability to different input resolutions and downstream tasks."},"soundness":{"value":4},"confidence":{"value":5},"questions":{"value":"- Could you provide more detailed comparisons with other state-space models (SSMs) used in the vision domain, such as S4ND and Vim? Specifically, how does VMamba perform in terms of accuracy, computational efficiency, and memory usage compared to these models?\n- How well does VMamba generalize to other types of visual tasks beyond image classification, object detection, and segmentation? Have you considered evaluating VMamba on tasks such as video analysis, 3D vision, or more complex scene understanding? How does it scale on these tasks?\n- How sensitive is VMamba to various hyperparameters? It would be helpful to know if specific hyperparameters are critical to achieving the reported performance and if there are guidelines or best practices for tuning them.\n- How easily can VMamba be integrated into existing deep learning frameworks and pipelines? Are there any specific requirements or modifications needed for seamless integration?"},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"* 2D-Selective-Scan Module: The introduction of the 2D-Selective-Scan (SS2D) module is a creative solution to bridge the gap between 1D selective scan and 2D vision data.\n* Comprehensive Experiments: The paper provides extensive experimental results on multiple benchmarks, including ImageNet-1K, MSCOCO, and ADE20K, demonstrating the effectiveness and robustness of VMamba across various tasks.\n* Clear Explanation: The paper is well-written, with clear explanations. The authors provide detailed descriptions of the architecture, modules, and experimental setups, making it accessible to readers.\n* Visualization: The use of visualizations, such as activation maps and effective receptive fields (ERF), helps in understanding the SS2D mechanism and the model's behavior, which is very important part of all the ablation studies.\n* Impact on Visual Representation Learning: VMamba addresses a critical issue in vision models by reducing computational complexity from quadratic to linear, which can significantly impact the field of visual representation learning."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* Limited Comparison with Other SSM-based Models: While the paper does compare VMamba with several benchmark models, it would benefit from a more detailed comparison with other state-space models (SSM) in the vision domain. Specifically, models like S4ND and Vim are mentioned, but the comparisons are somewhat brief. Providing more in-depth analysis and results would strengthen the argument for VMamba's superiority.\n* Adding more interesting works to Related Work section: There are some interesting works on neuromorphic vision and processing with SSMs that authors should cite and mention:\n\n[1] State Space Models for Event Cameras. Nikola Zubić, Mathias Gehrig, Davide Scaramuzza - CVPR 2024, Spotlight\n\n[2] Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Models. Mark Schöne, Neeraj Mohan Sushma, Jingyue Zhuge, Christian Mayr, Anand Subramoney, David Kappel - ICONS 2024\n\n* Generalization to Other Tasks: The experiments focus mainly on standard benchmarks for image classification, object detection, and segmentation. However, it is not clear how well VMamba generalizes to other types of visual tasks such as video analysis, 3D vision, or more complex scene understanding. Including some preliminary results or at least discussions on these aspects could highlight the versatility of VMamba further.\n* Clarity in Mathematical Derivations: Some of the mathematical derivations, especially in the relationship between SS2D and self-attention, are complex and may not be easily accessible to all readers. Simplifying the explanations or providing more intuitive visual insights alongside the formal derivations could enhance understanding. Also, they are not rigorously mathematically proven."},"limitations":{"value":"Authors addressed everything regarding limitations section."}},"nonreaders":[],"tmdate":1730878832163,"tcdate":1719951811193,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission3213/Reviewer_c4kg"],"signatures":["NeurIPS.cc/2024/Conference/Submission3213/Reviewer_c4kg"],"forum":"ZgtLQQR1K7","number":1,"license":"CC BY 4.0","cdate":1719951811193,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission3213/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878832163,"domain":"NeurIPS.cc/2024/Conference","replyto":"ZgtLQQR1K7","id":"FsFsv0wrPE","forumContent":{"TLDR":{"value":"This paper introduces a new architecture with promising performance on visual perception tasks, which is based on mamba."},"venue":{"value":"NeurIPS 2024 spotlight"},"keywords":{"value":["State Space Model","Transformer","Computer Vision","Foundation Model"]},"supplementary_material":{"value":"/attachment/550be5322feb4a54bbb9b3794247a515d062212f.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Designing computationally efficient network architectures remains an ongoing necessity in computer vision. In this paper, we adapt Mamba, a state-space language model, into VMamba, a vision backbone with linear time complexity. At the core of VMamba is a stack of Visual State-Space (VSS) blocks with the 2D Selective Scan (SS2D) module. By traversing along four scanning routes, SS2D bridges the gap between the ordered nature of 1D selective scan and the non-sequential structure of 2D vision data, which facilitates the collection of contextual information from various sources and perspectives. Based on the VSS blocks, we develop a family of VMamba architectures and accelerate them through a succession of architectural and implementation enhancements. Extensive experiments demonstrate VMamba’s\n promising performance across diverse visual perception tasks, highlighting its superior input scaling efficiency compared to existing benchmark models. Source code is available at https://github.com/MzeroMiko/VMamba"},"_bibtex":{"value":"@inproceedings{\nliu2024vmamba,\ntitle={{VM}amba: Visual State Space Model},\nauthor={Yue Liu and Yunjie Tian and Yuzhong Zhao and Hongtian Yu and Lingxi Xie and Yaowei Wang and Qixiang Ye and Jianbin Jiao and Yunfan Liu},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=ZgtLQQR1K7}\n}"},"title":{"value":"VMamba: Visual State Space Model"},"pdf":{"value":"/pdf/496ccac4131e6ee0bc83b0ca167de274006e7b1a.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"liu|vmamba_visual_state_space_model"},"authorids":{"value":["~Yue_Liu19","~Yunjie_Tian1","~Yuzhong_Zhao1","~Hongtian_Yu1","~Lingxi_Xie1","~Yaowei_Wang1","~Qixiang_Ye1","~Jianbin_Jiao1","~Yunfan_Liu3"]},"authors":{"value":["Yue Liu","Yunjie Tian","Yuzhong Zhao","Hongtian Yu","Lingxi Xie","Yaowei Wang","Qixiang Ye","Jianbin Jiao","Yunfan Liu"]}},"version":2},{"content":{"summary":{"value":"The paper addresses learning from continuous data streams by formalizing multi-stream sequence learning, where training proceeds in blocks but the model’s hidden state persists across updates and is reset only at episode/document boundaries (unlike IID chunking). To exploit this regime, the authors propose Memora, a recurrent-only architecture built on a Gated Linear Recurrent Unit (GLRU) that supports both parallel T-BPTT via scan and true online learning via RTRL. Across selective copying, byte-level language modeling and DNA modeling, the approach shows strong performance."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Is there a reason why the authors chose to explore language modelling at byte level and not token level?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper is clearly written and easy to read.\n\nThe experiment suite is very broad evaluating the proposed approach across many domains and tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I think my main concern is that the components the authors describe here have been present in literature. For instance, multi-stream learning is not new and was adopted in TransformersXL (https://arxiv.org/abs/1901.02860). However, this is okay since they don't claim they introduce this framework, they mainly formalize it. However, for the memora architecture, the idea of gated recurrence has been known in literature for a long time with LSTM and GRU networks. I am curious how memora differs from them (apart from the formulation)?\n\nFrom the language modelling results, this approach seems to help at low sequence lengths (8-32) but matches performance (or performs worse?) of transformers at high sequence lengths (512-1024). This reduces the effectiveness of the approach as at scale ideally one may want an approach that retains performance gain as we scale sequence length as at scale we generally don't train models at <100 sequence length.\n\nI think there should be a transformerXL baseline for the multi-stream setting. Similarly, even for Mamba and Hawk in the multi-stream setting, have the authors provided a state which persists across sequences? Also there are many approaches to maintain context for transformers beyond the current context window such as sink attention (https://arxiv.org/abs/2309.17453). I believe the authors should compare against either one of these baselines."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942090986,"tcdate":1761334424310,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22149/Reviewer_Sv1G"],"signatures":["ICLR.cc/2026/Conference/Submission22149/Reviewer_Sv1G"],"forum":"YgvIjdzR4C","number":1,"license":"CC BY 4.0","cdate":1761334424310,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22149/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942090986,"domain":"ICLR.cc/2026/Conference","replyto":"YgvIjdzR4C","id":"W4RxsVYiGa","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["sequence learning","recurrent learning","streaming learning","language models"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Sequence data are inherently dependent, yet sequence learners (e.g., language models) are often trained as if samples were independent and identically distributed (IID) by segmenting long streams into short, shuffled chunks, breaking natural continuity and undermining long-range credit assignment. We formalize multi-stream sequence learning, a continuity-preserving training framework that presents multiple streams in their natural order, a setting that has been conflated with solution methods and remains underexplored. To support this paradigm, we propose Memora, a recurrent-only architecture with persistent hidden states, making it more suitable for sequence learning than architectures trained with IID chunking. Memora is built around our Gated Linear Recurrent Unit (GLRU), a lightweight unit designed for efficient parallel training and robust temporal reasoning. It achieves effective learning on long byte-level sequences and remains reliable even in the strict streaming setting, where data arrive online one byte at a time. Our experiments highlight that continuity-preserving training outperforms IID chunking, underscoring the importance of continuity in sequence learning."},"_bibtex":{"value":"@misc{\nelsayed2025sequence,\ntitle={Sequence Learning from Continuous Streams of Data},\nauthor={Mohamed Elsayed and A. Rupam Mahmood},\nyear={2025},\nurl={https://openreview.net/forum?id=YgvIjdzR4C}\n}"},"title":{"value":"Sequence Learning from Continuous Streams of Data"},"pdf":{"value":"/pdf/43159e0f0e77ec8b143d140d8bc41b0f6ef019f2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"elsayed|sequence_learning_from_continuous_streams_of_data"},"authorids":{"value":["~Mohamed_Elsayed2","~A._Rupam_Mahmood1"]},"authors":{"value":["Mohamed Elsayed","A. Rupam Mahmood"]}},"version":2},{"content":{"summary":{"value":"The paper presents a theoretical analysis of learning and generalization for a simplified Mamba block (single-layer selective SSM with input-dependent gating followed by a 2-layer MLP) trained by gradient descent on structured sequence data. Two scenarios are studied: majority-voting and locality-structured sequences with token-level noise. The authors prove non-asymptotic sample complexity and convergence guarantees under width and noise conditions, show that the gating vector aligns with class-relevant features while suppressing irrelevant ones, and provide synthetic experiments supporting the theory. They also position Mamba’s dynamics relative to Transformers, arguing that Mamba can match the majority-voting setting and outperform attention baselines on locality-structured data due to selective recurrence."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- What would the theoretical results be if a more complex setting of the Mamba block were used? For example, what happens if a normalization layer is added or if the Mamba block is repeated?\n\n- Are the assumptions in Theorems 1 and 2 regarding width and small noise required only for the proofs to work, or are they fundamentally unavoidable? What happens if these assumptions do not hold?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- This paper contains a clear novel theoretical results. In particular, two formal and non-asymptotic bounds are presented for both sample complexity and iteration complexity in two scenarios (Theorems 1 and 2), with interpretable dependence on signal gap, locality separations, step size, sequence length, and noise.\n\n- In addition, the authors provide a rigorous characterization that the gate prioritizes class-relevant features and suppresses confusing ones; connects to attention’s feature-selection role while highlighting differences arising from multiplicative recurrence.\n\n- Synthetic experiments are closely tied to theory, including comparisons where Mamba outperforms Transformer/local attention for locality."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- My first main concern is the simplified assumptions (single layer, single head, simplified recurrence), which make the considered setting far from practical Mamba stacks (with depth, multi-head structure, residuals, normalization, and learned discretization). However, I understand that a simplified setting is commonly used in theoretical analysis, as the practical setting is usually too complicated to allow for a thorough analysis.\n\n- My second concern is the strong assumptions required in Theorems 1 and 2: the width $m$ must be larger than or equal to the square of the dimension $d$, and the noise level must satisfy $\\tau = O(1/d)$. These assumptions weaken the novelty of the theoretical results. A justification for these assumptions is needed.\n\n- The experimental results are quite limited. The empirical comparison to Transformers and local attention is restricted to synthetic settings, with no controlled ablations for optimizer choice, learning rate scaling, or positional encoding. It is unclear whether attention could close the gap with appropriately tuned locality biases."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764358770067,"tcdate":1762001327981,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19648/Reviewer_8FgG"],"signatures":["ICLR.cc/2026/Conference/Submission19648/Reviewer_8FgG"],"forum":"hvpKqEYJjj","number":3,"license":"CC BY 4.0","cdate":1762001327981,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19648/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764358770067,"domain":"ICLR.cc/2026/Conference","replyto":"hvpKqEYJjj","id":"Ak28zAMTOW","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Training Dynamics","Feature Learning","Generalization","Mamba","Learning Theory","Selective State Space Models"]},"supplementary_material":{"value":"/attachment/c9fcb2ad298f51c4efaa9f51d0c97c0844014e27.zip"},"primary_area":{"value":"learning theory"},"abstract":{"value":"The recent empirical success of Mamba and other selective state space models (SSMs) has renewed interest in non-attention architectures for sequence modeling, yet their theoretical foundations remain underexplored. We present a first-step analysis of generalization and learning dynamics for a simplified but representative Mamba block: a single-layer, single-head selective SSM with input-dependent gating, followed by a two-layer MLP trained via gradient descent (GD). Our study adopts a structured data model with tokens that include both class-relevant and class-irrelevant patterns under token-level noise and examines two canonical regimes: majority-voting and locality-structured data sequences. We prove that the model achieves guaranteed generalization by establishing non-asymptotic sample complexity and convergence rate bounds, which improve as the effective signal increases and the noise decreases.\nFurthermore, we show that the gating vector aligns with class-relevant features while ignoring irrelevant ones, thereby formalizing a feature-selection role similar to attention but realized through selective recurrence. Numerical experiments on synthetic data justify our theoretical results. Overall, our results provide principled insight into when and why Mamba-style selective SSMs learn efficiently, offering a theoretical counterpoint to Transformer-centric explanations."},"_bibtex":{"value":"@inproceedings{\nshandirasegaran2026a,\ntitle={A Theoretical Analysis of Mamba{\\textquoteright}s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models},\nauthor={Mugunthan Shandirasegaran and Hongkang Li and Songyang Zhang and Meng Wang and Shuai Zhang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=hvpKqEYJjj}\n}"},"title":{"value":"A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models"},"pdf":{"value":"/pdf/7ff125c33f63bfb0017d10d5e63ca5d19e737a02.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"shandirasegaran|a_theoretical_analysis_of_mambas_training_dynamics_filtering_relevant_features_for_generalization_in_state_space_models"},"authorids":{"value":["~Mugunthan_Shandirasegaran1","~Hongkang_Li1","~Songyang_Zhang2","~Meng_Wang4","~Shuai_Zhang6"]},"authors":{"value":["Mugunthan Shandirasegaran","Hongkang Li","Songyang Zhang","Meng Wang","Shuai Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper:\n\n1. Identifies three types of dependencies in multivariate time series data. \n2. Simplifies the Mamba activation functions to eliminate non-linearity. \n3. Proposes a dependency encoding strategy to disentangle these dependencies, minimizing interference between the time and variate dimensions."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"1. About order dependency:\n\nThe statement and evidences to prove that Transformer-based models are unsuitable for modeling order information seems insufficient. \n\nFirst, improving a model's perception of order information can depend on two factors: the tokenization and the training method. In tokenization, if different lag timesteps are treated as distinct features inside a token (e.g., series embedding as in Linear models and iTransformer, or patch embedding as in PatchTST), shuffling values across timesteps will clearly impact performance, as the linear projection layer automatically considers them as separate input features. However, if timestep values are projected into the same latent space during tokenization, the model must rely on additional structures (such as attention + positional encoding) to learn this order information. Can this order be learned with pure Transformers? The success of LLMs proves that models can learn precise positional information. LLMs accurately identify and utilize context positions for next-token prediction without making sequencing errors, thanks to the learning of mechanisms like induction heads, which has been widely validated. Autoregressive training method enforces models to learn these temporal ordering algorithms.\n\nThe choice of tokenization and training method is largely independent of the model architecture. Both Transformers and Mamba can use these methods to improve the capture of positional information in LTSF tasks. Experimental results in the paper also show that patch tokenization enhances performance across different architectures. It separates timesteps within a local period into distinct features inside a token, rather than treating all timesteps as equivalent tokens in the attention mechanism.\n\nConstructing algorithms between tokens in attention (or SSM in Mamba) involves learning the sequence's semantic information. The richer the semantic information, the more we need to model algorithms at a finer granularity (shorter patch sizes). The paper suggests that patching enhances semantic dependency learning, but this seems contradictory. Patching likely reduces the need for complex semantic relationship modeling, functioning as a form of attention regularization that improves performance on datasets with simpler semantics, contrary to Assumption 2. Correspondingly, Linear models treat the entire series as embeddings and all timesteps as lag features, omitting the need to construct semantic relationships between timesteps, which aligns with the statement in the paper. However, the analysis of patching seems inconsistent. The authors could experiment on datasets with richer semantic information (e.g., ODE-based datasets with underlying dynamics) to see if pure timestep embedding outperforms patch embedding.\n\nIn summary, does Mamba have an advantage over Transformer-based models in modeling order information as claimed? It seems so, but this advantage may come from patching tokenization and possibly from Mamba's SSM handling of sequential/causal token information. Previous LTSF encoder-only Transformers may lack the sequential structure. The authors could compare Mamba to Transformers using causal masks/training methods to further establish Mamba's structural superiority in LTSF. Additionally, more discussion is needed to support the definitions and statements of semantic dependency.\n\n\n2. About the dataset selection：\n\nThe experiments in Tables 1-3 and Figure 1 are conducted solely on the ETTm1 dataset, which seems insufficient to draw general conclusions. The nature of the datasets significantly impacts the performance of different models with varying degrees of regularization.\n\nThe performance comparison in Table 1 using shuffling on a single dataset appears insufficient (actually, I hold a similar view regarding these experiments conducted in the DLinear paper). Here's a simple counterexample: \n\nConsider an invertible MA(1) process, $ X_t = \\mu + \\varepsilon_t + \\theta \\varepsilon_{t-1} $, with two sub-optimal predictors: one using the global average of the input and the other using the last value as the prediction. It can be shown that the first predictor is better or equal to the second one under invertible condition. However, if the input is shuffled, the first predictor’s performance does not deteriorate, while the second, less optimal predictor's MSE decreases when $ \\theta > 0 $, remains unchanged at $ \\theta = 0 $, and increases when $ \\theta < 0 $. This indicates that comparing performance drops due to shuffling may not be sufficient to prove a model’s sensitivity to order, as better models may also learn predictors insensitive to position.\n\nThe conclusions from Table 2 may also be dataset-sensitive. For datasets with underlying dynamics and shifting multivariate effects, exposing more temporal tokens for algorithm construction in attentions could be more advantageous. The authors could extend experiments to datasets like Solar and Exchange to strengthen these claims.\n\nTable 3 should be a foundational point of the paper, but it requires validation across more datasets. The choice between model linearity and complexity largely depends on the dataset. When clear non-linear relationships exist, the model may need to build more complex algorithms between temporal tokens, leading to different conclusions.\n\nFigure 1 shows that removing activation functions increases the model’s regularization. The authors could consider testing this on more complex datasets, such as Traffic (or maybe PEMS?), to confirm that stronger regularization is indeed beneficial.\n\n\n### Conclusion\n\nEven though I have some concerns about the authors’ claims and experimental methods, the authors have put in a substantial amount of work throughout the paper. Therefore, I look forward to further discussion with the authors on these points before giving a final score."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":3},"strengths":{"value":"1. The authors provide a clear and comprehensive description of the key dependencies in LTSF modeling, including order dependency, semantic dependency, and cross-variate dependency. The model architecture is well-designed to capture these dependencies.\n\n2. The architecture introduced in this paper to capture relationships across both temporal and variate dimensions enhances Mamba's ability to effectively model cross-variate relationships.\n\n3. The experiments conducted demonstrate that the proposed architecture outperforms previous methods and can be effectively transferred to other models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper contains several spelling and formatting errors. The authors should carefully check them before submission. Examples include: \n   a) Inconsistent abbreviations, with “LTSF” sometimes written as “LTST”; \n   b) Formatting issues in Theorem 1; \n   c) Various spelling errors.\n\n2. The title and model name \"Simplified Mamba, SAMBA\" emphasize a simplification of Mamba, specifically through the removal of non-linearity. However, according to the paper, the performance improvements from this regularization method are not particularly significant compared to other strategies, such as patch tokenization. Additionally, the authors mention that the elimination is not fully implemented, as non-linear activation function remains in the gating mechanism. Furthermore, the overall SAMBA architecture actually employs Mamba along the temporal dimension and bidirectional Mamba along the variate dimension, making it more complex rather than simplified, compared to the original Mamba for sequence modeling. Thus, the authors might consider rebranding their methods to avoid misleading readers."}},"nonreaders":[],"tmdate":1732666007056,"tcdate":1730602432088,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission767/Reviewer_WkCP"],"signatures":["ICLR.cc/2025/Conference/Submission767/Reviewer_WkCP"],"forum":"9VRFPC29nb","number":3,"license":"CC BY 4.0","cdate":1730602432088,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission767/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732666007056,"domain":"ICLR.cc/2025/Conference","replyto":"9VRFPC29nb","id":"t5V2oqH6IW","forumContent":{"TLDR":{"value":"propose a simplifed mamba and a theoretically sound disentangled encoding strategy for long-term time series forecasting"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["long-term time series forecasting","time series modeling","mamba"]},"supplementary_material":{"value":"/attachment/f88a207f308111754eeb7cbc865e432708675aab.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent advances in deep learning have led to the development of numerous models for Long-term Time Series Forecasting (LTSF). However, most approaches still struggle to comprehensively capture reliable and informative dependencies inherent in time series data. In this paper, we identify and formally define three critical dependencies essential for improving forecasting accuracy: the order dependency and semantic dependency in the time dimension as well as cross-variate dependency in the variate dimension. Despite their significance, these dependencies are rarely considered holistically in existing models. Moreover, improper handling of these dependencies can introduce harmful noise that significantly impairs forecasting performance. To address these challenges, we explore the potential of Mamba for LTSF, highlighting its three key advantages to capture three dependencies, respectively. We further empirically observe that nonlinear activation functions used in vanilla Mamba are redundant for semantically sparse time series data. Therefore, we propose SAMBA, a Simplified Mamba with disentangled dependency encoding. Specifically, we first eliminate the nonlinearity of vanilla Mamba to make it more suitable for LTSF. Along this line, we propose a disentangled dependency encoding strategy to endow Mamba with efficient cross-variate dependency modeling capability while minimizing the interference between time and variate dimensions. We also provide rigorous theory as a justification for our design. Extensive experiments on nine real-world datasets demonstrate the effectiveness of SAMBA over state-of-the-art forecasting models."},"_bibtex":{"value":"@misc{\nweng2025simplified,\ntitle={Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting},\nauthor={Zixuan Weng and Jindong Han and Wenzhao Jiang and Hao Liu},\nyear={2025},\nurl={https://openreview.net/forum?id=9VRFPC29nb}\n}"},"title":{"value":"Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting"},"pdf":{"value":"/pdf/cb12f563a9e3d09396284efb314db4d48c961f42.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"weng|simplified_mamba_with_disentangled_dependency_encoding_for_longterm_time_series_forecasting"},"authorids":{"value":["~Zixuan_Weng1","~Jindong_Han1","~Wenzhao_Jiang1","~Hao_Liu17"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zixuan Weng","Jindong Han","Wenzhao Jiang","Hao Liu"]}},"version":2},{"content":{"summary":{"value":"The paper defines $\\mathcal F_\\theta=(\\mathcal S\\circ\\mathcal T)^L\\circ \\mathcal E$, where $\\mathcal T$ is a bidirectional selective SSM (Mamba) operating along time and $\\mathcal S$ is a graph SSM that propagates non-local spatial information; the model maps a single snapshot to an entire future trajectory and is trained with A-MSE over $P$ uniformly sampled times. The encoder repeats the initial graph $P$ times and injects a learned or sinusoidal time embedding, and the theory section proves permutation equivariance of the spatial update and discusses stability under spectral-radius bounds. Experiments report A-MSE/F-MSE on N-Body, CMU MoCap subsets, MD17 molecules (heavy-atom only), and AdK equilibrium."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"First, how would results change if the temporal block were strictly causal, and could you include such a variant to compare with sequential EGNN rollouts; this could clarify whether bidirectional smoothing is a source of gains. Second, can you enforce the stability conditions you prove, for example, by parameterizing $W$ with a spectral-norm cap or adding an explicit damping factor $\\alpha \\hat A$, and show ablations on long-horizon drift. Third, can you evaluate on rMD17 with force/energy MAE (and, for AdK, report energies or contact/dihedral metrics) to align with standard practice. Fourth, where do you stand against Graph-Mamba baselines that use node ordering or walk-based scans; please run at least one representative method to demonstrate the benefit of your order-free spatial SSM. Fifth, could you include GeoTDM and GF-NODE as baselines on the molecular and motion tasks and a diffusion baseline like MDM on MoCap, with shared splits and identical horizons. Sixth, can you report efficiency (tokens/s, memory, params) and show how GraMO scales with $P$ and $L$; the efficiency pitch is currently unsubstantiated. Finally, because the encoder repeats $\\mathcal G^{(0)}$ with time embeddings, can you test non-uniform or extrapolated time grids and report robustness across different $P$."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"From an originality perspective, presenting a trajectory operator that composes temporal SSM blocks with a spatial SSM is a clean architectural statement, and the decision to predict the whole rollout in one shot is aligned with prior operator work but still practical for long horizons. The paper offers clear expository sections, including an explicit definition of S2S vs S2T metrics and detailed hyperparameters; this improves clarity and reproducibility. The theoretical part, though modest, correctly establishes permutation equivariance for the spatial update and gives a stability discussion via spectral-radius control, which, if enforced, would be meaningful for long rollouts. Empirically, the MoCap results suggest that the temporal SSM can help in human-motion settings where long-range temporal structure matters, which hints at potential significance if strengthened."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"In terms of novelty, the contribution overlaps strongly with two existing streams. First, operator-learning with SE(3)-equivariant temporal convolutions already models state-to-trajectory mappings end-to-end (EGNO), where temporal kernels are parameterized in the Fourier domain; GraMO feels like swapping those kernels for SSMs. Second, there is now a growing pool of Graph-Mamba variants (Graph-Mamba, SpoT-Mamba, DyG-Mamba, etc.) that adapt selective SSMs to graphs; the manuscript would need to position itself precisely against these works and run them as baselines. At present, the paper reads as stitching together Mamba and EGNO, with the specific SSM parameterization being the main delta. \n\nOn evaluation quality, several missing baselines materially weaken the claims. For geometric trajectories with long horizons, GeoTDM is the natural diffusion baseline with SE(3)-equivariant temporal kernels; for continuous-time modeling on graphs, GF-NODE is directly relevant; for MoCap generation/prediction, a diffusion model such as MDM is expected. None appear in the tables, so the comparative story is incomplete.\n\nThe symmetry story is underpowered for 3D dynamics. The method proves permutation equivariance but does not encode SE(3) invariance/equivariance in space or time; this is a known requirement for robust molecular and rigid-body dynamics and is already present in strong baselines (EGNN/SE(3)-Tr., EGNO). The paper should either add an SE(3)-equivariant variant or carefully delimit its scope to settings where coordinate frames are fixed or augmented. \n\nSome design choices complicate interpretation. The temporal block is bidirectional, which is fine for offline trajectory reconstruction but undermines any causal forecasting claims; the paper should make this explicit and, ideally, report a causal variant for fair comparison with rollouts. The input construction repeats the initial graph $P$ times and relies on time embeddings; this conflates learning actual dynamics with extrapolating from positional encodings and deserves sensitivity analyses to $P$, to embedding type, and to non-uniform time grids."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926649004,"tcdate":1761625590073,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16568/Reviewer_8hXD"],"signatures":["ICLR.cc/2026/Conference/Submission16568/Reviewer_8hXD"],"forum":"6CO8dwHl4F","number":1,"license":"CC BY 4.0","cdate":1761625590073,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16568/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926649004,"domain":"ICLR.cc/2026/Conference","replyto":"6CO8dwHl4F","id":"wSwlpStNAA","forumContent":{"TLDR":{"value":"Introduces a new graph neural operator that leverages state-space models (SSMs) for long range modelling."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Graph Neural Network","Neural Operator","State Space Models"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"abstract":{"value":"Modelling complex 3D dynamics---from molecular conformations to particle interactions and human motion---requires capturing dependencies spanning long temporal horizons and non-local spatial interactions. Graph neural networks (GNNs) have shown promise in spatio-temporal settings but often suffer from instability and degraded accuracy in long-range forecasting. We propose the Graph Mamba Operator (GraMO), a neural operator that integrates state-space models (SSMs) with geometric learning to capture spatio-temporal correlations jointly. To jointly model complex dynamics, GraMO integrates a stable, SSM-based temporal backbone with an SSM-parameterized graph update to capture long-range spatial dependencies. Unlike stepwise predictors that accumulate errors over time, GraMO learns entire trajectories in a single forward pass. Across diverse benchmarks ranging from molecular dynamics to human motion capture, GraMO shows notable improvements in trajectory fidelity, stability, and robustness over strong baselines with relative improvements of over 26.3\\% on motion capture benchmarks and 25.2\\% on MD17 final-state prediction. Ablation studies reveal that temporal SSM components consistently improve performance, while spatial SSM updates show task-dependent benefits---helping with long-range interactions in large molecules but potentially hindering performance on systems with primarily local dependencies. Altogether, these results suggest that selective integration of SSM components with graph neural networks can improve performance on particle-based systems, with applications in molecular simulations, articulated rigid body dynamics, and particle systems."},"_bibtex":{"value":"@misc{\ntiwari2025graph,\ntitle={Graph Mamba Operator for Learning the Dynamics of Particle-based Systems},\nauthor={Karn Tiwari and Niladri Dutta and Prathosh AP and N M Anoop Krishnan},\nyear={2025},\nurl={https://openreview.net/forum?id=6CO8dwHl4F}\n}"},"title":{"value":"Graph Mamba Operator for Learning the Dynamics of Particle-based Systems"},"pdf":{"value":"/pdf/2f19ba46da22220db7180ecf5ababb98f9ad45eb.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"tiwari|graph_mamba_operator_for_learning_the_dynamics_of_particlebased_systems"},"authorids":{"value":["~Karn_Tiwari1","~Niladri_Dutta1","~Prathosh_AP1","~N_M_Anoop_Krishnan1"]},"authors":{"value":["Karn Tiwari","Niladri Dutta","Prathosh AP","N M Anoop Krishnan"]}},"version":2},{"content":{"comment":{"value":"**W4: In Remark 4, a comparison between Mamba and the linear Transformer is reported... However, is there any empirical evidence to support the summaries? For example, consider two LLMs with the same architecture and trained with different hyperparameters. Do the empirical evaluations support your theoretical results?**\n\n**A3**: The conclusions in Remarks 4 and 5 show that, to achieve comparable performance, the linear attention requires weaker conditions during training, while the additional stronger condition required for linear attention appears only in Remark 5, i.e., that during testing, $\\alpha$, the fraction of outliers, must not exceed $1/2$. Mamba does not require this condition. This is because our theory shows that the effect of Mamba’s nonlinear gating is to remove outliers (Corollary 2), allowing Mamba models to tolerate a high outlier fraction. In contrast, the linear Transformer cannot learn the outlier at all, and therefore needs the outlier fraction to be the minority to ensure correct final predictions.\n\nThis insight is supported by real-world experiments. In vision tasks, Mamba and other SSM models have been shown to exhibit stronger robustness than Transformer architectures against outliers such as adversarial attacks and common corruptions [Park et al., 2024; Du et al., 2024; Malik et al., 2025]. In particular, [Du et al., 2024] finds that the parameter $\\Delta$ are key to Mamba’s defense capability under white-box attacks. Since in our setting $\\Delta$ are formulated as functions of the nonlinear gating parameters $w$ (line 135), this observation is consistent with our theoretical conclusion in Corollary 2, i.e., nonlinear gating removes outliers and thereby improves robustness.\n\nBecause Mamba-based LLM are still under development, comparisons between the two types of LLMs on robustness are still less explored. However, the empirical findings mentioned above can provide experimental support for our theoretical results to some extent.\n\nPark et al., 2024. Can mamba learn how to learn? a comparative study on in-context learning tasks. At ICML.\n\nDu et al., 2024. Understanding Robustness of Visual State Space Models for Image Classification.\n\nMalik et al., 2025. Towards Evaluating the Robustness of Visual State Space Models. At CVPRW.\n\n\n**Q1: What is A in Lines 130 and 132?**\n\n**A4**: $A$ is the parameter used to compute the $\\bar{A}$ and $\\bar{B}$ for state update. It is introduced in Eqn 4 and Algorithm 2 of the original Mamba paper [Gu & Dao, 2023].\n\nGu & Dao, 2023. Mamba: Linear-Time Sequence Modeling with Selective State Spaces\n\n\n\n\n\n**Q2: Lines 163 & 310, the references have the wrong format.**\n\n**A5**: Thank you for pointing them out. We have revised them in the updated version."},"title":{"value":"Response to Reviewer 1e2f Part III"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764019529285,"tcdate":1764019529285,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7901/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7901/Authors"],"forum":"tswBfpkwHn","number":6,"license":"CC BY 4.0","cdate":1764019529285,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7901/-/Official_Comment"],"mdate":1764019529285,"domain":"ICLR.cc/2026/Conference","replyto":"0gJbRINZxN","id":"MtnXWrHFvb","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"This paper provide the first theoretical analysis of the training dynamics of a one-layer Mamba model and characterizes its in-context learning generaization ability with additive outliers in the prompt"},"keywords":{"value":["Mamba","in-context learning","Transformer","generalization","training dynamics","deep learning theory"]},"primary_area":{"value":"learning theory"},"abstract":{"value":"The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks. Like Transformers, Mamba exhibits in-context learning (ICL) capabilities, i.e.,  making predictions for new tasks based on a prompt containing input-label pairs and a query, without requiring fine-tuning.\nDespite its empirical success, the theoretical understanding of Mamba remains limited, largely due to the nonlinearity introduced by its gating mechanism. To the best of our knowledge, this paper presents the first theoretical analysis of the training dynamics of a one-layer Mamba model, which consists of a linear attention component followed by a nonlinear gating layer, and its ICL generalization on unseen binary classification tasks, even when the prompt includes additive outliers. Our analysis shows that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layer to suppress the influence of outliers. By establishing and comparing to the analysis of linear Transformers under the same setting, we show that although Mamba may require more training iterations to converge, it maintains accurate predictions even when the proportion of outliers exceeds the threshold that a linear Transformer can tolerate. These theoretical findings are supported by empirical experiments."},"_bibtex":{"value":"@misc{\nli2026can,\ntitle={Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis},\nauthor={Hongkang Li and Songtao Lu and Xiaodong Cui and Pin-Yu Chen and Meng Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=tswBfpkwHn}\n}"},"title":{"value":"Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis"},"pdf":{"value":"/pdf/771e925a0afdf6afc4b8a950bc4b771d6afabad8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"li|can_mamba_learn_in_context_with_outliers_a_theoretical_generalization_analysis"},"authorids":{"value":["~Hongkang_Li1","~Songtao_Lu1","~Xiaodong_Cui1","~Pin-Yu_Chen1","~Meng_Wang4"]},"authors":{"value":["Hongkang Li","Songtao Lu","Xiaodong Cui","Pin-Yu Chen","Meng Wang"]}},"version":2},{"content":{"venue":{"value":"AAAI 2025"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/34361/36516"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"yao|selective_visual_prompting_in_vision_mamba"},"html":{"value":"https://doi.org/10.1609/aaai.v39i21.34361"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/YaoLCPZ25,\n  author={Yifeng Yao and Zichen Liu and Zhenyu Cui and Yuxin Peng and Jiahuan Zhou},\n  title={Selective Visual Prompting in Vision Mamba},\n  year={2025},\n  cdate={1735689600000},\n  pages={22083-22091},\n  url={https://doi.org/10.1609/aaai.v39i21.34361},\n  booktitle={AAAI},\n  crossref={conf/aaai/2025}\n}\n"},"abstract":{"value":"Pre-trained Vision Mamba~(Vim) models have demonstrated exceptional performance across various computer vision tasks in a computationally efficient manner, attributed to their unique design of selective state space models. To further extend their applicability to diverse downstream vision tasks, Vim models can be adapted using the efficient fine-tuning technique known as visual prompting. However, existing visual prompting methods are predominantly tailored for Vision Transformer (ViT)-based models that leverage global attention, neglecting the distinctive sequential token-wise compression and propagation characteristics of Vim. Specifically, existing prompt tokens prefixed to the sequence are insufficient to effectively activate the input and forget gates across the entire sequence, hindering the extraction and propagation of discriminative information. To address this limitation, we introduce a novel Selective Visual Prompting (SVP) method specifically for the efficient fine-tuning of Vim. To prevent the loss of discriminative information during state space propagation, SVP employs lightweight selective prompters for token-wise prompt generation, ensuring adaptive activation of the update and forget gates within Mamba blocks to promote discriminative information propagation. Moreover, considering that Vim propagates both shared cross-layer information and specific inner-layer information, we further refine SVP with a dual-path structure: Cross-Prompting and Inner-Prompting. Cross-Prompting utilizes shared parameters across layers, while Inner-Prompting employs distinct parameters, promoting the propagation of both shared and specific information, respectively. Extensive experimental results on various large-scale benchmarks demonstrate that our proposed SVP significantly outperforms state-of-the-art methods."},"title":{"value":"Selective Visual Prompting in Vision Mamba"},"authors":{"value":[{"fullname":"Yifeng Yao","username":""},{"fullname":"Zichen Liu","username":""},{"fullname":"Zhenyu Cui","username":"~Zhenyu_Cui1"},{"fullname":"Yuxin Peng","username":""},{"fullname":"Jiahuan Zhou","username":""}]}},"tmdate":1784537405062,"pdate":1767139200000,"externalIds":["dblp:conf/aaai/YaoLCPZ25"],"tcdate":1784537399211,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Zhenyu_Cui1"],"forum":"3cQOs3zeoV","license":"CC BY-SA 4.0","number":67990,"cdate":1735689600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784537405062,"domain":"OpenReview.net/Public_Article","id":"3cQOs3zeoV","version":2},{"content":{"TLDR":{"value":"The paper derives generalization bounds for selective SSMs using connections to self-attention, showing that spectral properties of the state matrix influence generalization."},"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["Generalization Bounds","State-Space Models","Selective Scan","Stability","Transformers","Rademacher Complexity","Covering Numbers"]},"primary_area":{"value":"theory"},"abstract":{"value":"State-space models (SSMs) have recently emerged as a compelling alternative to Transformers for sequence modeling tasks. This paper presents a theoretical generalization analysis of selective SSMs, the core architectural component behind the Mamba model. We derive a novel covering number-based generalization bound for selective SSMs, building upon recent theoretical advances in the analysis of Transformer models. Using this result, we analyze how the spectral abscissa of the continuous-time state matrix influences the model’s stability during training and its ability to generalize across sequence lengths. We empirically validate our findings on a synthetic majority task, the IMDb sentiment classification benchmark, and the ListOps task, demonstrating how our theoretical insights translate into practical model behavior."},"_bibtex":{"value":"@inproceedings{\nhonarpisheh2025generalization,\ntitle={Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention},\nauthor={Arya Honarpisheh and Mustafa Bozdag and Octavia Camps and Mario Sznaier},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=YVZbaVikBp}\n}"},"title":{"value":"Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention"},"pdf":{"value":"/pdf/5c4b7076e7711c8642942198bf94cb670d9962ec.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"honarpisheh|generalization_error_analysis_for_selective_statespace_models_through_the_lens_of_attention"},"authorids":{"value":["~Arya_Honarpisheh1","~Mustafa_Bozdag1","~Octavia_Camps1","~Mario_Sznaier1"]},"authors":{"value":["Arya Honarpisheh","Mustafa Bozdag","Octavia Camps","Mario Sznaier"]}},"tmdate":1783627375699,"pdate":1758216978709,"tcdate":1746825581291,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission13619/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission13619/Authors"],"forum":"YVZbaVikBp","license":"CC BY-NC-SA 4.0","number":13619,"cdate":1746825581291,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission13619/-/Full_Submission","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission13619/-/Camera_Ready_Revision"],"mdate":1783627375699,"odate":1761704895857,"domain":"NeurIPS.cc/2025/Conference","id":"YVZbaVikBp","version":2},{"content":{"summary":{"value":"This paper proposes a hybrid mamba network (HMNet) to capture inter-sequence dependencies for few-shot segmentation tasks (FSS). The authors identify two issues when applying the original Mamba network to cross-attention scenarios: the support forgetting issue and the intra-class gap issue. The proposed HMNet includes: 1) Support Recapped Mamba (SRM), which periodically recaps the support features when scanning the query to ensure the hidden state contains rich support information; and 2) Query Intercepted Mamba (QIM), which intercepts the mutual interactions among query pixels when propagating the hidden state (with pure support FG) to encourage the fusion of more information from support features. Extensive experiments on PASCAL-5^i and COCO-20^i demonstrate the superiority of the proposed method."},"soundness":{"value":3},"confidence":{"value":2},"questions":{"value":"See above."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. This paper indicates the support forgetting and intra-class gap issue when applying mamba to FSS.\n2. The proposed HMB is more effective for the fusion of support-query features, leading to improved FSS performance.\n3. Experimental results on PASCAL-5^i and COCO-20^i verify its effectiveness."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The Mamba model exhibits significantly enhanced efficiency over the attention-based models as the sequence length increases, such as the million-length real-world data mentioned in the Mamba paper. In FSS, is there a comparative analysis that evaluates efficiency and effectiveness across different patch quantity settings?\n2. The proposed SRM strategy includes inserting support patches periodically among query patches. Could this approach inadvertently lead to forgetting the query patches in earlier positions? \n3. For consistency in terminology, consider distinguishing between \"feature\" and \"patch,\" as in the caption of Figure 3: \"...alternatively appeared support features and query patches...\"."},"limitations":{"value":"It would be beneficial if the author could address the potential risk of query patch forgetting and include comparisons with attention-based methods across different patch settings."}},"nonreaders":[],"tmdate":1730878780258,"tcdate":1720691236571,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2498/Reviewer_iSLu"],"signatures":["NeurIPS.cc/2024/Conference/Submission2498/Reviewer_iSLu"],"forum":"Qe2BKeCEBC","number":2,"license":"CC BY 4.0","cdate":1720691236571,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2498/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878780258,"domain":"NeurIPS.cc/2024/Conference","replyto":"Qe2BKeCEBC","id":"DyhnT2hdYS","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["few-shot segmentation","mamba","intra-sequence dependencies"]},"supplementary_material":{"value":"/attachment/c2e47c3a1b58104b90d7e4c87b1052435097d66e.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Many few-shot segmentation (FSS) methods use cross attention to fuse support foreground (FG) into query features, regardless of the quadratic complexity. A recent advance Mamba can also well capture intra-sequence dependencies, yet the complexity is only linear. Hence, we aim to devise a cross (attention-like) Mamba to capture inter-sequence dependencies for FSS. A simple idea is to scan on support features to selectively compress them into the hidden state, which is then used as the initial hidden state to sequentially scan query features. Nevertheless, it suffers from (1) support forgetting issue: query features will also gradually be compressed when scanning on them, so the support features in hidden state keep reducing, and many query pixels cannot fuse sufficient support features; (2) intra-class gap issue: query FG is essentially more similar to itself rather than to support FG, i.e., query may prefer not to fuse support features but their own ones from the hidden state, yet the success of FSS relies on the effective use of support information. To tackle them, we design a hybrid Mamba network (HMNet), including (1) a support recapped Mamba to periodically recap the support features when scanning query, so the hidden state can always contain rich support information; (2) a query intercepted Mamba to forbid the mutual interactions among query pixels, and encourage them to fuse more support features from the hidden state. Consequently, the support information is better utilized, leading to better performance. Extensive experiments have been conducted on two public benchmarks, showing the superiority of HMNet. The code is available at https://github.com/Sam1224/HMNet."},"_bibtex":{"value":"@inproceedings{\nxu2024hybrid,\ntitle={Hybrid Mamba for Few-Shot Segmentation},\nauthor={Qianxiong Xu and Xuanyi Liu and Lanyun Zhu and Guosheng Lin and Cheng Long and Ziyue Li and Rui Zhao},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=Qe2BKeCEBC}\n}"},"title":{"value":"Hybrid Mamba for Few-Shot Segmentation"},"pdf":{"value":"/pdf/7fca1561230feeb691e2dbccd7f1ef8f79dfa5ec.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"xu|hybrid_mamba_for_fewshot_segmentation"},"authorids":{"value":["~Qianxiong_Xu1","~Xuanyi_Liu3","~Lanyun_Zhu1","~Guosheng_Lin1","~Cheng_Long1","~Ziyue_Li2","~Rui_Zhao6"]},"authors":{"value":["Qianxiong Xu","Xuanyi Liu","Lanyun Zhu","Guosheng Lin","Cheng Long","Ziyue Li","Rui Zhao"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.21535v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"dong|ecmamba_consolidating_selective_state_space_model_with_retinex_guidance_for_efficient_multiple_exposure_correction"},"authorids":{"value":["~Wei_Dong9","https://dblp.org/search/pid/api?q=author:Han_Zhou_0003:","https://dblp.org/search/pid/api?q=author:Yulun_Zhang_0001:","https://dblp.org/search/pid/api?q=author:Xiaohong_Liu_0001:","https://dblp.org/search/pid/api?q=author:Jun_Chen_0005:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.21535"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-21535,\n  publtype={informal},\n  author={Wei Dong and Han Zhou and Yulun Zhang and Xiaohong Liu and Jun Chen},\n  title={ECMamba: Consolidating Selective State Space Model with Retinex Guidance for Efficient Multiple Exposure Correction},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.21535},\n  url={https://doi.org/10.48550/arXiv.2410.21535}\n}\n"},"abstract":{"value":"Exposure Correction (EC) aims to recover proper exposure conditions for images captured under over-exposure or under-exposure scenarios. While existing deep learning models have shown promising results, few have fully embedded Retinex theory into their architecture, highlighting a gap in current methodologies. Additionally, the balance between high performance and efficiency remains an under-explored problem for exposure correction task. Inspired by Mamba which demonstrates powerful and highly efficient sequence modeling, we introduce a novel framework based on Mamba for Exposure Correction (ECMamba) with dual pathways, each dedicated to the restoration of reflectance and illumination map, respectively. Specifically, we firstly derive the Retinex theory and we train a Retinex estimator capable of mapping inputs into two intermediary spaces, each approximating the target reflectance and illumination map, respectively. This setup facilitates the refined restoration process of the subsequent Exposure Correction Mamba Module (ECMM). Moreover, we develop a novel 2D Selective State-space layer guided by Retinex information (Retinex-SS2D) as the core operator of ECMM. This architecture incorporates an innovative 2D scanning strategy based on deformable feature aggregation, thereby enhancing both efficiency and effectiveness. Extensive experiment results and comprehensive ablation studies demonstrate the outstanding performance and the importance of each component of our proposed ECMamba. Code is available at https://github.com/LowlevelAI/ECMamba."},"title":{"value":"ECMamba: Consolidating Selective State Space Model with Retinex Guidance for Efficient Multiple Exposure Correction"},"authors":{"value":["Wei Dong","Han Zhou","Yulun Zhang","Xiaohong Liu","Jun Chen"]}},"tmdate":1759421267937,"pdate":1704067200000,"externalIds":["dblp:journals/corr/abs-2410-21535"],"tcdate":1759421257918,"writers":["~"],"signatures":["~Wei_Dong9"],"forum":"2SFUl4G7oP","license":"CC BY-SA 4.0","number":633785,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1759421267937,"domain":"DBLP.org","id":"2SFUl4G7oP","version":2},{"content":{"review":{"value":"This proposal effectively explores the implementation of a small-scale Mamba architecture for protein sequence generation, addressing computational efficiency, long-sequence handling, and biological plausibility. The primary strengths include (1) innovative use of selective state-space updates, yielding linear complexity, (2) comprehensive evaluation across structural and functional metrics, and (3) accessibility through reduced computational demands. However, potential weaknesses are (1) limited scalability compared to larger models and (2) reliance on specific datasets, which may affect generalization. Overall, the proposal promises meaningful contributions to resource-efficient protein modeling."},"confidence":{"value":4},"rating":{"value":10},"title":{"value":"Proposal of paper \"Protein Sequence Generation Model\""}},"nonreaders":[],"tmdate":1731382751110,"tcdate":1731382751110,"writers":["tsinghua.edu.cn/THU/2024/Fall/AML","~Ruilin_Hu2"],"signatures":["~Ruilin_Hu2"],"forum":"I2FbCxyYVS","number":6,"license":"CC BY 4.0","cdate":1731382751110,"readers":["everyone"],"invitations":["tsinghua.edu.cn/THU/2024/Fall/AML/Submission52/-/Official_Review"],"mdate":1731382751110,"domain":"tsinghua.edu.cn/THU/2024/Fall/AML","replyto":"I2FbCxyYVS","id":"z6wpZsDkiw","forumContent":{"TLDR":{"value":"This proposal explores implementing a Mamba architecture for efficient protein sequence generation, aiming to balance performance with reduced computational resource demands and making the architecture more suited to biological dynamics."},"venue":{"value":"THU 2024 Fall AML Submission"},"pdf":{"value":"/pdf/8d5ee7449a5695662b51c83289c76642879f2be4.pdf"},"keywords":{"value":["protein sequence generation","Mamba architecture","efficient modeling","protein language models","biological dynamics"]},"venueid":{"value":"tsinghua.edu.cn/THU/2024/Fall/AML/Submission"},"paperhash":{"value":"jiang|[proposalml]protein_sequence_generation_model"},"authorids":{"value":["~pan_jiang1","zeeshanzulfiqar79@outlook.com","ivanyazykov.mipt@gmail.com"]},"abstract":{"value":"This proposal outlines a novel approach for protein sequence generation, a key area in computational biology focused on generating sequences with specific functional and structural characteristics. Despite recent advances with deep learning models, challenges remain in handling long sequences and capturing biological dynamics. We propose implementing a Mamba architecture that leverages selective state-space updates to achieve efficient protein sequence generation. With linear computational complexity and effective handling of long-range dependencies, this architecture is designed to be more suited to biological dynamics while reducing resource requirements. A comprehensive evaluation on the UniRef50 dataset will demonstrate its potential to deliver competitive performance, providing a viable solution for research environments with limited computational resources."},"_bibtex":{"value":"@inproceedings{\njiang2024proposalmlprotein,\ntitle={[Proposal-{ML}]Protein Sequence Generation Model},\nauthor={pan jiang and Zeeshan Zulfiqar and Ivan Iazykov},\nbooktitle={Submitted to Tsinghua University Course: Advanced Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=I2FbCxyYVS},\nnote={under review}\n}"},"title":{"value":"[Proposal-ML]Protein Sequence Generation Model"},"authors":{"value":["pan jiang","Zeeshan Zulfiqar","Ivan Iazykov"]}},"version":2},{"content":{"venue":{"value":"IEEE Transactions on Medical Imaging"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/42/11165775/10980210.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"cheng|mambasea_a_mambabased_framework_with_globaltolocal_sequence_augmentation_for_generalizable_medical_image_segmentation"},"html":{"value":"https://doi.org/10.1109/TMI.2025.3564765"},"abstract":{"value":"To segment medical images with distribution shifts, domain generalization (DG) has emerged as a promising setting to train models on source domains that can generalize to unseen target domains. Existing DG methods are mainly based on CNN or ViT architectures. Recently, advanced state space models, represented by Mamba, have shown promising results in various supervised medical image segmentation. The success of Mamba is primarily owing to its ability to capture long-range dependencies while keeping linear complexity with input sequence length, making it a promising alternative to CNNs and ViTs. Inspired by the success, in the paper, we explore the potential of the Mamba architecture to address distribution shifts in DG for medical image segmentation. Specifically, we propose a novel Mamba-based framework, Mamba-Sea, incorporating global-to-local sequence augmentation to improve the model’s generalizability under domain shift issues. Our Mamba-Sea introduces a global augmentation mechanism designed to simulate potential variations in appearance across different sites, aiming to suppress the model’s learning of domain-specific information. At the local level, we propose a sequence-wise augmentation along input sequences, which perturbs the style of tokens within random continuous sub-sequences by modeling and resampling style statistics associated with domain shifts. To our best knowledge, Mamba-Sea is the first work to explore the generalization of Mamba for medical image segmentation, providing an advanced and promising Mamba-based architecture with strong robustness to domain shifts. Remarkably, our proposed method is the first to surpass a Dice coefficient of 90% on the Prostate dataset, which exceeds previous SOTA of 88.61%. The code is available at https://github.com/orange-czh/Mamba-Sea."},"title":{"value":"Mamba-Sea: A Mamba-Based Framework With Global-to-Local Sequence Augmentation for Generalizable Medical Image Segmentation"},"authors":{"value":[{"fullname":"Zihan Cheng","username":"~Zihan_Cheng4"},{"fullname":"Jintao Guo"},{"fullname":"Jian Zhang"},{"fullname":"Lei Qi"},{"fullname":"Luping Zhou"},{"fullname":"Yinghuan Shi"},{"fullname":"Yang Gao"}]}},"tmdate":1789092531754,"pdate":1756684800000,"externalIds":["doi:10.1109/tmi.2025.3564765"],"tcdate":1762588054744,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Zihan_Cheng4"],"forum":"D9QTddfEav","license":"CC BY-SA 4.0","number":11199,"cdate":1758084782887,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/-/Edit"],"mdate":1789092531754,"domain":"OpenReview.net/Public_Article","id":"D9QTddfEav","version":2},{"content":{"venue":{"value":"IEEE Trans. Medical Imaging 2025"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/42/11165775/10980210.pdf"},"venueid":{"value":"dblp.org/journals/TMI/2025"},"paperhash":{"value":"cheng|mambasea_a_mambabased_framework_with_globaltolocal_sequence_augmentation_for_generalizable_medical_image_segmentation"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zihan_Cheng_0001:","https://dblp.org/search/pid/api?q=author:Jintao_Guo:","https://dblp.org/search/pid/api?q=author:Jian_Zhang_0090:","~Lei_Qi1","https://dblp.org/search/pid/api?q=author:Luping_Zhou:","https://dblp.org/search/pid/api?q=author:Yinghuan_Shi:","https://dblp.org/search/pid/api?q=author:Yang_Gao_0001:"]},"html":{"value":"https://doi.org/10.1109/TMI.2025.3564765"},"_bibtex":{"value":"@article{DBLP:journals/tmi/ChengGZQZSG25,\n  author={Zihan Cheng and Jintao Guo and Jian Zhang and Lei Qi and Luping Zhou and Yinghuan Shi and Yang Gao},\n  title={Mamba-Sea: A Mamba-Based Framework With Global-to-Local Sequence Augmentation for Generalizable Medical Image Segmentation},\n  year={2025},\n  month={September},\n  cdate={1756684800000},\n  journal={IEEE Trans. Medical Imaging},\n  volume={44},\n  number={9},\n  pages={3741-3755},\n  url={https://doi.org/10.1109/TMI.2025.3564765}\n}\n"},"abstract":{"value":"To segment medical images with distribution shifts, domain generalization (DG) has emerged as a promising setting to train models on source domains that can generalize to unseen target domains. Existing DG methods are mainly based on CNN or ViT architectures. Recently, advanced state space models, represented by Mamba, have shown promising results in various supervised medical image segmentation. The success of Mamba is primarily owing to its ability to capture long-range dependencies while keeping linear complexity with input sequence length, making it a promising alternative to CNNs and ViTs. Inspired by the success, in the paper, we explore the potential of the Mamba architecture to address distribution shifts in DG for medical image segmentation. Specifically, we propose a novel Mamba-based framework, Mamba-Sea, incorporating global-to-local sequence augmentation to improve the model’s generalizability under domain shift issues. Our Mamba-Sea introduces a global augmentation mechanism designed to simulate potential variations in appearance across different sites, aiming to suppress the model’s learning of domain-specific information. At the local level, we propose a sequence-wise augmentation along input sequences, which perturbs the style of tokens within random continuous sub-sequences by modeling and resampling style statistics associated with domain shifts. To our best knowledge, Mamba-Sea is the first work to explore the generalization of Mamba for medical image segmentation, providing an advanced and promising Mamba-based architecture with strong robustness to domain shifts. Remarkably, our proposed method is the first to surpass a Dice coefficient of 90% on the Prostate dataset, which exceeds previous SOTA of 88.61%. The code is available at https://github.com/orange-czh/Mamba-Sea."},"title":{"value":"Mamba-Sea: A Mamba-Based Framework With Global-to-Local Sequence Augmentation for Generalizable Medical Image Segmentation"},"authors":{"value":["Zihan Cheng","Jintao Guo","Jian Zhang","Lei Qi","Luping Zhou","Yinghuan Shi","Yang Gao"]}},"tmdate":1769333293447,"pdate":1767139200000,"externalIds":["dblp:journals/tmi/ChengGZQZSG25"],"tcdate":1769333286930,"writers":["~"],"signatures":["~Lei_Qi1"],"forum":"cBmSuUNrQ6","license":"CC BY-SA 4.0","number":796297,"cdate":1756684800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769333293447,"domain":"DBLP.org","id":"cBmSuUNrQ6","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2504.17515v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"cheng|mambasea_a_mambabased_framework_with_globaltolocal_sequence_augmentation_for_generalizable_medical_image_segmentation"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zihan_Cheng_0001:","https://dblp.org/search/pid/api?q=author:Jintao_Guo:","https://dblp.org/search/pid/api?q=author:Jian_Zhang_0090:","~Lei_Qi1","https://dblp.org/search/pid/api?q=author:Luping_Zhou:","https://dblp.org/search/pid/api?q=author:Yinghuan_Shi:","https://dblp.org/search/pid/api?q=author:Yang_Gao_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2504.17515"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2504-17515,\n  publtype={informal},\n  author={Zihan Cheng and Jintao Guo and Jian Zhang and Lei Qi and Luping Zhou and Yinghuan Shi and Yang Gao},\n  title={Mamba-Sea: A Mamba-based Framework with Global-to-Local Sequence Augmentation for Generalizable Medical Image Segmentation},\n  year={2025},\n  month={April},\n  cdate={1743465600000},\n  journal={CoRR},\n  volume={abs/2504.17515},\n  url={https://doi.org/10.48550/arXiv.2504.17515}\n}\n"},"abstract":{"value":"To segment medical images with distribution shifts, domain generalization (DG) has emerged as a promising setting to train models on source domains that can generalize to unseen target domains. Existing DG methods are mainly based on CNN or ViT architectures. Recently, advanced state space models, represented by Mamba, have shown promising results in various supervised medical image segmentation. The success of Mamba is primarily owing to its ability to capture long-range dependencies while keeping linear complexity with input sequence length, making it a promising alternative to CNNs and ViTs. Inspired by the success, in the paper, we explore the potential of the Mamba architecture to address distribution shifts in DG for medical image segmentation. Specifically, we propose a novel Mamba-based framework, Mamba-Sea, incorporating global-to-local sequence augmentation to improve the model's generalizability under domain shift issues. Our Mamba-Sea introduces a global augmentation mechanism designed to simulate potential variations in appearance across different sites, aiming to suppress the model's learning of domain-specific information. At the local level, we propose a sequence-wise augmentation along input sequences, which perturbs the style of tokens within random continuous sub-sequences by modeling and resampling style statistics associated with domain shifts. To our best knowledge, Mamba-Sea is the first work to explore the generalization of Mamba for medical image segmentation, providing an advanced and promising Mamba-based architecture with strong robustness to domain shifts. Remarkably, our proposed method is the first to surpass a Dice coefficient of 90% on the Prostate dataset, which exceeds previous SOTA of 88.61%. The code is available at https://github.com/orange-czh/Mamba-Sea."},"title":{"value":"Mamba-Sea: A Mamba-based Framework with Global-to-Local Sequence Augmentation for Generalizable Medical Image Segmentation"},"authors":{"value":["Zihan Cheng","Jintao Guo","Jian Zhang","Lei Qi","Luping Zhou","Yinghuan Shi","Yang Gao"]}},"tmdate":1769333291405,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2504-17515"],"tcdate":1769333286604,"writers":["~"],"signatures":["~Lei_Qi1"],"forum":"C1uAUFLauB","license":"CC BY-SA 4.0","number":796287,"cdate":1743465600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769333291405,"domain":"DBLP.org","id":"C1uAUFLauB","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2411.15459v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"liu|mambavlt_timeevolving_multimodal_state_space_model_for_visionlanguage_tracking"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xinqi_Liu:","~Li_Zhou6","https://dblp.org/search/pid/api?q=author:Zikun_Zhou:","https://dblp.org/search/pid/api?q=author:Jianqiu_Chen:","https://dblp.org/search/pid/api?q=author:Zhenyu_He_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2411.15459"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2411-15459,\n  publtype={informal},\n  author={Xinqi Liu and Li Zhou and Zikun Zhou and Jianqiu Chen and Zhenyu He},\n  title={MambaVLT: Time-Evolving Multimodal State Space Model for Vision-Language Tracking},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2411.15459},\n  url={https://doi.org/10.48550/arXiv.2411.15459}\n}\n"},"abstract":{"value":"The vision-language tracking task aims to perform object tracking based on various modality references. Existing Transformer-based vision-language tracking methods have made remarkable progress by leveraging the global modeling ability of self-attention. However, current approaches still face challenges in effectively exploiting the temporal information and dynamically updating reference features during tracking. Recently, the State Space Model (SSM), known as Mamba, has shown astonishing ability in efficient long-sequence modeling. Particularly, its state space evolving process demonstrates promising capabilities in memorizing multimodal temporal information with linear complexity. Witnessing its success, we propose a Mamba-based vision-language tracking model to exploit its state space evolving ability in temporal space for robust multimodal tracking, dubbed MambaVLT. In particular, our approach mainly integrates a time-evolving hybrid state space block and a selective locality enhancement block, to capture contextual information for multimodal modeling and adaptive reference feature update. Besides, we introduce a modality-selection module that dynamically adjusts the weighting between visual and language references, mitigating potential ambiguities from either reference type. Extensive experimental results show that our method performs favorably against state-of-the-art trackers across diverse benchmarks."},"title":{"value":"MambaVLT: Time-Evolving Multimodal State Space Model for Vision-Language Tracking"},"authors":{"value":["Xinqi Liu","Li Zhou","Zikun Zhou","Jianqiu Chen","Zhenyu He"]}},"tmdate":1747105391348,"pdate":1704067200000,"tcdate":1747105388070,"writers":["~"],"signatures":["~Li_Zhou6"],"forum":"uTOhkJ56Oe","license":"CC BY-SA 4.0","number":433297,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747105391348,"domain":"DBLP.org","id":"uTOhkJ56Oe","version":2},{"content":{"comment":{"value":"# How sensitive are results to state size?\nWe use a state size of 8 (images), 32 (3D data), 64 (videos},  and apply the same state size across all S3K layers for each setting. Empirically, increasing the state size in the first S3K layer meaningfully enhances the performance, likely because this layer is the primary opportunity to capture high-frequency spatial details and a larger state size helps preserve this. In contrast, increasing the state size in later layers yields only marginal gains. For simplicity and consistency, we adopt the first layer’s optimal state size for all layers. In our experiments, performance improves as the state size increases but begins to saturate once the state size is sufficiently large relative to the signal complexity. In practice, this occurs around 8 for images, around 32 for 3D data, and around 64 for videos–values we therefore use throughout the model.\nWe will add the complete ablations on state size as soon as we finish the experiment.\n\n# Does this work for very high resolution images/videos on realistic training budgets?\nYes—within practical limits. Our experiments already include the CLIC2020 dataset, which contains 2K‐resolution images (2048×1080), and our model trains on these without difficulty. This demonstrates that S3K-LPNet is capable of handling high-resolution inputs under standard training budgets. From 4K-resolution images (3840 x 2160), LPNet-S3K exceeds our 48 GB memory budget.\n\n### References \n[1] Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (Gu et al., NeurIPS 2021)\n\n[2] Efficiently Modeling Sequences with Structured State Space (Gu et al., ICLR 2022)\n\n[3] Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Gu et al., COLM 2024)\n\n[4] On the parameterization and initialization of diagonal state space models (Gu et al., NeurIPS 2022)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763731848518,"tcdate":1763731848518,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission967/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission967/Authors"],"forum":"R5xBLfD9Dv","number":3,"license":"CC BY 4.0","cdate":1763731848518,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission967/-/Official_Comment"],"mdate":1763731848518,"domain":"ICLR.cc/2026/Conference","replyto":"XjT4FplkgU","id":"05a91rbQo0","forumContent":{"TLDR":{"value":"This paper first applies the compressive property of state-space models to data-specific neural representations, which aim to represent a datum by overfitting a neural network."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["state-space model"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"This paper studies the problem of data-specific neural representations, aiming for compact, flexible, and modality-agnostic storage of individual visual data using neural networks. Our approach considers a visual datum as a set of discrete observations of an underlying continuous signal, thus requiring models capable of capturing the inherent structure of the signal. For this purpose, we investigate state-space models (SSMs), which are well-suited for modeling latent signal dynamics. We first explore the appealing properties of SSMs for data-specific neural representation and then present a novel framework that integrates SSMs into the representation pipeline. The proposed framework achieved compact representations and strong reconstruction performance across a range of visual data formats, suggesting the potential of SSMs for data-specific neural representations."},"_bibtex":{"value":"@inproceedings{\nlee2026exploring,\ntitle={Exploring State-Space Models for Data-Specific Neural Representations},\nauthor={Jinsung Lee and Suha Kwak},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=R5xBLfD9Dv}\n}"},"title":{"value":"Exploring State-Space Models for Data-Specific Neural Representations"},"pdf":{"value":"/pdf/0bede07d70300c7e50981706917e67543dcd39ed.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lee|exploring_statespace_models_for_dataspecific_neural_representations"},"authorids":{"value":["~Jinsung_Lee1","~Suha_Kwak3"]},"authors":{"value":["Jinsung Lee","Suha Kwak"]}},"version":2},{"content":{"summary":{"value":"This paper explores the similarities and differences between the Mamba and linear attention Transformer models. It redefines Mamba as a variant of linear attention Transformer with six key distinctions. The paper also studies each design's impact, identifying the forget gate and block design as key to Mamba's success. Based on these findings, the paper introduces a Mamba-like linear Attention (MLLA) model, which outperforms other vision Mamba models on various tasks while maintaining fast computation and inference speed."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"I have some concerns (see weaknesses) about this paper. I am inclined to raise my score if my concerns are fully resolved.\n\n**Reference:**  \n\n[1],Diffusion Models Without Attention.CVPR24. \n\n[2],ZIGMA: A DiT-style Zigzag Mamba Diffusion Model,ECCV24. \n\n[3],Scalable Diffusion Models with State Space Backbone,Arxiv."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"* The paper is well-written and well-motivated from the perspective of linear attention.\n\n* The findings are interesting to me."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* Several works using Mamba for image synthesis should be discussed [1][2][3]. In Line 68, I wonder whether the exploration in this paper is orthogonal to previous explorations on Mamba. For example, Zigma considers layerwise scanpaths with 8 directions. Can the proposed method still yield some improvements on that result?\n\n* Another consideration is that the authors should discuss something with this paper: https://arxiv.org/abs/2406.06484. Check Table 4; it provides a much more general framework that includes most of the linear attention models.\n\n* My third concern is whether the exploration in this work can extend to other linear attention-based models, such as RWKV and xLSTM."},"limitations":{"value":"Yes, they are addressed and discussed."}},"nonreaders":[],"tmdate":1730878968063,"tcdate":1720776946404,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission4979/Reviewer_ii6x"],"signatures":["NeurIPS.cc/2024/Conference/Submission4979/Reviewer_ii6x"],"forum":"LvJ1R88KAk","number":3,"license":"CC BY 4.0","cdate":1720776946404,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission4979/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878968063,"domain":"NeurIPS.cc/2024/Conference","replyto":"LvJ1R88KAk","id":"XvTLGsFiGv","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","Linear Attention Transformer"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba’s success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba’s success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-\nInspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https://github.com/LeapLabTHU/MLLA."},"_bibtex":{"value":"@inproceedings{\nhan2024demystify,\ntitle={Demystify Mamba in Vision: A Linear Attention Perspective},\nauthor={Dongchen Han and Ziyi Wang and Zhuofan Xia and Yizeng Han and Yifan Pu and Chunjiang Ge and Jun Song and Shiji Song and Bo Zheng and Gao Huang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=LvJ1R88KAk}\n}"},"title":{"value":"Demystify Mamba in Vision: A Linear Attention Perspective"},"pdf":{"value":"/pdf/419f5b34c2a7aefa14ac7e991c160ab502bd73f5.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"han|demystify_mamba_in_vision_a_linear_attention_perspective"},"authorids":{"value":["~Dongchen_Han1","~Ziyi_Wang21","~Zhuofan_Xia2","~Yizeng_Han1","~Yifan_Pu1","~Chunjiang_Ge1","~Jun_Song5","~Shiji_Song1","~Bo_Zheng5","~Gao_Huang1"]},"authors":{"value":["Dongchen Han","Ziyi Wang","Zhuofan Xia","Yizeng Han","Yifan Pu","Chunjiang Ge","Jun Song","Shiji Song","Bo Zheng","Gao Huang"]}},"version":2},{"content":{"TLDR":{"value":"Uniform-RU corrects resolution-induced recurrent and residual-trajectory mismatches in frozen pretrained visual Mamba models, enabling robust cross-resolution transfer with lightweight label-free calibration."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["State Space Models","Visual Mamba","Resolution Generalization","Model Calibration","Cross-Resolution Transfer"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Mamba has emerged as a leading paradigm for long-sequence modeling and has been widely extended to vision. Despite its linear complexity and appeal for high-resolution representations, pretrained visual Mamba models often degrade severely at unseen resolutions. In this paper, we reveal that resolution scaling in visual Mamba is not merely a one-dimensional sequence extension, but entails a fundamental change in the two-dimensional spatial rate of pretrained selective dynamics. The resulting structural mismatch propagates through state-dependent residual updates and hierarchical transitions. To quantify this resolution-induced trajectory drift, we formulate Cross-Resolution Trajectory Discrepancy (CRTD). Pre-calibration analysis reveals a strong association between raw trajectory discrepancy and cross-resolution accuracy degradation. Based on these insights, we propose Architecture-Native Uniform Residual/Update Calibration (Uniform-RU), a lightweight and label-free framework that calibrates recurrent dynamics and residual trajectories using only a small fraction of the training images while keeping the pretrained backbone frozen. Across diverse visual Mamba architectures and resolutions, Uniform-RU substantially restores cross-resolution accuracy on image classification and generalizes well to detection and segmentation tasks. These results demonstrate that recurrent calibration can effectively extend pretrained visual Mamba models, offering a dynamics-centric perspective for resolution transfer beyond sequence scaling. Code is available at https://anonymous.4open.science/r/Uniform-RU/."},"_bibtex":{"value":"@inproceedings{\nanonymous2026beyond,\ntitle={Beyond Linear Scaling: Recurrent Calibration for Cross-Resolution Transfer in Pretrained Visual Mamba Models},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=1FnGjuTsxq},\nnote={under review}\n}"},"title":{"value":"Beyond Linear Scaling: Recurrent Calibration for Cross-Resolution Transfer in Pretrained Visual Mamba Models"},"pdf":{"value":"/pdf/d4480a47f29f1cae3905a7692885a14b2a247714.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230159405,"tcdate":1789540127871,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission23630/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission23630/Authors"],"forum":"1FnGjuTsxq","license":"CC BY 4.0","number":23630,"cdate":1789540127871,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission23630/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791230159405,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"1FnGjuTsxq","version":2},{"content":{"summary":{"value":"The paper proposes Numerion, a time-series forecaster that maps inputs into multiple hypercomplex spaces (complex, quaternion, octonion, sedenion) and learns with a unified Real–Hypercomplex–Real MLP (RHR-MLP). The key idea is that higher-dimensional hypercomplex spaces naturally bias toward lower-frequency components, enabling a simple, parallel decomposition and an adaptive fusion head to combine per-space predictions. Across standard long-term forecasting benchmarks, Numerion reports SOTA or competitive results, and provides visual/quantitative analyses indicating specialization of higher dimensions for trends and lower dimensions for fluctuations. The method generalizes linear layers and tanh to arbitrary power-of-two dimensions and supplies ablations, sensitivity, and efficiency studies."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"How sensitive is Numerion to patch levels/embedding dims under noisy domains (e.g., Exchange, where MAE improvement is only 0.005), beyond the Electricity study?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- This paper extends linear/activation layers to arbitrary hypercomplex dimensions and composes them into a parallel multi-space forecaster with adaptive fusion—distinct from typical decomposition or spectral methods.\n- Offers a simple, backbone-agnostic path to multi-frequency modeling without heavy architectural complexity; could inspire broader use of hypercomplex representations for TS.\n- Comprehensive experiments across 10 datasets with consistent improvements and honest discussion where variable-fusion models can outperform on high-dimensional data; includes ablations, visualization, and sensitivity."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- This paper lacks comparisons against strong, explicit frequency or time–frequency baselines, such as FBM[1], to support the claimed advantage brought by operations in higher-dimensional hypercomplex spaces. \n\n- It remains unclear whether improvements stem from genuinely hypercomplex algebra or simply from vector factorization at dyadic sizes ($n \\in$ \\{$1,2,4,8,16$\\}). The Cayley–Dickson construction does not mathematically guarantee a low-pass (frequency-selective) behavior; thus the observed “low-frequency preference” might arise from architecture/parameterization effects rather than algebraic properties. The authors may add controls that replace hypercomplex layers with *grouped real MLPs* (same splits $n $); also visualize learned frequency responses to disambiguate where the bias truly comes from.\n\n\n[1] Yang, R., Cao, L., & YANG, J. (2024). Rethinking Fourier transform from a basis functions perspective for long-term time series forecasting. Advances in Neural Information Processing Systems, 37, 8515-8540."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925588375,"tcdate":1761576667284,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15291/Reviewer_ugew"],"signatures":["ICLR.cc/2026/Conference/Submission15291/Reviewer_ugew"],"forum":"2hCwLqx5gm","number":1,"license":"CC BY 4.0","cdate":1761576667284,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15291/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925588375,"domain":"ICLR.cc/2026/Conference","replyto":"2hCwLqx5gm","id":"Og6Alpg0lM","forumContent":{"TLDR":{"value":"We propose Numerion, a hypercomplex space-based model, decomposes and forecasts time series using multi-dimensional RHR-MLPs, achieving state-of-the-art results."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Time Series Forecasting","Hypercomplex Numbers","Hypercomplex Time Series Models","Multi-Hypercomplex Space"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Many methods aim to enhance time series forecasting by decomposing the series through intricate model structures and prior knowledge, yet they are inevitably limited by computational complexity and the robustness of the assumptions. Our research uncovers that in the complex domain and higher-order hypercomplex spaces, the characteristic frequencies of time series naturally decrease. Leveraging this insight, we propose Numerion, a time series forecasting model based on multiple hypercomplex spaces. Specifically, grounded in theoretical support, we generalize linear layers and activation functions to hypercomplex spaces of arbitrary power-of-two dimensions and introduce a novel Real-Hypercomplex-Real Domain Multi-Layer Perceptron (RHR-MLP) architecture. Numerion utilizes multiple RHR-MLPs to map time series into hypercomplex spaces of varying dimensions, naturally decomposing and independently modeling the series, and adaptively fuses the latent patterns exhibited in different spaces through a dynamic fusion mechanism. Experiments validate the model’s performance, achieving state-of-the-art results on multiple public datasets. Visualizations and quantitative analyses comprehensively demonstrate the ability of multi-dimensional RHR-MLPs to naturally decompose time series and reveal the tendency of higher-dimensional hypercomplex spaces to capture lower-frequency features."},"_bibtex":{"value":"@inproceedings{\ncao2026numerion,\ntitle={Numerion: A Multi-Hypercomplex Model for Time Series Forecasting},\nauthor={Hanzhong Cao and WenBo Yan and Ying Tan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=2hCwLqx5gm}\n}"},"title":{"value":"Numerion: A Multi-Hypercomplex Model for Time Series Forecasting"},"pdf":{"value":"/pdf/0b40c1391b1d3dadb27f73e92392b8e4eb6c195b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"cao|numerion_a_multihypercomplex_model_for_time_series_forecasting"},"authorids":{"value":["~Hanzhong_Cao1","~WenBo_Yan1","~Ying_Tan1"]},"authors":{"value":["Hanzhong Cao","WenBo Yan","Ying Tan"]}},"version":2},{"content":{"TLDR":{"value":"Systematic study of linear diagonal RNNs' length generalization failure and memory capacity in language modeling and contextual information retrieval."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["RNN","foundation models","long-context"]},"supplementary_material":{"value":"/attachment/d71dbd3e7ce14131c3720da88c049fa27def3e94.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"One essential advantage of recurrent neural networks (RNNs) over transformer-based language models is their linear computational complexity concerning the sequence length, which makes them much faster in handling long sequences during inference. However, most publicly available RNNs (e.g., Mamba and RWKV) are trained on sequences with less than 10K tokens, and their effectiveness in longer contexts remains largely unsatisfying so far. In this paper, we study the cause of the inability to process long context for RNNs and suggest critical mitigations. First, we investigate *state explosion* (SE) in Mamba-2 when processing long sequences, a phenomenon where some channels of the state exhibit exploding values that cause severe performance degradation. With controlled experiments, we discover that the model fails to forget the earlier tokens when there is more information than it can remember. We attribute this to overfitting due to the recurrent state being overparameterized for the training length, thereby establishing a relationship between SE and the capacity of the state. To support this hypothesis, we make an important empirical observation: for any given state size, there exists a training length threshold such that SE is exhibited if and only if the training length is greater than this threshold. Empirically searching for this threshold for different state sizes reveals that it is a linear function of the state size. We also search for the maximum context length at which the model can recall contextual information and find that this context length scales exponentially to the state size. Based on this, we empirically train a Mamba-2 370M with near-perfect passkey retrieval accuracy on 256K context length. This suggests a promising future for RNN-based long-context modeling. Code and model checkpoints will be publicly released."},"_bibtex":{"value":"@misc{\nchen2025stuffed,\ntitle={Stuffed Mamba: State Collapse and State Capacity of {RNN}-Based Long-Context Modeling},\nauthor={Yingfa Chen and Xinrong Zhang and Shengding Hu and Xu Han and Zhiyuan Liu and Maosong Sun},\nyear={2025},\nurl={https://openreview.net/forum?id=cu2CT2VAvs}\n}"},"title":{"value":"Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling"},"pdf":{"value":"/pdf/81b99123fa4ffe7fe2c3aa7b6d9993ea1a3889f6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chen|stuffed_mamba_state_collapse_and_state_capacity_of_rnnbased_longcontext_modeling"},"authorids":{"value":["~Yingfa_Chen1","~Xinrong_Zhang1","~Shengding_Hu2","~Xu_Han2","~Zhiyuan_Liu1","~Maosong_Sun1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yingfa Chen","Xinrong Zhang","Shengding Hu","Xu Han","Zhiyuan Liu","Maosong Sun"]}},"tmdate":1738735849218,"tcdate":1727472130652,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12121/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission12121/Authors"],"forum":"cu2CT2VAvs","license":"CC BY 4.0","number":12121,"cdate":1727472130652,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission12121/-/Full_Submission","ICLR.cc/2025/Conference/Submission12121/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735849218,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"cu2CT2VAvs","version":2},{"content":{"TLDR":{"value":"This paper presents a novel method for counterfactual prediction over time series with state-space models."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series; State-space Models; Treatment Effect Estimation"]},"primary_area":{"value":"causal reasoning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Time-varying counterfactual prediction (TCP) from observational data supports the answer of when and how to assign multiple sequential treatments, yielding importance in various applications. Despite the progress achieved by recent advances, e.g., LSTM or Transformer based causal approaches, their capability of capturing interactions in long sequences remains to be improved in both prediction performance and running efficiency. In parallel with the development of TCP, the success of the state-space models (SSMs) has achieved remarkable progress toward long-sequence modeling with saved running time. Consequently, studying how Mamba simultaneously benefits the effectiveness and efficiency of TCP  becomes a compelling research direction. In this paper, we propose to exploit advantages of the SSMs to tackle the TCP task, by introducing a counterfactual Mamba model with Covariate-based Decorrelation towards Selective Parameters (Mamba-CDSP). Motivated by the over-balancing problem in TCP of the direct covariate balancing methods, we propose to de-correlate between the current treatment and the representation of historical covariates, treatments, and outcomes, which can mitigate the confounding bias while preserve more covariate information. In addition, we show that the overall de-correlation in TCP is equivalent to regularizing the selective parameters of Mamba over each time step, which leads our approach to be effective and lightweight. We conducted extensive experiments on both synthetic and real-world datasets, demonstrating that Mamba-CDSP not only outperforms baselines by a large margin, but also exhibits prominent running efficiency."},"_bibtex":{"value":"@inproceedings{\nwang2025effective,\ntitle={Effective and Efficient Time-Varying Counterfactual Prediction with State-Space Models},\nauthor={Haotian Wang and Haoxuan Li and Hao Zou and Haoang Chi and Long Lan and Wanrong Huang and Wenjing Yang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=yheQRc5xWB}\n}"},"title":{"value":"Effective and Efficient Time-Varying Counterfactual Prediction with State-Space Models"},"pdf":{"value":"/pdf/6543649eb8bf53c6ca75c251a0032d3e63a00c35.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wang|effective_and_efficient_timevarying_counterfactual_prediction_with_statespace_models"},"authorids":{"value":["~Haotian_Wang2","~Haoxuan_Li6","~Hao_Zou1","~Haoang_Chi1","~Long_Lan2","~Wanrong_Huang1","~Wenjing_Yang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Haotian Wang","Haoxuan Li","Hao Zou","Haoang Chi","Long Lan","Wanrong Huang","Wenjing Yang"]}},"tmdate":1740971471191,"pdate":1737562523087,"tcdate":1727364175335,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission7314/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission7314/Authors"],"forum":"yheQRc5xWB","license":"CC BY 4.0","number":7314,"cdate":1727364175335,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission7314/-/Full_Submission","ICLR.cc/2025/Conference/Submission7314/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit","ICLR.cc/2025/Conference/Submission7314/-/Camera_Ready_Revision"],"mdate":1740971471191,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"yheQRc5xWB","version":2},{"content":{"venue":{"value":"TIME 2016"},"pdf":{"value":"https://ieeexplore.ieee.org/iel7/7774233/7774630/07774643.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2016"},"paperhash":{"value":"mousa|optimal_control_for_simple_linear_hybrid_systems"},"authorids":{"value":["","~Sven_Schewe1",""]},"html":{"value":"https://doi.org/10.1109/TIME.2016.9"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/MousaSW16,\n  author={Mahmoud A. A. Mousa and Sven Schewe and Dominik Wojtczak},\n  title={Optimal Control for Simple Linear Hybrid Systems},\n  year={2016},\n  cdate={1451606400000},\n  pages={12-20},\n  url={https://doi.org/10.1109/TIME.2016.9},\n  booktitle={TIME},\n  crossref={conf/time/2016}\n}\n"},"abstract":{"value":"This paper studies optimal time-bounded control in a simple subclass of linear hybrid systems, which consists of one continuous variable and global constraints. Each state has a continuous cost attached to it, which is linear in the sojourn time, while a discrete cost is attached to each transition taken. We show the corresponding decision problem to be NP-complete and develop an FPTAS for finding an approximate solution. We have implemented a small prototype to compare the performance of these approximate and precise algorithms for this problem. Our results indicate that the proposed approximation schemes scale. Furthermore, we show that the same problem with infinite time horizon is in LOGSPACE."},"title":{"value":"Optimal Control for Simple Linear Hybrid Systems"},"authors":{"value":["Mahmoud A. A. Mousa","Sven Schewe","Dominik Wojtczak"]}},"tmdate":1775124543337,"pdate":1483142400000,"tcdate":1775124531663,"writers":["~"],"signatures":["~Sven_Schewe1"],"forum":"9zrSeVx4VD","license":"CC BY-SA 4.0","number":863474,"cdate":1451606400000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1775124543337,"domain":"DBLP.org","id":"9zrSeVx4VD","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2608.08256v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"aliane|arassm_a_bidirectional_statespace_encoder_for_arabic_masked_language_modeling"},"html":{"value":"https://doi.org/10.48550/arXiv.2608.08256"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2608-08256,\n  publtype={informal},\n  author={Ahmed Amine Aliane and Hassina Aliane and Nasredine Semmar},\n  title={AraSSM: A bidirectional state-space encoder for Arabic masked language modeling},\n  year={2026},\n  month={August},\n  cdate={1785542400000},\n  journal={CoRR},\n  volume={abs/2608.08256},\n  url={https://doi.org/10.48550/arXiv.2608.08256}\n}\n"},"abstract":{"value":"Pretrained Transformer encoders such as AraBERT, MARBERT, and CAMeLBERT have become the standard backbone for Arabic natural language understanding, but their self-attention mechanism scales quadratically with sequence length, which limits efficiency on long documents. Mamba, a selective state-space model (SSM), offers linear-time sequence modeling as a competitive alternative to attention, yet no dedicated bidirectional Mamba encoder pretrained specifically for Arabic currently exists. We introduce AraSSM, a bidirectional Mamba encoder pretrained via masked language modeling on a corpus combining Arabic Wikipedia and CulturaX text, trained end-to-end on four consumer-grade NVIDIA RTX 2080Ti GPUs (11GB) over approximately ten days. We evaluate AraSSM by fine-tuning on four established Arabic NLU benchmarks covering sentiment classification (HARD), named entity recognition (ANERcorp), extractive question answering (ARCD), and natural language inference (XNLI-ar), following the per-task evaluation protocol introduced by AraBERT, and report results as mean +/- standard deviation across three fine-tuning seeds. AraSSM matches or exceeds published base-sized Transformer baselines on sentiment classification (96.37 +/- 0.03% accuracy on HARD), is competitive on extractive QA (32.19 +/- 1.07 EM, 63.79 +/- 0.25 F1 on ARCD) and named entity recognition (81.54 +/- 0.30 entity-level F1 on ANERcorp), and trails the base-sized Transformer range on natural language inference (72.83 +/- 0.07% accuracy on XNLI-ar), despite being trained entirely from scratch on consumer hardware rather than large-scale accelerator clusters."},"title":{"value":"AraSSM: A bidirectional state-space encoder for Arabic masked language modeling"},"authors":{"value":[{"fullname":"Ahmed Amine Aliane","username":""},{"fullname":"Hassina Aliane","username":""},{"fullname":"Nasredine Semmar","username":"~Nasredine_Semmar2"}]}},"tmdate":1791991199959,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2608-08256"],"tcdate":1791991193764,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Nasredine_Semmar2"],"forum":"FA5tvn0L04","license":"CC BY-SA 4.0","number":184112,"cdate":1785542400000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1791991199959,"domain":"OpenReview.net/Public_Article","id":"FA5tvn0L04","version":2},{"content":{"summary":{"value":"This paper proposed **ParallelTime** to combine two dominant architectures, Transformer and Mamba. While prior work in NLP has combined these models, it typically does so by simple averaging, which assigns equal weight to both short- and long-term features. The authors argue this is suboptimal for time series data.\n\nThe core contributions are:\n1. **ParallelTime Architecture**: A decoder-only model that tokenizes the input multivariable time-series data as patchifed univariable time-series and processes them through a combination of Mamba blocks and window-attention blocks by **ParallelTime Weighter** instead of simply summing or averaging.\n1. **ParallelTime Weighter**: A novel, dynamic, and token-specific weighting mechanism to combine the output of the window-attention block and output attention.\n\nDetailed experimental results show ParallelTime achieves SOTA performance on real-world benchmarks."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Is there any prior work that applies window-attention with global registers in time-series data (I did not see one in the related works from line 108 to line 113)? If so, has the paper compared such a model? If not, will it work well without a combination of mamba blocks? Also in Appendix 9.3, it's stated that global registers yield slight performance enhancements. Are there any experimental results comparing ParallelTime with and without global registers?\n1. What is the reason to compress the dimensionality from $\\text{dim}$ to $\\sqrt{\\text{dim}}$ in ParallelTime Weighter other than some other dimensionalities?\n1. Efficiency comparison shows that ParallelTime has fewer FLOPs (and parameters), but how about the inference latency?\n1. ParallelTime Weighter outperforms simple sum or average, but how about concat (in dimension)?\n1. Since window-attention has a global register, can it also capture long-term dependencies (see statements in lines 247, the author says window-attention with global registers only captures short-term dependencies)."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. This paper is the first to combine the mamba and transformer architectures for time-series, expecting short- and long-term dependencies gains from their architectures.\n1. A novel Parallel Weighter is proposed to combine the outputs from the mamba blocks and the local attention blocks, and experiment results show that it benefits the performance by simply summing or averaging (with ablation studies for justification).\n1. Detailed experiments are conducted to show that the Parallel Time model achieves state-of-the-art performance, with nearly half the FLOPs compared to PatchTST.\n1. The writing is detailed and easy to follow."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"This paper is solid and well-written, except for some minor weaknesses:\n1. Missing definition of $\\mathbf{y}_t$ in line 222, and missing relationship between $\\text{Mamba}(\\cdot)$ and $\\mathbf{x}, \\mathbf{y}, \\mathbf{h}$ in line 227.\n1. Missing comparison of pure mamba and pure window-attention in Figure 6.\n1. Efficiency comparison only compares with PatchTST, missing some potentially more lightweight models like DLinear.\n1. ParallelTime was designed to capture more long-time dependencies (or global dependencies), but the result in Table 1 shows it only performs the best half of the dataset in prediction with a prediction length of 720.\n1. Figure 4 shows the visualization of the weighting outputs of the Parallel Weighter to justify that the Parallel Weighter can dynamically adjust weights of short- and long-term dependences. However, the prediction length of 192 is not long enough (compared to 720) to show long-term dependencies."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933740314,"tcdate":1761791799805,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20257/Reviewer_vQqx"],"signatures":["ICLR.cc/2026/Conference/Submission20257/Reviewer_vQqx"],"forum":"dHhCBcsKet","number":3,"license":"CC BY 4.0","cdate":1761791799805,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20257/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933740314,"domain":"ICLR.cc/2026/Conference","replyto":"dHhCBcsKet","id":"TDh5yXw091","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["time series","forecasting","mamba","transformer","attention","dynamic weighting of temporal dependencies"]},"supplementary_material":{"value":"/attachment/f418e63cf50b4432fd7216d67f70d587a0855cff.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Modern multivariate time series forecasting primarily relies on two architectures: the Transformer with attention mechanism and Mamba. In natural language processing, an approach has been used that combines local window attention for capturing short-term dependencies and Mamba for capturing long-term dependencies, with their outputs averaged to assign equal weight to both. We find that for time-series forecasting tasks, assigning equal weight to long-term and short-term dependencies is not optimal. To mitigate this, we propose a dynamic weighting mechanism, ParallelTime Weighter, which calculates interdependent weights for long-term and short-term dependencies for each token based on the input and the model's knowledge. Furthermore, we introduce the ParallelTime architecture, which incorporates the ParallelTime Weighter mechanism to deliver state-of-the-art performance across diverse benchmarks. Our architecture demonstrates robustness, achieves lower FLOPs, requires fewer parameters, scales effectively to longer prediction horizons, and significantly outperforms existing methods. These advances highlight a promising path for future developments of parallel Attention-Mamba in time series forecasting. The implementation is readily available at the GitHub link."},"_bibtex":{"value":"@misc{\nkatav2026paralleltime,\ntitle={ParallelTime: Dynamically Weighting the Balance of Short- and Long-Term Temporal Dependencies},\nauthor={Itay Katav and Aryeh Kontorovich},\nyear={2026},\nurl={https://openreview.net/forum?id=dHhCBcsKet}\n}"},"title":{"value":"ParallelTime: Dynamically Weighting the Balance of Short- and Long-Term Temporal Dependencies"},"pdf":{"value":"/pdf/081743e6eebba527c6ec57e6c1dd07b426500338.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"katav|paralleltime_dynamically_weighting_the_balance_of_short_and_longterm_temporal_dependencies"},"authorids":{"value":["~Itay_Katav1","~Aryeh_Kontorovich1"]},"authors":{"value":["Itay Katav","Aryeh Kontorovich"]}},"version":2},{"content":{"comment":{"value":"> The authors could strengthen their contribution by providing more extensive ablation studies on their architectural choices. \n\nIn our work, the primary goal was to demonstrate that simplified RNN architectures, such as minLSTM and minGRU, can perform comparably to modern state-of-the-art sequence models. To achieve this, we intentionally designed a minimalistic architecture, following standard practices such as residual connections, normalization, and a downprojection layer for the RNN's expanded hidden states. For more complex tasks like language modeling and Long Range Arena, standard components (convolutional layer and MLP) are added.\n\nTo provide further insight into the impact of these architectural choices, we performed an ablation study on the ListOps (Long Range Arena) dataset. The results, averaged over 3 seeds, are shown below:\n\n\n|          Model         | Accuracy |\n|:----------------------:|:--------:|\n|     Random Baseline    |   0.10   |\n|         minLSTM        |   0.46   |\n|    minLSTM (+ conv)    |   0.45   |\n|     minLSTM (+ mlp)    |   0.52   |\n| minLSTM (+ conv + mlp) |   0.59   |\n\nWe have updated Appendix C.1 to better explain the architectural choices and rationale behind these components.\n\n> Finally, more detailed inference speed comparisons would strengthen the paper. How do minGRU and minLSTM compare with Mamba and traditional LSTM/GRU implementations across different sequence lengths and batch sizes?\n\nThank you for your suggestion to include more detailed inference speed comparisons. We have now included inference speed results for GRU, LSTM, minGRU, minLSTM, and Mamba (using the official implementation). However, we would like to highlight that inference speed can vary depending on hardware and implementation.\n\nFor these experiments, we varied both batch size (8, 16, 32, 64) and sequence length (up to 2048). In Figure 3, we report the average inference speed across 50 runs, with context tokens considered before performing inference. Since GRU and LSTM models process context tokens sequentially, their inference times are significantly slower compared to minGRU, minLSTM, and Mamba, all of which benefit from parallel processing.\n\nOverall, minLSTM and minGRU show inference speeds comparable to Mamba. Specifically, minGRU was 6.6%, 4.1%, 4.9%, and 2.9% faster than Mamba for batch sizes of 8, 16, 32, and 64, respectively. On the other hand, minLSTM was slightly slower than Mamba by 3.6%, 2.9%, 0%, and 1.3% for the same batch sizes. \n\nSince minLSTM and minGRU are simplifications of LSTM and GRU, we expect them to generally perform faster, including during inference. This is demonstrated in Figure 4, where we compare the inference speed of minLSTM and minGRU with traditional LSTM and GRU models across varying batch sizes. As expected, minGRU and minLSTM are 19.6% and 41.5% faster than GRU and LSTM for a batch size of 64, respectively.\n\nThese detailed inference speed results and plots are included in the Appendix.\n\n\n---\n\nMartin, Eric and Chris Cundy \"Parallelizing Linear Recurrent Neural Nets over Sequence Length.\" ICLR (2018).\n\nGu, Albert and Tri Dao. \"Mamba: Linear-Time Sequence Modeling with Selective State Spaces.\"  arXiv preprint arXiv:2312.00752 (2023).\n\nTay, Yi, et al. \"Long Range Arena: A Benchmark for Efficient Transformers.\" ICLR (2021).\n\nDelétang, Grégoire, et al. \"Neural Networks and the Chomsky Hierarchy.\" ICLR (2023)."},"title":{"value":"Response to Reviewer jHLY (4/4)"}},"tmdate":1732172868577,"tcdate":1732169724874,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8241/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission8241/Authors"],"forum":"GrmFFxGnOR","number":10,"license":"CC BY 4.0","cdate":1732169724874,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8241/-/Official_Comment"],"mdate":1732172868577,"domain":"ICLR.cc/2025/Conference","replyto":"ZLKbCUEU7B","id":"oWkXuahpzC","forumContent":{"TLDR":{"value":"Revisiting decade-old RNNs (LSTMs and GRUs), we introduce minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters, (2) are parallelizable during training, and (3) match the performance of recent sequence models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Recurrent Neural Networks","RNNs","Sequence Modelling","Efficiency","LSTMs","GRUs","Parallel Scan"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers—particularly with respect to sequence length—have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively.\nIn this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers."},"_bibtex":{"value":"@misc{\nfeng2025were,\ntitle={Were {RNN}s All We Needed?},\nauthor={Leo Feng and Frederick Tung and Mohamed Osama Ahmed and Yoshua Bengio and Hossein Hajimirsadeghi},\nyear={2025},\nurl={https://openreview.net/forum?id=GrmFFxGnOR}\n}"},"title":{"value":"Were RNNs All We Needed?"},"pdf":{"value":"/pdf/a56137c0e88e5c5b4003b438b48e61ee149bd102.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"feng|were_rnns_all_we_needed"},"authorids":{"value":["~Leo_Feng1","~Frederick_Tung1","~Mohamed_Osama_Ahmed2","~Yoshua_Bengio1","~Hossein_Hajimirsadeghi1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Leo Feng","Frederick Tung","Mohamed Osama Ahmed","Yoshua Bengio","Hossein Hajimirsadeghi"]}},"version":2},{"content":{"venue":{"value":"SCOPE - ICLR 2025 Oral"},"keywords":{"value":["Linear Sequence Modeling","Mixture-of-Experts","Distributed Training"]},"abstract":{"value":"Linear Sequence Modeling (LSM) and Mixture-of-Experts (MoEs) have recently emerged as effective architectural improvements. In this paper, we introduce Linear-MoE, a production-level system for modeling and training large-scale models that integrate LSM with MoEs. Linear-MoE leverages the advantages of both LSM modules for linear-complexity sequence modeling and MoE layers for sparsely activation, aiming to offer high performance with efficient training and deployment. The Linear-MoE system comprises two primary subsystems: Modeling and Training. The Modeling subsystem provides a unified framework supporting multiple types of LSM methods, including linear attention, SSM, and linear RNN. The Training subsystem facilitates efficient training by incorporating advanced parallelism techniques like Tensor, Pipeline, and Expert Parallelism, along with LASP-based Sequence Parallelism for managing very-long input sequences. The system is designed to be extensible for integrating more sequence modeling and training abilities in the future. Additionally, we explore hybrid Linear-MoE models that combine Linear-MoE layers with standard Transformer-MoE layers to further enhance model flexibility and performance. Experimental evaluations on two model series, A0.3B-2B and A1B-7B, demonstrate that Linear-MoE achieves efficiency gains while maintaining competitive performance on various benchmarks. The code is released at: \\url{https://github.com/OpenSparseLLMs/Linear-MoE}."},"_bibtex":{"value":"@inproceedings{\nsun2025linearmoe,\ntitle={Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts},\nauthor={Weigao Sun and Disen Lan and Tong Zhu and Xiaoye Qu and Yu Cheng},\nbooktitle={First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models},\nyear={2025},\nurl={https://openreview.net/forum?id=HKIvuZxGbl}\n}"},"title":{"value":"Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts"},"Anonymization":{"value":"This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs."},"pdf":{"value":"/pdf/1cde5bcf966d473d1cc29ef74640cb00824610e0.pdf"},"venueid":{"value":"ICLR.cc/2025/Workshop/SCOPE"},"paperhash":{"value":"sun|linearmoe_linear_sequence_modeling_meets_mixtureofexperts"},"authorids":{"value":["~Weigao_Sun1","~Disen_Lan1","~Tong_Zhu2","~Xiaoye_Qu1","~Yu_Cheng1"]},"Track":{"value":"Main paper track (up to 5 pages excluding references and appendix)"},"authors":{"value":["Weigao Sun","Disen Lan","Tong Zhu","Xiaoye Qu","Yu Cheng"]}},"tmdate":1744697715650,"pdate":1741193735623,"tcdate":1739188025475,"writers":["ICLR.cc/2025/Workshop/SCOPE","ICLR.cc/2025/Workshop/SCOPE/Submission98/Authors"],"signatures":["ICLR.cc/2025/Workshop/SCOPE/Submission98/Authors"],"forum":"HKIvuZxGbl","license":"CC BY 4.0","number":98,"cdate":1739188025475,"readers":["everyone"],"invitations":["ICLR.cc/2025/Workshop/SCOPE/-/Submission","ICLR.cc/2025/Workshop/SCOPE/-/Post_Submission","ICLR.cc/2025/Workshop/SCOPE/-/Edit","ICLR.cc/2025/Workshop/SCOPE/Submission98/-/Camera-Ready_Revision"],"mdate":1744697715650,"odate":1741193735623,"domain":"ICLR.cc/2025/Workshop/SCOPE","id":"HKIvuZxGbl","version":2},{"content":{"summary":{"value":"This paper proposes Successor Flow Features (SF²), a representation learning framework that integrates flow matching with Successor Measure (SM) to bridge SM estimation and online policy optimization, to enhance sample efficiency of RL training. SF² enforces a structured linear decomposition into a time-invariant embedding and a time-dependent projection, enabling seamless integration with off-policy algorithms like TD3 and SAC. Empirical results on DeepMind Control Suite tasks show improved sample efficiency and training stability over successor feature baselines. The core idea of combining flow-based generative modeling with successor representations is theoretically novel, but several critical gaps limit its comprehensiveness and practical impact."},"soundness":{"value":3},"confidence":{"value":2},"questions":{"value":"1. Does SF² bring better generalization performance?\n2. In what RL scenarios is SF² most effective?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The integration of flow matching with SM for online RL is a creative extension of both generative modeling and successor representation paradigms. The structured linear decomposition (time-invariant embedding + time-dependent projection) addresses SM’s lack of compact, online-friendly representations, which is a non-trivial technical contribution.\n2. Experiments on 7 DeepMind Control Suite tasks demonstrate consistent performance gains over standard TD3/SAC and SF-based baselines. The detailed hyperparameter analysis (EMA coefficient, denoising steps, feature size) and ablation studies add rigor to the results.\n3. The paper establishes meaningful links between SF² and existing methods (Successor Representation, Diffusion Spectral Representation), providing conceptual clarity and grounding the framework in prior literature."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Insufficient Justification for Successor Representations in Online RL: The paper frames sample efficiency as a key advantage, but successor representations are inherently designed for reward-dynamics decoupling (enabling generalization/zero-shot transfer). No evidence is provided that SF² outperforms state-of-the-art sample-efficient representation methods (e.g., SPR, CURL) that do not rely on successor paradigms. This raises questions about whether successor representations are necessary for online RL’s core goal of sample-efficient single-task learning.\n2. Missing Comparisons with Critical Baselines: The baselines are limited to standard TD3/SAC and SF variants (TD3Sim/SACSim). The absence of comparisons with representative sample-efficient RL methods—particularly SPR (Self-Predictive Representations), which directly targets data efficiency via self-supervised representations—undermines claims of advancing the state-of-the-art in online RL.\n3. Unproven Generalization to Pixel-Based Environments: All experiments use structured state features (e.g., joint angles, velocities) from DeepMind Control Suite. The paper provides no analysis or modifications for pixel-based inputs (high-dimensional, unstructured data), which are common in real-world online RL. It remains unclear if SF²’s linear decomposition and flow matching components can adapt to image data.\n4. No Evaluation on Discrete Action Spaces: The work exclusively focuses on continuous control tasks with TD3/SAC. Successor representations (e.g., SR) originated in discrete state-action spaces, but SF²’s performance and adaptability to discrete actions are unaddressed. This limits the framework’s generality, as discrete action scenarios (e.g., game playing, recommendation systems) are central to online RL.\n5. Complexity: The paper notes SF²’s ~2x longer running time than standard TD3 (e.g., 1300s vs. 659s on AcrobotSwingup).\n\nReference\n\nSPR: Schwarzer, Max, et al. \"Data-Efficient Reinforcement Learning with Self-Predictive Representations.\" ICLR 2021.\n\nCURL: Laskin, Michael, Aravind Srinivas, and Pieter Abbeel. \"Curl: Contrastive unsupervised representations for reinforcement learning.\" ICML 2020."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764358710723,"tcdate":1761981178980,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5664/Reviewer_Cb48"],"signatures":["ICLR.cc/2026/Conference/Submission5664/Reviewer_Cb48"],"forum":"jA3KmR18S7","number":3,"license":"CC BY 4.0","cdate":1761981178980,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5664/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764358710723,"domain":"ICLR.cc/2026/Conference","replyto":"jA3KmR18S7","id":"hqJ9kS0nuN","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["reinforcement learning","representation learning","flow matching"]},"primary_area":{"value":"reinforcement learning"},"abstract":{"value":"The Successor Measure (SM), a powerful method in reinforcement learning (RL), describes discounted future state distributions under a policy, and it has recently been studied using generative modeling techniques. Although SM is a powerful predictive object, it lacks compact representations tailored for online RL. To address this, we introduce Successor Flow Features (SF2), a representation learning framework that bridges SM estimation with policy optimization. SF2 leverages flow-matching generative models to approximate successor measures, while enforcing a structured linear decomposition into a time-invariant embedding and a time-dependent projection. This yields compact, policy-aware state-action features that integrate readily into standard off-policy algorithms like TD3 and SAC. Experiments on DeepMind Control Suite tasks show that SF2 improves sample efficiency and training stability compared to strong successor feature baselines. We attribute these gains to the compact representation induced by flow matching, which reduces compounding errors in long-horizon predictions. The code is available on https://github.com/Shiien/successor-flow-representation-implementation ."},"_bibtex":{"value":"@inproceedings{\nshi2026bridging,\ntitle={Bridging Successor Measure and Online Policy Learning with Flow Matching-Based Representations},\nauthor={Haosen Shi and Jianda Chen and Sinno Jialin Pan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=jA3KmR18S7}\n}"},"title":{"value":"Bridging Successor Measure and Online Policy Learning with Flow Matching-Based Representations"},"pdf":{"value":"/pdf/6f81f0d3814654ff2b2cf9760e14a6eab755cc57.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"shi|bridging_successor_measure_and_online_policy_learning_with_flow_matchingbased_representations"},"authorids":{"value":["~Haosen_Shi1","~Jianda_Chen1","~Sinno_Jialin_Pan1"]},"authors":{"value":["Haosen Shi","Jianda Chen","Sinno Jialin Pan"]}},"version":2},{"content":{"summary":{"value":"This paper introduces KambaAD, a model for time series anomaly detection that incorporates an Encoder network with (1) a Kolmogorov-Arnold Network (KAN), (2) a Selective Structured State Space Model (MAMBA), and (3) an Attention mechanism. The hybrid encoder employs a two-stage feature extraction process: After the sequence is processed by KAN, it is sent to the subsequent Attention and Mamba modules for further feature extraction. There is also a simple Reconstructor network for linear reconstruction from extracted features using channel independence and patch partition. Experimental results demonstrate the benefits of the proposed solution against several baselines on multiple datasets."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"(1) It is necessary to elaborate more on the interpretation of the tables and plots. For example, in Table 2 KambaAD performs less well in GECCO dataset under Aff-R. What is the reason behind this phenomenon? What does this indicate? In Table 4, why does the Reconstructor perform better than the integration with GECCO, and why does Encoder perform better with Mulvar? \n\n(2) Section 4.3.5 compares the performance of CI and CD methods. How does the CD method work? It is better to analyze the reason why the CD negatively impacts the overall results.\n\n(3) The experiments mainly compare the accuracy of KambaAD with the baselines. However, the authors did not provide any result on the running speed of KambaAD. Could the authors include a runtime analysis or comparison to evaluate the efficiency of KambaAD？\n\n(4) From Section 1: “Meanwhile, MAMBA focuses on detecting local, context-specific anomalies.” The statement claims that Mamba is mainly used for local anomaly detection, but the experiments did not demonstrate this. In fact, research [2] shows that Mamba is more suitable for long sequence modeling. I would like to know how the authors view this problem. If Mamba and attention have overlapping effects, then considering Mamba's lower complexity, why not just use two Mamba modules instead? It would be more convincing if the author could include an ablation study to demonstrate the superiority of the Mamba-attention design.\n\n[2] Yu, Weihao, and Xinchao Wang. \"MambaOut: Do We Really Need Mamba for Vision?.\"\n\n(5) Similarly, the KAN in MambaAD is designed to capture evident physical anomalies. However, from Eqn. (3) in Section 3.2.1, it is difficult to find out how X_{KAN} can capture such anomalies in the data. The experiments did not demonstrate this as well. Could the author include an example or visualization demonstrating how X_{KAN} captures the anomalies? I also suggest an ablation study comparing the performance with and without KAN to show its effect in MambaAD.\n\n(6) As shown in Table 4, the performance of a single reconstructor is quite good, while its total parameters should be much lower than that of the encoder or/and KambaAD. Can the authors provide additional comparisons under the same model size?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- The main ideas are clearly presented, and the writing is easy to follow.  \n- The model performance is very promising on multiple multivariate datasets.   \n- This paper also provides a series of ablation experiments to test the effect of each design."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The proposal is a fusion of existing ideas, for example, KAN, MAMBA, and attention. Taking channels independently and breaking them into patches are also well-known techniques in the time series domain [1]. While it looks like the integration is executed somehow effectively, the paper itself offers limited innovation.\n\n[1] Nie, Yuqi, et al. \"A time series is worth 64 words: Long-term forecasting with transformers.\"\n\n- The discussion of the experimental results is inadequate, and the ablation experiments are not sufficient to highlight the necessity of the model design."}},"nonreaders":[],"tmdate":1732627227421,"tcdate":1730204700828,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4490/Reviewer_terY"],"signatures":["ICLR.cc/2025/Conference/Submission4490/Reviewer_terY"],"forum":"XmnTfSX5Az","number":1,"license":"CC BY 4.0","cdate":1730204700828,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4490/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732627227421,"domain":"ICLR.cc/2025/Conference","replyto":"XmnTfSX5Az","id":"5H1yRooXVY","forumContent":{"TLDR":{"value":"We introduce KambaAD, a framework combining MAMBA and KAN to enhance time series anomaly detection, outperforming state-of-the-art models across multiple datasets."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Anomaly Detection","Mamba","KAN","Time series"]},"supplementary_material":{"value":"/attachment/a7877f9569c65354bff9efad6c6e06a7477733f9.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Time series anomaly detection is critical in numerous practical applications, yet existing deep learning methods often fall short of real-world demands. These models fail to swiftly filter out physically implausible anomalies, insufficiently address distributional shifts, and lack a comprehensive approach that integrates both global and local perspectives for anomaly detection. Moreover, most successful models rely on channel-dependent methods that tend to treat all features at the same timestamp as a single token and then focus on finding relationships between these tokens. This approach overlooks the unique periodicities, trends, and lagged relationships between different features, leading to suboptimal performance. To address these limitations, we propose KambaAD, a model comprised of an Encoder and Reconstructor. The Encoder integrates the strengths of the Kolmogorov-Arnold Network (KAN), the attention mechanism, and the Selective Structured State Space Model (MAMBA). Specifically, KAN is employed to swiftly enforce data consistency, enabling rapid detection of anomalies that violate physical laws. The attention mechanism ensures balanced processing of global information while enhancing the representation of key data characteristics. We leverage MAMBA's capability as a sequence model to capture anomalies caused by local variations. Additionally, its internal selection mechanism allows the model to effectively handle distribution shifts, ensuring robustness and adaptability in the presence of changing data distributions. Additionally, the framework incorporates a time-series-specific Reconstructor, which reduces computational complexity through patch-based operations that exploit local consistency in time series data. It also employs channel-independent linear reconstruction to prevent interference between different features. Through extensive experiments on multiple multivariate datasets, KambaAD consistently outperforms state-of-the-art models, demonstrating its superior performance in anomaly detection."},"_bibtex":{"value":"@misc{\nchen2025kambaad,\ntitle={Kamba{AD}: Enhancing State Space Models with Kolmogorov{\\textendash}Arnold for time series Anomaly Detection},\nauthor={Yiyun Chen and Jia Guo and Xinchun Yu and Defu Cao and Xiao-Ping Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=XmnTfSX5Az}\n}"},"title":{"value":"KambaAD: Enhancing State Space Models with Kolmogorov–Arnold for time series Anomaly Detection"},"pdf":{"value":"/pdf/91e241263b9d771776b2d96ac63471b17016dc63.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chen|kambaad_enhancing_state_space_models_with_kolmogorovarnold_for_time_series_anomaly_detection"},"authorids":{"value":["~Yiyun_Chen2","~Jia_Guo9","~Xinchun_Yu1","~Defu_Cao1","~Xiao-Ping_Zhang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yiyun Chen","Jia Guo","Xinchun Yu","Defu Cao","Xiao-Ping Zhang"]}},"version":2},{"content":{"venue":{"value":"NeurIPS 2024"},"pdf":{"value":"https://proceedings.neurips.cc/paper_files/paper/2024/file/850e8063d902e0825d3c5504d183bafe-Paper-Conference.pdf"},"venueid":{"value":"dblp.org/conf/NIPS/2024"},"paperhash":{"value":"huang|decision_mamba_reinforcement_learning_via_hybrid_selective_sequence_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Sili_Huang:","https://dblp.org/search/pid/api?q=author:Jifeng_Hu:","https://dblp.org/search/pid/api?q=author:Zhejian_Yang:","https://dblp.org/search/pid/api?q=author:Liwei_Yang:","https://dblp.org/search/pid/api?q=author:Tao_Luo:","~Hechang_Chen2","https://dblp.org/search/pid/api?q=author:Lichao_Sun_0001:","https://dblp.org/search/pid/api?q=author:Bo_Yang_0002:"]},"html":{"value":"http://papers.nips.cc/paper_files/paper/2024/hash/850e8063d902e0825d3c5504d183bafe-Abstract-Conference.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/nips/HuangHYYLC0Y24,\n  author={Sili Huang and Jifeng Hu and Zhejian Yang and Liwei Yang and Tao Luo and Hechang Chen and Lichao Sun and Bo Yang},\n  title={Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling},\n  year={2024},\n  cdate={1704067200000},\n  url={http://papers.nips.cc/paper_files/paper/2024/hash/850e8063d902e0825d3c5504d183bafe-Abstract-Conference.html},\n  booktitle={NeurIPS},\n  crossref={conf/nips/2024}\n}\n"},"abstract":{"value":"Recent works have shown the remarkable superiority of transformer models in reinforcement learning (RL), where the decision-making problem is formulated as sequential generation. Transformer-based agents could emerge with self-improvement in online environments by providing task contexts, such as multiple trajectories, called in-context RL. However, due to the quadratic computation complexity of attention in transformers, current in-context RL methods suffer from huge computational costs as the task horizon increases. In contrast, the Mamba model is renowned for its efficient ability to process long-term dependencies, which provides an opportunity for in-context RL to solve tasks that require long-term memory. To this end, we first implement Decision Mamba (DM) by replacing the backbone of Decision Transformer (DT). Then, we propose a Decision Mamba-Hybrid (DM-H) with the merits of transformers and Mamba in high-quality prediction and long-term memory. Specifically, DM-H first generates high-value sub-goals from long-term memory through the Mamba model. Then, we use sub-goals to prompt the transformer, establishing high-quality predictions. Experimental results demonstrate that DM-H achieves state-of-the-art in long and short-term tasks, such as D4RL, Grid World, and Tmaze benchmarks. Regarding efficiency, the online testing of DM-H in the long-term task is 28$\\times$ times faster than the transformer-based baselines."},"title":{"value":"Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling"},"authors":{"value":["Sili Huang","Jifeng Hu","Zhejian Yang","Liwei Yang","Tao Luo","Hechang Chen","Lichao Sun","Bo Yang"]}},"tmdate":1747577675215,"pdate":1704067200000,"tcdate":1747577672520,"writers":["~"],"signatures":["~Hechang_Chen2"],"forum":"92kDlYiVgI","license":"CC BY-SA 4.0","number":521898,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747577675215,"domain":"DBLP.org","id":"92kDlYiVgI","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.00079v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"huang|decision_mamba_reinforcement_learning_via_hybrid_selective_sequence_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Sili_Huang:","https://dblp.org/search/pid/api?q=author:Jifeng_Hu:","https://dblp.org/search/pid/api?q=author:Zhejian_Yang:","https://dblp.org/search/pid/api?q=author:Liwei_Yang:","https://dblp.org/search/pid/api?q=author:Tao_Luo:","~Hechang_Chen2","https://dblp.org/search/pid/api?q=author:Lichao_Sun:","~Bo_Yang6"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.00079"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-00079,\n  publtype={informal},\n  author={Sili Huang and Jifeng Hu and Zhejian Yang and Liwei Yang and Tao Luo and Hechang Chen and Lichao Sun and Bo Yang},\n  title={Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.00079},\n  url={https://doi.org/10.48550/arXiv.2406.00079}\n}\n"},"abstract":{"value":"Recent works have shown the remarkable superiority of transformer models in reinforcement learning (RL), where the decision-making problem is formulated as sequential generation. Transformer-based agents could emerge with self-improvement in online environments by providing task contexts, such as multiple trajectories, called in-context RL. However, due to the quadratic computation complexity of attention in transformers, current in-context RL methods suffer from huge computational costs as the task horizon increases. In contrast, the Mamba model is renowned for its efficient ability to process long-term dependencies, which provides an opportunity for in-context RL to solve tasks that require long-term memory. To this end, we first implement Decision Mamba (DM) by replacing the backbone of Decision Transformer (DT). Then, we propose a Decision Mamba-Hybrid (DM-H) with the merits of transformers and Mamba in high-quality prediction and long-term memory. Specifically, DM-H first generates high-value sub-goals from long-term memory through the Mamba model. Then, we use sub-goals to prompt the transformer, establishing high-quality predictions. Experimental results demonstrate that DM-H achieves state-of-the-art in long and short-term tasks, such as D4RL, Grid World, and Tmaze benchmarks. Regarding efficiency, the online testing of DM-H in the long-term task is 28$\\times$ times faster than the transformer-based baselines."},"title":{"value":"Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling"},"authors":{"value":["Sili Huang","Jifeng Hu","Zhejian Yang","Liwei Yang","Tao Luo","Hechang Chen","Lichao Sun","Bo Yang"]}},"tmdate":1747577672824,"pdate":1704067200000,"tcdate":1722968225887,"writers":["~"],"signatures":["~Bo_Yang6"],"forum":"BGHl33WmIj","license":"CC BY-SA 4.0","number":57710,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747577672824,"domain":"DBLP.org","id":"BGHl33WmIj","version":2},{"content":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["In-context Reinforcement Learning","Mamba"]},"primary_area":{"value":"reinforcement_learning"},"abstract":{"value":"Recent works have shown the remarkable superiority of transformer models in reinforcement learning (RL), where the decision-making problem is formulated as sequential generation. Transformer-based agents could emerge with self-improvement in online environments by providing task contexts, such as multiple trajectories, called in-context RL. However, due to the quadratic computation complexity of attention in transformers, current in-context RL methods suffer from huge computational costs as the task horizon increases. In contrast, the Mamba model is renowned for its efficient ability to process long-term dependencies, which provides an opportunity for in-context RL to solve tasks that require long-term memory. To this end, we first implement Decision Mamba (DM) by replacing the backbone of Decision Transformer (DT). Then, we propose a Decision Mamba-Hybrid (DM-H) with the merits of transformers and Mamba in high-quality prediction and long-term memory. Specifically, DM-H first generates high-value sub-goals from long-term memory through the Mamba model. Then, we use sub-goals to prompt the transformer, establishing high-quality predictions. Experimental results demonstrate that DM-H achieves state-of-the-art in long and short-term tasks, such as D4RL, Grid World, and Tmaze benchmarks. Regarding efficiency, the online testing of DM-H in the long-term task is 28$\\times$ times faster than the transformer-based baselines."},"_bibtex":{"value":"@inproceedings{\nhuang2024decision,\ntitle={Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling},\nauthor={Sili Huang and Jifeng Hu and Zhejian Yang and Liwei Yang and Tao Luo and Hechang Chen and Lichao Sun and Bo Yang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=wFzIMbTsY7}\n}"},"title":{"value":"Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling"},"pdf":{"value":"/pdf/f79cbc369ef6968176c7cc958c79839cb99e59b0.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"huang|decision_mamba_reinforcement_learning_via_hybrid_selective_sequence_modeling"},"authorids":{"value":["~Sili_Huang1","~Jifeng_Hu1","~Zhejian_Yang1","~Liwei_Yang2","~Tao_Luo2","~Hechang_Chen2","~Lichao_Sun1","~Bo_Yang6"]},"authors":{"value":["Sili Huang","Jifeng Hu","Zhejian Yang","Liwei Yang","Tao Luo","Hechang Chen","Lichao Sun","Bo Yang"]}},"tmdate":1730873888521,"pdate":1727287796045,"tcdate":1715571097377,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission5823/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission5823/Authors"],"forum":"wFzIMbTsY7","license":"CC BY 4.0","number":5823,"cdate":1715571097377,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/-/Submission","NeurIPS.cc/2024/Conference/-/Edit","NeurIPS.cc/2024/Conference/-/Post_Submission","NeurIPS.cc/2024/Conference/Submission5823/-/Revision","NeurIPS.cc/2024/Conference/Submission5823/-/Camera_Ready_Revision"],"mdate":1730873888521,"odate":1730873888510,"domain":"NeurIPS.cc/2024/Conference","id":"wFzIMbTsY7","version":2},{"content":{"summary":{"value":"1. Addressing the main issue in the existing model:\n   Mamba, despite its success in long sequence tasks, faces mixed opinions and challenges in visual tasks due to inherent flaws and suboptimal performance. Understanding these flaws and optimizing Mamba's performance in the visual domain are critical research questions.\n\n2. Proposed solution for the issue:\n   The paper proposes Vision Mamba Mender, a systematic approach to enhance Mamba's performance in visual recognition tasks. This approach involves predictive correlation analysis of Mamba's hidden states, both internally and externally, to identify and address flaws. Tailored repair methods are then applied to optimize model performance.\n\n3. How did the author evaluate the proposed solution on various standard dataset:\n   The efficacy of Vision Mamba Mender is validated through extensive experiments on prevalent Mamba architectures. These experiments demonstrate significant improvements in model performance across various standard datasets, showcasing the practical impact and effectiveness of the proposed methods. The algorithm code is also provided for transparency and reproducibility."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please refer to the Weaknesses block"},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Innovative Post-Hoc Optimization: Vision Mamba Mender introduces a novel approach to optimize Mamba models post-training, focusing on identifying and rectifying operational flaws rather than predefining architectures. This method is innovative in its systematic approach to improving model performance.\n\n2. Comprehensive State Analysis: The paper introduces a detailed state correlation analysis method that evaluates Mamba's hidden states from both external and internal perspectives. This comprehensive analysis helps in pinpointing specific flaws that affect prediction accuracy in visual recognition tasks.\n\n3. Tailored Repair Methods: Tailored repair methods are proposed for both external and internal state flaws identified through the analysis. By imposing constraints on state correlations within Mamba modules, these methods effectively enhance the model's ability to make accurate predictions.\n\n4. Applicability to Existing Architectures: Vision Mamba Mender is designed to be applicable across various state-of-the-art Vision Mamba architectures. This versatility ensures that the optimization approach can be seamlessly integrated into different implementations without significant modifications.\n\n5. Experimental Validation: Extensive experiments validate the effectiveness of Vision Mamba Mender across different benchmarks and datasets. The approach consistently demonstrates improvements in model accuracy, showcasing its practical utility in real-world applications.\n\n6. Transparency and Reproducibility: The availability of algorithm code in the supplementary material and commitment to making it publicly accessible enhance the transparency and reproducibility of the research findings. This openness facilitates further validation and adoption of the proposed methods by the research community."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Limitation in Methodology: The paper mentions the introduction of state correlation analysis and repair methods for Mamba, but it lacks clarity on the specific algorithms or mathematical formulations used. This could hinder reproducibility and transparency, as other researchers may find it challenging to implement or verify the proposed methods without detailed methodology descriptions.\n\n2. Assumptions on Flaws and Repair Methods: The paper suggests that Vision Mamba Mender identifies and rectifies anomalies in Mamba's mechanisms, focusing on external and internal state correlations. However, without empirical evidence or case studies illustrating the nature of these flaws across different datasets and scenarios, the validity and generality of these assumptions remain unclear."},"limitations":{"value":"Please refer to the Weaknesses block"}},"nonreaders":[],"tmdate":1730879147310,"tcdate":1720901718465,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission7275/Reviewer_4vzy"],"signatures":["NeurIPS.cc/2024/Conference/Submission7275/Reviewer_4vzy"],"forum":"9VnevS2YoR","number":4,"license":"CC BY 4.0","cdate":1720901718465,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission7275/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879147310,"domain":"NeurIPS.cc/2024/Conference","replyto":"9VnevS2YoR","id":"BxNLVX1qo5","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","State Space Models","Computer Vision"]},"supplementary_material":{"value":"/attachment/231f16a7aedf14ace21ad17af99f0f84a5a06ee6.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Mamba, a state-space model with selective mechanisms and hardware-aware architecture, has demonstrated outstanding performance in long sequence modeling tasks, particularly garnering widespread exploration and application in the field of computer vision. While existing works have mixed opinions of its application in visual tasks, the exploration of its internal workings and the optimization of its performance remain urgent and worthy research questions given its status as a novel model. Existing optimizations of the Mamba model, especially when applied in the visual domain, have primarily relied on predefined methods such as improving scanning mechanisms or integrating other architectures, often requiring strong priors and extensive trial and error. In contrast to these approaches, this paper proposes the Vision Mamba Mender, a systematic approach for understanding the workings of Mamba, identifying flaws within, and subsequently optimizing model performance. Specifically, we present methods for predictive correlation analysis of Mamba's hidden states from both internal and external perspectives, along with corresponding definitions of correlation scores, aimed at understanding the workings of Mamba in visual recognition tasks and identifying flaws therein. Additionally, tailored repair methods are proposed for identified external and internal state flaws to eliminate them and optimize model performance. Extensive experiments validate the efficacy of the proposed methods on prevalent Mamba architectures, significantly enhancing Mamba's performance."},"_bibtex":{"value":"@inproceedings{\nhu2024vision,\ntitle={Vision Mamba Mender},\nauthor={Jiacong Hu and Anda Cao and Zunlei Feng and Shengxuming Zhang and Yi Wang and Lingxiang Jia and Mingli Song},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=9VnevS2YoR}\n}"},"title":{"value":"Vision Mamba Mender"},"pdf":{"value":"/pdf/8a5b7efdabc5e5544e1b8f3b86095c84a05b91e1.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"hu|vision_mamba_mender"},"authorids":{"value":["~Jiacong_Hu1","~Anda_Cao1","~Zunlei_Feng1","~Shengxuming_Zhang1","~Yi_Wang56","~Lingxiang_Jia1","~Mingli_Song1"]},"authors":{"value":["Jiacong Hu","Anda Cao","Zunlei Feng","Shengxuming Zhang","Yi Wang","Lingxiang Jia","Mingli Song"]}},"version":2},{"content":{"summary":{"value":"The paper presents SegMaST, a spatio-temporal framework based on the Mamba state-space model for longitudinal medical image analysis. The model is designed to jointly segment baseline images and localize progression (new or enlarging lesions) in follow-up scans. To address the prevalence of stable cases in clinical data, the authors introduce an imbalance-aware loss accumulation strategy. Evaluation is conducted on Multiple Sclerosis (MS) and diffuse glioma datasets, demonstrating improvements in lesion detection and segmentation consistency over standard CNN and attention-based baselines."},"justification_of_final_rating":{"value":"While the paper still presents considerable room for improvement, I strongly encourage the authors to genuinely pursue the directions outlined in their future work. However, given the authors' honest acknowledgment of the current limitations and their conscientious response during the rebuttal, I am willing to increase my rating."},"confidence":{"value":5},"final_rating":{"value":4},"justification_of_the_preliminary_rating":{"value":"While the problem is important and timely, the current task formulation appears somewhat idealized relative to the complexities of real-world oncology and neurology. Consequently, the technical novelty also leaves substantial room for improvement."},"confidentiality_llm_acknowledgment":{"value":"Yes"},"strengths":{"value":"1. Investigating longitudinal disease progression is fundamental for quantifying treatment efficacy and disease evolution, yet it remains significantly more challenging than cross-sectional analysis.\n2. The use of a Mamba-based selective state-space model (SSM) addresses the quadratic complexity  of standard attention, allowing for global receptive fields with linear complexity.\n3. The introduction of a filtering mechanism to balance \"progression\" and \"no-progression\" samples in training batches is a well-motivated response to the skewed distribution of real-world clinical datasets."},"weaknesses":{"value":"1. While the joint modeling of two timepoints is technically sound, it relies on a specific \"paired input\" requirement that can be difficult to satisfy in standard clinical practice. In many settings, perfectly registered, high-quality longitudinal pairs are rare or difficult to obtain due to varying acquisition protocols and patient follow-up inconsistency. The model's utility may be limited by this high bar for input data.\n2. In the UCSF-ALTPD glioma dataset, progression is not merely a biological spatio-temporal process; it is heavily influenced by clinical interventions such as surgery (resection cavities), radiation, and chemotherapy. The current task setting treats these changes as purely visual cues. Similar to the BraTS challenge, which distinguishes between pre-treatment and post-treatment segmentation tasks, the authors should have addressed how treatment effects confound the detection of \"true\" disease progression.\n3. Disease progression is a multi-factor phenomenon. The current model relies solely on image features , ignoring demographic data (age, gender) or treatment status. Without these non-imaging covariates, the model lacks the necessary context to reliably distinguish between aggressive progression and treatment-induced pseudoprogression, a critical distinction in neurological care.\n4. The technical novelty of this work appears limited. Although the authors incorporate the Mamba architecture and an imbalance-aware loss accumulation strategy, these changes remain largely incremental. Moreover, recent studies have leveraged VLM–based or diffusion-based segmentation paradigms and have reported more substantial gains in segmentation performance."},"detailed_comments":{"value":"See Weaknesses."},"questions_to_address_in_the_rebuttal":{"value":"1. Regarding the UCSF-ALTPD dataset, did the authors consider the presence of resection cavities or post-surgical changes as a specific class or confounding variable?\n2. Could the authors elaborate on how the model could incorporate clinical metadata (e.g., time since last treatment) to improve progression detection?\n3. How does the performance compare if the model is extended to handle more than two timepoints, which is common in chronic MS monitoring?"},"preliminary_rating":{"value":4}},"parentInvitations":"MIDL.io/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1771530033169,"tcdate":1767926750734,"writers":["MIDL.io/2026/Conference","MIDL.io/2026/Conference/Submission117/Reviewer_CpuJ"],"signatures":["MIDL.io/2026/Conference/Submission117/Reviewer_CpuJ"],"forum":"aJU5lkPRao","number":2,"license":"CC BY 4.0","cdate":1767926750734,"readers":["everyone"],"invitations":["MIDL.io/2026/Conference/Submission117/-/Official_Review","MIDL.io/2026/Conference/-/Edit"],"mdate":1771530033169,"domain":"MIDL.io/2026/Conference","replyto":"aJU5lkPRao","id":"dAp6tB2wEJ","forumContent":{"TLDR":{"value":"SegMaST is a Mamba-based spatio-temporal model that jointly segments baseline and new lesions while handling real-world progression imbalance."},"venue":{"value":"MIDL 2026 Poster"},"midl_latex_submission_checklist":{"value":["The paper compiles correctly using the pdflatex compiler.","Created a single midl26_NNN.zip file with midl26_NNN.tex, midl26_NNN.bib, all necessary figures and files.","The LaTeX file includes the correct header commands before \\title.","Hyperref is preloaded; do not modify it.","The times package is not used.","All co-authors are listed with correct firstname, lastname, and any suffix/prefix.","Math in the title and abstract uses valid LaTeX.","References are provided through the .bib file only.","Tables and figures stay within the page margins.","The zip archive contains all necessary figures and no unused files.","Special formatting from rebuttal has been removed.","All special characters use LaTeX commands.","Appendices and supplementary material are included in the same PDF after references.","The main paper does not exceed 12 pages."]},"keywords":{"value":["Longitudinal Segmentation","Spatio-Temporal Attention","Mamba","Imbalance Aware Learning"]},"read_cfp_and_author_instructions":{"value":"Yes"},"originality_policy":{"value":"Yes"},"abstract":{"value":"Longitudinal medical image segmentation is fundamental for quantifying disease progression and evaluating treatment efficacy. However, two critical challenges persist: First, methods that jointly segment baseline and follow-up images remain underexplored, often missing the contextual benefits of simultaneous assessment and lacking longitudinal consistency. Second, real-world datasets typically exhibit severe class imbalance between stable and progressive scans — an issue frequently neglected by existing models. To address these limitations, we propose SegMaST, a novel Mamba-based spatio-temporal framework. Unlike conventional approaches that treat timepoints in isolation, SegMaST leverages cross-temporal information and spatial correspondences to jointly segment the initial baseline mask and explicitly localize new pathologies in follow-up scans. Additionally, we introduce an imbalance-aware loss accumulation strategy to enhance robustness in realistic clinical settings. On longitudinal Multiple Sclerosis and Glioma cohorts, SegMaST outperforms established CNN- and attention-based baselines for follow-up segmentation (mean follow-up Dice MS in-house $0.536$, MSSEG-2 $0.620$, and Glioma $0.631$) and lesion detection (F1 in-house $0.688$, MSSEG-2 $0.723$), while maintaining state-of-the-art accuracy in baseline segmentation (Dice: $0.617$ MS, $0.844$ Glioma)."},"_bibtex":{"value":"@inproceedings{\nvarma2026segmast,\ntitle={SegMa{ST}: Mamba-based Spatio-Temporal Modeling to Improve Longitudinal Detection and Segmentation},\nauthor={Aswathi Varma and Jonas Weidner and Laurin Lux and Cosmin I. Bercea and Mark M{\\\"u}hlau and Jan Kirschke and Benedikt Wiestler and Daniel Rueckert},\nbooktitle={Medical Imaging with Deep Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=aJU5lkPRao}\n}"},"title":{"value":"SegMaST: Mamba-based Spatio-Temporal Modeling to Improve Longitudinal Detection and Segmentation"},"latex_code":{"value":"/attachment/5d283c2218c40181460bb009ef9e77537a0f54a3.zip"},"secondary_subject_area":{"value":"Application: Neuroimaging"},"pdf":{"value":"/pdf/7c5e1aaad8e5ab5e2b923fd5fbe31c182485e6e1.pdf"},"copyright_form":{"value":"/attachment/53abf4ef5f566104f288bd3ae5145783b5fb9571.pdf"},"visa":{"value":"Yes"},"single_blind_notice":{"value":"Yes"},"venueid":{"value":"MIDL.io/2026/Conference"},"paperhash":{"value":"varma|segmast_mambabased_spatiotemporal_modeling_to_improve_longitudinal_detection_and_segmentation"},"primary_subject_area":{"value":"Segmentation"},"authorids":{"value":["~Aswathi_Varma1","~Jonas_Weidner1","~Laurin_Lux1","~Cosmin_I._Bercea1","~Mark_Mühlau1","~Jan_Kirschke1","~Benedikt_Wiestler1","~Daniel_Rueckert2"]},"registration":{"value":"Yes"},"authors":{"value":["Aswathi Varma","Jonas Weidner","Laurin Lux","Cosmin I. Bercea","Mark Mühlau","Jan Kirschke","Benedikt Wiestler","Daniel Rueckert"]},"llm_policy_acknowledgment":{"value":"Yes"}},"version":2},{"content":{"summary":{"value":"This paper attempts to extend the orthogonal basis projection in the standard SSM to a multivariate orthogonal basis projection and proposes a variant called Poly-mamba, analyzing its practical advantages through various experiments. However, the paper has significant shortcomings in the methodology, evaluation, and writing."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"* **Q1**: In the MOPA module, the author simplifies operations using dot product. How can one ensure that the complete multivariate dependency relationships can still be learned?\n\n* **Q2:**: Upon reviewing the code, I discovered that Poly-Mamba does not utilize parallel scanning for accelerated computation but instead computes in a recursive manner. This discovery raises doubts regarding the efficiency analysis presented in Table 3. Why is Poly-Mamba faster than S-Mamba if it does not employ parallel scanning for computation?\n\n* **Q3:**: Poly-mamba still utilizes the standard SSM A matrix, which essentially is a measure kernel function with Laplacian decay. How can one ensure that in this scenario, the dot product operations in MOPA can still model the relationships between multivariate polynomial coefficients effectively?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":1},"strengths":{"value":"The motivation behind the proposed method is meaningful, as the standard SSM is limited to the projection and reconstruction of univariate functions and lacks the ability to model relationships between multivariate time series. Extending the SSM to a multivariate orthogonal polynomial projection provides an **elegant solution**."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* **W1**: Firstly, the writing of this article needs improvement. The overall impression of this article is rather mediocre, with numerous lines consisting of only one or two words and substantial white spaces. The model diagram and experimental figures are also quite rudimentary. Furthermore, I recommend relocating large tables such as Table 1 & 2 to the appendix, including only summarized results in the main body of the text.\n\n* **W2**: Attention should be paid to certain details. For instance, in line 123, the S4 model utilized a diagonal plus low-rank form of the A matrix, transitioning to using a diagonal A matrix starting from DSS[1] and S4D[2].\n\n* **W3**: This article only conducted standard experiments in the field of time series forecasting, lacking further experimental analysis. For example, (a) performing other time series analysis tasks to explore the potential of Poly-Mamba as a foundational model for time series analysis; (b) conducting experiments with different input lengths to verify that Poly-Mamba still retains its ability to model long-range dependencies. I believe that given the current stage of development in the field of time series analysis, these are essential considerations.\n\n* **W4**: Recent work[3,4,5] has shown that the components of Mamba are detrimental to time series forecasting tasks. The authors should consider improving from the SSM core rather than Mamba.\n\n* **W5**: The performance improvement of Poly-Mamba is marginal, significantly lagging behind existing variants [6,7,8,9] of Mamba for time series forecasting. Despite the paper's claim of modeling inter-variable relationships through multivariate polynomial projection, its effectiveness is even inferior to simply using the SSM core along with an additional channel attention mechanism [10].\n\n* **W6**: The premise of this article is commendable, highlighting the inadequacy of unit polynomial approximations in handling relationships within multivariate time series. However, I believe that the method proposed by the authors does not effectively address this issue: Most modules are designed based on empirical guidance, lacking corresponding theoretical foundations (the theoretical analysis lacks rigorous mathematical proofs and resembles more of a storytelling approach), which is my primary concern regarding this work. For more details, please refer to the **Questions** section.\n\n* **W7**: I think this article should be written following the approach of Hippo, starting from specific orthogonal polynomial families such as Legendre, Chebyshev, Laguerre, etc., to derive the matrix representation of A in the multivariate case, combined with multi-input multi-output SSM systems. Additionally, the article structure could be organized by referencing Koopa[11] and examining the components of time-varying and time-invariant SSMs.\n\n[1] Gupta A, Gu A, Berant J. Diagonal state spaces are as effective as structured state spaces[J]. Advances in Neural Information Processing Systems, 2022, 35: 22982-22994.\n\n[2] Gu A, Goel K, Gupta A, et al. On the parameterization and initialization of diagonal state space models[J]. Advances in Neural Information Processing Systems, 2022, 35: 35971-35983.\n\n[3] Wang S, Li B Z, Khabsa M, et al. Linformer: Self-attention with linear complexity[J]. arXiv preprint arXiv:2006.04768, 2020.\n\n[4] Hou H, Yu F R. Rwkv-ts: Beyond traditional recurrent neural network for time series tasks[J]. arXiv preprint arXiv:2401.09093, 2024.\n\n[5] Sun Y, Dong L, Huang S, et al. Retentive network: A successor to transformer for large language models (2023)[J]. URL http://arxiv. org/abs/2307.08621 v1, 15.\n\n[6] Patro B N, Agneeswaran V S. Simba: Simplified mamba-based architecture for vision and multivariate time series[J]. arXiv preprint arXiv:2403.15360, 2024.\n\n[7] Ahamed M A, Cheng Q. Timemachine: A time series is worth 4 mambas for long-term forecasting[J]. arXiv preprint arXiv:2403.09898, 2024.\n\n[8 ]Zeng C, Liu Z, Zheng G, et al. C-Mamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting[J]. arXiv preprint arXiv:2406.05316, 2024.\n\n[9] Behrouz A, Santacatterina M, Zabih R. Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models[J]. arXiv preprint arXiv:2406.04320, 2024.\n\n[10] Hu J, Lan D, Zhou Z, et al. Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting[J]. arXiv preprint arXiv:2405.16312, 2024.\n\n[11] Liu Y, Li C, Wang J, et al. Koopa: Learning non-stationary time series dynamics with koopman predictors[J]. Advances in Neural Information Processing Systems, 2024, 36."}},"nonreaders":[],"tmdate":1731428117367,"tcdate":1730279402030,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9204/Reviewer_xKbE"],"signatures":["ICLR.cc/2025/Conference/Submission9204/Reviewer_xKbE"],"forum":"nclyFUZpX9","number":2,"license":"CC BY 4.0","cdate":1730279402030,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9204/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428117367,"domain":"ICLR.cc/2025/Conference","replyto":"nclyFUZpX9","id":"xYVkD1hPpc","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forecasting；Deep Learning； State Space Model"]},"supplementary_material":{"value":"/attachment/f064e3aed57d3d1812e4a9ee5e1f4ea5b44219ea.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"For multivariate time series (MTS) tasks, previous state space models (SSMs) followed the modeling paradigm of Transformer-based methods. However, none of them explicitly model the complex dependencies of MTS: the Channel Dependency variations with Time (CDT). In view of this, we delve into the derivation of SSM, which involves approximating continuously updated functions by orthogonal function basis. We then develop Poly-Mamba, a novel method for MTS forecasting. Its core concept is to expand the original orthogonal function basis space into a multivariate orthogonal function space containing variable mixing terms, and make a projection on this space so as to explicitly describe the CDT by weighted coefficients. In Poly-Mamba, we propose the Multivariate Orthogonal Polynomial Approximation (MOPA) as a simplified implementation of this concept. For the simple linear relationship between channels, we propose Linear Channel Mixing (LCM) and generate CDT patterns adaptively for different channels through a proposed Order Combining method. Experiments on six real-world datasets demonstrate that Poly-Mamba outperforms the SOTA methods, especially when dealing with datasets having a large number of channels and complex correlations. The codes and log files are in the supplementary."},"_bibtex":{"value":"@misc{\nwu2025a,\ntitle={A {SSM} is Polymerized from Multivariate Time Series},\nauthor={Haixiang Wu},\nyear={2025},\nurl={https://openreview.net/forum?id=nclyFUZpX9}\n}"},"title":{"value":"A SSM is Polymerized from Multivariate Time Series"},"pdf":{"value":"/pdf/56e1fd4acec2afe967a5ef8777c48a18b660772d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"wu|a_ssm_is_polymerized_from_multivariate_time_series"},"authorids":{"value":["~Haixiang_Wu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Haixiang Wu"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Mamba-3, a followup sequence model architecture that builds upon the Mamba family of SSMs. The work is motivated by an \"inference-first\" perspective, aiming to create a model that excels across three axes: quality, capability, and inference efficiency, addressing limitations in prior sub-quadratic models like Mamba-2.  Basically, this paper propose three core, major improvements derived from classical SSM theory and also validate Mamba-3 through extensive experiments, showing that it outperforms strong baselines in language modeling, possesses new state-tracking capabilities on synthetic tasks, and establishes a new Pareto-frontier for performance versus inference cost.   It presents a clear, significant, and well-executed advancement in the design of efficient sequence models. The paper is strong for three main reasons:\n\n1.  **Principled Improvements over existing SSMs/Mamba family:** The core contributions are not arbitrary architectural tweaks but are grounded in control theory and SSM principles. The connection drawn between complex SSMs and data-dependent RoPE is very elegant IMO and provides a clear theoretical justification for the observed gains in capability.\n\n2.  **Holistic Improvement:** The work addresses the multi-faceted challenge of building a practical LLM. It simultaneously pushes forward model quality, unlocks new fundamental capabilities (state-tracking issue which has been rooted in RNNs), and improves hardware-level efficiency(MIMO formulation). This is a rare and impactful combination. (Though it will be more impactful if all artifacts will be open-sourced. \n\n3.  **Compelling and Rigorous Empirical Evidence:** The experimental validation is thorough and persuasive. The head-to-head comparisons on language modeling, the stark success/failure results on formal language tasks, and the Pareto-frontier analysis collectively provide a powerful and convincing case for the superiority of the Mamba-3 design."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- The results on retrieval (Table 2) are improved but still trail the Transformer baseline significantly. Do you view this as a fundamental ceiling for fixed-state models, or do you see pathways for future SSM-based architectures to further close this gap? Maybe authors can add some potential exploration about hybrid models? I think we are at the age that Hybrid models are becoming more important and can help alleviate the shortages of RNNs which we should acknowledge and make good use of.\n\n- The \"RoPE trick\" is a very efficient way to implement the complex SSM. Could you briefly comment on how the cumulative product of rotation matrices `(Π R_i)` is handled efficiently during both the parallel training scan and the sequential inference steps?\n\nAdditional questions:\n\n- The MIMO formulation is a compelling idea for improving hardware efficiency. Could you elaborate on the selection of the MIMO rank `r=4`? Have you explored the sensitivity of model performance and inference latency to different values of `r`? Is there a risk of diminishing returns or even performance degradation if `r` becomes too large?\n\n- The ablation in Table 4a convincingly shows that the trapezoidal discretization and BC bias make the short convolution redundant. Can you provide more intuition on *why* this is the case? Does the learned size-two convolution from the trapezoidal rule effectively capture the same local information as a dedicated convolution layer?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"- **Theoretical Elegance and Insight:** The paper's standout strength is its use of classical SSM theory to motivate architectural changes. The derivation of a data-dependent RoPE from a complex state-space is a beautiful and insightful result that bridges two important lines of work.\n\n- **Targeted Problem Solving:** Each of the three methodological changes directly targets a known weakness in prior models. Trapezoidal discretization for expressivity, complexification for state-tracking, and MIMO for hardware utilization. This focused approach makes the paper's narrative clear and its contributions easy to appreciate.\n\n- **Excellent Empirical Validation:** The experiments are top-notch. The Pareto-frontier plot (Figure 3) is a highly effective way to visualize the trade-off between performance and inference cost, clearly showing Mamba-3's dominance. The ablation on state-tracking capabilities (Table 4b) is a decisive demonstration of the complex SSM's impact.\n\n*   **Practicality and Inference-First Focus:** The introduction of the MIMO variant shows a deep understanding of the practical bottlenecks in LLM deployment. By focusing on arithmetic intensity, the authors address a subtle but critical aspect of real-world performance that goes beyond theoretical FLOPs.\n\n-  **High-Quality Presentation:** The paper is extremely well-written, with complex ideas explained clearly and supported by well-designed figures and tables."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"My major concern is that retrieval Capabilities Still Lag Transformers: While the paper is honest about this, and Mamba-3 shows improvement over Mamba-2, the results in Table 2 confirm that a fundamental gap in retrieval performance versus Transformer models remains. This is an inherent challenge for fixed-state recurrent models and represents a key limitation. Maybe authors can add some potential exploration about hybrid models? I think we are at the age that Hybrid models are becoming more important and can help alleviate the shortages of RNNs which we should acknowledge and make good use of."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924146534,"tcdate":1762144851920,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13549/Reviewer_kFu5"],"signatures":["ICLR.cc/2026/Conference/Submission13549/Reviewer_kFu5"],"forum":"HwCvaJOiCj","number":4,"license":"CC BY 4.0","cdate":1762144851920,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13549/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924146534,"domain":"ICLR.cc/2026/Conference","replyto":"HwCvaJOiCj","id":"BFJlpId6sY","forumContent":{"venue":{"value":"ICLR 2026 Oral"},"TLDR":{"value":"Mamba-3, an inference-first SSM that pushes on core SSM principles: improved discretization for better quality, complex dynamics for new capabilities, and MIMO updates for efficient inference."},"keywords":{"value":["State Space Models","Mamba","LLMs","Subquadratic Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While current Transformer models deliver strong quality, their quadratic compute and linear memory requirements make inference expensive. This has spurred the development of sub-quadratic models with reduced compute and constant memory requirements.\nHowever, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice.\nGuided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule enabling richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation that improves model performance without increasing decode latency.\nTogether with architectural refinements, Mamba-3 achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with the MIMO variant further improving accuracy by an additional 1.2 points, for a total gain of 1.8 points.\nAcross state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half the state size. These results demonstrate that Mamba-3 advances the performance–efficiency frontier."},"_bibtex":{"value":"@inproceedings{\nlahoti2026mamba,\ntitle={Mamba-3: Improved Sequence Modeling using State Space Principles},\nauthor={Aakash Lahoti and Kevin Li and Berlin Chen and Caitlin Wang and Aviv Bick and J Zico Kolter and Tri Dao and Albert Gu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=HwCvaJOiCj}\n}"},"title":{"value":"Mamba-3: Improved Sequence Modeling using State Space Principles"},"pdf":{"value":"/pdf/09c67b76b7d1ad254ea30eeba839b8592de9bd78.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lahoti|mamba3_improved_sequence_modeling_using_state_space_principles"},"authorids":{"value":["~Aakash_Lahoti1","~Kevin_Li3","~Berlin_Chen1","~Caitlin_Wang1","~Aviv_Bick1","~J_Zico_Kolter1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Aakash Lahoti","Kevin Li","Berlin Chen","Caitlin Wang","Aviv Bick","J Zico Kolter","Tri Dao","Albert Gu"]}},"version":2},{"content":{"research_area_keywords":{"value":"text-to-text generation, model architectures, efficient models, inference methods"},"languages_studied":{"value":"english"},"venue":{"value":"ACL ARR 2024 December Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025remamba,\ntitle={ReMamba: Equip Mamba with Effective Long-Sequence Modeling},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - December 2024},\nyear={2025},\nurl={https://openreview.net/forum?id=RMjyNzYv2K},\nnote={under review}\n}"},"title":{"value":"ReMamba: Equip Mamba with Effective Long-Sequence Modeling"},"contribution_types":{"value":["NLP engineering experiment","Approaches low compute settings-efficiency"]},"abstract":{"value":"While the Mamba architecture demonstrates superior inference efficiency and competitive performance on short-context natural language processing (NLP) tasks, empirical evidence suggests its capacity to comprehend long contexts is limited compared to transformer-based models. In this study, we investigate the long-context efficiency issues of the Mamba models and propose ReMamba, which enhances Mamba's ability to comprehend long contexts. ReMamba incorporates selective compression and adaptation techniques within a two-stage re-forward process, incurring minimal additional inference costs overhead. Experimental results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy, improving over the baselines by 3.2 and 1.6 points, respectively, and attaining performance almost on par with same-size transformer models."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/97e46c3608b5c2f8d562cea7f0ee264954c178a5.pdf"},"research_area":{"value":"Generation"},"venueid":{"value":"aclweb.org/ACL/ARR/2024/December/Submission"}},"tmdate":1782962106635,"tcdate":1734268240814,"writers":["aclweb.org/ACL/ARR/2024/December","aclweb.org/ACL/ARR/2024/December/Submission849/Authors"],"signatures":["aclweb.org/ACL/ARR/2024/December/Submission849/Authors"],"forum":"RMjyNzYv2K","license":"CC BY 4.0","number":849,"cdate":1734268240814,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2024/December/-/Submission","aclweb.org/ACL/ARR/2024/December/-/Edit","aclweb.org/ACL/ARR/2024/December/-/Post_Submission","aclweb.org/ACL/ARR/2024/December/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2024/December/-/Preprint_Post_Submission"],"mdate":1782962106635,"odate":1738780930893,"domain":"aclweb.org/ACL/ARR/2024/December","id":"RMjyNzYv2K","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.15496v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"yuan|remamba_equip_mamba_with_effective_longsequence_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Danlong_Yuan:","https://dblp.org/search/pid/api?q=author:Jiahao_Liu:","https://dblp.org/search/pid/api?q=author:Bei_Li:","https://dblp.org/search/pid/api?q=author:Huishuai_Zhang:","https://dblp.org/search/pid/api?q=author:Jingang_Wang:","~Xunliang_Cai1","~Dongyan_Zhao1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.15496"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-15496,\n  publtype={informal},\n  author={Danlong Yuan and Jiahao Liu and Bei Li and Huishuai Zhang and Jingang Wang and Xunliang Cai and Dongyan Zhao},\n  title={ReMamba: Equip Mamba with Effective Long-Sequence Modeling},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.15496},\n  url={https://doi.org/10.48550/arXiv.2408.15496}\n}\n"},"abstract":{"value":"While the Mamba architecture demonstrates superior inference efficiency and competitive performance on short-context natural language processing (NLP) tasks, empirical evidence suggests its capacity to comprehend long contexts is limited compared to transformer-based models. In this study, we investigate the long-context efficiency issues of the Mamba models and propose ReMamba, which enhances Mamba's ability to comprehend long contexts. ReMamba incorporates selective compression and adaptation techniques within a two-stage re-forward process, incurring minimal additional inference costs overhead. Experimental results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy, improving over the baselines by 3.2 and 1.6 points, respectively, and attaining performance almost on par with same-size transformer models."},"title":{"value":"ReMamba: Equip Mamba with Effective Long-Sequence Modeling"},"authors":{"value":["Danlong Yuan","Jiahao Liu","Bei Li","Huishuai Zhang","Jingang Wang","Xunliang Cai","Dongyan Zhao"]}},"tmdate":1768916568223,"pdate":1704067200000,"tcdate":1731927469809,"writers":["~"],"signatures":["~Xunliang_Cai1"],"forum":"GeWZa7TsnQ","license":"CC BY-SA 4.0","number":249924,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1768916568223,"domain":"DBLP.org","id":"GeWZa7TsnQ","version":2},{"content":{"summary":{"value":"This paper proposes a method to address the memory bottleneck in long video sequence processing by applying the TTT concept to video streams. As a result, the network shows adaptive performance improvement during test time even with longer frame sequences, and is capable of processing videos of over 10K frames within a fixed memory budget. The method also features a Hessian-Free optimizer that enables efficient updates during testing."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please refer the weaknesses part."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The model can handle long video sequences about 10K frames, mitigating the scalability limitation in long-term sequence modeling.\n- The application of TTT to the video streaming environment is interesting, as it enables adaptive behavior at test time."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The paper claims to preserve information via TTTHF instead of hard drop, but in practice, a cosine similarity based token discard step still exists. Because of this, the approach cannot be considered entirely different from traditional token-dropping methods, and the empirical evidence for actual information preservation remains unclear.\n- Training is limited to a maximum of 1024 frames, and there is no quantitative analysis of memory efficiency (frame count vs memory usage) during training. The memory aspect thus remains insufficiently analyzed.\n- The TTT concept has already been explored in prior video research (Test-Time Training on Video Streams), and similar adaptive mechanisms exist in RNN and Mamba architectures through state updates. It should be clarified whether the proposed approach's distinction is limited to the HF-based second-order optimization, or if there are structural differences beyond that.\n- Experimental comparison with existing sequence compression approaches is lacking. For example, methods such as VideoLLM-online or Q-former could have been included for a fair evaluation against non-TTT token-reduction techniques, making it difficult to judge whether the proposed method is truly more effective."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924900193,"tcdate":1761935234306,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14504/Reviewer_Axud"],"signatures":["ICLR.cc/2026/Conference/Submission14504/Reviewer_Axud"],"forum":"cO1PLFtIxe","number":2,"license":"CC BY 4.0","cdate":1761935234306,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14504/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924900193,"domain":"ICLR.cc/2026/Conference","replyto":"cO1PLFtIxe","id":"df4hTrjB5j","forumContent":{"TLDR":{"value":"Audio-visual LLM for streaming understanding long videos"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["large language model","audio-visual","long video","memory","streaming"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Continuous, high-frame-rate, high-resolution processing of long video streams is critical for future AI agents, yet current video-understanding LLMs struggle to scale. Offline, fixed-frame-number methods require the stream length to adapt frame rates; streaming methods constrain memory by merging or discarding tokens, losing information. We propose video-SALMONN S, a streaming audio–visual LLM that processes arbitrary-length videos at 360p resolution under a fixed memory budget. The framework introduces (i) a test-time-training (TTT) memory module that continually updates token representations to capture long-range dependencies by replacing token merging, and (ii) a prompt-dependent memory reader that selectively retrieves context-relevant content from fixed-size memory. The TTT module is optimised with a Hessian-free conjugate-gradient procedure (TTT-HF) for efficient adaptation. On long-video benchmarks (Video-MME, LVBench, VideoEvalPro), video-SALMONN-S sustains high-quality understanding on multi-hour videos with >10k frames and ~1M tokens. Our 8B-parameter model achieves 74.2% overall and 67.8% on the Video-MME long split, significantly outperforming both offline and streaming baselines."},"_bibtex":{"value":"@misc{\nsun2026videosalmonn,\ntitle={video-{SALMONN} S: Streaming Audio-Visual {LLM}s Beyond Length Limits via Memory},\nauthor={Guangzhi Sun and Yixuan Li and Xiaodong Wu and Yudong Yang and Wei Li and Zejun MA and Chao Zhang},\nyear={2026},\nurl={https://openreview.net/forum?id=cO1PLFtIxe}\n}"},"title":{"value":"video-SALMONN S: Streaming Audio-Visual LLMs Beyond Length Limits via Memory"},"pdf":{"value":"/pdf/d002f360e532346f7b66c22b75187f8e37b5ee50.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"sun|videosalmonn_s_streaming_audiovisual_llms_beyond_length_limits_via_memory"},"authorids":{"value":["~Guangzhi_Sun1","~Yixuan_Li13","~Xiaodong_Wu4","~Yudong_Yang1","~Wei_Li78","~Zejun_MA1","~Chao_Zhang20"]},"authors":{"value":["Guangzhi Sun","Yixuan Li","Xiaodong Wu","Yudong Yang","Wei Li","Zejun MA","Chao Zhang"]}},"version":2},{"content":{"summary":{"value":"The paper presents Visual 2-Dimensional Mamba (V2M), a novel framework for image representation learning that adapts the state space model (SSM), specifically the Mamba model, to the two-dimensional structure of image data. This paper extends the 1d SSM to a 2d form to fundamentally accommodate the structure of image data. As a non-hierarchical vision mamba method, it achieves better results than its baseline Vim."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please refer to the weakness part."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- The consideration of 2d spatial relationship in the visual domain is intuitive and sounds reasonable.\n- Different from other hierarchical vision mamba methods, this paper handles the visual perception with a plain, non-hierarchical architecture, maintaining the ability in multimodality applications.\n- Compared with the baseline method, this paper presents significant improvements in both classification and dense prediction tasks.\n- This paper aims at a good question. Scanning strategy with SSM methods is a key problem because the 1d original scanning does not treat all tokens equally."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The character corners in Figure 1 require further explanation.\n- Note that function names in formulas are usually typeset properly in Roman font, e.g., `rot`, `concat`, `SSM`, `Linear`, `sum` in Eq. 13~15.\n- The paper references preprints and arXiv versions of significant works, such as Mamba (COLM), Vision Mamba (ICML), and VMamba (NeurIPS). The authors should update these citations to their final published versions to reflect the current state of the literature."}},"nonreaders":[],"tmdate":1731427657826,"tcdate":1730703289224,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2798/Reviewer_pHLf"],"signatures":["ICLR.cc/2025/Conference/Submission2798/Reviewer_pHLf"],"forum":"FowFLhUTgO","number":3,"license":"CC BY 4.0","cdate":1730703289224,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2798/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427657826,"domain":"ICLR.cc/2025/Conference","replyto":"FowFLhUTgO","id":"xONtb3nfG3","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Visual Backbone","Representation Learning"]},"supplementary_material":{"value":"/attachment/5af7971476d880f0927c713ccfbd3afb094e976e.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has garnered widespread attention due to its flexible design and efficient hardware performance to process 1D sequences based on the state space model (SSM). Recent studies have attempted to apply Mamba to the visual domain by flattening 2D images into patches and then regarding them as a 1D sequence. To compensate for the 2D structure information loss (e.g., local similarity) of the original image, most existing methods focus on designing different orders to sequentially process the tokens, which could only alleviate this issue to some extent. In this paper, we propose a Visual 2-Dimensional Mamba (V2M) model as a complete solution, which directly processes image tokens in the 2D space. We first generalize SSM to the 2-dimensional space which generates the next state considering two adjacent states on both dimensions (e.g., columns and rows). We then construct our V2M based on the 2-dimensional SSM formulation and incorporate Mamba to achieve hardware-efficient parallel processing. The proposed V2M effectively incorporates the 2D locality prior yet inherits the efficiency and input-dependent scalability of Mamba. Extensive experimental results on ImageNet classification and downstream visual tasks including object detection and instance segmentation on COCO and semantic segmentation on ADE20K demonstrate the effectiveness of our V2M compared with other visual backbones."},"_bibtex":{"value":"@misc{\nwang2025vm,\ntitle={V2M: Visual 2-Dimensional Mamba for Image Representation Learning},\nauthor={Chengkun Wang and Wenzhao Zheng and Yuanhui Huang and Jie Zhou and Jiwen Lu},\nyear={2025},\nurl={https://openreview.net/forum?id=FowFLhUTgO}\n}"},"title":{"value":"V2M: Visual 2-Dimensional Mamba for Image Representation Learning"},"pdf":{"value":"/pdf/e5c0ef44b4174b06f68eb3a20282f4958621aa79.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"wang|v2m_visual_2dimensional_mamba_for_image_representation_learning"},"authorids":{"value":["~Chengkun_Wang1","~Wenzhao_Zheng1","~Yuanhui_Huang1","~Jie_Zhou3","~Jiwen_Lu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chengkun Wang","Wenzhao Zheng","Yuanhui Huang","Jie Zhou","Jiwen Lu"]}},"version":2},{"content":{"venue":{"value":"Crossref"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-031-73232-4_15.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"zhang|motion_mamba_efficient_and_long_sequence_motion_generation"},"html":{"value":"https://doi.org/10.1007/978-3-031-73232-4_15"},"abstract":{"value":"Human motion generation stands as a significant pursuit in generative computer vision, while achieving long-sequence and efficient motion generation remains challenging. Recent advancements in state space models (SSMs), notably Mamba, have showcased considerable promise in long sequence modeling with an efficient hardware-aware design, which appears to be a significant direction upon building motion generation model. Nevertheless, adapting SSMs to motion generation faces hurdles since the lack of a specialized design architecture to model motion sequence. To address these challenges, we propose Motion Mamba, a simple yet efficient approach that presents the pioneering motion generation model utilized SSMs. Specifically, we design a Hierarchical Temporal Mamba (HTM) block to process temporal data by ensemble varying numbers of isolated SSM modules across a symmetric U-Net architecture aimed at preserving motion consistency between frames. We also design a Bidirectional Spatial Mamba (BSM) block to bidirectionally process latent poses, to enhance accurate motion generation within a temporal frame. Our proposed method achieves up to 50% FID improvement and up to 4 times faster on the HumanML3D and KIT-ML datasets compared to the previous best diffusion-based method, which demonstrates strong capabilities of high-quality long sequence motion modeling and real-time human motion generation."},"title":{"value":"Motion Mamba: Efficient and Long Sequence Motion Generation"},"authors":{"value":[{"fullname":"Zeyu Zhang","username":"~Zeyu_Zhang11"},{"fullname":"Akide Liu"},{"fullname":"Ian Reid"},{"fullname":"Richard Hartley"},{"fullname":"Bohan Zhuang"},{"fullname":"Hao Tang"}]}},"tmdate":1789090284816,"pdate":1735689600000,"externalIds":["doi:10.1007/978-3-031-73232-4_15"],"tcdate":1762162578778,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Zeyu_Zhang11"],"forum":"pYqslaCDun","license":"CC BY-SA 4.0","number":2514,"cdate":1732226338583,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/-/Edit"],"mdate":1789090284816,"domain":"OpenReview.net/Public_Article","id":"pYqslaCDun","version":2},{"content":{"venue":{"value":"ECCV (1) 2024"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-031-73232-4_15.pdf"},"venueid":{"value":"dblp.org/conf/ECCV/2024"},"paperhash":{"value":"zhang|motion_mamba_efficient_and_long_sequence_motion_generation"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zeyu_Zhang_0006:","https://dblp.org/search/pid/api?q=author:Akide_Liu:","https://dblp.org/search/pid/api?q=author:Ian_D._Reid_0001:","~Richard_I._Hartley1","~Bohan_Zhuang1","https://dblp.org/search/pid/api?q=author:Hao_Tang_0005:"]},"html":{"value":"https://doi.org/10.1007/978-3-031-73232-4_15"},"_bibtex":{"value":"@inproceedings{DBLP:conf/eccv/ZhangLRHZT24,\n  author={Zeyu Zhang and Akide Liu and Ian D. Reid and Richard I. Hartley and Bohan Zhuang and Hao Tang},\n  title={Motion Mamba: Efficient and Long Sequence Motion Generation},\n  year={2024},\n  cdate={1704067200000},\n  pages={265-282},\n  url={https://doi.org/10.1007/978-3-031-73232-4_15},\n  booktitle={ECCV (1)},\n  crossref={conf/eccv/2024-1}\n}\n"},"abstract":{"value":"Human motion generation stands as a significant pursuit in generative computer vision, while achieving long-sequence and efficient motion generation remains challenging. Recent advancements in state space models (SSMs), notably Mamba, have showcased considerable promise in long sequence modeling with an efficient hardware-aware design, which appears to be a significant direction upon building motion generation model. Nevertheless, adapting SSMs to motion generation faces hurdles since the lack of a specialized design architecture to model motion sequence. To address these challenges, we propose Motion Mamba, a simple yet efficient approach that presents the pioneering motion generation model utilized SSMs. Specifically, we design a Hierarchical Temporal Mamba (HTM) block to process temporal data by ensemble varying numbers of isolated SSM modules across a symmetric U-Net architecture aimed at preserving motion consistency between frames. We also design a Bidirectional Spatial Mamba (BSM) block to bidirectionally process latent poses, to enhance accurate motion generation within a temporal frame. Our proposed method achieves up to 50% FID improvement and up to 4 times faster on the HumanML3D and KIT-ML datasets compared to the previous best diffusion-based method, which demonstrates strong capabilities of high-quality long sequence motion modeling and real-time human motion generation."},"title":{"value":"Motion Mamba: Efficient and Long Sequence Motion Generation"},"authors":{"value":["Zeyu Zhang","Akide Liu","Ian D. Reid","Richard I. Hartley","Bohan Zhuang","Hao Tang"]}},"tmdate":1767793104876,"pdate":1704067200000,"tcdate":1741397782065,"writers":["~"],"signatures":["~Richard_Hartley1"],"forum":"cGgne8xwm0","license":"CC BY-SA 4.0","number":366317,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1767793104876,"domain":"DBLP.org","id":"cGgne8xwm0","version":2},{"content":{"summary":{"value":"This paper introduces **Natural Product Chemical Language Models (NPCLMs)**, addressing the underrepresentation of natural products (NPs) in molecular language modeling. Using a curated dataset of ~1M NP SMILES, the authors pre-train state-space models (Mamba, Mamba-2) and compare them with transformer baselines (GPT), systematically evaluating eight tokenization strategies. Experiments span **molecule generation** (validity, uniqueness, novelty, scaffold diversity) and **property prediction** (anti-cancer activity, taste, and peptide permeability). Results show that **Mamba** yields more valid and unique molecules with fewer long-range syntax errors, while **GPT** generates slightly more novel scaffolds. For property prediction, Mamba and Mamba-2 modestly outperform GPT under random splits but converge under scaffold splits. Crucially, **tokenization choice** strongly impacts outcomes, with smaller, chemically informed vocabularies (AIS, NP-specific BPE) outperforming large vocabularies. The study highlights that domain-specific pretraining on NPs can match or exceed general chemical LMs trained on 100× more data, emphasizing the importance of data quality and relevance over sheer scale."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. **Motivation for NP-specific training**: Could the authors elaborate on the motivation for training exclusively on natural products? Intuitively, pretraining on a broader dataset that includes but is not limited to NPs might improve generalization, while still capturing NP-specific features. Why was NP-only pretraining chosen instead of a mixed strategy?\n2. **Validity of generated molecules**: In Table 1, the reported validity ratios are relatively low (even with atom-level tokenization). Prior works on SMILES generation typically report validity above 90% [1]. Could the authors clarify why validity is lower here?\n3. **Effect of model scaling**: What is the impact of scaling model size in this setting? Given that GPT-type transformers often benefit strongly from scaling, do the authors expect GPT to outperform Mamba under larger-scale setups?\n\n[1] Molecular de-novo design through deep reinforcement learning."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. **Essential field and motivation**: The paper addresses an important and underexplored problem: developing chemical language models specifically for natural products (NPs). Since NPs are chemically more complex and biologically significant than typical synthetic molecules, focusing on them fills a meaningful gap in current research and aligns well with drug discovery applications. By targeting this space, the paper contributes to an essential field with high scientific and practical relevance.\n2. **Systematic and comprehensive experiments**: The study is notable for its breadth and rigor in experimental design. It benchmarks three different model families (Mamba, Mamba-2, GPT) across eight tokenization strategies, under both random and scaffold splits, using a curated dataset of ~1M NPs. The authors evaluate on molecule generation and property prediction tasks with multiple metrics, conduct error analysis, and even compare with general CLMs like ChemBERTa-2 and MoLFormer. This systematic approach makes the results highly credible and useful as a benchmark reference.\n3. **Clear writing and presentation**: The paper is well-written and accessible, with a clear motivation, logical structure, and smooth narrative flow from background to experiments to conclusions. The figures and tables effectively summarize key findings, while detailed reproducibility notes in the appendices strengthen transparency. Overall, the writing quality and presentation make the technical contributions easy to follow for both machine learning researchers and domain experts in chemistry."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The main limitation of the paper is that the contribution largely reduces to an extensive set of empirical comparisons, but without introducing new methodological innovations or deriving deeper insights that could guide future work.\n\n1. **Model comparison (Mamba vs GPT)**: The comparison between Mamba and GPT models offers limited value. The results mainly show that Mamba tends to generate molecules with higher validity and uniqueness, while GPT produces more novel ones. However, this trade-off between validity and novelty is easily tunable through standard decoding hyperparameters (temperature, top-k, top-p) in both model families, and outcomes are also highly sensitive to training settings such as the number of steps. As a result, the comparison does not provide clear guidance on which architecture is better suited for molecular tasks.\n2. **Tokenizer comparison**: The tokenizer study similarly contributes little novelty. Prior work [1] has already shown the superiority of atom-level tokenization over generic BPE approaches. Moreover, the models here are trained only on canonical SMILES without augmentation by randomized SMILES, which is known to benefit large-vocabulary BPE tokenizers. This design choice biases the results and weakens the conclusions.\n3. **Connection to natural products**: Although the paper positions itself around natural products, the methods and analyses are only weakly tied to NP-specific characteristics. The dataset merely provides a source of training molecules, and the main conclusions (validity vs novelty trade-off, tokenizer effects) appear general to SMILES modeling rather than unique to natural products.\n4. **Choice of architectures for property prediction**: While autoregressive models are natural for molecule generation, they are not state-of-the-art for molecular property prediction, where graph neural networks and 3D-aware models currently dominate. The paper’s detailed comparison of autoregressive architectures in this context therefore seems of limited relevance.\n\n[1] Comparing SMILES and SELFIES tokenization for enhanced chemical language modeling."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926480811,"tcdate":1760952060773,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16342/Reviewer_8tF2"],"signatures":["ICLR.cc/2026/Conference/Submission16342/Reviewer_8tF2"],"forum":"TcmsiSkN5T","number":1,"license":"CC BY 4.0","cdate":1760952060773,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16342/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926480811,"domain":"ICLR.cc/2026/Conference","replyto":"TcmsiSkN5T","id":"iAYpc6zmUK","forumContent":{"TLDR":{"value":"We apply the latest state space models to chemical language models for Natural Products, comparing Mamba, Mamba-2, and GPT for molecule generation and property prediction, discussing the impact of architecture, tokenization, and training strategies."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Chemical Language Models (CLMs)","Natural Products (NPs)","State Space Models","Mamba","Mamba-2","GPT","molecule generation","tokenization","property prediction"]},"supplementary_material":{"value":"/attachment/0b9f3890801a6017a19b69f8ce20f17c78d1268c.zip"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"abstract":{"value":"Language models are increasingly applied in scientific domains such as chemistry, where chemical language models (CLMs) are well-established for predicting molecular properties or generating de novo compounds for small molecules. However, Natural Products (NPs)---such as penicillin, morphine, and quinine, which have driven major breakthroughs in medicine---have received limited attention in CLM research. This gap limits the potential of NPs as a source of new therapeutics. To bridge this gap, we develop Natural Product–specific CLMs (NPCLMs) by pre-training the latest state-space model variants, Mamba and Mamba-2, which have shown great potential in modeling information-dense sequences, and compare them with transformer baselines (GPT). Using the largest known collection of $\\sim$1M NPs, we provide the first extensive experimental comparison of selective state-space models (S6) and transformers in NP-focused tasks, along with a comparison of eight tokenization strategies, including character-level, Atom-in-SMILES (AIS), general byte-pair encoding (BPE) and NP-specific byte-pair encoding (NPBPE). Model performance is evaluated on two tasks: molecule generation, measured by validity, uniqueness, and novelty, and property prediction (peptide membrane permeability, taste, and anti-cancer activity), evaluated using Matthews Correlation Coefficient (MCC) and Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The results show that Mamba consistently generates 1–2\\% more valid and unique molecules than Mamba-2 and GPT, while making 3-6\\% less long-range dependency errors; however, GPT produces $\\sim$2\\% more novel structures. In property prediction, both Mamba and Mamba-2 outperform GPT by a modest but consistent 0.02 to 0.04 improvement in MCC under random splitting. Under stricter scaffold splitting, which groups molecules by core structure to better assess generalization to new scaffolds, all models perform comparably. In addition, chemically informed tokenization further enhances performance. For comparison, we include general-domain CLMs (ChemBERTa-2 and MoLFormer) and found that pre-training on $\\sim$1M NPs achieves results on par with general CLMs trained on datasets over 100 times larger, emphasizing the value of domain-specific pre-training and data quality over scale in chemical language modeling."},"_bibtex":{"value":"@misc{\nwang2026chemical,\ntitle={Chemical Language Models for Natural Products: A State-Space Model Approach},\nauthor={Ho-Hsuan Wang and Afnan Sultan and Andrea Volkamer and Dietrich Klakow},\nyear={2026},\nurl={https://openreview.net/forum?id=TcmsiSkN5T}\n}"},"title":{"value":"Chemical Language Models for Natural Products: A State-Space Model Approach"},"pdf":{"value":"/pdf/226c332cceedd7a1e72611651a0a7ec6481ea17c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wang|chemical_language_models_for_natural_products_a_statespace_model_approach"},"authorids":{"value":["~Ho-Hsuan_Wang1","~Afnan_Sultan1","~Andrea_Volkamer1","~Dietrich_Klakow1"]},"authors":{"value":["Ho-Hsuan Wang","Afnan Sultan","Andrea Volkamer","Dietrich Klakow"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2412.08947v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"yao|selective_visual_prompting_in_vision_mamba"},"html":{"value":"https://doi.org/10.48550/arXiv.2412.08947"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-08947,\n  publtype={informal},\n  author={Yifeng Yao and Zichen Liu and Zhenyu Cui and Yuxin Peng and Jiahuan Zhou},\n  title={Selective Visual Prompting in Vision Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2412.08947},\n  url={https://doi.org/10.48550/arXiv.2412.08947}\n}\n"},"abstract":{"value":"Pre-trained Vision Mamba (Vim) models have demonstrated exceptional performance across various computer vision tasks in a computationally efficient manner, attributed to their unique design of selective state space models. To further extend their applicability to diverse downstream vision tasks, Vim models can be adapted using the efficient fine-tuning technique known as visual prompting. However, existing visual prompting methods are predominantly tailored for Vision Transformer (ViT)-based models that leverage global attention, neglecting the distinctive sequential token-wise compression and propagation characteristics of Vim. Specifically, existing prompt tokens prefixed to the sequence are insufficient to effectively activate the input and forget gates across the entire sequence, hindering the extraction and propagation of discriminative information. To address this limitation, we introduce a novel Selective Visual Prompting (SVP) method specifically for the efficient fine-tuning of Vim. To prevent the loss of discriminative information during state space propagation, SVP employs lightweight selective prompters for token-wise prompt generation, ensuring adaptive activation of the update and forget gates within Mamba blocks to promote discriminative information propagation. Moreover, considering that Vim propagates both shared cross-layer information and specific inner-layer information, we further refine SVP with a dual-path structure: Cross-Prompting and Inner-Prompting. Cross-Prompting utilizes shared parameters across layers, while Inner-Prompting employs distinct parameters, promoting the propagation of both shared and specific information, respectively. Extensive experimental results on various large-scale benchmarks demonstrate that our proposed SVP significantly outperforms state-of-the-art methods. Our code is available at https://github.com/zhoujiahuan1991/AAAI2025-SVP."},"title":{"value":"Selective Visual Prompting in Vision Mamba"},"authors":{"value":[{"fullname":"Yifeng Yao","username":""},{"fullname":"Zichen Liu","username":""},{"fullname":"Zhenyu Cui","username":"~Zhenyu_Cui1"},{"fullname":"Yuxin Peng","username":""},{"fullname":"Jiahuan Zhou","username":""}]}},"tmdate":1784537407079,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2412-08947"],"tcdate":1784537399334,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Zhenyu_Cui1"],"forum":"gm1kFsnAs1","license":"CC BY-SA 4.0","number":67986,"cdate":1704067200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784537407079,"domain":"OpenReview.net/Public_Article","id":"gm1kFsnAs1","version":2},{"content":{"comment":{"value":"We sincerely thank the reviewer for the timely feedback. We apologize if our previous response lacked clarity. Actually, we have already responded to the concern about benchmark evaluation results. Since several reviewers shared this common concern, we addressed it collectively in the **Downstream task evaluation of 1B Model trained with optimized LR schedule** part in **General Response**. Here, we are glad to clarify this issue for you again, including a table summarizing downstream evaluation. For your convenience, we reproduce the table below.\n\n**Q2:** Could you add benchmark evaluation results, to see whether the loss improvement with optimized learning schedule indeed improves the accuracy?\n\n**A:** **The optimized LR schedule consistently outperforms the cosine LR schedule across multiple downstream tasks**. In the experiment of the 1B model, we derived the optimized LR schedule based on its corresponding MPL. Following the evaluation criteria outlined in the Mamba paper[1], we selected the first four downstream tasks to compare the performance of the cosine and optimized LR schedules. The comparison results are shown in the table below, demonstrating that the optimized LR schedule performs better than the cosine schedule. \n| **Downstream Dataset** | **LAMBADA** | **HellaSwag** | **PIQA** | **ARC-E** |\n|-------------------------|-------------|---------------|----------|-----------|\n| **Cosine Schedule**     | 46.54       | 37.12         | **65.13** | 43.56     |\n| **Optimized Schedule**  | **48.71** ($\\uparrow$ 2.17%) | **37.74** ($\\uparrow$ 0.62%) | 65.07 ($\\downarrow$ 0.06%) | **44.09** ($\\uparrow$ 0.53%) |\n\n[1] [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/pdf/2312.00752)"},"title":{"value":"Response to Reviewer ic14 [1/2]"}},"tmdate":1732796485013,"tcdate":1732796485013,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13754/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13754/Authors"],"forum":"KnoS9XxIlK","number":25,"license":"CC BY 4.0","cdate":1732796485013,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13754/-/Official_Comment"],"mdate":1732796485013,"domain":"ICLR.cc/2025/Conference","replyto":"uLcN56xx80","id":"8AzM4Dj6Y1","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"Loss curve prediction and optimized learning rate schedule"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large language model","Learning rate scheduler","Scaling Law","Hyperparameter optimization"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Training large models is both resource-intensive and time-consuming, making it crucial to understand the quantitative relationship between model performance and hyperparameters. In this paper, we derive an empirical law that predicts pretraining loss for large language models for every intermediate training step across various learning rate schedules, including constant, cosine, and step decay schedules. Our proposed law takes a multi-power form, combining a power law based on the sum of learning rates and additional power laws to account for a loss reduction effect as learning rate decays. We validate this law extensively on Llama-2 models of varying sizes and demonstrate that, after fitting on a few learning rate schedules, it accurately predicts the loss curves for unseen schedules of different shapes and horizons. Moreover, by minimizing the predicted final pretraining loss across learning rate schedules, we are able to find a schedule that outperforms the widely-used cosine learning rate schedule. Interestingly, this automatically discovered schedule bears some resemblance to the recently proposed Warmup-Stable-Decay (WSD) schedule (Hu et al, 2024) but achieves a slightly lower final loss. We believe these results could offer valuable insights for understanding the dynamics of pretraining and for designing learning rate schedules to improve efficiency."},"_bibtex":{"value":"@inproceedings{\nluo2025a,\ntitle={A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules},\nauthor={Kairong Luo and Haodong Wen and Shengding Hu and Zhenbo Sun and Maosong Sun and Zhiyuan Liu and Kaifeng Lyu and Wenguang Chen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=KnoS9XxIlK}\n}"},"title":{"value":"A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules"},"pdf":{"value":"/pdf/212cefb36307e11021cced0a49c98a00931e6431.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"luo|a_multipower_law_for_loss_curve_prediction_across_learning_rate_schedules"},"authorids":{"value":["~Kairong_Luo1","~Haodong_Wen1","~Shengding_Hu2","~Zhenbo_Sun1","~Zhiyuan_Liu1","~Maosong_Sun1","~Kaifeng_Lyu2","~Wenguang_Chen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Kairong Luo","Haodong Wen","Shengding Hu","Zhenbo Sun","Zhiyuan Liu","Maosong Sun","Kaifeng Lyu","Wenguang Chen"]}},"version":2},{"content":{"TLDR":{"value":"Stable SSMs do not forget purely exponentially. We prove memory decays as $\\Theta(k^{m-1}\\rho^k)$, where Jordan block structure creates long transient memory, improving recall but hurting counting-based reasoning tasks."},"venue":{"value":"DEMO 2026 Poster"},"pdf":{"value":"/pdf/2130ec241f89fc450d0f91fccc0afa3744d7ef04.pdf"},"keywords":{"value":["State Space Models","Memory Decay","Long-Range Dependencies","Jordan Normal Form","Spectral Theory","Mamba","S4","Sequence Modeling","Forecasting","Dynamical Systems","Expressivity","Control Theory"]},"venueid":{"value":"ICML.cc/2026/Workshop/DEMO"},"paperhash":{"value":"dadwal|the_illusion_of_state_sharp_memorydecay_bounds_in_linear_ssms"},"authorids":{"value":["~Aryan_Dadwal1"]},"abstract":{"value":"Linear State Space Models (SSMs) such as S4 and Mamba are widely believed to possess exponentially decaying memory whenever the transition matrix is stable. We show that this characterization is incomplete. By analyzing the Jordan normal form of the transition operator, we prove that memory retention in linear SSMs follows the sharp asymptotic law $\\Theta(k^{m-1}\\rho^k)$, where $\\rho$ is the spectral radius and $m$ is the size of the dominant Jordan block. This reveals that defective, non-diagonalizable dynamics induce a polynomial transient phase that can dramatically extend the effective context horizon beyond predictions based solely on spectral radius. We further derive matching upper and lower bounds for both hidden-state and input-output memory, establish a controllability--observability characterization through Gramians, and generalize the analysis to time-varying gated architectures including Mamba-style selective SSMs. Empirically, we uncover a fundamental tradeoff: defective transition matrices substantially improve associative recall but degrade formal language counting tasks that require independent state tracking. We additionally show that LayerNorm suppresses transient numerical explosion while preserving the memory extension effect. Our results provide a sharp constructive theory of memory in SSMs and yield concrete architectural principles for long-horizon sequence modeling and forecasting."},"_bibtex":{"value":"@inproceedings{\ndadwal2026the,\ntitle={The Illusion of State: Sharp Memory-Decay Bounds in Linear {SSM}s},\nauthor={Aryan Dadwal},\nbooktitle={Decision-Making from Offline Datasets to Online Adaptation: Black-Box Optimization to Reinforcement Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=8Kg8lL3L58}\n}"},"title":{"value":"The Illusion of State: Sharp Memory-Decay Bounds in Linear SSMs"},"authors":{"value":["Aryan Dadwal"]}},"tmdate":1783646465866,"pdate":1779747018388,"tcdate":1778280890353,"writers":["ICML.cc/2026/Workshop/DEMO","ICML.cc/2026/Workshop/DEMO/Submission153/Authors"],"signatures":["ICML.cc/2026/Workshop/DEMO/Submission153/Authors"],"forum":"8Kg8lL3L58","license":"CC BY 4.0","number":153,"cdate":1778280890353,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/DEMO/-/Submission","ICML.cc/2026/Workshop/DEMO/-/Post_Submission","ICML.cc/2026/Workshop/DEMO/-/Edit","ICML.cc/2026/Workshop/DEMO/Submission153/-/Revision","ICML.cc/2026/Workshop/DEMO/Submission153/-/Camera_Ready_Submission"],"mdate":1783646465866,"odate":1779903472120,"domain":"ICML.cc/2026/Workshop/DEMO","id":"8Kg8lL3L58","version":2},{"content":{"summary":{"value":"This work establishes density results of input-controlled differential equations, which formulate different types of state-space models such as S4 and Mamba in the continuous-time idealization. Under this framework, different closures of these models are derived, indicating their distinct inductive biases in the (universal) approximation sense, particularly implying the advantage of data-dependent modeling."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Please provide more details of questions raised in the weaknesses section above. \n2. In formulation, the mentioned \"gating\" mechanism in this work seems to have a different meaning as usual. Here, gating refers to the transformation of inputs (i.e. data pre-processing). However, in practice, gating often means the (multiplicative) interaction between the hidden states and readout layers (i.e., to get outputs). Can authors explain more about this?\n3. Minor issues: What is the definition of (linear) NCDE (see line 311, 312, 315, 317 and 643)."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. This work provides fundamental theoretical justifications on the expressivity of different types of SSMs. \n2. The derived inductive biases clearly distinguish the recent new data-dependent modeling (e.g. Mamba) from the classic original data-independent modeling (e.g. S4) thorough a input-selective framework, which shows rigorously the superiority of the former architecture. \n3. Other theoretical results also lead to useful insights. For instance, diagonal recurrence weakens the approximation capability, but this can be alleviated via stacking. \n4. All these insights are basically verified by numerical experiments."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. In theory, only the density type of results is provided. That is, although models can rates certain targets universally when models' parameters go to infinity, the convergence *rates* are not characterized. This is much more important since the convergence can  be rather slow in certain situations (e.g. curse of dimension), leading to possibly vacuous bounds. The theoretical part would be strengthened and more convincing if authors can discuss more on the approximation rates of SSMs, especially the improvements (in parameter efficiency) of data-dependent modeling. \n2. In experiments, the simulations are conducted on low-dimensional (2 & 3) and synthetic datasets. Is it due to the ability of path signature? Can the path signature be powerful and efficient when handling high-dimensional (and real-world) input data?"},"limitations":{"value":"As is stated by authors, this work applies a continuous-time idealization in the formulation. The role of concrete discretization schemes is also worthy of explorations, particularly under the case of data-dependent discretization."}},"nonreaders":[],"tmdate":1730879555170,"tcdate":1720302433034,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission12481/Reviewer_1RLC"],"signatures":["NeurIPS.cc/2024/Conference/Submission12481/Reviewer_1RLC"],"forum":"3SzrqwupUx","number":1,"license":"CC BY 4.0","cdate":1720302433034,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission12481/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879555170,"domain":"NeurIPS.cc/2024/Conference","replyto":"3SzrqwupUx","id":"lJS87Q48hJ","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["theory","deep learning theory","SSM","RNN","neural controlled differential equations","signatures","kernels"]},"supplementary_material":{"value":"/attachment/416fe722331399eafa4acc22b5fce96ef3d6cd13.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Structured state-space models (SSMs) are gaining popularity as effective foundational architectures for sequential data, demonstrating outstanding performance across a diverse set of domains alongside desirable scalability properties. Recent developments show that if the linear recurrence powering SSMs allows for a selectivity mechanism leveraging multiplicative interactions between inputs and hidden states (e.g. Mamba, GLA, Hawk/Griffin, HGRN2), then the resulting architecture can surpass attention-powered foundation models trained on text in both accuracy and efficiency, at scales of billion parameters. In this paper, we give theoretical grounding to the selectivity mechanism, often linked to in-context learning, using tools from Rough Path Theory. We provide a framework for the theoretical analysis of generalized selective SSMs, fully characterizing their expressive power and identifying the gating mechanism as the crucial architectural choice. Our analysis provides a closed-form description of the expressive powers of modern SSMs, such as Mamba, quantifying theoretically the drastic improvement in performance from the previous generation of models, such as S4. Our theory not only motivates the success of modern selective state-space models, but also provides a solid framework to understand the expressive power of future SSM variants. In particular, it suggests cross-channel interactions could play a vital role in future improvements."},"_bibtex":{"value":"@inproceedings{\ncirone2024theoretical,\ntitle={Theoretical Foundations of Deep Selective State-Space Models},\nauthor={Nicola Muca Cirone and Antonio Orvieto and Benjamin Walker and Cristopher Salvi and Terry Lyons},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=3SzrqwupUx}\n}"},"title":{"value":"Theoretical Foundations of Deep Selective State-Space Models"},"pdf":{"value":"/pdf/4e86fe9ae93de98a547f68ad2934a6a01ebc450e.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"cirone|theoretical_foundations_of_deep_selective_statespace_models"},"authorids":{"value":["~Nicola_Muca_Cirone1","~Antonio_Orvieto3","~Benjamin_Walker1","~Cristopher_Salvi1","~Terry_Lyons2"]},"authors":{"value":["Nicola Muca Cirone","Antonio Orvieto","Benjamin Walker","Cristopher Salvi","Terry Lyons"]}},"version":2},{"content":{"summary":{"value":"The authors introduce Voxel Mamba, a 3D object detection method based on state-space models.  Voxel Mamba introduces a efficient group-free approach to avoid inefficient grouping operations and prevent limitations on the receptive field for 3D object detection in point clouds. The authors also present Asymmetrical State Space Models (ASSMs) to capture multi-scale context information, offering each voxel data-dependent global contexts. Voxel Mamba enables voxel interactions throughout the scene and captures cloud locality using space-filling curves. Extensive experiments validate its performance, positioning Voxel Mamba as a viable alternative for 3D object detection."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"See Weakness. Besides, there are few other suggestions:\n\n1. Could the authors provide more description of Window Sweep in Table 5(c)? Does it include the Shifted Regional Grouping Shifted Token Groups operation used in SST?\n1. Is a Linear for the up dimension missing from the SSM part of Fig. 2?"},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper is well-organized and easy to follow.\n2. The authors first time delve into the potential of Mamba in 3D object detection, which is an intriguing subject."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Motivation can be improved. The authors must clarify why SSM is essential for their motivation and quantify the advantages it offers over transformers for this task. The question in lines 55-56 does not demonstrate SSM's superiority. The author's design for SSM is also hardly inspiring for the 3D detection community.\n2. Some of the method's design confuse me. Traversing the downsampled feature maps with a Hilbert curve in the opposite direction can misalign the feature receptive fields. Does this challenge model training with varying inputs, even with IWPE included? Additionally, given its use of multi-scale features, how much does ASSM outperform FPN?\n3. The experimental comparisons may be inadequate. The authors present a pillar version for Voxel Mamba but fail to compare its performance and efficiency against current state-of-the-art pillar-based methods in the main table.\n4. The experiment's details should be more clearly documented. For example, does Voxel Mamba adopts IoU-rectification scheme? The comparative setup in the ablation study could have been accounted for in more detail and analyzed why the method designed by the authors was effective.\n5. How does Voxel Mamba integrate with multi-frame temporal 3D detection?"},"limitations":{"value":"The authors have discussed the limitations."}},"nonreaders":[],"tmdate":1730878757585,"tcdate":1721111924536,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2210/Reviewer_kB9g"],"signatures":["NeurIPS.cc/2024/Conference/Submission2210/Reviewer_kB9g"],"forum":"gHYhVSCtDH","number":4,"license":"CC BY 4.0","cdate":1721111924536,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2210/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878757585,"domain":"NeurIPS.cc/2024/Conference","replyto":"gHYhVSCtDH","id":"mOcKqsRJVR","forumContent":{"TLDR":{"value":"A simple and effective 3D backbone for point cloud based 3D object detection."},"venue":{"value":"NeurIPS 2024 spotlight"},"keywords":{"value":["Point Cloud based 3D Object Detection","State Space Models","Group-free"]},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Serialization-based methods, which serialize the 3D voxels and group them into multiple sequences before inputting to Transformers, have demonstrated their effectiveness in 3D object detection. However, serializing 3D voxels into 1D sequences will inevitably sacrifice the voxel spatial proximity. Such an issue is hard to be addressed by enlarging the group size with existing serialization-based methods due to the quadratic complexity of Transformers with feature sizes. Inspired by the recent advances of state space models (SSMs), we present a Voxel SSM, termed as Voxel Mamba, which employs a group-free strategy to serialize the whole space of voxels into a single sequence. The linear complexity of SSMs encourages our group-free design, alleviating the loss of spatial proximity of voxels. To further enhance the spatial proximity, we propose a Dual-scale SSM Block to establish a hierarchical structure, enabling a larger receptive field in the 1D serialization curve, as well as more complete local regions in 3D space. Moreover, we implicitly apply window partition under the group-free framework by positional encoding, which further enhances spatial proximity by encoding voxel positional information. Our experiments on Waymo Open Dataset and nuScenes dataset show that Voxel Mamba not only achieves higher accuracy than state-of-the-art methods, but also demonstrates significant advantages in computational efficiency. The source code is available at https://github.com/gwenzhang/Voxel-Mamba."},"_bibtex":{"value":"@inproceedings{\nzhang2024voxel,\ntitle={Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection},\nauthor={Guowen Zhang and Lue Fan and Chenhang HE and Zhen Lei and Zhaoxiang Zhang and Lei Zhang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=gHYhVSCtDH}\n}"},"title":{"value":"Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection"},"pdf":{"value":"/pdf/0cd37d7602d988cb025ea9d85f62abe79ef5139d.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"zhang|voxel_mamba_groupfree_state_space_models_for_point_cloud_based_3d_object_detection"},"authorids":{"value":["~Guowen_Zhang1","~Lue_Fan1","~Chenhang_HE1","~Zhen_Lei2","~Zhaoxiang_Zhang3","~Lei_Zhang2"]},"authors":{"value":["Guowen Zhang","Lue Fan","Chenhang HE","Zhen Lei","Zhaoxiang Zhang","Lei Zhang"]}},"version":2},{"content":{"summary":{"value":"Summary: The paper presents an investigation into normalization techniques within the Mamba architecture, addressing an important aspect of deep learning model stability. While the work makes some valuable contributions to understanding normalization in Mamba networks, there are several significant limitations that need to be addressed."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"-"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":1},"strengths":{"value":"- Takes on the important challenge of improving training stability in Mamba networks.\n- Provides practical insights into normalization layer positioning and combination."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Major issues:\n\n1. The examination of normalization effects is restricted to only two tasks, which is insufficient to support broad conclusions. Findings appear task-dependent, with contradictory results between vision and sequence tasks.\n2. The focus on long sequence modeling for normalization position analysis lacks justification.\n3. The Weight Domination hypothesis lacks sufficient supporting evidence. Contradictory results regarding BN→None performance weaken the Weight Domination hypothesis. The relationship between findings in vision and sequence tasks is not  dequately explained.\n\n\nMinor issues:\n\n1. Inconsistent visualization approaches in Figure 3 (varying x-axes and plotting styles) hamper interpretation.\n2. Results from Vision Mamba (showing LN sufficiency) contradict findings from sequence modeling.\n3. The work falls short of providing general insights that could replace task-specific ablation studies."}},"nonreaders":[],"tmdate":1731428661570,"tcdate":1730652623867,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9166/Reviewer_aHU3"],"signatures":["ICLR.cc/2025/Conference/Submission9166/Reviewer_aHU3"],"forum":"YK8eO7BEkJ","number":2,"license":"CC BY 4.0","cdate":1730652623867,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9166/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428661570,"domain":"ICLR.cc/2025/Conference","replyto":"YK8eO7BEkJ","id":"Yeujk7kBMd","forumContent":{"TLDR":{"value":"We conduct an emprical study on the role of extra normalization layers in enhancing model performance and stability during the training for Mamba architecture"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","long-sequence modeling","normalization","performance","stability"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Normalization layers are crucial for improving the training efficiency and stability of deep neural network architectures. The recently proposed Mamba network has demonstrated significant potential in competing with Transformers. However, as with many deep architectures, the training stability of Mamba remains a significant challenge, and normalization techniques are key to addressing this issue. In this paper, we systematically investigate the effects of normalization type, position and combinations on the Mamba Block. On the one hand, we conducted extensive experiments to evaluate the impact of applying various normalization layers before or after the SSM module(the core module of Mamba Block). On the other hand, we performed thorough experiments to assess the effects of combining diverse normalization techniques before and after the SSM module. Our analysis encompasses both long sequence modeling and image classification tasks. The results show that applying normalization layers after the SSM module (if used only once) and combining different normalization layers before and after the SSM module can enhance training stability and improve Mamba performance. Furthermore, we provide practical recommendations for selecting appropriate normalization techniques in designing Mamba  architectures and validated them on other datasets. We hope that our insights will help mitigate training instabilities in deep learning and foster the development of more robust architectures. All codes and models used in this study will be open-sourced on GitHub."},"_bibtex":{"value":"@misc{\nfeng2025an,\ntitle={An Empirical Study on Normalization in Mamba},\nauthor={Peilin Feng and Yuanshuai Wang and Yunhao Ni and Zekun Li and Wenjun Wu and Lei Huang},\nyear={2025},\nurl={https://openreview.net/forum?id=YK8eO7BEkJ}\n}"},"title":{"value":"An Empirical Study on Normalization in Mamba"},"pdf":{"value":"/pdf/9e73f11d8e2a5213c03f3049e0d3b4489169eff2.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"feng|an_empirical_study_on_normalization_in_mamba"},"authorids":{"value":["~Peilin_Feng1","~Yuanshuai_Wang1","~Yunhao_Ni1","~Zekun_Li11","~Wenjun_Wu2","~Lei_Huang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Peilin Feng","Yuanshuai Wang","Yunhao Ni","Zekun Li","Wenjun Wu","Lei Huang"]}},"version":2},{"content":{"summary":{"value":"This paper improves context compression in linear attention models with row-sparse update and sparse state expansion (SSE). The row-sparse update learns a sparse mask with top-k and softmax to update only a few rows in the contextual states, which utilizes the state space more effectively than dense update. Since different tokens are associated with different rows, the row-sparse update eliminates the need of gating at each step, thereby avoiding the limited receptive field caused by gating. The authors then propose SSE to extend row-sparse update to larger state space. SSE first divides a large state space into N partitions, and then perform dense updates on k partitions. By sharing the attention parameters across partitions, SSE decouples the parameters size and the memory capacity, thereby solving the capacity bottleneck of linear attention models. Experiments on language modeling, needle-in-a-haystack and reasoning benchmarks show that SSE and the hybrid SSE-H achieves state-of-the-art performance among linear attention models."},"soundness":{"value":4},"confidence":{"value":3},"questions":{"value":"1. Is the cosine similarity computed for a single step then averaged over a sequence or the whole dataset?\n2. The logic in Line 174-178 is hard to understand. It’s hard to understand what Propositions 2-4 are without looking into the Appendix. Propositions 2-4 use the row-sparse update, which is introduced only in later sections. Besides, it’s not very clear how the conclusion of decay in gated variants is derived before looking into Proposition 4. As row-sparse update is complementary to gating and they may co-exist, it’s also hard to understand why SSE solves the issue caused by gating. You may discuss eliminating gating as a benefit of SSE after introducing the method.\n3. Line 184-186: By theoretical analysis, do you mean Proposition 2? It’s a little bit hard to understand this without looking into the appendix. You need to add more details in the main paper.\n4. The functions softmax and top-k produce a $k$ dimension vector by their definition, which is not correct. Do you mean softmax on the $k$ non-zero elements and keep the rest as 0? Then you need to re-define your softmax function.\n5. Line 263-265: This key insight should be brought to early paragraphs of Sec 4.1. Otherwise, it’s hard to understand how SSE is connected with row-sparse update. You’re essentially factorizing a row-sparse update for N*c rows into two small parameter matrices.\n6. Font size in Figure 3 & 4 should be increased.\n7. Line 284 & 286: Why can the always-selected partition capture local interactions? The inputs are accumulated by addition, which doesn’t model interactions.\n8. Line 294-296: Please explain why singular value entropy reflects the difficulty of compression. It’s not a commonsense to audience.\n9. Line 301: Sequential computation is never explained in the main paper.\n10. Line 308-310: I would recommend to draw a figure for grouping and varlen technique.\n11. In Table 1, linear attention models are worse than Transformer. Is it because language modeling requires more pairwise interactions? Then why SSE becomes on par with Transformer in Table 3 & 4?\n12. Figure 5: Which task is it?\n13. Figure 6: Does SSE-Shared refer to the always-selected partition? Please be consistent in the terms.\n14. Line 457: Is the sparsity measured within the selected partitions? For n4k1, I understand the upper bound of sparsity for all partitions should be 25%, right?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":4},"strengths":{"value":"1. This paper tackles the memory capacity problem, a key issue in linear attention models with clear motivations. The row-wise sparse update aims to improve the utilization of a fixed size state space, while the SSE extends the memory capacity without increasing the number of parameters.\n2. Experiments are thorough and solid. The authors visualizes the cosine similarity of the state space and show that row-sparse update significantly improves state space utilization compared to existing designs. The final SSE model is evaluated against Transformer and linear attention baselines on a wide range of benchmarks, with a rigorous setup of fixed number of parameters.\n3. The efficiency of SSE is not confined to theory. The authors implemented SSE by grouping tokens into subsequences according to their partitions and executing them with a linear attention kernel call. This makes SSE practical for real-world long context use."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The writing of this paper may be largely improved. The authors used quite a few terminologies or preliminaries without enough explanations, which make this paper hard to follow. For example, the authors didn’t explain how they computed the cosine similarity in Figure 1. Line 176 compares SSE against gated linear attention, but the form of gated linear attention is never mentioned in the paper. There aren’t ground truth classes nor a classification task, but the authors keep using the term classification to refer to state space utilization. In Line 245, the term segmented clustering is used, but this isn’t a commonsense for audience. Captions of figure 1 and 3 need to be extended with their implications. See more in questions.\n2. The title doesn’t exactly reflect the contribution of this paper. “Scaling linear attention\" sounds like this paper studies the model performance under different linear attention sizes. I would suggest modifying it to be “Extending linear attention capacity with sparse state expansion”.\n3. While SSE focuses on improving linear attention models, there isn’t any comparison of wall time for SSE and baselines. Could you please report the performance-time trade-off curve for SSE, Transformer and other linear attention models? That will help audience know which model to use given a specific context length."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762920679751,"tcdate":1761451662833,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8934/Reviewer_heMc"],"signatures":["ICLR.cc/2026/Conference/Submission8934/Reviewer_heMc"],"forum":"R6DrJ4tnGV","number":2,"license":"CC BY 4.0","cdate":1761451662833,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8934/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762920679751,"domain":"ICLR.cc/2026/Conference","replyto":"R6DrJ4tnGV","id":"2EE5YHbpQy","forumContent":{"TLDR":{"value":"SSE is a partitioned state expansion method under a row-sparse update framework for linear attention, improving retrieval and reasoning."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Linear Attention","Language Model"]},"supplementary_material":{"value":"/attachment/6e3475d7248015327464c0fe2192bc1fb8d0142d.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The Transformer architecture, despite its widespread success, struggles with long-context scenarios due to quadratic computation and linear memory growth. While various linear attention variants mitigate these efficiency constraints by compressing context into fixed-size states, they often degrade performance in tasks such as in-context retrieval and reasoning. To address this limitation and achieve more effective context compression, we propose two key innovations. First, we introduce a row-sparse update formulation for linear attention by conceptualizing state updating as information categorization. This enables sparse state updates via softmax-based top-$k$ row selection, thereby extending receptive fields and reducing information interference. Second, we present Sparse State Expansion (SSE) within the sparse framework, which expands the contextual state into multiple partitions, effectively decoupling parameter size from state capacity while maintaining the sparse row-selection paradigm. Supported by efficient parallelized implementations, our design achieves highly discriminative state representations. We extensively validate SSE in both pure linear and hybrid (SSE-H) architectures across language modeling, in-context retrieval, and mathematical reasoning benchmarks. SSE demonstrates strong retrieval performance and scales favorably with state size. Moreover, after reinforcement learning (RL) training, our 2B SSE-H model achieves state-of-the-art mathematical reasoning performance among small reasoning models, scoring 64.5 on AIME24 and 50.2 on AIME25, significantly outperforming similarly sized open-source Transformers. These results highlight SSE as a promising and efficient architecture for long-context modeling."},"_bibtex":{"value":"@inproceedings{\npan2026scaling,\ntitle={Scaling Linear Attention Capacity with Sparse State Expansion},\nauthor={Yuqi Pan and Yongqi An and Zheng Li and Yuhong Chou and Rui-Jie Zhu and Xiaohui Wang and Mingxuan Wang and Jinqiao Wang and Guoqi Li},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=R6DrJ4tnGV}\n}"},"title":{"value":"Scaling Linear Attention Capacity with Sparse State Expansion"},"pdf":{"value":"/pdf/73d9d62767b69cd0649b8caaa9d65c613f858782.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"pan|scaling_linear_attention_capacity_with_sparse_state_expansion"},"authorids":{"value":["~Yuqi_Pan2","~Yongqi_An1","~Zheng_Li48","~Yuhong_Chou1","~Rui-Jie_Zhu2","~Xiaohui_Wang4","~Mingxuan_Wang1","~Jinqiao_Wang1","~Guoqi_Li1"]},"authors":{"value":["Yuqi Pan","Yongqi An","Zheng Li","Yuhong Chou","Rui-Jie Zhu","Xiaohui Wang","Mingxuan Wang","Jinqiao Wang","Guoqi Li"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2503.18970v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"somvanshi|a_survey_on_structured_state_space_sequence_s4_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Shriyank_Somvanshi:","https://dblp.org/search/pid/api?q=author:Md._Monzurul_Islam:","https://dblp.org/search/pid/api?q=author:Mahmuda_Sultana_Mimi:","https://dblp.org/search/pid/api?q=author:Sazzad_Bin_Bashar_Polock:","~Gaurab_Chhetri1","https://dblp.org/search/pid/api?q=author:Subasish_Das:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2503.18970"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-18970,\n  publtype={informal},\n  author={Shriyank Somvanshi and Md. Monzurul Islam and Mahmuda Sultana Mimi and Sazzad Bin Bashar Polock and Gaurab Chhetri and Subasish Das},\n  title={A Survey on Structured State Space Sequence (S4) Models},\n  year={2025},\n  month={March},\n  cdate={1740787200000},\n  journal={CoRR},\n  volume={abs/2503.18970},\n  url={https://doi.org/10.48550/arXiv.2503.18970}\n}\n"},"abstract":{"value":"Recent advancements in sequence modeling have led to the emergence of Structured State Space Models (SSMs) as an efficient alternative to Recurrent Neural Networks (RNNs) and Transformers, addressing challenges in long-range dependency modeling and computational efficiency. While RNNs suffer from vanishing gradients and sequential inefficiencies, and Transformers face quadratic complexity, SSMs leverage structured recurrence and state-space representations to achieve superior long-sequence processing with linear or near-linear complexity. This survey provides a comprehensive review of SSMs, tracing their evolution from the foundational S4 model to its successors like Mamba, Simplified Structured State Space Sequence Model (S5), and Jamba, highlighting their improvements in computational efficiency, memory optimization, and inference speed. By comparing SSMs with traditional sequence models across domains such as natural language processing (NLP), speech recognition, vision, and time-series forecasting, we demonstrate their advantages in handling long-range dependencies while reducing computational overhead. Despite their potential, challenges remain in areas such as training optimization, hybrid modeling, and interpretability. This survey serves as a structured guide for researchers and practitioners, detailing the advancements, trade-offs, and future directions of SSM-based architectures in AI and deep learning."},"title":{"value":"A Survey on Structured State Space Sequence (S4) Models"},"authors":{"value":["Shriyank Somvanshi","Md. Monzurul Islam","Mahmuda Sultana Mimi","Sazzad Bin Bashar Polock","Gaurab Chhetri","Subasish Das"]}},"tmdate":1753676862294,"pdate":1735689600000,"tcdate":1753676858832,"writers":["~"],"signatures":["~Gaurab_Chhetri1"],"forum":"gztN8uXuwF","license":"CC BY-SA 4.0","number":595711,"cdate":1740787200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1753676862294,"domain":"DBLP.org","id":"gztN8uXuwF","version":2},{"content":{"summary":{"value":"This paper introduces ST-VLO, a novel visual-LiDAR odometry framework that unifies spatial and temporal feature modeling to achieve accurate and low-drift motion estimation in dynamic environments. Unlike conventional pairwise odometry methods that only process consecutive frame pairs, ST-VLO leverages multi-frame temporal information through a Mamba-based spatio-temporal fusion module called MMG (MaxPooling-Mamba-gMLP). This module jointly models correlations across visual and LiDAR modalities over time, providing richer motion context.\nTo further mitigate cumulative localization errors, the authors propose a Temporal Drift Compensation mechanism that iteratively corrects pose predictions using residuals from historical frames. In addition, a Keypoint-Aware Auxiliary Loss emphasizes static, high-salience regions to enhance feature robustness and reduce the influence of dynamic objects.\nExperiments on the KITTI and Argoverse benchmarks demonstrate that ST-VLO achieves state-of-the-art performance, reducing translation and rotation errors by about 19% and 22%, respectively, compared to the best prior deep visual-LiDAR odometry methods. The framework also runs efficiently (≈74 ms per frame), showing strong potential for real-time autonomous driving applications."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. How is the history length (Th) chosen? How sensitive is performance to this parameter?\n2. How would ST-VLO handle large temporal gaps or asynchronous visual/LiDAR frequencies?\n3. Does the system require calibration refinement during training, or is extrinsic calibration fixed?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The proposed MMG (MaxPooling-Mamba-gMLP) module is an interesting design that harmonizes visual and LiDAR features across space and time. The formulation is well motivated by recent developments in SSMs and avoids the high computational burden of Transformers.\n2. The compensation mechanism that iteratively refines poses over historical windows is original and practically valuable for odometry tasks. It provides an elegant, differentiable alternative to traditional loop closure. \n3. The real-time efficiency is remarkable. Achieving 74 ms inference per frame on a 4090 GPU (≈13 Hz) demonstrates strong engineering quality and practical relevance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. It is an interesting endeavor to apply Mamba to odometry estimation. However, while Mamba-based modeling and temporal compensation are well-executed, each component is adapted from existing paradigms (Deformable DETR, PWC-LO, Mamba SSM). The overall contribution is more of an incremental integration than a fundamentally new principle. \n\n2. Table 6 evaluates the three main modules but lacks quantitative breakdowns for parameters like temporal window sizes (Th, Tg), compensation frequency, or loss weighting. These details are essential for understanding the stability and generality of the approach. Moreover, the performance gaps between different ablation variants are not very appreciable.\n\n3. The pipeline description is dense and formula-heavy; key intuitions behind design decisions (e.g., why use MaxPooling in MMG, why gMLP before Mamba) are missing. The paper would benefit from schematic clarity or pseudocode.\n\n4. Figure 6 highlights static regions for keypoints but doesn’t quantify how this improves robustness in dynamic scenes (e.g., removing moving object influence).\n\n5. The discussion on unsupervised methods is missing. I have listed a few for reference: \n[1] Generalizing Unsupervised Lidar Odometry Model from Normal to Snowy Weather Conditions\n[2] Self-supervised learning of lidar odometry for robotic applications\n[3] Robust selfsupervised lidar odometry via representative structure discovery and 3d inherent error modeling,\n[4] Hpplo-net: Unsupervised lidar odometry using a hierarchical point-to-plane solve"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915832923,"tcdate":1761444830019,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1611/Reviewer_vUSs"],"signatures":["ICLR.cc/2026/Conference/Submission1611/Reviewer_vUSs"],"forum":"XwFXJ38jET","number":1,"license":"CC BY 4.0","cdate":1761444830019,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1611/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915832923,"domain":"ICLR.cc/2026/Conference","replyto":"XwFXJ38jET","id":"K6aNvxJeN2","forumContent":{"TLDR":{"value":"We propose an effective and efficient visual-LiDAR odometry framework named ST-VLO, which establishes the unified spatio-temporal correlation with Mamba models and addresses the long-standing cumulative drift problem with temporal compensation."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Robot Localization","Visual-LiDAR Odometry","SLAM"]},"supplementary_material":{"value":"/attachment/2e30a5d18282c243952e26080c102d721a26f934.zip"},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"We propose an effective and efficient visual-LiDAR odometry framework named ST-VLO, which establishes the unified spatio-temporal correlation with Mamba models and addresses the long-standing cumulative drift problem with temporal compensation for the localization in 4D dynamic environments. \nSpecifically, ST-VLO includes a novel unified spatial-temporal correlation module established on Mamba to fuse heterogeneous visual and LiDAR information across multi-frame video clips, overcoming the insufficient temporal information exploration in previous pairwise odometry methods. Furthermore, a Temporal Drift Compensation module is designed to minimize cumulative drifts by iteratively learning correction residuals from multiple history frames. To strengthen the spatial feature representation on salient features, we also propose a Keypoint-Aware Auxiliary Loss with a winner-takes-all strategy.\nST-VLO achieves state-of-the-art performance on two commonly-used autonomous driving datasets, surpassing previous methods with a 19\\% \\( t_{rel} \\) and 22\\% \\( r_{rel} \\) reduction on KITTI, and a 18\\% ATE and 16\\% RPE reduction on Argoverse."},"_bibtex":{"value":"@misc{\nliu2025stvlo,\ntitle={{ST}-{VLO}: Unified Spatio-Temporal Correlation for Visual-Li{DAR} Odometry with Temporal Drift Compensation},\nauthor={Mengmeng Liu and Jiuming Liu and Michael Ying Yang and Chaokang Jiang and Jiangtao Li and Yunpeng Zhang and Hesheng Wang and Francesco Nex and Hao Cheng},\nyear={2025},\nurl={https://openreview.net/forum?id=XwFXJ38jET}\n}"},"title":{"value":"ST-VLO: Unified Spatio-Temporal Correlation for Visual-LiDAR Odometry with Temporal Drift Compensation"},"pdf":{"value":"/pdf/13b0daf705357d333b50ccb130ec4ea621c7318e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"liu|stvlo_unified_spatiotemporal_correlation_for_visuallidar_odometry_with_temporal_drift_compensation"},"authorids":{"value":["~Mengmeng_Liu2","~Jiuming_Liu1","~Michael_Ying_Yang2","~Chaokang_Jiang1","~Jiangtao_Li1","~Yunpeng_Zhang2","~Hesheng_Wang1","~Francesco_Nex2","~Hao_Cheng12"]},"authors":{"value":["Mengmeng Liu","Jiuming Liu","Michael Ying Yang","Chaokang Jiang","Jiangtao Li","Yunpeng Zhang","Hesheng Wang","Francesco Nex","Hao Cheng"]}},"version":2},{"content":{"venue":{"value":"ENLSP 2024"},"pdf":{"value":"https://raw.githubusercontent.com/mlresearch/v262/main/assets/qiao24a/qiao24a.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"qiao|vlmamba_exploring_state_space_models_for_multimodal_learning"},"html":{"value":"https://proceedings.mlr.press/v262/qiao24a.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/enlsp/QiaoYZCSGWL24,\n  author={Yanyuan Qiao and Zheng Yu and Zijia Zhao and Sihan Chen and Mingzhen Sun and Longteng Guo and Qi Wu and Jing Liu},\n  title={VL-Mamba: Exploring State Space Models for Multimodal Learning},\n  year={2024},\n  cdate={1704067200000},\n  pages={102-113},\n  url={https://proceedings.mlr.press/v262/qiao24a.html},\n  booktitle={ENLSP},\n  crossref={conf/enlsp/2024}\n}\n"},"abstract":{"value":"Multimodal large language models (MLLMs) have gained considerable attention due to their ability to integrate visual and textual information, enhancing understanding and providing context for complex tasks. While Transformer-based architectures have been the dominant framework for MLLMs, recent studies suggest that state space models (SSMs) like Mamba can achieve competitive or even superior performance. However, no prior research has investigated the potential of SSMs to replace Transformers in multimodal tasks, which are inherently more challenging due to the heterogeneity of visual and language data and the complexities of aligning these modalities. In this paper, we introduce VL-Mamba, the first study to explore the application of state space models in multimodal learning tasks. VL-Mamba leverages a pretrained Mamba language model as its core, and we propose a novel MultiModal Connector (MMC) that incorporates a Vision Selective Scan (VSS) module to improve visual sequence modeling. We empirically explore how to effectively apply the 2D vision selective scan mechanism for multimodal learning and the combinations of different vision encoders and variants of pretrained Mamba language models. Our experiments across multiple multimodal benchmarks demonstrate that VL-Mamba achieves competitive performance against small MLLMs of similar size, and in some cases, surpasses larger models such as the 7B and 13B versions of LLaVA-1.5. These results suggest that state space models have the potential to serve as an alternative to Transformers in multimodal learning tasks."},"title":{"value":"VL-Mamba: Exploring State Space Models for Multimodal Learning"},"authors":{"value":[{"fullname":"Yanyuan Qiao","username":""},{"fullname":"Zheng Yu","username":"~Zheng_Yu2"},{"fullname":"Zijia Zhao","username":""},{"fullname":"Sihan Chen","username":""},{"fullname":"Mingzhen Sun","username":""},{"fullname":"Longteng Guo","username":""},{"fullname":"Qi Wu","username":""},{"fullname":"Jing Liu","username":""}]}},"tmdate":1784537495305,"pdate":1735603200000,"externalIds":["dblp:conf/enlsp/QiaoYZCSGWL24"],"tcdate":1784537493450,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Zheng_Yu2"],"forum":"iD1kWDerrq","license":"CC BY-SA 4.0","number":68063,"cdate":1704067200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784537495305,"domain":"OpenReview.net/Public_Article","id":"iD1kWDerrq","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2405.16605v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"han|demystify_mamba_in_vision_a_linear_attention_perspective"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Dongchen_Han:","https://dblp.org/search/pid/api?q=author:Ziyi_Wang:","https://dblp.org/search/pid/api?q=author:Zhuofan_Xia:","https://dblp.org/search/pid/api?q=author:Yizeng_Han:","https://dblp.org/search/pid/api?q=author:Yifan_Pu:","~Chunjiang_Ge1","https://dblp.org/search/pid/api?q=author:Jun_Song:","https://dblp.org/search/pid/api?q=author:Shiji_Song:","https://dblp.org/search/pid/api?q=author:Bo_Zheng:","~Gao_Huang1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2405.16605"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2405-16605,\n  publtype={informal},\n  author={Dongchen Han and Ziyi Wang and Zhuofan Xia and Yizeng Han and Yifan Pu and Chunjiang Ge and Jun Song and Shiji Song and Bo Zheng and Gao Huang},\n  title={Demystify Mamba in Vision: A Linear Attention Perspective},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2405.16605},\n  url={https://doi.org/10.48550/arXiv.2405.16605}\n}\n"},"abstract":{"value":"Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba's success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba's success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-Inspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https://github.com/LeapLabTHU/MLLA."},"title":{"value":"Demystify Mamba in Vision: A Linear Attention Perspective"},"authors":{"value":["Dongchen Han","Ziyi Wang","Zhuofan Xia","Yizeng Han","Yifan Pu","Chunjiang Ge","Jun Song","Shiji Song","Bo Zheng","Gao Huang"]}},"tmdate":1760579127211,"pdate":1704067200000,"tcdate":1736663700774,"writers":["~"],"signatures":["~Chunjiang_Ge1"],"forum":"HPS5fLi7MC","license":"CC BY-SA 4.0","number":264766,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1760579127211,"domain":"DBLP.org","id":"HPS5fLi7MC","version":2},{"content":{"venue":{"value":"NeurIPS 2024"},"pdf":{"value":"https://proceedings.neurips.cc/paper_files/paper/2024/file/e618724ac897c6cf3fbfb273f8695d67-Paper-Conference.pdf"},"venueid":{"value":"dblp.org/conf/NIPS/2024"},"paperhash":{"value":"han|demystify_mamba_in_vision_a_linear_attention_perspective"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Dongchen_Han:","https://dblp.org/search/pid/api?q=author:Ziyi_Wang:","https://dblp.org/search/pid/api?q=author:Zhuofan_Xia:","https://dblp.org/search/pid/api?q=author:Yizeng_Han:","https://dblp.org/search/pid/api?q=author:Yifan_Pu:","https://dblp.org/search/pid/api?q=author:Chunjiang_Ge:","","https://dblp.org/search/pid/api?q=author:Shiji_Song:","https://dblp.org/search/pid/api?q=author:Bo_Zheng:","~Gao_Huang1"]},"html":{"value":"http://papers.nips.cc/paper_files/paper/2024/hash/e618724ac897c6cf3fbfb273f8695d67-Abstract-Conference.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/nips/HanWXHPGSSZ024,\n  author={Dongchen Han and Ziyi Wang and Zhuofan Xia and Yizeng Han and Yifan Pu and Chunjiang Ge and Jun Song and Shiji Song and Bo Zheng and Gao Huang},\n  title={Demystify Mamba in Vision: A Linear Attention Perspective},\n  year={2024},\n  cdate={1704067200000},\n  url={http://papers.nips.cc/paper_files/paper/2024/hash/e618724ac897c6cf3fbfb273f8695d67-Abstract-Conference.html},\n  booktitle={NeurIPS},\n  crossref={conf/nips/2024}\n}\n"},"abstract":{"value":"Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba’s success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba’s success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-Inspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https://github.com/LeapLabTHU/MLLA."},"title":{"value":"Demystify Mamba in Vision: A Linear Attention Perspective"},"authors":{"value":["Dongchen Han","Ziyi Wang","Zhuofan Xia","Yizeng Han","Yifan Pu","Chunjiang Ge","Jun Song","Shiji Song","Bo Zheng","Gao Huang"]}},"tmdate":1760579124827,"pdate":1704067200000,"externalIds":["dblp:conf/nips/HanWXHPGSSZ024"],"tcdate":1760158008642,"writers":["~"],"signatures":["~Jun_Song5"],"forum":"MCTenjqW0S","license":"CC BY-SA 4.0","number":642754,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1760579124827,"domain":"DBLP.org","id":"MCTenjqW0S","version":2},{"content":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","Linear Attention Transformer"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba’s success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba’s success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-\nInspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https://github.com/LeapLabTHU/MLLA."},"_bibtex":{"value":"@inproceedings{\nhan2024demystify,\ntitle={Demystify Mamba in Vision: A Linear Attention Perspective},\nauthor={Dongchen Han and Ziyi Wang and Zhuofan Xia and Yizeng Han and Yifan Pu and Chunjiang Ge and Jun Song and Shiji Song and Bo Zheng and Gao Huang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=LvJ1R88KAk}\n}"},"title":{"value":"Demystify Mamba in Vision: A Linear Attention Perspective"},"pdf":{"value":"/pdf/419f5b34c2a7aefa14ac7e991c160ab502bd73f5.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"han|demystify_mamba_in_vision_a_linear_attention_perspective"},"authorids":{"value":["~Dongchen_Han1","~Ziyi_Wang21","~Zhuofan_Xia2","~Yizeng_Han1","~Yifan_Pu1","~Chunjiang_Ge1","~Jun_Song5","~Shiji_Song1","~Bo_Zheng5","~Gao_Huang1"]},"authors":{"value":["Dongchen Han","Ziyi Wang","Zhuofan Xia","Yizeng Han","Yifan Pu","Chunjiang Ge","Jun Song","Shiji Song","Bo Zheng","Gao Huang"]}},"tmdate":1736600097600,"pdate":1727287766591,"tcdate":1715491786004,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission4979/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission4979/Authors"],"forum":"LvJ1R88KAk","license":"CC BY 4.0","number":4979,"cdate":1715491786004,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/-/Submission","NeurIPS.cc/2024/Conference/-/Edit","NeurIPS.cc/2024/Conference/-/Post_Submission","NeurIPS.cc/2024/Conference/Submission4979/-/Revision","NeurIPS.cc/2024/Conference/Submission4979/-/Camera_Ready_Revision"],"mdate":1736600097600,"odate":1730873880816,"domain":"NeurIPS.cc/2024/Conference","id":"LvJ1R88KAk","version":2},{"content":{"summary":{"value":"The paper presents a series of hybrid state-space models (SSMs), inspired by the design of convnets. It achieves this by generalizing 1D kernels to 2D and designing the depthwise, group, and full convolution counterparts for SSMs. Experiments on several sequence modeling tasks validate the effectiveness of the proposed method."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"My concerns are listed in the weaknesses section. \n\nI have one additional question: For the generalized SSM blocks with group/full convolution patterns, can I understand that these blocks integrate both token-mixing and channel-mixing within a single block, making other channel-mixing blocks, such as FFNs, unnecessary?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The concept of generalized SSM blocks, configurable with flexible connectivity structures, demonstrates good soundness and could complement existing SSM designs.\n\n2. The proposed method is supported by both theoretical analysis and empirical experiments."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The major concern is that the real-device training/inference efficiency of the proposed generalized SSM blocks is not provided. It is unclear whether the flexibility of these building blocks comes at the cost of reduced real-device efficiency.\n\n2. Another major concern is that the position of the proposed method among the latest SOTA SSMs is unclear. For example, Mamba has introduced input adaptivity in their selective state design and employed an advanced macro-structure with gating mechanisms and FFNs to enhance channel mixing in addition to token mixing. These advancements also address the limited expressiveness of vanilla SSMs, which are mostly depthwise-separable. The authors are expected to demonstrate whether the proposed method is a better choice or is orthogonal and can be combined with these advanced designs.\n\n3. The proposed method is evaluated on small-scale sequence modeling tasks. It would be highly desirable for it to be evaluated on language modeling and commonsense reasoning tasks.\n\n4. The writing clarity could be improved by better explaining certain terms. For example, the concept of \"depthwise-separable\" in the context of SSMs is important to the logic of this paper and should be better clarified."}},"nonreaders":[],"tmdate":1732612422974,"tcdate":1730819817215,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5569/Reviewer_Td3n"],"signatures":["ICLR.cc/2025/Conference/Submission5569/Reviewer_Td3n"],"forum":"PkpNRmBZ32","number":4,"license":"CC BY 4.0","cdate":1730819817215,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5569/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732612422974,"domain":"ICLR.cc/2025/Conference","replyto":"PkpNRmBZ32","id":"yjUulr43QJ","forumContent":{"TLDR":{"value":"using deep SSMs like ConvNets to do audio processing"},"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["state-space models; convolution; tensor networks; audio processing; speech recognition"]},"supplementary_material":{"value":"/attachment/fa4ef6cda9d84c821dc523c54c102112f6637a45.zip"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We introduce Centaurus, a class of networks composed of generalized state-space model (SSM) blocks, where the SSM operations can be treated as tensor contractions during training. The optimal order of tensor contractions can then be systematically determined for every SSM block to maximize training efficiency. This allows more flexibility in designing SSM blocks beyond the depthwise-separable configuration commonly implemented. The new design choices will take inspiration from classical convolutional blocks including group convolutions, full convolutions, and bottleneck blocks. We architect the Centaurus network with a mixture of these blocks, to balance between network size and performance, as well as memory and computational efficiency during both training and inference. We show that this heterogeneous network design outperforms its homogeneous counterparts in raw audio processing tasks including keyword spotting, speech denoising, and automatic speech recognition (ASR). For ASR, Centaurus is the first network with competitive performance that can be made fully state-space based, without using any nonlinear recurrence (LSTMs), explicit convolutions (CNNs), or (surrogate) attention mechanism."},"_bibtex":{"value":"@inproceedings{\npei2025let,\ntitle={Let {SSM}s be ConvNets: State-space Modeling with Optimal Tensor Contractions},\nauthor={Yan Ru Pei},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=PkpNRmBZ32}\n}"},"title":{"value":"Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions"},"pdf":{"value":"/pdf/caca2cd85f75e7097a42034b5e78add24868bd37.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"pei|let_ssms_be_convnets_statespace_modeling_with_optimal_tensor_contractions"},"authorids":{"value":["~Yan_Ru_Pei1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yan Ru Pei"]}},"version":2},{"content":{"summary":{"value":"This proposal outlines a project to automate the detection of exoplanet transits from stellar light curves using Time-Series Intelligence techniques. The authors specifically target smaller exoplanets with weak or non-periodic signals, which are notoriously difficult to detect using existing architectures. The methodology proposes using a hybrid Mamba and Attention architecture, cross-domain training on Kepler and TESS data, and data augmentation with synthetic data. The data will be sourced from the MAST archives utilizing Target Pixel Files (TPF) and Full Frame Images (FFI)."},"strengths":{"value":"The proposal provides an excellent, comprehensive background on transit photometry, including a clear explanation of false positives like eclipsing binaries and stellar variability. \nThe metrics (SNR, SDE, SES, MES) and diagnostic tests (centroid analysis, difference image) are well-defined and demonstrate a deep understanding of the astronomical domain. \nThe proposed methodology introduces a modern approach by suggesting a Mamba-Attention hybrid, which is highly relevant for capturing the nuances of astronomical time series.  \nThe inclusion of cross-domain training using both Kepler and TESS datasets is a strong strategy to reduce instrumental bias and improve generalization."},"weaknesses":{"value":"While the proposal mentions using a \"Mamba and Attention hybrid architecture,\" it lacks technical details on exactly how these two distinct sequence modeling paradigms will be integrated."},"confidence":{"value":"4: Confident"},"questions":{"value":"How specifically do you plan to combine the Mamba and Attention modules within your proposed architecture to handle these specific light curves?   \n\nRegarding data augmentation, what specific generative techniques will be used to create realistic \"artificial data\" and inject false labels without introducing unrealistic artifacts into the training set?"},"rating":{"value":"8: Very Good"}},"nonreaders":[],"tmdate":1779869441373,"tcdate":1779022714705,"writers":["tsinghua.edu.cn/THU/2026/Spring/ANM"],"signatures":["tsinghua.edu.cn/THU/2026/Spring/ANM/Submission8/Reviewer_Syd7"],"forum":"1Per9rP2px","number":1,"license":"CC BY 4.0","cdate":1779022714705,"readers":["everyone"],"invitations":["tsinghua.edu.cn/THU/2026/Spring/ANM/Submission8/Reviewer_Syd7/-/Official_Review"],"mdate":1779869441373,"domain":"tsinghua.edu.cn/THU/2026/Spring/ANM","replyto":"1Per9rP2px","id":"TkruG6Rd6b","forumContent":{"venue":{"value":"THU 2026 Spring ANM Submission"},"pdf":{"value":"/pdf/c01c4b37fd02f5360173780690808a2cb6a50a62.pdf"},"keywords":{"value":["Time Series Intelligence","Transit Photometry"]},"venueid":{"value":"tsinghua.edu.cn/THU/2026/Spring/ANM/Submission"},"paperhash":{"value":"dallamico|automated_exoplanet_transit_detection_from_stellar_light_curves"},"authorids":{"value":["~Tommaso_Dall'Amico1","~Mikhail_Kuvakin1","~Lamat_Foschini_Antichi1"]},"abstract":{"value":"The discovery of exoplanets is a crucial frontier in modern astrophysics. It expands our understanding\nof planetary formation, contextualizes the uniqueness of our own solar system, and drives the search\nfor potentially habitable worlds. Just as ancient explorers like Herodotus gathered knowledge about\nislands of habitability outside the known world, modern astronomers seek out habitable zones beyond\nour local cosmic neighborhood. Until recent years, the field relied heavily on classical approaches\nand manual interpretation of potential exoplanet candidates. These methods are labor-intensive,\nsubject to human error, and often struggle to isolate weak signals hidden in noise. The recent\nintroduction of Machine Learning has driven significant progress, leading to the automated discovery\nof many new potential exoplanets. In this course project, we aim to contribute to this intersection of\nPlanetary Science and ML by bringing state-of-the-art Time-Series Intelligence techniques into the\nexisting framework of exoplanet discovery. Specifically, we plan to focus on the automated search for\nsmaller exoplanets with weak or non-periodic transit signals, which are currently difficult for existing\narchitectures to recognize."},"_bibtex":{"value":"@inproceedings{\nkuvakin2026automated,\ntitle={Automated Exoplanet Transit Detection from Stellar Light Curves},\nauthor={Mikhail Kuvakin},\nbooktitle={Submitted to Advanced Networking Management (Time Series Intelligence) - Spring 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=1Per9rP2px},\nnote={under review}\n}"},"title":{"value":"Automated Exoplanet Transit Detection from Stellar Light Curves"},"authors":{"value":["Tommaso Dall'Amico","Mikhail Kuvakin","Lamat Foschini Antichi"]}},"version":2},{"content":{"TLDR":{"value":"We propose CASSM, a content-aware state space model that groups and reorders visual tokens based on learned content relationships, enabling Mamba to model interactions beyond conventional spatial scanning for UHD low-light image enhancement."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Low-level vision","Low-light image enhancement","Ultra-High-Definition image restoration"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Mamba-based architectures have shown promising results in low-light image enhancement by modeling visual tokens through selective, linear-complexity scanning. However, most existing approaches rely on predefined spatial scanning strategies that determine token order mainly based on spatial locations, ignoring content similarity among visual tokens. Consequently, content-similar yet spatially distant tokens may be placed far apart in the scanning sequence, limiting effective long-range dependency modeling, particularly for ultra-high-definition (UHD) images with millions of pixels. To address this limitation, we propose a Content-Aware State Space Model (CASSM) for UHD low-light image enhancement, which constructs content-aware visual sequences with linear complexity. Specifically, CASSM leverages a learned low-frequency codebook and a lightweight feature alignment adapter to efficiently generate a content-aware assignment map. Guided by these assignments, the Content-Aware Mamba Block performs token-level content grouping to gather content-similar tokens, followed by group-level reordering using 1D t-SNE projections of the codebook entries. This hierarchical organization brings spatially distant yet visually similar tokens closer in the resulting sequence, enabling more effective state-space modeling. Extensive experiments demonstrate that CASSM achieves an average PSNR improvement of 0.75 dB over the state-of-the-art methods across four benchmarks."},"_bibtex":{"value":"@inproceedings{\nanonymous2026cassm,\ntitle={{CASSM}: Content-Aware State Space Modeling for Ultra-High-Definition Low-Light Image Enhancement},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=ftRKIj2h41},\nnote={under review}\n}"},"title":{"value":"CASSM: Content-Aware State Space Modeling for Ultra-High-Definition Low-Light Image Enhancement"},"pdf":{"value":"/pdf/43dafbc8fa22651c06f71fa66132feeb81290830.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791227391858,"tcdate":1789289678456,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission15857/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission15857/Authors"],"forum":"ftRKIj2h41","license":"CC BY 4.0","number":15857,"cdate":1789289678456,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission15857/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791227391858,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"ftRKIj2h41","version":2},{"content":{"venue":{"value":"TechRxiv"},"pdf":{"value":"/pdf/dba8e9b075b00d4871d3c09b9df3a7ad02aecb72.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"anand|implementation_and_evaluation_of_mamba_selective_state_space_models_for_shakespearian_text_synthesis"},"authorids":{"value":["~Suryanshu_Anand1"]},"html":{"value":"https://doi.org/10.36227/techrxiv.177078556.69521973/v1"},"abstract":{"value":"This study evaluates the training efficiency and scaling behaviour of the Mamba Selective State Space Model (SSM). We implement a character-level language model trained on the TinyShakespeare dataset using JAX and the experimental Metal backend on an Apple M2 Pro. Our findings demonstrate near-perfect linear scaling (50.5%) when doubling model depth and identify critical hardware-constrained hyperparameter boundaries to prevent GPU buffer errors."},"title":{"value":"Implementation and Evaluation of Mamba Selective State Space Models for Shakespearian Text Synthesis"},"authors":{"value":["Suryanshu Anand"]}},"tmdate":1778058497170,"pdate":1770748200000,"tcdate":1778058497170,"writers":["~Suryanshu_Anand1"],"signatures":["~Suryanshu_Anand1"],"forum":"5aDqvoorwh","license":"CC BY 4.0","number":48952,"cdate":1778058497170,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1778058497170,"domain":"OpenReview.net/Archive","id":"5aDqvoorwh","version":2},{"content":{"summary":{"value":"This paper introduces CYLON,  a hardware-aware linear attention architecture, which partitions the recurrent state across the registers of multiple GPU processors and asynchronously combines the partitions. When applying CYLON linear attention to popular architectures, such as Hedgehog and Mamba-2,  it unlocks 3× higher throughput compared to prior linear attention algorithms for these architectures on both Hopper and Blackwell GPUs. Additionally, the proposed  CYLON makes large states available to model designers by unlocking\nsizes (e.g., ≥ 131 MB) that are not by the existing linear attention kernels."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. The paper claims that asynchronous updates between queries and key-values improve efficiency and representation quality. Could the authors elaborate on the mechanistic reason behind this improvement? For instance, does asynchrony introduce beneficial temporal diversity or implicit regularization, or does it merely approximate a form of delayed gradient update? Some empirical or theoretical clarification would help justify why asynchrony leads to better results beyond computational scheduling advantages.\n\n2. Since state-space models (e.g., Mamba, Hyena) also enable efficient long-sequence modeling with sub-quadratic complexity, how does CYLON compare conceptually and practically to these approaches?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. The paper introduces an asynchronous linear attention mechanism that allows decoupling of query and key-value updates. This improves efficiency and makes the model more flexible in streaming or autoregressive scenarios—an interesting departure from traditional synchronous designs.\n\n2. The formulation appears mathematically grounded, with clear derivations showing how the asynchronous update maintains stability and approximate equivalence to standard attention under certain conditions.\n\n3. The paper includes solid experimental validation on multiple benchmarks, showing that CYLON achieves better or comparable performance with less computation and memory, demonstrating both effectiveness and scalability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Most experiments are on standard benchmarks. The benefits of asynchrony (e.g., in real-time or streaming settings) are not thoroughly demonstrated in practical large-scale or latency-critical applications\n'\n2. The intuition behind why asynchronous updates yield better representations is somewhat under-explained. The paper could clarify whether the asynchronous mechanism introduces a temporal inductive bias, improves stability, or reduces redundancy — and under what conditions this matters. It would be better to include visualizations (e.g., attention maps over time, update trajectories) or toy examples could help convey the underlying intuition.\n\n3. While comparisons with standard linear attention methods are included, it may lack evaluation against the latest state-space models (like Mamba or Hyena), which could provide a stronger empirical contrast to justify the novelty and advantage."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942902019,"tcdate":1761964151873,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission24023/Reviewer_mZng"],"signatures":["ICLR.cc/2026/Conference/Submission24023/Reviewer_mZng"],"forum":"va4FWa5TDN","number":2,"license":"CC BY 4.0","cdate":1761964151873,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission24023/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942902019,"domain":"ICLR.cc/2026/Conference","replyto":"va4FWa5TDN","id":"pPag3SYYdx","forumContent":{"TLDR":{"value":"Hardware-aware linear attention algorithm for large state sizes."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["systems","recurrent","architectures","kernels","cuda"]},"supplementary_material":{"value":"/attachment/e29454619ea90326bcf1a9a03536ce2321ccf44c.zip"},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Sequence models face stark tradeoffs between recall quality and memory efficiency. Recall -- the ability to use information over long sequences -- is critical for sequence modeling tasks ranging from information extraction to reasoning. \nPrior work has shown that in theory, \\textit{linear} attention models with sufficient recurrent state sizes can expand the Pareto frontier of the recall-memory tradeoff space beyond alternative architectures such as softmax attention and state space models.\nHowever, it is difficult to scale the linear attention state size due to hardware bottlenecks. I/O aware algorithms store the linear attention states in thread registers, however state sizes beyond $\\approx 3$ megabytes exhaust register memory and trigger expensive register spills. \nIn this work, we introduce CYLON, a hardware-aware strategy for partitioning linear attention's recurrent state across the registers of multiple GPU processors and asynchronously combining the partitions. When applying CYLON to popular architectures, such as Hedgehog and Mamba-2, we unlock 3$\\times$ higher throughput compared to prior linear attention algorithms for these architectures on both Hopper and Blackwell GPUs. Finally, CYLON makes large states available to model designers by unlocking sizes (e.g., $\\geq$ 131 MB) that are not achievable by the existing linear attention kernels."},"_bibtex":{"value":"@misc{\nwaitz2026cylon,\ntitle={Cylon: Asynchronous Linear Attention},\nauthor={Alexander Waitz and Benjamin Frederick Spector and Christopher Re and Simran Arora},\nyear={2026},\nurl={https://openreview.net/forum?id=va4FWa5TDN}\n}"},"title":{"value":"Cylon: Asynchronous Linear Attention"},"pdf":{"value":"/pdf/7f2eee22004cc8e3675440d39926a9b5f5d263d2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"waitz|cylon_asynchronous_linear_attention"},"authorids":{"value":["~Alexander_Waitz1","~Benjamin_Frederick_Spector1","~Christopher_Re1","~Simran_Arora1"]},"authors":{"value":["Alexander Waitz","Benjamin Frederick Spector","Christopher Re","Simran Arora"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.11451v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"liu|bidirectional_gated_mamba_for_sequential_recommendation"},"authorids":{"value":["~Ziwei_Liu9","~Qidong_Liu2","~Yejing_Wang1","~Wanyu_Wang1","~Pengyue_Jia1","~Maolin_Wang1","~Zitao_Liu1","~Yi_Chang4","~Xiangyu_Zhao8"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.11451"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-11451,\n  publtype={informal},\n  author={Ziwei Liu and Qidong Liu and Yejing Wang and Wanyu Wang and Pengyue Jia and Maolin Wang and Zitao Liu and Yi Chang and Xiangyu Zhao},\n  title={Bidirectional Gated Mamba for Sequential Recommendation},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.11451},\n  url={https://doi.org/10.48550/arXiv.2408.11451}\n}\n"},"abstract":{"value":"In various domains, Sequential Recommender Systems (SRS) have become essential due to their superior capability to discern intricate user preferences. Typically, SRS utilize transformer-based architectures to forecast the subsequent item within a sequence. Nevertheless, the quadratic computational complexity inherent in these models often leads to inefficiencies, hindering the achievement of real-time recommendations. Mamba, a recent advancement, has exhibited exceptional performance in time series prediction, significantly enhancing both efficiency and accuracy. However, integrating Mamba directly into SRS poses several challenges. Its inherently unidirectional nature may constrain the model's capacity to capture the full context of user-item interactions, while its instability in state estimation can compromise its ability to detect short-term patterns within interaction sequences. To overcome these issues, we introduce a new framework named Selective Gated Mamba (SIGMA) for Sequential Recommendation. This framework leverages a Partially Flipped Mamba (PF-Mamba) to construct a bidirectional architecture specifically tailored to improve contextual modeling. Additionally, an input-sensitive Dense Selective Gate (DS Gate) is employed to optimize directional weights and enhance the processing of sequential information in PF-Mamba. For short sequence modeling, we have also developed a Feature Extract GRU (FE-GRU) to efficiently capture short-term dependencies. Empirical results indicate that SIGMA outperforms current models on five real-world datasets. Our implementation code is available at https://github.com/ziwliu-cityu/SIMGA to ease reproducibility."},"title":{"value":"Bidirectional Gated Mamba for Sequential Recommendation"},"authors":{"value":["Ziwei Liu","Qidong Liu","Yejing Wang","Wanyu Wang","Pengyue Jia","Maolin Wang","Zitao Liu","Yi Chang","Xiangyu Zhao"]}},"tmdate":1769056872092,"pdate":1704067200000,"tcdate":1728272493573,"writers":["~"],"signatures":["~Zitao_Liu1"],"forum":"xaJx6aRwRG","license":"CC BY-SA 4.0","number":140912,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769056872092,"domain":"DBLP.org","id":"xaJx6aRwRG","version":2},{"content":{"summary":{"value":"The paper “Harnessing Untrained Dynamics: A Reservoir Computing Approach to State-Space Models” introduces the Reservoir State Space Model (RSSM), a hybrid neural architecture that merges State Space Models (SSMs) with Reservoir Computing (RC) to efficiently capture long-term dependencies in sequential data. RSSM leverages the structured linear dynamics of SSMs as a fixed, untrained convolutional reservoir, training only a lightweight feed-forward readout layer to drastically reduce computational cost and energy usage. The authors provide theoretical analyses of stability and representational capacity, demonstrating how the untrained dynamics yield rich, hierarchical temporal features. Empirical results on benchmarks such as sMNIST, psMNIST, sCIFAR-10, and Long Range Arena show that RSSM achieves accuracy comparable to state-of-the-art trainable models like S4 and GRU while offering superior efficiency and scalability, establishing it as a compelling approach for resource-constrained sequence modeling."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please address the weaknesses above."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper’s strengths lie in its thoughtful synthesis, solid theoretical framing, and focus on efficiency and sustainability, an increasingly important dimension in modern machine learning. In terms of originality, the authors take a creative step by combining State Space Models (SSMs) and Reservoir Computing (RC), two areas that have evolved largely in parallel, into a unified architecture (RSSM) that inherits the stability and long-memory properties of SSMs while preserving the training simplicity of RC. Regarding quality, the paper provides a theoretical analysis of stability and representational capacity, connecting spectral properties of SSMs to memory and expressive power in deep untrained reservoirs. The experimental section, while not exhaustive, demonstrates a convincing efficiency–accuracy trade-off, showing that RSSM can approach the performance of trainable models such as S4 while requiring significantly less computation and energy. In terms of clarity, the paper is well organized, the derivations are accessible, and the diagrams clearly illustrate architectural design choices (e.g., removal of the mixing layer, layer stacking). Finally, in terms of significance, the work contributes to the growing discourse on sustainable and low-resource AI, offering an alternative path to high-efficiency sequence modeling without massive training overhead."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While the paper presents a clean synthesis of SSMs and reservoir computing, its novelty and relevance are somewhat limited given the current landscape. The proposed RSSM builds directly on ideas already explored in DeepESNs (Gallicchio et al., 2017) and diagonal SSMs like S4D (Gupta et al., 2022), but the paper does not convincingly explain how its fixed, untrained dynamics fundamentally improve upon or differ from these prior models. Moreover, the experimental comparisons are outdated: while S4 and GRUs are useful baselines, the field has moved toward more powerful and adaptive state-space architectures such as Mamba (Gu & Dao, 2024), which combine dynamic gating with learnable state updates. It remains unclear how an untrained reservoir model like RSSM competes with these modern approaches in both accuracy and scalability. Evaluating RSSM against Mamba or related architectures—or at least discussing whether the reservoir idea could be integrated into these adaptive frameworks—would make the contribution more relevant to ICLR 2026 audiences. In addition, the theoretical results, though well presented, mostly restate known stability and frequency-domain properties from SSM literature without empirically validating how they translate to improved modeling capacity or generalization. Finally, the efficiency and sustainability claims, while appealing, rely on coarse CO₂ estimates and do not provide standardized measurement conditions or normalized baselines. To strengthen the paper, the authors should include stronger contemporary baselines (e.g., Mamba, RWKV), analyze sensitivity to spectral initialization, and clarify how untrained structured reservoirs might complement or extend recent learnable state-space models."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931618548,"tcdate":1761889618601,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19773/Reviewer_bBrt"],"signatures":["ICLR.cc/2026/Conference/Submission19773/Reviewer_bBrt"],"forum":"azNpwiObPp","number":4,"license":"CC BY 4.0","cdate":1761889618601,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19773/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931618548,"domain":"ICLR.cc/2026/Conference","replyto":"azNpwiObPp","id":"Glf0Ix291i","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["echo state networks","reservoir computing","recurrent neural networks"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"We introduce the Reservoir State Space Model (RSSM), a novel neural architecture that integrates the structured dynamics of State Space Models (SSMs) with the efficiency of reservoir computing to address long-term dependencies in sequence modeling. Leveraging the linear structure of SSMs, RSSMs implement efficient convolutional operations that maintain a latent internal state, akin to Recurrent Neural Networks (RNNs), while enabling fast and parallelizable computation. We conduct a stability analysis of the underlying SSMs to extend the memory capacity of the model, ensuring rich and expressive hidden representations.\nA key innovation of RSSM is its use of untrained, structured convolutional dynamics as a fixed reservoir, with learning confined to a lightweight feed-forward readout layer. This design drastically reduces training complexity and computational overhead, making RSSMs well-suited for low-resource or real-time applications.\nEmpirical evaluations on standard sequence modeling benchmarks demonstrate that RSSMs achieve competitive accuracy while offering significant efficiency gains compared to traditional trainable architectures. \nOur results establish RSSMs as a new class of sequence models that combines the strengths of structured, fixed dynamics with the flexibility of learned representations, offering a compelling trade-off between performance and efficiency."},"_bibtex":{"value":"@misc{\nlombardi2025harnessing,\ntitle={Harnessing Untrained Dynamics: A Reservoir Computing Approach to State-Space Models},\nauthor={Giuseppe Lombardi and Andrea Ceni and Claudio Gallicchio},\nyear={2025},\nurl={https://openreview.net/forum?id=azNpwiObPp}\n}"},"title":{"value":"Harnessing Untrained Dynamics: A Reservoir Computing Approach to State-Space Models"},"pdf":{"value":"/pdf/a190efc43a43817bee23c86366874d9f7854d99a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"lombardi|harnessing_untrained_dynamics_a_reservoir_computing_approach_to_statespace_models"},"authorids":{"value":["~Giuseppe_Lombardi1","~Andrea_Ceni1","~Claudio_Gallicchio1"]},"authors":{"value":["Giuseppe Lombardi","Andrea Ceni","Claudio Gallicchio"]}},"version":2},{"content":{"comment":{"value":"**Conclusion and Future Work**\n\nIn this paper, we propose DenseAttention Network -- a general architecture which simplifies the Transformer block and can serve as a drop-in replacement in every model architecture using it. We conduct experiments on the diverse modalities spanning from logic to language modeling and image classification and from short to extremely long sequence lengths using the LRA suite of benchmarks and MLM-style language model pre-training on text data. The results show that DenseAttention is capable of generalizing to many different tasks and context sizes and achieving favorable performance in comparison with standard Transformer and its augmented variants while being faster and more computationally efficient even with no specialized, low-level computation algorithms such as in [1].\n\nWe acknowledge that there are other modalities and specialized architectures that would benefit from long-context efficiency improvements if the DenseAttention is ported or applied to them, such as ViT [2] and SAM [3] for Computer Vision tasks, and LLAMA \n[4] for decoder-style language modeling. We hope to address them in future work. In particular, we look forward to adapting DenseAttention architecture to causal LLAMA-style LLMs and studying their scaling laws at billions of parameters range. \n\nReferences:\n\n[1] Dao et al., \"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.\" NeurIPS 2022 \n\n[2] Dosovitskiy et al., \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.\" ICLR 2021\n\n[3] Kirillov et al., \"Segment Anything.\" ICCV 2023\n\n[4] Touvron et al., \"LLaMA: Open and Efficient Foundation Language Models.\" arXiv preprint arXiv:2302.13971, 2023.\n\n---\n\n**Other sub-quadratic algorithms for sequence processing**\n\nAnother promising line of work focuses on applying deep State Space Models (SSMs) [1-4] and Linear RNNs [5-7] to long-range sequence and language modeling. Fundamentally, these architectures model interactions in sequence dimension by a linear recurrence:\n\n$\n\\begin{align*}\n&x_t = \\mathbf{\\overline{A}}x_{t-1}  + \\mathbf{\\overline{B}}u_t, \\\\\n& y_t = \\mathbf{\\overline{C}}x_t + \\mathbf{\\overline{D}}u_t,\n\\end{align*}\n$\n\nwhere recurrence matrix $\\mathbf{\\overline{A}}$ and other parameters are data-independent matrices which form and initialization are defining properties for a particular SSM/ RNN architecture. \nThe linear recurrence is advantageous during inference as it runs in $O(N)$ time. For training, it also can be unrolled into a convolutional kernel $\\mathbf{K} = \\begin{bmatrix} \\mathbf{\\overline{C}  \\overline{B}}, & \\mathbf{\\overline{C} \\overline{A} \\overline{B}}, & \\ldots, & \\mathbf{\\overline{C} \\overline{A}}^{N-1}\\mathbf{\\overline{B}} \\end{bmatrix} $ to compute $y = \\mathbf{K} * u$ via Fast Fourier Transform (FFT) in $O (N \\log N)$ time. Here, we set $\\mathbf{D}=0$ for ease of exposition, but in practice it's usually set to identity to act as a skip-connection ubiquitous in modern deep NN architectures.\n\nAmong other novel algorithms which rely on FFT or its generalizations such as Monarch matrices [8], are Long Convolutions [9], Hyena [10], and Monarch Mixer [11], with the latter using sub-quadratic primitives both for computations along the sequence length and the model dimension. \n\nWhile being sub-quadratic, these algorithms are still slower than linear time as in DenseAttention. However, recently [4] introduced data-dependent gating for SSM parameters and low-level, hardware efficient CUDA implementation for parallel-scan operation which allows for fast linear-time processing both during training and inference. And [12] adopt a similar gating mechanism for causal linear attention which allows to drop the denominator in the linear attention formula but also admits no parallel training without resorting to low-level implementations.\n\nReferences: \n\n[1] Gu et al., \"Efficiently Modeling Long Sequences with Structured State Spaces.\" ICLR 2022\n\n[2] Gupta et al., \"Diagonal State Spaces are as Effective as Structured State Spaces.\" NeurIPS 2022\n\n[3] Ma et al., \"Mega: Moving Average Equipped Gated Attention.\" ICLR 2023\n\n[4] Gu and Dao, \"Mamba: Linear-Time Sequence Modeling with Selective State Spaces.\" CoLM 2024\n\n[5] Beck et al., \"xLSTM: Extended Long Short-Term Memory.\" NeurIPS 2024\n\n[6] Orvieto et al., \"Resurrecting Recurrent Neural Networks for Long Sequences.\" ICML 2023\n\n[7] Peng et al., \"RWKV: Reinventing RNNs for the Transformer Era.\" Findings of EMNLP 2023\n\n[8] Dao et al., \"Monarch: Expressive Structured Matrices for Efficient and Accurate Training.\" ICML 2022 \n\n[9] Fu et al., \"Simple Hardware-Efficient Long Convolutions for Sequence Modeling.\" ICML 2023\n\n[10] Poli et al., \"Hyena Hierarchy: Towards Larger Convolutional Language Models.\" ICML 2023\n\n[11] Fu et al., \"Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture.\" NeurIPS 2023\n\n[12] Yang et al., \"Gated Linear Attention Transformers with Hardware-Efficient Training.\" ICML 2024"},"title":{"value":"General Response. Part 3"}},"tmdate":1733279650170,"tcdate":1733202884768,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13608/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13608/Authors"],"forum":"2bIQBDSfRk","number":42,"license":"CC BY 4.0","cdate":1733202884768,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13608/-/Official_Comment"],"mdate":1733279650170,"domain":"ICLR.cc/2025/Conference","replyto":"zDdAMW0NMx","id":"tUvBdkhdo2","forumContent":{"TLDR":{"value":"We propose a novel DenseAttention architecture which achieves 1) favorable computational efficiency;  2) linear time & space complexity by simplification and reduction of standard Transformer architecture."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["self-attention","deep learning","transformer architecture","nlp","efficient transformers","DenseAttention","long context","Long Range Arena"]},"supplementary_material":{"value":"/attachment/78e4b43b1d574d7e27aaf0271c25bf639f64491b.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The ubiquitous Transformer architecture suffers from two main bottlenecks: 1) low computational and memory efficiency, leading to suboptimal hardware utilization, and 2) quadratic time complexity with respect to sequence length $N$, making it slow and costly for large data contexts. We propose a novel DenseAttention Network architecture, a straightforward simplification of the standard Transformer block that addresses these issues and serves as a drop-in replacement for language modeling tasks. We eliminate memory-bound components in DenseAttention, including Softmax, masking, one skip connection, and both LayerNorms, as well as key, value, and output projection matrices, as they become redundant. Despite these removals, it maintains exact $N \\times N$ pairwise interactions between tokens. By exploiting the associativity of matrix multiplications, DenseAttention can be computed with $O(N^2d)$ or $O(Nd^2)$ time and space complexity, depending on the context. To handle the absence of Softmax and prevent numerical instability, we introduce MaxNormActivation at both ends of the Transformer block. We also devise Cosine Relative Positional Embeddings as a computationally efficient replacement for RoPE, and simple LocalAttention variations of the block to help the model focus on details in extremely long contexts. \n\nDenseAttention competes with FlashAttention in speed on small sequences and outperforms it by orders of magnitude on large contexts. We pre-train encoder language models on sequences up to 16K in length, which perform similarly or better than baseline BERT-large, while significantly improving speed and efficiency.  Finally, we achieve state-of-the-art on the LRA benchmark among the Transformer-based architectures."},"_bibtex":{"value":"@misc{\nargatkiny2025denseattention,\ntitle={DenseAttention: No-Compromise Exact All \\$N {\\textbackslash}times N\\$  Interactions Algorithm with \\$O(N)\\$ Space and Time Complexity},\nauthor={Andrew Argatkiny},\nyear={2025},\nurl={https://openreview.net/forum?id=2bIQBDSfRk}\n}"},"title":{"value":"DenseAttention: No-Compromise Exact All $N \\times N$  Interactions Algorithm with $O(N)$ Space and Time Complexity"},"pdf":{"value":"/pdf/efeb6787f0c4f6394dce7d2e3d58e8e6dae7e3ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"argatkiny|denseattention_nocompromise_exact_all_n_\\times_n_interactions_algorithm_with_on_space_and_time_complexity"},"authorids":{"value":["~Andrew_Argatkiny1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Andrew Argatkiny"]}},"version":2},{"content":{"decision":{"value":"Accept (regular)"},"comment":{"value":"The authors present a masked discrete diffusion language model that replaces the standard quadratic-cost Transformer backbone with bidirectional Mamba-2 layers to achieve linear-time sequence modeling. To balance efficiency with global dependency modeling, the authors also propose a hybrid variant which interleaves attention layers at fixed intervals within the Mamba stack. Evaluated at scales up to 1.3B parameters, the architecture demonstrates competitive perplexity and inference throughput compared to Transformer baselines, particularly on long sequences. The work further provides a theoretical and empirical analysis of various inference strategies.\n\nThe reviewers found the topic important and relevant to today's research efforts on DLLMs, and were convinced by the method's efficiency and performance. There were some issues raised in the initial reviews:\n- Evaluating the model only up to 1.3B parameters, whereas modern DLLM architectures are often scaled to ~30B parameters\n- The proposed method uses more parameters than the baseline Diffusion Transformers which could make an unfair comparison\n- The proposed method underperformed the standard Transformer baseline at the 240M scale\n- Lack of an ablation study to justify the fixed hybrid interval design\n- Token-level perplexity is insufficient as a generation quality metric\n- Inference speed tested at BS=1 only.\n- The reviewers disagreed about the level or novelty/originality involved in incorporating Mamba into DLLMs\n\nGenerally these issues were resolved. The reviewer who left their score as \"Weak Reject\" also noted \"all concerns resolved\".\n- While the authors did not provide new results for models beyond 1.3B, they discussed the performance trends they observed\n- Param count mismatches was acknowledged, and a new study produced to examine FLOPs per token\n- Underperformance at 240M was acknowledged and attributed to the more difficult information compression task that Mamba does, which requires larger architectures to take advantage of. This is not a fundamental reason to reject the method.\n- The choice of hybrid interval setting was explained as coming from Nemotron, and an ablation was done with new results\n- The MAUVE metric was added\n- BS=8 was tested\n- There can be some disagreement on originality, but given that the authors have appropriately acknowledged prior work, this is not a fundamental issue.\n\nSince reviewers agreed the work is sound, I am recommending acceptance.\n\nThe authors must incorporate the new studies that address reviewer comments into their final version of the work.\n\n---\nNote to authors - One reference in your paper appears to be hallucinated or incorrectly entered:\n\nGu, A. and Goel, T. D. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint\narXiv:2312.00752, 2023.\n\nAuthors who use AI tools are responsible for checking the quality of outputs. Hallucinated references are not acceptable."},"title":{"value":"Paper Decision"},"reference_correctness_check":{"value":"Some references in the submitted paper were flagged by an automated checker. As you are preparing the camera-ready version, please check the flagged references and correct any inaccuracies (authors, title, venue, arXiv ID, etc.). Note that the automated checker can produce false positives.\n\nFlagged references:\n\n- Gu, A. and Goel, T. D. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.\n  #### **Issue:** authors mismatch with arXiv"}},"parentInvitations":"ICML.cc/2026/Conference/-/Decision","nonreaders":[],"tmdate":1782344295600,"tcdate":1777565032403,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Program_Chairs"],"signatures":["ICML.cc/2026/Conference/Program_Chairs"],"forum":"hLizFuTBsA","number":1,"license":"CC BY 4.0","cdate":1777565032403,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission16782/-/Decision","ICML.cc/2026/Conference/-/Edit"],"mdate":1782344295600,"domain":"ICML.cc/2026/Conference","replyto":"hLizFuTBsA","id":"iZTeNxlHN0","forumContent":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Diffusion language models","State space models","Mamba"]},"_bibtex":{"value":"@inproceedings{\nsingh2026diffumamba,\ntitle={DiffuMamba: High-Throughput Diffusion {LM}s with Mamba Backbone},\nauthor={Vaibhav Singh and Oleksiy Ostapenko and Pierre-Andre Noel and Eugene Belilovsky and Torsten Scholak},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=hLizFuTBsA}\n}"},"title":{"value":"DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone"},"paperhash":{"value":"singh|diffumamba_highthroughput_diffusion_lms_with_mamba_backbone"},"originally_submitted_PDF":{"value":"/pdf/b1cc609bb7116035e8f92ea140a2dc75bbb8ad61.pdf"},"TLDR":{"value":"Diffusion language model with mamba backbone for faster inference at long sequence lengths."},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2× and 4.3× higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants, combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems."},"pdf":{"value":"/pdf/2a0726f8dba38b2e0a314e73cb5a94e4bfeaad5f.pdf"},"lay_summary":{"value":"Today's AI chatbots write one word at a time, like reading a book aloud,  fast for short notes but slow for long ones. Diffusion language models instead generate whole paragraphs in parallel by polishing gibberish into text, like a sculptor at work. But each polishing pass uses attention, which compares every word to every other word, making the cost explode at long lengths. We replaced attention with Mamba, an engine that reads text in a single sweep instead of cross-comparing every pair of words. Our two models: DiffuMamba and the hybrid DiffuMamba-H plug this engine into the same diffusion recipe, trained up to 1.3B parameters. On standard benchmarks, they match or beat the attention baseline, generate text up to 8.2× faster on long inputs, and stay stable at lengths where the attention baseline collapses."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Vaibhav_Singh1","~Oleksiy_Ostapenko1","~Pierre-Andre_Noel1","~Eugene_Belilovsky1","~Torsten_Scholak1"]},"authors":{"value":["Vaibhav Singh","Oleksiy Ostapenko","Pierre-Andre Noel","Eugene Belilovsky","Torsten Scholak"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a new variant of vision-language model (VLM) called MambaVLM, which introduces multiple improvements to previous VLM with Mamba method, Cobra. Specifically, the paper proposes to concat the visual features from DINOv2 and SigLIP by sequence axis instead of the channel-axis in Cobra, followed by a new Mamba-based projector. Performance is validated on various VLM benchmarks."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See weaknesses."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper is easy to understand with clear illustrations on the proposed methods.\n2. According to the experiments, MambaVLM achieves overall better performance compared to previous VLMs, such as Qwen-VL, LLaVA-1.5, and Cobra."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The novelty is limited with the following reasons: (1) The model is based on Cobra, with minor changes on the concatenation of visual features and projector. (2) The scan directions are from VMamba, only the stitch-scan is novel.\n\n2. Although the sequence-level concatenation improves the performance, it poses a great concern on the efficiency of the model, but the authors did not provide the inference speed, computational cost, and memory cost comparisons. Though the Mamba has linear computational complexity, longer sequence indeed increases the FLOPs and memory consuption, and the heavy projector also introduces additional costs. As a result, directly compare the model with existing methods such as Cobra without comparing the efficiency is **unfair**.\n\n3. In Figure 1, directly comparing LLaVA-1.5 with MambaVLM to demonstrate the effective of Mamba and the superiority on training time is unfair, as MambaVLM uses better DINOv2-SigLIP encoder.\n\n4. In lines 215~235, \"regardless of how many channels ... loss of visual information\" is overstated, lacking precise theoretical evidence to support the claims. Bottleneck-structures are widely used in networks such as ResNet, and according to information bottleneck principle, it is no clear evidence to state that the compression of channels will definitely lose the valuable information. Please reword.\n\n5. In Table 1, some results (62.6, 76.3) of MambaVLM is not the best and should not be bolded. Please correct them."}},"nonreaders":[],"tmdate":1731428096734,"tcdate":1730646686860,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4294/Reviewer_7YET"],"signatures":["ICLR.cc/2025/Conference/Submission4294/Reviewer_7YET"],"forum":"0A6f1b66pE","number":2,"license":"CC BY 4.0","cdate":1730646686860,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4294/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428096734,"domain":"ICLR.cc/2025/Conference","replyto":"0A6f1b66pE","id":"C77aBujYsP","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Vision-Language Models; Mamba;"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"While emerging multi-modal large language models (MLLM) have demonstrated impressive advances, the quadratic complexity of their Transformer-based LLMs (3B or larger) inevitably leads to considerable computational overhead. On the other hand, the recently proposed selective state space model (i.e., Mamba) enjoys both model capacity and computational efficiency, making it an ideal component to enhance MLLM's efficiency and performance. However, recent attempts to introduce Mamba into MLLMs simply replace their LLMs with Mamba, ignoring the unique characteristics of either side. We argue that such a naive combination cannot exhibit the potential of Mamba in MLLMs. In this paper, we delve into harnessing Mamba's unique properties, and propose tailored designs from both multi-modal input and architectural perspectives to unleash its true power. First, we fully utilize Mamba's linear complexity to construct visual long sequences for a thorough perception at a minor efficiency burden. To integrate the scanning mechanism with the built visual long sequence, we devise a novel cross-stitch scanning approach to capture and fuse spatial and semantic properties simultaneously, enhancing the interaction of visual information and the vision-language alignment. Built upon these designs, we propose MambaVLM, a simple yet effective MLLM framework that exhibits highly competitive results across multiple benchmarks. Moreover, our framework is also compatible with Transformer-based LLMs (e.g., Vicuna), demonstrating remarkable training and inference efficiency. Notably, with only 0.66M data and 14 hours training on a single A800 node, our MambaVLM outperforms LLaVA-1.5 by significant margins and performs on par or even better than the 1.4B data trained Qwen-VL. The appealing results from both effectiveness and efficiency aspects indicate the promising prospects of Mamba in MLLMs."},"_bibtex":{"value":"@misc{\nchen2024unleashing,\ntitle={Unleashing the Power of Selective State Space Models in Vision-Language Models},\nauthor={Honghao Chen and Yibing Song and Shoufa Chen and Chongjian GE and Kaiqi Huang},\nyear={2024},\nurl={https://openreview.net/forum?id=0A6f1b66pE}\n}"},"title":{"value":"Unleashing the Power of Selective State Space Models in Vision-Language Models"},"pdf":{"value":"/pdf/1183a88cc6949631a83ff61a73451d323b4592a6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"chen|unleashing_the_power_of_selective_state_space_models_in_visionlanguage_models"},"authorids":{"value":["~Honghao_Chen1","~Yibing_Song1","~Shoufa_Chen1","~Chongjian_GE1","~Kaiqi_Huang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Honghao Chen","Yibing Song","Shoufa Chen","Chongjian GE","Kaiqi Huang"]}},"version":2},{"content":{"summary":{"value":"This paper builds upon the previous Markov-ICL framework to empirically and theoretically investigate the ICL ability of Mamba architecture.\nSpecifically, they design experiments to train Mamba for next-token prediction loss on random Markov chains, and find one layer Mamba match the performance of the statistical optimal estimator, and convolution layer play an important role to ahcieve this performance. Theoretically, the authors provide a constructive proof showing how Mamba's components work in concert to implement the laplace smoothingy: convolution identifies the $k^{th}$-order transition, recurrence (with $a_t \\approx 1$) accumulates all transition counts into the hidden state, and selectivity reads out the specific counts needed for the current prediction. Finally, the paper proves a lower bound showing that for Mamba the hidden dimension $d$ must scale exponentially with the Markov order $k$."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Does the result in Theorem 2 imply that the transformer is more powerful than Mamba in in-context learn the complex task (higher order Markov process)? Can we conclude transformer is more parameter-efficient than Mamba when solving complex tasks? Do authors have any insight into a way to design a Mamba block that sidesteps this trade-off?\n\n2. In the theoretical proof, the author omits the gating mechanism in Mamba, but later found in the NLP task that gating is an important component. If we want to add gating back to theoretical analysis, how does it interact with the \"counting\" mechanism described in Theorem 1?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. This is the first work to investigate theoretically the ICL ability of the Mamba architecture.\n\n2. The paper is well-written; the logic and arrangement are easy to follow, and the author provides enough background in the main text to help readers understand the scientific question being studied. It starts from an empirical observation, finds convolution plays an important role, and then echoes this importance in the theoretical analysis.\n\n3. The paper is perfectly designed and written to shed light not only on the theoretical analysis but also on the real-world application and the Mamba architecture. The theory echoes the key component and mechanism in the original Mamba paper (convolution and the selective mechanism) and is supported by experiments on synthetic data."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. One possible weakness is that, to the best of our current understanding, Mamba has not demonstrated the strong ICL ability seen in large transformer-based language models. While this is not a weakness of the theoretical analysis itself, Mamba's actual ICL capability in real-world applications will influence the impact of this theoretical work on the community.\n\n2. The real-world experiment is only on the language modeling task (e.g., perplexity on WikiText-103) instead of a dedicated, real-world ICL experiment, which limits the generality of the findings."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927221740,"tcdate":1761964804389,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17277/Reviewer_6hFD"],"signatures":["ICLR.cc/2026/Conference/Submission17277/Reviewer_6hFD"],"forum":"kmK3WSCOCT","number":2,"license":"CC BY 4.0","cdate":1761964804389,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17277/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927221740,"domain":"ICLR.cc/2026/Conference","replyto":"kmK3WSCOCT","id":"0EtcU2071F","forumContent":{"TLDR":{"value":"We uncover an interesting phenomenon where a single-layer Mamba represents the Bayes optimal Laplacian smoothing estimator when trained on Markov chains and we demonstrate it theoretically and empirically."},"venue":{"value":"ICLR 2026 Oral"},"keywords":{"value":["State-space models","Markov chains","In-context learning","Laplacian smoothing"]},"supplementary_material":{"value":"/attachment/7481132bbc16e969531b0db2b1a4e3d8dbafeefc.zip"},"primary_area":{"value":"interpretability and explainable AI"},"abstract":{"value":"While transformer-based language models have driven the AI revolution thus far, their computational complexity has spurred growing interest in viable alternatives, such as structured state space sequence models (SSMs) and Selective SSMs. Among these, Mamba (S6) and its variant Mamba-2 have shown remarkable inference speed-ups over transformers while achieving comparable or superior performance on complex language modeling tasks. However, despite these architectural innovations and empirical successes, the fundamental learning capabilities of Mamba remain poorly understood. In this paper, we address this gap by studying in-context learning (ICL) on Markov chains and uncovering an interesting phenomenon: even a single-layer Mamba efficiently learns the in-context Laplacian smoothing estimator, which is both Bayes and minimax optimal. To explain this, we theoretically characterize the representation capacity of Mamba and reveal the fundamental role of convolution in enabling it to represent the optimal Laplacian smoothing. These theoretical insights align strongly with empirical results and, to the best of our knowledge, represent the first formal connection between Mamba and optimal statistical estimators. Finally, we outline promising research directions inspired by these findings."},"_bibtex":{"value":"@inproceedings{\nbondaschi2026from,\ntitle={From Markov to Laplace: How Mamba In-Context Learns Markov Chains},\nauthor={Marco Bondaschi and Nived Rajaraman and Xiuying Wei and Razvan Pascanu and Caglar Gulcehre and Michael Gastpar and Ashok Vardhan Makkuva},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=kmK3WSCOCT}\n}"},"title":{"value":"From Markov to Laplace: How Mamba In-Context Learns Markov Chains"},"pdf":{"value":"/pdf/7c5974f3688d2ba6f4817747b376897b745b3197.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"bondaschi|from_markov_to_laplace_how_mamba_incontext_learns_markov_chains"},"authorids":{"value":["~Marco_Bondaschi1","~Nived_Rajaraman1","~Xiuying_Wei1","~Razvan_Pascanu1","~Caglar_Gulcehre1","~Michael_Gastpar1","~Ashok_Vardhan_Makkuva1"]},"authors":{"value":["Marco Bondaschi","Nived Rajaraman","Xiuying Wei","Razvan Pascanu","Caglar Gulcehre","Michael Gastpar","Ashok Vardhan Makkuva"]}},"version":2},{"content":{"summary":{"value":"This paper revisits sequence modeling for offline reinforcement learning (RL) by exploring design choices like architecture and context length across diverse datasets, moving beyond the limited evaluations typically used in this area. The paper makes use of max-return sequence modeling, which predicts maximized returns to guide actions, removing the need for manually set target returns and allowing for broader dataset applicability. Key findings highlight that dataset characteristics often impact performance more than model architecture, providing insights into when sequence modeling or traditional RL methods may be preferable."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"What dataset characteristics specifically correlate with optimal context lengths? The paper mentions variations in context length without thoroughly linking these findings to specific dataset properties. It would strengthen the findings to identify specific dataset characteristics that correlate with optimal context lengths. Again, properties such as trajectory length, state/action space dimensionality, and reward sparsity could be analyzed to determine how they influence context length choices. Exploring these aspects could reveal generalizable patterns for determining optimal context lengths in other datasets and applications, potentially providing valuable tuning guidelines.\n\nDo you anticipate that the observed effects of architecture and context length on performance will hold for other reinforcement learning tasks or domains not included in the study? Any insight into the potential for generalizing these findings would strengthen the conclusions. Or do you have any marginal evidence that might suggest this?\n\nWill you provide code, model checkpoints, or more detailed instructions for setting up experiments? This would help others replicate and validate your findings and would enhance the practical impact of your contributions."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The max-return sequence modeling framework addresses the limitation of manually specified target returns effectively.\n\nThe paper explores nine datasets, three model architectures, and various context lengths, which is a notable improvement over previous work limited to simpler datasets.\n\nThe findings on architecture and context length provide useful guidelines for practitioners selecting models for different offline RL tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"To enhance reproducibility, it would be highly beneficial to release code and resources. Specifically, providing implementations of key model components (e.g., max-return sequence modeling framework), data preprocessing scripts, and evaluation protocols would allow researchers to replicate and build on your work effectively. Including guidelines on setting hyperparameters and selecting context lengths for different dataset characteristics could further assist reproducibility and practical application.\n\nAlthough the paper mentions that dataset characteristics significantly impact model performance, it does not delve deeply into specific dataset factors that might inform architecture or hyperparameter tuning. The analysis of dataset characteristics could be deepened by exploring specific factors that might guide architecture or hyperparameter decisions. For example, aspects such as trajectory length distribution, reward sparsity, and the dimensionality of the state and action spaces could provide insights into selecting context length or architecture. A discussion on how these characteristics might impact model performance would help clarify the tuning process for specific tasks or dataset types.\n\nWhile the paper makes a valuable effort to explore underrepresented datasets, a more explicit discussion about why specific datasets were chosen and any limitations of these datasets would strengthen the robustness of the study, especially concerning sequence modeling applicability.\n\nVery minor typo at line 060: “Overall, the dataset characteristics of the have greater …”\n(you should remove “of the”)\n\nLine 067: This is very awkwardly phrased and should be more clearly stated. As is the first remark on context length being minor to performance:\n\n“The impact of context length on performance is relatively minor.”\n\nSeems contradictory to the following two sentences:\n\n “A shorter context length is more advantageous for trajectory stitching. Moreover, it is surprising to find that models trained on long sequences perform exceptionally well during inference with short sequences, significantly enhancing their trajectory stitching capabilities.”"}},"nonreaders":[],"tmdate":1731428082915,"tcdate":1730134843538,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4253/Reviewer_z4Tj"],"signatures":["ICLR.cc/2025/Conference/Submission4253/Reviewer_z4Tj"],"forum":"IUwqJ8VT4F","number":1,"license":"CC BY 4.0","cdate":1730134843538,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4253/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428082915,"domain":"ICLR.cc/2025/Conference","replyto":"IUwqJ8VT4F","id":"FpHSMV7e6u","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Sequence Modeling in Offline Reinforcement Learning"]},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Decision Transformer (DT), free from optimal value functions fitting and policy gradient computation, attempts to solve offline reinforcement learning (RL) via supervised sequence modeling. During inference, sequence modeling requires an initial target returns assigned with expert knowledge, which blocks comprehensive evaluation on more diverse datasets. As a result, existing sequence modeling only focuses on limited evaluation on Gym datasets and some understanding is severely biased. In this paper, we aim to revisit the design choices, including architecture and context length, in sequence modeling on more diverse datasets. We utilize the max-return sequence modeling that replaces the manual target returns with maximized returns predicted by itself. We systematically investigate the impact of 1) architectural choices and 2) context lengths in max-return sequence modeling on nine datasets with varying data distributions. Abundant experiments and thorough analyses reveal that design choices are highly influenced by the dataset characteristics, which further underscores the significance of more diverse evaluation."},"_bibtex":{"value":"@misc{\nzhuang2025revisiting,\ntitle={Revisiting the Design Choices in Max-Return Sequence Modeling},\nauthor={Zifeng Zhuang and Dengyun Peng and Donglin Wang and Jiacheng Liu and Xing Lei and Diyuan Shi and Ziqi Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=IUwqJ8VT4F}\n}"},"title":{"value":"Revisiting the Design Choices in Max-Return Sequence Modeling"},"pdf":{"value":"/pdf/61caa124244e3bd60019b3a4b213d64a4af3d57d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhuang|revisiting_the_design_choices_in_maxreturn_sequence_modeling"},"authorids":{"value":["~Zifeng_Zhuang1","~Dengyun_Peng1","~Donglin_Wang1","~Jiacheng_Liu13","~Xing_Lei2","~Diyuan_Shi1","~Ziqi_Zhang7"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zifeng Zhuang","Dengyun Peng","Donglin Wang","Jiacheng Liu","Xing Lei","Diyuan Shi","Ziqi Zhang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.02013v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"cao|mamba_as_decision_maker_exploring_multiscale_sequence_modeling_in_offline_reinforcement_learning"},"authorids":{"value":["~Jiahang_Cao1","~Qiang_Zhang10","~Ziqing_Wang1","https://dblp.org/search/pid/api?q=author:Jiaxu_Wang:","~Hao_Cheng15","https://dblp.org/search/pid/api?q=author:Yecheng_Shao:","https://dblp.org/search/pid/api?q=author:Wen_Zhao:","https://dblp.org/search/pid/api?q=author:Gang_Han:","https://dblp.org/search/pid/api?q=author:Yijie_Guo:","~Renjing_Xu1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.02013"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-02013,\n  publtype={informal},\n  author={Jiahang Cao and Qiang Zhang and Ziqing Wang and Jiaxu Wang and Hao Cheng and Yecheng Shao and Wen Zhao and Gang Han and Yijie Guo and Renjing Xu},\n  title={Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.02013},\n  url={https://doi.org/10.48550/arXiv.2406.02013}\n}\n"},"abstract":{"value":"Sequential modeling has demonstrated remarkable capabilities in offline reinforcement learning (RL), with Decision Transformer (DT) being one of the most notable representatives, achieving significant success. However, RL trajectories possess unique properties to be distinguished from the conventional sequence (e.g., text or audio): (1) local correlation, where the next states in RL are theoretically determined solely by current states and actions based on the Markov Decision Process (MDP), and (2) global correlation, where each step's features are related to long-term historical information due to the time-continuous nature of trajectories. In this paper, we propose a novel action sequence predictor, named Mamba Decision Maker (MambaDM), where Mamba is expected to be a promising alternative for sequence modeling paradigms, owing to its efficient modeling of multi-scale dependencies. In particular, we introduce a novel mixer module that proficiently extracts and integrates both global and local features of the input sequence, effectively capturing interrelationships in RL datasets. Extensive experiments demonstrate that MambaDM achieves state-of-the-art performance in Atari and OpenAI Gym datasets. Furthermore, we empirically investigate the scaling laws of MambaDM, finding that increasing model size does not bring performance improvement, but scaling the dataset amount by 2x for MambaDM can obtain up to 33.7% score improvement on Atari dataset. This paper delves into the sequence modeling capabilities of MambaDM in the RL domain, paving the way for future advancements in robust and efficient decision-making systems. Our code will be available at https://github.com/AndyCao1125/MambaDM."},"title":{"value":"Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning"},"authors":{"value":["Jiahang Cao","Qiang Zhang","Ziqing Wang","Jiaxu Wang","Hao Cheng","Yecheng Shao","Wen Zhao","Gang Han","Yijie Guo","Renjing Xu"]}},"tmdate":1747748701566,"pdate":1704067200000,"tcdate":1724556348635,"writers":["~"],"signatures":["~Jiahang_Cao1"],"forum":"hY9rw9YELf","license":"CC BY-SA 4.0","number":72191,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747748701566,"domain":"DBLP.org","id":"hY9rw9YELf","version":2},{"content":{"venue":{"value":"NeurIPS 2024"},"pdf":{"value":"https://proceedings.neurips.cc/paper_files/paper/2024/file/d13a3eae72366e61dfdc7eea82eeb685-Paper-Conference.pdf"},"venueid":{"value":"dblp.org/conf/NIPS/2024"},"paperhash":{"value":"yang|parallelizing_linear_transformers_with_the_delta_rule_over_sequence_length"},"authorids":{"value":["~Songlin_Yang1","https://dblp.org/search/pid/api?q=author:Bailin_Wang:","https://dblp.org/search/pid/api?q=author:Yu_Zhang:","https://dblp.org/search/pid/api?q=author:Yikang_Shen:","https://dblp.org/search/pid/api?q=author:Yoon_Kim:"]},"html":{"value":"http://papers.nips.cc/paper_files/paper/2024/hash/d13a3eae72366e61dfdc7eea82eeb685-Abstract-Conference.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/nips/YangWZSK24,\n  author={Songlin Yang and Bailin Wang and Yu Zhang and Yikang Shen and Yoon Kim},\n  title={Parallelizing Linear Transformers with the Delta Rule over Sequence Length},\n  year={2024},\n  cdate={1704067200000},\n  url={http://papers.nips.cc/paper_files/paper/2024/hash/d13a3eae72366e61dfdc7eea82eeb685-Abstract-Conference.html},\n  booktitle={NeurIPS},\n  crossref={conf/nips/2024}\n}\n"},"abstract":{"value":"Transformers with linear attention (i.e., linear transformers) and state-space models have recently been suggested as a viable linear-time alternative to transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive variants of linear transformers which replace the additive update in linear transformers with the delta rule (DeltaNet) have been found to be more effective at associative recall, existing algorithms for training such models do not parallelize over sequence length and are thus inefficient to train on modern hardware. This work describes a hardware-efficient algorithm for training linear transformers with the delta rule, which exploits a memory-efficient representation for computing products of Householder matrices. This algorithm allows us to scale up DeltaNet to standard language modeling settings. We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines such as Mamba and GLA in terms of perplexity and zero-shot performance on downstream tasks. We also experiment with two hybrid models which combine DeltaNet layers with (1) sliding-window attention layers every other layer or (2) two global attention layers, and find that these hybrids outperform strong transformer baselines."},"title":{"value":"Parallelizing Linear Transformers with the Delta Rule over Sequence Length"},"authors":{"value":["Songlin Yang","Bailin Wang","Yu Zhang","Yikang Shen","Yoon Kim"]}},"tmdate":1740189697647,"pdate":1704067200000,"tcdate":1740189695138,"writers":["~"],"signatures":["~Songlin_Yang1"],"forum":"Ly95EDSqLi","license":"CC BY-SA 4.0","number":338309,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1740189697647,"domain":"DBLP.org","id":"Ly95EDSqLi","version":2},{"content":{"TLDR":{"value":"DeltaNet, i.e., Linear Transformer with delta rule, suffers from slow recurrent training. This work develops a novel algorithm that enables the parallelization of training over sequence length, allowing for large-scale experiments"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["linear transformer","recurrent neural network","hardware-aware algorithm","state space model","sequence modeling","efficient training"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Transformers with linear attention (i.e., linear transformers) and state-space models have recently been suggested as a viable linear-time alternative to transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive variants of linear transformers which replace the additive update in linear transformers with the delta rule (DeltaNet) have been found to be more effective at associative recall, existing algorithms for training such models do not parallelize over sequence length and are thus inefficient to train on modern hardware. This work describes a hardware-efficient algorithm for training linear transformers with the delta rule, which exploits a memory-efficient representation for computing products of Householder matrices. This algorithm allows us to scale up DeltaNet to standard language modeling settings. We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines such as Mamba and GLA in terms of perplexity and zero-shot performance on downstream tasks. We also experiment with two hybrid models which combine DeltaNet layers with (1) sliding-window attention layers every other layer or (2) two global attention layers, and find that these hybrids outperform strong transformer baselines."},"_bibtex":{"value":"@inproceedings{\nyang2024parallelizing,\ntitle={Parallelizing Linear Transformers with the Delta Rule over Sequence Length},\nauthor={Songlin Yang and Bailin Wang and Yu Zhang and Yikang Shen and Yoon Kim},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=y8Rm4VNRPH}\n}"},"title":{"value":"Parallelizing Linear Transformers with the Delta Rule over Sequence Length"},"pdf":{"value":"/pdf/397e5724c60c7bb0691c6436dc8a56f5a0336f4f.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"yang|parallelizing_linear_transformers_with_the_delta_rule_over_sequence_length"},"authorids":{"value":["~Songlin_Yang1","~Bailin_Wang3","~Yu_Zhang36","~Yikang_Shen1","~Yoon_Kim1"]},"authors":{"value":["Songlin Yang","Bailin Wang","Yu Zhang","Yikang Shen","Yoon Kim"]}},"tmdate":1736918075039,"pdate":1727287990092,"tcdate":1715714221245,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission11970/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission11970/Authors"],"forum":"y8Rm4VNRPH","license":"CC BY 4.0","number":11970,"cdate":1715714221245,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/-/Submission","NeurIPS.cc/2024/Conference/-/Post_Submission","NeurIPS.cc/2024/Conference/Submission11970/-/Revision","NeurIPS.cc/2024/Conference/-/Edit","NeurIPS.cc/2024/Conference/Submission11970/-/Camera_Ready_Revision"],"mdate":1736918075039,"odate":1730873944189,"domain":"NeurIPS.cc/2024/Conference","id":"y8Rm4VNRPH","version":2},{"content":{"summary":{"value":"This manuscript proposes Memba, a membrane-driven Parameter-Efficient Fine-Tuning (PEFT) approach tailored for Mamba models, addressing the limitation that traditional Transformer-focused PEFT methods fail to account for Mamba’s unique temporal processing dynamics. Memba integrates Leaky Integrate Membrane neurons with strategic Low-Rank Adaptations (LoRA) on input/output projections and cross-layer membrane transfer, enhancing selective information retention without modifying Mamba’s core state-space components. Extensive experiments on commonsense reasoning and VTAB-1k demonstrate that Memba outperforms existing PEFT methods  across various model sizes while maintaining parameter efficiency, with theoretical analysis confirming LIM’s role in stable temporal integration and adaptive regularization."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Will this method or idea be applied to other linear models, like MoR or Gated networks?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The LIM neuron brings biologically inspired temporal processing into Mamba’s gating, filling the gap left by its simpler gates and helping the model learn what to remember or forget over time.\n2. The paper backs this up with clear analysis (like loss decomposition and bounded regularization), explaining how LIM adapts over time and smooths the loss landscape.\n3. The writing and figure are both well illustrated."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The performance depends on tuning things like the number of chunks (T), the leak factor (τ), and the threshold (Vth), so LIM likely need task-specific tuning.\n2. The theoretical analysis shows that the LIM neuron acts as a regularizer to smooth the loss landscape. However, the paper fails to explain the causal link between this optimization-level effect and the claimed practical benefits of enhanced temporal modeling and selective attention.\n3. The evaluation's focus on classification-style benchmarks, instead of long-sequence generative tasks, fails to fully validate the method's claimed improvements to Mamba's core temporal modeling capabilities."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916293922,"tcdate":1762012391043,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2594/Reviewer_wqXu"],"signatures":["ICLR.cc/2026/Conference/Submission2594/Reviewer_wqXu"],"forum":"eDZjLPfsin","number":4,"license":"CC BY 4.0","cdate":1762012391043,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2594/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916293922,"domain":"ICLR.cc/2026/Conference","replyto":"eDZjLPfsin","id":"nMWo2zprrh","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Brain-inspired computing","Mamba","Fine-tuning"]},"supplementary_material":{"value":"/attachment/ecb3d9aaa894996771e7b4905ff726146159e301.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"State Space Models (SSMs) have emerged as powerful alternatives to attention-based Transformers, with Mamba demonstrating impressive efficiency and scalability. As these models grow increasingly larger, the need for Parameter-Efficient Fine-Tuning (PEFT) methods becomes critical to adapt pre-trained Mamba to downstream tasks without prohibitive computational costs. However, previous approaches simply apply traditional Transformer-tailored PEFT methods without addressing the unique temporal processing dynamics of SSMs. To address this limitation, we propose ***Memba***, a membrane-driven PEFT approach specifically designed for Mamba. ***Memba*** introduces Leaky Integrate Membrane (LIM) neurons as bio-inspired gating mechanisms that naturally accumulate membrane potentials over time, enhancing selective information retention. By strategically combining LIM neurons with Low-Rank Adaptations (LoRA) and cross-layer membrane transfer, our approach significantly improves Mamba's temporal modeling capabilities. Extensive experiments across language and vision tasks demonstrate that ***Memba*** achieves substantial improvements over existing PEFT methods."},"_bibtex":{"value":"@inproceedings{\nlee2026memba,\ntitle={Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba},\nauthor={Donghyun Lee and Yuhang Li and Ruokai Yin and Shiting Xiao and Priyadarshini Panda},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=eDZjLPfsin}\n}"},"title":{"value":"Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba"},"pdf":{"value":"/pdf/fe3c7fdba0014b6df3c27084c7315fde71c17ca0.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lee|memba_membranedriven_parameterefficient_finetuning_for_mamba"},"authorids":{"value":["~Donghyun_Lee5","~Yuhang_Li1","~Ruokai_Yin1","~Shiting_Xiao2","~Priyadarshini_Panda1"]},"authors":{"value":["Donghyun Lee","Yuhang Li","Ruokai Yin","Shiting Xiao","Priyadarshini Panda"]}},"version":2},{"content":{"summary":{"value":"The paper proposes a hybrid sequence–graph framework for equity forecasting that couples a bidirectional Mamba (Bi-Mamba) state-space encoder with a Bayesian Multi-head Adaptive Graph Attention Convolution (MAGAC) layer. The Bi-Mamba encoder processes price windows in both forward and reverse directions and feeds sequence features to MAGAC; MAGAC builds an adaptive adjacency by blending a Gaussian kernel on learnable node embeddings with attention-based scores, then applies Chebyshev spectral filtering and multi-head aggregation. A lightweight Bayesian treatment is introduced by applying MC-Dropout to node embeddings and DropEdge at inference to obtain posterior predictive means/variances optimized via heteroscedastic Gaussian NLL. Experiments on NASDAQ/NYSE/DJIA report very large gains in RMSE/IC/RIC over LSTM/Transformer/GNN baselines, with modest MACs and parameter counts claimed for the proposed model."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please see above W1-W5."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1.   A linear-time temporal encoder (Mamba) with an adaptive spectral/attention GNN is a coherent and potentially scalable combination for long horizons. The bidirectional Mamba block and residual/normalization design are clearly described.\n2.   The MAGAC construction (Gaussian + attention blend, Chebyshev supports, head mixing) is laid out step-by-step with equations that are easy to implement.\n3.    The paper spells out how MC-Dropout on node embeddings and inference-time DropEdge are used to derive mean/variance and the closed-form variance propagation through a linear head, optimizing a heteroscedastic Gaussian NLL. This is clearly written and easy to reproduce technically.\n4.   The paper reports MACs/params/epoch time and positions the model against Transformers with respect to efficiency (albeit for S=1). Consistent, large improvements on all benchmarks, with excellent calibration.\n5.   Consistent, large improvements was reported on all benchmarks, with excellent calibration."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. The reported IC values are extraordinarily high for daily single-day returns (e.g., IC=0.9413 on NASDAQ). The paper defines IC/RIC as correlations with realized single-day returns, but offers no safeguards against common leakage vectors (e.g., improper chronological splits, cross-sectional standardization using future info, or target leakage through feature engineering). Please justify these numbers or audit the pipeline.\nAdditionally, the evaluation uses a fixed L=5 with bidirectional processing; while the text claims “causality is preserved,” there is no precise description of how labels are formed and how windows are cut to guarantee no look-ahead within the window (e.g., predicting $r_{t+1}$ from $[t−4,t]$). Provide a rigorous data construction diagram and masking rules.\nW2. The dataset description states N=82 features per day; however, the graph notion (nodes = assets) is not specified (how many tickers? which universe? how are nodes aligned over time? dynamic graph cadence?). The text also equates $E = d_{model}$ with “number of graph nodes (model width),” which is dimensionally inconsistent—E is typically feature width, not the number of assets. Later equations sum over E as if it were the node count. This confusion must be resolved with explicit tensor shapes (batch, nodes, time, channels) and a clear asset universe size.\nW3. The paper claims “well-calibrated uncertainty” and “without inflating computational cost,” yet no calibration metrics (NLL comparisons, CRPS, PIT/QQ plots, ECE, PICP/ACE) are reported. Moreover, Table-reported MACs are for S=1 while the method elsewhere enables S=10 at inference; the “no overhead” claim is therefore misleading. Also, DropEdge is applied only at inference, effectively evaluating a different model than trained. Provide proper calibration evaluation and fair compute accounting for the posterior estimates.\nW4. There is no ablation disentangling the gains from bidirectionality, Gaussian+attention blend, Chebyshev order, number of heads, Bayesian sampling, or DropEdge. Without such analysis, the source of improvements is unclear.\nW5. The authors acknowledge restrictions to daily closing prices and the need for theory on Bayesian MAGAC stability, but neither robustness checks (e.g., regime shifts, crisis periods) nor sensitivity analyses are presented."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918902534,"tcdate":1761799570907,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6564/Reviewer_Pwni"],"signatures":["ICLR.cc/2026/Conference/Submission6564/Reviewer_Pwni"],"forum":"KqCU5rfcMm","number":3,"license":"CC BY 4.0","cdate":1761799570907,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6564/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918902534,"domain":"ICLR.cc/2026/Conference","replyto":"KqCU5rfcMm","id":"b6Xo4evNmf","forumContent":{"TLDR":{"value":"Bi-Mamba & Bayesian MAGAC combines bidirectional selective state-space models with Bayesian graph attention for scalable, uncertainty-aware financial forecasting in linear time."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["mamba","transformer","bayesian networks"]},"primary_area":{"value":"probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)"},"abstract":{"value":"This paper presents \\textbf{BIMAMBA \\& Bayesian-MAGAC}, a unified framework that integrates bidirectional Selective State-Space Models with Bayesian Multi-head Adaptive Graph Attention Convolution for uncertainty-aware financial forecasting. The framework addresses two fundamental challenges: capturing long-range temporal dependencies across volatile market regimes while maintaining linear complexity, and learning adaptive cross-sectional structure with calibrated predictive uncertainty. BIMAMBA processes sequences bidirectionally via reversible state-space filters, extracting complementary temporal features while preserving strict causality. MAGAC constructs dynamic adjacencies through Gaussian kernel and attention blending, followed by Chebyshev spectral filtering for multi-scale aggregation. The Bayesian extension treats adjacencies and spectral filters as stochastic variables via Monte Carlo Dropout and DropEdge, yielding posterior predictive distributions with closed-form variance propagation at $\\mathcal{O}(N)$ complexity. Comprehensive evaluations on U.S. equity indices demonstrate that the architecture achieves substantial improvements in both point prediction accuracy and uncertainty calibration compared to established baselines, with statistically significant correlation between predicted uncertainty and prediction difficulty, suggesting practical utility for risk-aware decision making in financial applications.\n\\textit{The code is available on GitHub but has been hidden to preserve anonymity during the review process.}"},"_bibtex":{"value":"@misc{\ndang2026integrating,\ntitle={Integrating Selective State-Space Models and Bayesian Graph Attention for Uncertainty-aware Time-Series Analysis},\nauthor={Quang-Vinh Dang and SonAn Nguyen Ngoc},\nyear={2026},\nurl={https://openreview.net/forum?id=KqCU5rfcMm}\n}"},"title":{"value":"Integrating Selective State-Space Models and Bayesian Graph Attention for Uncertainty-aware Time-Series Analysis"},"pdf":{"value":"/pdf/2df5bc4bd83d8fe264929f03c5c341ba93bd102f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"dang|integrating_selective_statespace_models_and_bayesian_graph_attention_for_uncertaintyaware_timeseries_analysis"},"authorids":{"value":["~Quang-Vinh_Dang1","~SonAn_Nguyen_Ngoc1"]},"authors":{"value":["Quang-Vinh Dang","SonAn Nguyen Ngoc"]}},"version":2},{"content":{"summary":{"value":"This paper addresses enhancements to the Vision Mamba (ViM) architecture, a state-space paradigm that has been customized for vision tasks. The primary improvement entails the integration of register tokens into the input sequence, which resolves the severe artifact issue that affects semantic representation in ViM's feature maps. Cleaner feature maps and improved contextual comprehension are enabled by these registers. The enhanced model, known as Mamba®, surpasses its predecessor in critical benchmarks such as ImageNet classification and semantic segmentation on ADE20K, ensuring that efficiency is maintained while improving accuracy and scalability."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Clarification of Register Token Placement Strategy: Could the authors offer a more comprehensive explanation or justification for selecting register tokens that are equitably distributed as the most effective placement strategy? Furthermore, have other configurations, such as adaptive or context-aware placement strategies, been investigated?\n\n2. Ablation Study on Multiple Datasets: I think you can broaden the evaluation to include supplementary datasets, such as COCO, CIFAR-10/100, or ADE20K, in order to verify the proposed method's robustness?\n\n3. Analysis of Persistent Artifacts in Deeper Layers: The paper indicates that high-norm artifacts are still prevalent in deeper layers of the model, albeit to a lesser extent. Could the authors provide further details on the reasons for the persistence of these anomalies at a deeper level, as well as potential mitigation strategies?\n\n4. Comparative Benchmarking with Other SSM-Based Architectures: Although the paper offers comparisons with Vision Transformers and previous versions of Vision Mamba, there is a dearth of discussion regarding the performance of Mamba® in comparison to other state-space models or hybrid SSM architectures that have been modified for vision tasks. Are there any intentions to compare these alternatives or benchmark them?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Innovative Application of Registers: The inventive and effective improvement of semantic focus and the mitigation of artifacts in feature maps is achieved through the introduction of register tokens.\n\n2. Improved Performance: Mamba® exhibits a substantial increase in accuracy compared to the base Vision Mamba models, and it is capable of scaling to larger models with competitive results.\n\n3. Scalability and Versatility: The model maintains robust performance across a variety of benchmarks, adapting to varying sizes and duties."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Register Placement Strategy Complexity: Although the paper underscores the advantages of uniformly dispersing register tokens throughout the input sequence, the specific placement strategy could introduce complications in the process of replicating or modifying the architecture. \n\n2. Artifact Sensitivity: Despite the reduction in artifact sensitivity, artifacts continue to present challenges, particularly in deeper layers where high-norm values could potentially impact feature extraction.\n\n3. Ablation Studies with a Limited Scope: The ablation studies that have been presented, while insightful, are limited to the ImageNet dataset. This limitation affects the generalizability of the findings, as a more comprehensive substantiation across multiple datasets is required to bolster the conclusions regarding the efficacy of register tokens and placement strategies. The authors could broaden the scope by conducting similar studies on a variety of datasets, including COCO, ADE20K, or other domain-specific data, to demonstrate the architecture's versatility and robustness in various visual contexts and data distributions.\n\n4. The name of Mamba® causes confusion of the many Mamba family of methods."}},"nonreaders":[],"tmdate":1731427468579,"tcdate":1730596742667,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1694/Reviewer_8Xkt"],"signatures":["ICLR.cc/2025/Conference/Submission1694/Reviewer_8Xkt"],"forum":"wxEASOHHdT","number":2,"license":"CC BY 4.0","cdate":1730596742667,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1694/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427468579,"domain":"ICLR.cc/2025/Conference","replyto":"wxEASOHHdT","id":"aod3Ccvan4","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["State Space Models","Mamba","Representation Learning"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Similar to Vision Transformers, this paper identifies artifacts also present within the feature maps of Vision Mamba. These artifacts, corresponding to high-norm tokens emerging in low-information background areas of images, appear much more severe in Vision Mamba---they exist prevalently even with the tiny-sized model and activate extensively across background regions. To mitigate this issue, we follow the prior solution of introducing register tokens into Vision Mamba. To better cope with Mamba blocks' uni-directional inference paradigm, two key modifications are introduced: 1) evenly inserting registers throughout the input token sequence, and 2) recycling registers for final decision predictions. We term this new architecture MambaReg. Qualitative observations suggest, compared to vanilla Vision Mamba, MambaReg's feature maps appear cleaner and more focused on semantically meaningful regions.  Quantitatively, MambaReg attains stronger performance and scales better.  For example, on the ImageNet benchmark, our MambaReg-B attains 83.0% accuracy, significantly outperforming Vim-B's 81.8%; furthermore, we provide the first successful scaling to the large model size (i.e., with 341M parameters), attaining a competitive accuracy of 83.6% (84.5% if finetuned with 384x384 inputs). Additional validation on the downstream semantic segmentation task also supports MambaReg's efficacy."},"_bibtex":{"value":"@misc{\nwang2024mambareg,\ntitle={Mamba-Reg: Vision Mamba Also Needs Registers},\nauthor={Feng Wang and Jiahao Wang and Sucheng Ren and Guoyizhe Wei and Jieru Mei and Wei Shao and Yuyin Zhou and Alan Yuille and Cihang Xie},\nyear={2024},\nurl={https://openreview.net/forum?id=wxEASOHHdT}\n}"},"title":{"value":"Mamba-Reg: Vision Mamba Also Needs Registers"},"pdf":{"value":"/pdf/85e2e7cb78a7d2c589e0ea139ce0b63bae5fc9cd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"wang|mambareg_vision_mamba_also_needs_registers"},"authorids":{"value":["~Feng_Wang14","~Jiahao_Wang5","~Sucheng_Ren1","~Guoyizhe_Wei1","~Jieru_Mei2","~Wei_Shao8","~Yuyin_Zhou1","~Alan_Yuille1","~Cihang_Xie3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Feng Wang","Jiahao Wang","Sucheng Ren","Guoyizhe Wei","Jieru Mei","Wei Shao","Yuyin Zhou","Alan Yuille","Cihang Xie"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.12245v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"li|scalable_autoregressive_image_generation_with_mamba"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Haopeng_Li:","https://dblp.org/search/pid/api?q=author:Jinyue_Yang:","https://dblp.org/search/pid/api?q=author:Kexin_Wang:","~Xuerui_Qiu1","~Yuhong_Chou1","https://dblp.org/search/pid/api?q=author:Xin_Li:","~Guoqi_Li1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.12245"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-12245,\n  publtype={informal},\n  author={Haopeng Li and Jinyue Yang and Kexin Wang and Xuerui Qiu and Yuhong Chou and Xin Li and Guoqi Li},\n  title={Scalable Autoregressive Image Generation with Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.12245},\n  url={https://doi.org/10.48550/arXiv.2408.12245}\n}\n"},"abstract":{"value":"We introduce AiM, an autoregressive (AR) image generative model based on Mamba architecture. AiM employs Mamba, a novel state-space model characterized by its exceptional performance for long-sequence modeling with linear time complexity, to supplant the commonly utilized Transformers in AR image generation models, aiming to achieve both superior generation quality and enhanced inference speed. Unlike existing methods that adapt Mamba to handle two-dimensional signals via multi-directional scan, AiM directly utilizes the next-token prediction paradigm for autoregressive image generation. This approach circumvents the need for extensive modifications to enable Mamba to learn 2D spatial representations. By implementing straightforward yet strategically targeted modifications for visual generative tasks, we preserve Mamba's core structure, fully exploiting its efficient long-sequence modeling capabilities and scalability. We provide AiM models in various scales, with parameter counts ranging from 148M to 1.3B. On the ImageNet1K 256*256 benchmark, our best AiM model achieves a FID of 2.21, surpassing all existing AR models of comparable parameter counts and demonstrating significant competitiveness against diffusion models, with 2 to 10 times faster inference speed. Code is available at https://github.com/hp-l33/AiM"},"title":{"value":"Scalable Autoregressive Image Generation with Mamba"},"authors":{"value":["Haopeng Li","Jinyue Yang","Kexin Wang","Xuerui Qiu","Yuhong Chou","Xin Li","Guoqi Li"]}},"tmdate":1731475566532,"pdate":1704067200000,"tcdate":1727677247470,"writers":["~"],"signatures":["~Yuhong_Chou1"],"forum":"5XN0SB2Cp4","license":"CC BY-SA 4.0","number":108985,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1731475566532,"domain":"DBLP.org","id":"5XN0SB2Cp4","version":2},{"content":{"summary":{"value":"The paper provides detailed analyses of memory utilization for many input-invariant and input-varying linear operators, including attention, convolutions, and recurrences. It introduces the metric called effective state size by leveraging rank or entropy. This metric is used to evaluate and compare the memory utilization of different linear operators from three model phases: initialization, mid-training, and post-training, with a lot of findings. The authors also showed using this to get better performance through better initialization, normalization, and distillation."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see the weakness part."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"* The paper proposes considering memory utilization to evaluate the performance of various linear operators. Specifically, it introduces two metrics to measure effective state size, which can serve as a proxy for memory utilization.\n\n* Empirical experiments are conducted across different phases of model training, yielding many interesting findings.\n\n* The analyses of memory utilization may help people better understand the behavior and performance of different models and inspire further improvements."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* For the recurrence, did the authors investigate the A matrix in the vectorized shape or the scalar value shape? The vectorized A is used in many classical RNN papers as well as in S4, S5, and S6. However, the scalar value A is used in the recent Mamba2 paper and RetNet, which directly builds the connection between attention and state-space models. I think this choice can affect the memory utilization and also efficiency.\n\n* Could the paper provide more discussion and analyses of memory utilization across different sequence lengths? I see the paper extends the analysis to a sequence length of 2048, but what about even longer sequences? Many state-space models highlight their long-context abilities more than attention, and long-context is also a crucial ability required in LLMs.\n\n* While there are many findings (9 in the main paper) related to memory utilization, I wonder which are the most important. Could the authors highlight the most meaningful findings, perhaps in the introduction, to better emphasize the paper's contribution?\n\n* What is the direct conclusion regarding the memory utilization of different linear operators? I didn’t find a direct conclusion or explanation in the findings.\n\n* The terms GLA and WLA are ambiguous. Why do the authors label the input-invariant A matrix as weighted linear attention? Because the softmax attention is also weighted. Also, in line 242, the authors categorize Mamba as a GLA layer. However, Mamba or S6 is not exactly linear attention, although Mamba2 can be."}},"nonreaders":[],"tmdate":1731429282231,"tcdate":1731201961620,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9386/Reviewer_o977"],"signatures":["ICLR.cc/2025/Conference/Submission9386/Reviewer_o977"],"forum":"DHVjLvSps6","number":4,"license":"CC BY 4.0","cdate":1731201961620,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9386/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429282231,"domain":"ICLR.cc/2025/Conference","replyto":"DHVjLvSps6","id":"FJUWNTvUe1","forumContent":{"TLDR":{"value":"We propose the Effective State-Size (ESS) metric to quantitatively analyze how sequence models utilize memory and context, offering insights into model in-context recall, initialization strategies, efficiency, and architecture design."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["model analysis","interpretability","linear systems","attention","state-space models","sequence models","memory utilization","context utilization"]},"primary_area":{"value":"interpretability and explainable AI"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"As the space of causal sequence modeling architectures continues to grow, the need to develop a general framework for their analysis becomes increasingly important. With this aim, we draw insights from classical signal processing and control theory, to develop a quantitative measure of *memory utilization*: the internal mechanisms through which a model stores past information to produce future outputs. This metric, which we call **effective state-size** (ESS), is tailored to the fundamental class of systems with *input-invariant* and *input-varying linear operators*, encompassing a variety of computational units such as variants of attention, convolutions, and recurrences. Unlike prior work on memory utilization, which either relies on raw operator visualizations (e.g. attention maps), or simply the total *memory capacity* (i.e. cache size) of a model, our metrics provide highly interpretable and actionable measurements. In particular, we show how ESS can be leveraged to improve initialization strategies, inform novel regularizers and advance the performance-efficiency frontier through model distillation. Furthermore, we demonstrate that the effect of context delimiters (such as end-of-speech tokens) on ESS highlights cross-architectural differences in how large language models utilize their available memory to recall information. Overall, we find that ESS provides valuable insights into the dynamics that dictate memory utilization, enabling the design of more efficient and effective sequence models."},"_bibtex":{"value":"@misc{\nparnichkun2025quantifying,\ntitle={Quantifying Memory Utilization with Effective State-Size},\nauthor={Rom Parnichkun and Neehal Tumma and Armin W Thomas and Alessandro Moro and Qi An and Taiji Suzuki and Atsushi Yamashita and Michael Poli and Stefano Massaroli},\nyear={2025},\nurl={https://openreview.net/forum?id=DHVjLvSps6}\n}"},"title":{"value":"Quantifying Memory Utilization with Effective State-Size"},"pdf":{"value":"/pdf/9de503dde3c70436d5a15c5372fb7d21e789bc5a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"parnichkun|quantifying_memory_utilization_with_effective_statesize"},"authorids":{"value":["~Rom_Parnichkun1","~Neehal_Tumma1","~Armin_W_Thomas1","~Alessandro_Moro1","~Qi_An6","~Taiji_Suzuki1","~Atsushi_Yamashita1","~Michael_Poli1","~Stefano_Massaroli1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Rom Parnichkun","Neehal Tumma","Armin W Thomas","Alessandro Moro","Qi An","Taiji Suzuki","Atsushi Yamashita","Michael Poli","Stefano Massaroli"]}},"version":2},{"content":{"summary":{"value":"This paper focuses on building theoretical understanding of the Mamba model for its powerful in-context learning capability. According to the results, the authors demonstrate that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layers to suppress the influence of outliers."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"See Weakness."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- This paper is the first paper to theoretically understand the in-context capability of Mamba, which is an essential contribution.\n- The paper is well written. \n- The empirical results are good."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The paper mainly focuses on the one-layer Mamba model, which cannot be seen as a comprehensive understanding of Mamba. However, this is not a significant drawback due to the complex large models.\n- As a following question, can a model with just one layer achieve the similar capability of large Mamba models, such as in-context learning?\n- The assumptions, which decompose the data into components of relevant and irrelevant parts, in Section 3.2 are interesting. I think such assumptions contribute to the theoretical analyses later. However, I am curious about whether such assumptions work in practice. In other words, can you provide some empirical or simple theoretical results to demonstrate that some assumptions are derived from the real cases?\n- In Remark 4, a comparison between Mamba and the linear Transformer is reported. Specifically, compared to Mamba, the linear Transformer should satisfy some specific conditions for similar performance as Mamba. This is interesting. However, is there any empirical evidence to support the summaries? For example, consider two LLMs with the same architecture and trained with different hyperparameters. Do the empirical evaluations support your theoretical results?\n\n**Minors**:\n- What is $A$ in Lines 130 and 132?\n- Lines 163 & 310, the references have the wrong format."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919929538,"tcdate":1762158578475,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7901/Reviewer_1e2F"],"signatures":["ICLR.cc/2026/Conference/Submission7901/Reviewer_1e2F"],"forum":"tswBfpkwHn","number":3,"license":"CC BY 4.0","cdate":1762158578475,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7901/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919929538,"domain":"ICLR.cc/2026/Conference","replyto":"tswBfpkwHn","id":"a2BM6XBeTf","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"This paper provide the first theoretical analysis of the training dynamics of a one-layer Mamba model and characterizes its in-context learning generaization ability with additive outliers in the prompt"},"keywords":{"value":["Mamba","in-context learning","Transformer","generalization","training dynamics","deep learning theory"]},"primary_area":{"value":"learning theory"},"abstract":{"value":"The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks. Like Transformers, Mamba exhibits in-context learning (ICL) capabilities, i.e.,  making predictions for new tasks based on a prompt containing input-label pairs and a query, without requiring fine-tuning.\nDespite its empirical success, the theoretical understanding of Mamba remains limited, largely due to the nonlinearity introduced by its gating mechanism. To the best of our knowledge, this paper presents the first theoretical analysis of the training dynamics of a one-layer Mamba model, which consists of a linear attention component followed by a nonlinear gating layer, and its ICL generalization on unseen binary classification tasks, even when the prompt includes additive outliers. Our analysis shows that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layer to suppress the influence of outliers. By establishing and comparing to the analysis of linear Transformers under the same setting, we show that although Mamba may require more training iterations to converge, it maintains accurate predictions even when the proportion of outliers exceeds the threshold that a linear Transformer can tolerate. These theoretical findings are supported by empirical experiments."},"_bibtex":{"value":"@misc{\nli2026can,\ntitle={Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis},\nauthor={Hongkang Li and Songtao Lu and Xiaodong Cui and Pin-Yu Chen and Meng Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=tswBfpkwHn}\n}"},"title":{"value":"Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis"},"pdf":{"value":"/pdf/771e925a0afdf6afc4b8a950bc4b771d6afabad8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"li|can_mamba_learn_in_context_with_outliers_a_theoretical_generalization_analysis"},"authorids":{"value":["~Hongkang_Li1","~Songtao_Lu1","~Xiaodong_Cui1","~Pin-Yu_Chen1","~Meng_Wang4"]},"authors":{"value":["Hongkang Li","Songtao Lu","Xiaodong Cui","Pin-Yu Chen","Meng Wang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.11144v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"wang|is_mamba_effective_for_time_series_forecasting"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zihan_Wang:","https://dblp.org/search/pid/api?q=author:Fanheng_Kong:","~Shi_Feng3","","https://dblp.org/search/pid/api?q=author:Han_Zhao:","~Daling_Wang2","~Yifei_Zhang12"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.11144"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-11144,\n  publtype={informal},\n  author={Zihan Wang and Fanheng Kong and Shi Feng and Ming Wang and Han Zhao and Daling Wang and Yifei Zhang},\n  title={Is Mamba Effective for Time Series Forecasting?},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.11144},\n  url={https://doi.org/10.48550/arXiv.2403.11144}\n}\n"},"abstract":{"value":"In the realm of time series forecasting (TSF), it is imperative for models to adeptly discern and distill hidden patterns within historical time series data to forecast future states. Transformer-based models exhibit formidable efficacy in TSF, primarily attributed to their advantage in apprehending these patterns. However, the quadratic complexity of the Transformer leads to low computational efficiency and high costs, which somewhat hinders the deployment of the TSF model in real-world scenarios. Recently, Mamba, a selective state space model, has gained traction due to its ability to process dependencies in sequences while maintaining near-linear complexity. For TSF tasks, these characteristics enable Mamba to comprehend hidden patterns as the Transformer and reduce computational overhead compared to the Transformer. Therefore, we propose a Mamba-based model named Simple-Mamba (S-Mamba) for TSF. Specifically, we tokenize the time points of each variate autonomously via a linear layer. A bidirectional Mamba layer is utilized to extract inter-variate correlations and a Feed-Forward Network is set to learn temporal dependencies. Finally, the generation of forecast outcomes through a linear mapping layer. Experiments on thirteen public datasets prove that S-Mamba maintains low computational overhead and achieves leading performance. Furthermore, we conduct extensive experiments to explore Mamba's potential in TSF tasks. Our code is available at https://github.com/wzhwzhwzh0921/S-D-Mamba."},"title":{"value":"Is Mamba Effective for Time Series Forecasting?"},"authors":{"value":["Zihan Wang","Fanheng Kong","Shi Feng","Ming Wang","Han Zhao","Daling Wang","Yifei Zhang"]}},"tmdate":1752855581434,"pdate":1704067200000,"tcdate":1724656261489,"writers":["~"],"signatures":["~Ming_Wang7"],"forum":"BQPfCbrJuC","license":"CC BY-SA 4.0","number":73755,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1752855581434,"domain":"DBLP.org","id":"BQPfCbrJuC","version":2},{"content":{"venue":{"value":"Neurocomputing 2025"},"venueid":{"value":"dblp.org/journals/IJON/2025"},"paperhash":{"value":"wang|is_mamba_effective_for_time_series_forecasting"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zihan_Wang:","~Fanheng_Kong1","~Shi_Feng3","https://dblp.org/search/pid/api?q=author:Ming_Wang_0006:","https://dblp.org/search/pid/api?q=author:Xiaocui_Yang:","https://dblp.org/search/pid/api?q=author:Han_Zhao:","~Daling_Wang2","~Yifei_Zhang12"]},"html":{"value":"https://doi.org/10.1016/j.neucom.2024.129178"},"_bibtex":{"value":"@article{DBLP:journals/ijon/WangKFWYZWZ25,\n  author={Zihan Wang and Fanheng Kong and Shi Feng and Ming Wang and Xiaocui Yang and Han Zhao and Daling Wang and Yifei Zhang},\n  title={Is Mamba effective for time series forecasting?},\n  year={2025},\n  cdate={1735689600000},\n  journal={Neurocomputing},\n  volume={619},\n  pages={129178},\n  url={https://doi.org/10.1016/j.neucom.2024.129178}\n}\n"},"abstract":{"value":"In the realm of time series forecasting (TSF), it is imperative for models to adeptly discern and distill hidden patterns within historical time series data to forecast future states. Transformer-based models exhibit formidable efficacy in TSF, primarily attributed to their advantage in apprehending these patterns. However, the quadratic complexity of the Transformer leads to low computational efficiency and high costs, which somewhat hinders the deployment of the TSF model in real-world scenarios. Recently, Mamba, a selective state space model, has gained traction due to its ability to process dependencies in sequences while maintaining near-linear complexity. For TSF tasks, these characteristics enable Mamba to comprehend hidden patterns as the Transformer and reduce computational overhead compared to the Transformer. Therefore, we propose a Mamba-based model named Simple-Mamba (S-Mamba) for TSF. Specifically, we tokenize the time points of each variate autonomously via a linear layer. A bidirectional Mamba layer is utilized to extract inter-variate correlations and a Feed-Forward Network is set to learn temporal dependencies. Finally, the generation of forecast outcomes through a linear mapping layer. Experiments on thirteen public datasets prove that S-Mamba maintains low computational overhead and achieves leading performance. Furthermore, we conduct extensive experiments to explore Mamba’s potential in TSF tasks. Our code is available at https://github.com/wzhwzhwzh0921/S-D-Mamba."},"title":{"value":"Is Mamba effective for time series forecasting?"},"authors":{"value":["Zihan Wang","Fanheng Kong","Shi Feng","Ming Wang","Xiaocui Yang","Han Zhao","Daling Wang","Yifei Zhang"]}},"tmdate":1752855577732,"pdate":1735689600000,"tcdate":1747248573720,"writers":["~"],"signatures":["~Fanheng_Kong1"],"forum":"g1XxM5Yyi2","license":"CC BY-SA 4.0","number":465202,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1752855577732,"domain":"DBLP.org","id":"g1XxM5Yyi2","version":2},{"content":{"summary":{"value":"The paper introduces a matrix mixer framework for sequence models which linearly applies an LxL matrix M  to a sequence representation X of length L. Popular sequence models can be framed within this context e.g. softmax self-attention or SSMs, according to different properties of M. The authors use their framework to identify desirable properties of M, such as data dependence or extendability (where the parameterisation is such that the sequence length L can change for different sequences), or efficient matmul. The authors then use their framework to design sequence models with Vandemonde or Cauchy mixing matrices that perform comparably to attention. The main contribution of the paper is the introduction of Hydra, a bi-directional variant of SSM like Mamba. To do this, the authors let M instead denote a quasiseparable matrix (which is a bidirectional variant of the causal semiseparable matrices introduced in the Mamba-2 paper [6]). The authors show that Hydra (i.e. using quasiseparable matrix mixers) outperforms other matrix mixers (4.1), outperforms other approaches for bidirectional SSMs (4.2), and outperform other standard sequence model arhchitectures like transformers on Masked Language Modelling and ImageNet classification."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"- Is there a theoretical argument for why data dependence/extendability are nice properties? Or just intuitive? Are there settings where these properties can hurt e.g long range?\n\nTypo:\n- linear 38 \"they often lack a systematic\" - remove \"and\""},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":3},"strengths":{"value":"1. The paper provides a general framework for matrix mixing and introduces desirable properties like Sequence Alignment.\n2. Introduces a bidirectional version of SSMs, called Hydra, which is motivated through their framework.\n3. Hydra seems to work really well, beats unidirectional Mamba and transformers across different settings where bidirectionality is desired.\n4. Shows their framework can motivate new sequence models using e.g. Vandermonde or Cauchy matrices, and provides insights into why existing methods may perform well (like low rank matrices in linear attention)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The writing is unclear in large parts. This detracts from the flow and readability, and ultimately makes it somewhat a frustrating read, as it seems there are nice ideas here but communicated poorly. For example:\n- What does “common data transformations” mean in definition 2.1.\n- The term “Structured matrices” is not standard terminology and is not formally defined. In my reading the closest thing to a definition is line 117: “... a structured matrix, which are known to possess sub-quadratic matrix multiplication”. Is a structured matrix defined by the ability to do sub-quadratic matrix multiplication, because line 117 suggests there are other properties involved. Also the reference to established “mathematical literature” in line 120 but no citations are provided nor examples of structured matrices.\n - “Data dependence” also doesn’t seem to have a theoretical definition, but is part of the theoretical results (proposition 2.3) so should be defined. What does “canonical” mean in the context of line 138? The authors write \"Although data-dependency is a popular notion that is widely regarded as being crucial to performance of Attention, it lacked any formal definition in the literature.\" but there also isn't a formal provided here in my reading. What does \"the third interpretation\" refer to in line 153?\n- What are $\\phi$ or $\\hat{P}$ in definition 2.2 of sequence alignment, are there any constraints on $\\hat{P}$, why/when is it necessary in the definition? I assume $\\phi$ is the empty set but this is not obvious nor defined.\n- The proof of proposition 2.6 does not prove the sequence alignment property.\n- Are Quasiseparable matrices established in the literature or have you defined them? There is no citation provided afaict.\n- Should the tilde be a hat on P in line 237 (over vice versa in line 131).\n- The 2 paragraphs from line 190 to line 205 make reference to results which are a lot later (e.g. table 2). I would move the table earlier or at least refer to table 2.\n- “Taming the Hydra” isn’t particularly insightful for a subsection title.\n- The advantages of Hydra (lines 246 to 249) are not justified until later in the subsection, so on first reading it seems like overclaiming (also see my 2nd point). It is also not clear what is being compared to with these advantages (“heuristic alternatives” is cited for the first one but is this also true for the second and third benefit?)\n- “It is known that quasiseparable matrices are closed under addition” - cite or prove.\n2. It is not clear to me if the theoretical motivation of Quasiseparable matrices is actually a big factor behind the practical gains: in Figure 4 it seems like you are discretising the forward and backward SSD differently, which seems like it will break the symmetry between the A matrix in the forward and backward SSD which is necessary in for the equivalence in proposition 3.2, which to my understanding states in layman terms: “a QS matrix is just two SS matrices that *share parameters*  with flips/shifts”. I could be wrong, but discretise_A is not defined so it is hard to check and the zipped code is different. But if I am right this seems to go against one of the motivations of the paper that previous bidirectional SSMs are heuristic or ad-hoc.\n3. Likewise, the arguments in line 250-258 that motivate Hydra as opposed to previous approaches for bidirectional SSMs make it seem like the only difference is the diagonal elements, which can be seen as skip connections (see e.g. https://arxiv.org/abs/2302.10322 in the example of diagonal elements of attention matrices replacing standard skip connections). Given that the previous methods presumably also use skip connections, this difference seems quite minimal. If anything, it might be better to pitch this framework as encapsulating and generalising previous works.\n4. Ablations to understand the empirical improvements made in Hydra are missing. In general the results seem great, but as discussed a reader may have unanswered questions why Hydra seems to work so well.  For example, related to above, what to performance happens if you remove the extra diagonal part, which should boil down to the Add method in table 3 right? Also what if you change the convolution (e.g. remove it or make it a causal conv) because it doesn’t seem to be connect to the theory here (of quasiseparable mixers)? Also what happens if you remove the parameter sharing between the two SSDs (which again should get closer to the previous methods)? What if you place the different mixers in a standard transformer block instead of a Mamba block in table 2 (does table 2 give an unfair advantage to QS)?\n5. The authors write that \"Hardware efficiency\" is a limitation in the appendix. Do the authors have throughput results for Hydra compared to transformers? If hardware efficiency is a concern then isn't \"fast implementations on hardware\" a more practical desiderata to design new sequence models as opposed to use \"sub-quadratic matmuls in theory\", which is the line taken by this work?"},"limitations":{"value":"There is a discussion of some limitations in the appendix."}},"nonreaders":[],"tmdate":1730878827244,"tcdate":1720566401138,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission3122/Reviewer_RwkG"],"signatures":["NeurIPS.cc/2024/Conference/Submission3122/Reviewer_RwkG"],"forum":"preo49P1VY","number":1,"license":"CC BY 4.0","cdate":1720566401138,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission3122/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878827244,"domain":"NeurIPS.cc/2024/Conference","replyto":"preo49P1VY","id":"uUzVIn4WU0","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Deep learning","sequence models","state space models","mamba"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"A wide array of sequence models are built on a framework modeled after Transformers, comprising alternating sequence mixer and channel mixer layers.  This paper studies a unifying *matrix mixer* view of sequence mixers that can be conceptualized as a linear map on the input sequence. This framework encompasses a broad range of well-known sequence models, including the self-attention of Transformers as well as recent strong alternatives such as structured state space models (SSMs), and allows understanding downstream characteristics such as efficiency and expressivity through properties of their structured matrix class. We identify a key axis of matrix parameterizations termed *sequence alignment*, which increases the flexibility and performance of matrix mixers, providing insights into the strong performance of Transformers and recent SSMs such as Mamba. Furthermore, the matrix mixer framework offers a systematic approach to developing sequence mixers with desired properties, allowing us to develop several new sub-quadratic sequence models. In particular, we propose a natural bidirectional extension of the Mamba model (**Hydra**), parameterized as a *quasiseparable matrix mixer*, which demonstrates superior performance over other sequence models including Transformers on non-causal tasks. As a drop-in replacement for attention layers, \\name outperforms BERT by 0.8 points on the GLUE benchmark and ViT by 2% Top-1 accuracy on ImageNet."},"_bibtex":{"value":"@inproceedings{\nhwang2024hydra,\ntitle={Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers},\nauthor={Sukjun Hwang and Aakash Lahoti and Ratish Puduppully and Tri Dao and Albert Gu},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=preo49P1VY}\n}"},"title":{"value":"Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers"},"pdf":{"value":"/pdf/975e174ed724f910ffb63bad51c2578e145b4c50.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"hwang|hydra_bidirectional_state_space_models_through_generalized_matrix_mixers"},"authorids":{"value":["~Sukjun_Hwang1","~Aakash_Lahoti1","~Ratish_Puduppully1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Sukjun Hwang","Aakash Lahoti","Ratish Puduppully","Tri Dao","Albert Gu"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a new paradigm called Active Speech Enhancement (ASE), which aims to go beyond traditional passive speech enhancement and active noise cancellation by actively shaping the speech signal. Unlike conventional approaches that only suppress noise or reverberation, ASE both attenuates interference and amplifies speech-relevant frequency components to improve perceptual quality. The authors propose a Transformer–Mamba–based model (ASE-TM) that generates an active modification signal through time–frequency Mamba blocks and an attention mechanism, coupled with a joint suppression–enrichment loss balancing denoising and signal enhancement. Comprehensive experiments across denoising, dereverberation, and declipping tasks show consistent improvements over classical and deep ANC baselines (e.g., DeepANC, ARN, THF-FxLMS) in PESQ, STOI, and NMSE metrics. The results demonstrate ASE-TM’s robustness under diverse acoustic and nonlinear conditions, suggesting its potential as a unified framework for active and intelligent speech restoration."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. What concrete benefits does the ASE formalization provide compared to conventional SE frameworks?\n\n2. What is the physical or algorithmic interpretation of a(n) in your system, given that it no longer functions as an anti-noise signal?\n\n3. Can you isolate the gain of Mamba2 over SE-Mamba through controlled experiments?\n\n4. Have you compared your method with recent diffusion-based or GAN-based SE systems?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Clear writing and decent experimental reproducibility.\n\n2. The idea of linking active control theory with speech enhancement could, in principle, be interesting if more rigorously developed.\n\n3. The adoption of Mamba2 might inspire follow-up studies in sequence modeling for SE."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Conceptual Ambiguity in Task Definition\nThe paper claims to “formalize the ASE task and describe appropriate evaluation metrics that capture both noise suppression and speech enhancement.” However, in Section 3 (Eq. 4), the formulation directly borrows the notation from Active Noise Control (ANC), where the primary signal d(n) and anti-signal a(n) have clear physical meanings. In the proposed ASE setting, a(n) no longer corresponds to an anti-noise component with opposite phase or amplitude, and there is no identifiable physical counterpart in the presented model. This raises fundamental doubts about whether ASE constitutes a truly independent task, rather than merely a reinterpretation of ANC applied to speech enhancement. The authors should clarify what specific theoretical or practical advantage the ANC framework offers over conventional speech enhancement formulations.\n\n2. Limited Novelty in Model Architecture\nThe proposed “Transformer–Mamba–based model” is nearly identical to SE-Mamba, with the main change being the replacement of Mamba with Mamba2. While Mamba2 has demonstrated strong modeling ability in ASR [1], its integration here is a straightforward substitution rather than a conceptually new design. The claimed novelty therefore lies more in model adoption than in architectural innovation.\n\n3. Minimal Contribution in Loss Design\nThe proposed “joint suppression–enrichment loss” combines six components, five of which are identical to those used in MP-SENet. The only addition, L_T, represents a combined time–frequency loss that has already appeared in earlier works (e.g., [2]). Thus, the contribution in objective design is marginal and largely derivative of prior art.\n\n4. Weak Empirical Performance\nAs shown in Table 1, the denoising results of the proposed method fall significantly short of current state-of-the-art systems [3]. This performance gap undermines the paper’s claim that the ASE formulation or the proposed model yields superior practical benefits.\n\n[1] Y. Masuyama, K. Miyazaki and M. Murata, \"Mamba-Based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition,\" 2024 IEEE Spoken Language Technology Workshop (SLT), Macao, 2024, pp. 1-6, doi: 10.1109/SLT61566.2024.10832186.\n[2] Chen, Hang, et al. \"Cross-attention among spectrum, waveform and SSL representations with bidirectional knowledge distillation for speech enhancement.\" Information Fusion (2025): 103218.\n[3] M. S. Khan, M. L. Quatra, K. -H. Hung, S. -W. Fu, S. M. Siniscalchi and Y. Tsao, \"Exploiting Consistency-Preserving Loss and Perceptual Contrast Stretching to Boost SSL-Based Speech Enhancement,\" 2024 IEEE 26th International Workshop on Multimedia Signal Processing (MMSP), West Lafayette, IN, USA, 2024, pp. 1-6, doi: 10.1109/MMSP61759.2024.10743615."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916013707,"tcdate":1762106976952,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2082/Reviewer_saHn"],"signatures":["ICLR.cc/2026/Conference/Submission2082/Reviewer_saHn"],"forum":"QN4WZ5QoqC","number":3,"license":"CC BY 4.0","cdate":1762106976952,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2082/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916013707,"domain":"ICLR.cc/2026/Conference","replyto":"QN4WZ5QoqC","id":"xnffUohrKh","forumContent":{"TLDR":{"value":"Active speech enhancement and noise suppression"},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Active Noise Cancellation","ANC","Transformer","Mamba","Dereverberation","Declipping","NLP","Speech"]},"supplementary_material":{"value":"/attachment/132cd6538b5aeb93ec56d7c7b07445bb3c5101c5.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"We introduce a new paradigm for active sound modification: Active Speech Enhancement (ASE). While Active Noise Cancellation (ANC) algorithms focus on suppressing external interference and traditional speech enhancement passively reconstructs degraded speech signals, ASE goes further by actively shaping the speech signal, both attenuating unwanted noise components and amplifying speech-relevant frequencies to improve intelligibility and perceptual quality. To enable this, we propose a novel Transformer-Mamba-based architecture, along with a task-specific loss function designed to jointly optimize interference suppression and signal enrichment in an acoustic environment. Our method outperforms existing baselines across multiple speech processing tasks, including denoising, dereverberation, and declipping, demonstrating the effectiveness of active, targeted modulation in challenging acoustic environments. A demo page and source code are provided in the Supplementary Materials."},"_bibtex":{"value":"@misc{\nyaish2025active,\ntitle={Active speech enhancement: beyond passive denoising declipping and dereverberation},\nauthor={Ofir Yaish and Yehuda Mishaly and Eliya Nachmani},\nyear={2025},\nurl={https://openreview.net/forum?id=QN4WZ5QoqC}\n}"},"title":{"value":"Active speech enhancement: beyond passive denoising declipping and dereverberation"},"pdf":{"value":"/pdf/dd6c854ee61972557d03d83f5fd2999c83ad442e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"yaish|active_speech_enhancement_beyond_passive_denoising_declipping_and_dereverberation"},"authorids":{"value":["~Ofir_Yaish1","~Yehuda_Mishaly1","~Eliya_Nachmani1"]},"authors":{"value":["Ofir Yaish","Yehuda Mishaly","Eliya Nachmani"]}},"version":2},{"content":{"summary":{"value":"The article includes a dynamic market-guided module and the SelectiveMamba module, effectively addressing the challenges posed by noisy data. It introduces Mamba into stock price forecasting and employs a 3D scan method. By integrating market sentiment, the article incorporates subjective factors. Experiments demonstrate its performance."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please refer to the \"Weaknesses\" for your response, especially the first three points."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1.The article introduces Mamba into stock price forecasting.\n\n2.The article proposes a 3D scan method to capture interactions across the dimensions of time, stock, and market state.\n\n3.The method proposed in the article performs well in experiments."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The article lacks novelty and sufficient discussion on related work:\n\n   a) While using GPT to process market sentiment is indeed innovative, there have been many previous works that utilized text data, including the use of pre-trained language models as in [1].\n\n   b) It is difficult to see the relationship between the Time-Spectral method in the article and stock data. It appears to merely apply a time series method to stock data without analyzing how the Time-Spectral method aids stock prediction or offers any special improvements for it. Additionally, frequency methods are also common in time series analysis, as in [2].\n\n   c) Inter-stock correlations have been used in many previous articles, as in [3].\n\n2. The article claims that the model can handle noisy data, but I do not see how or why noisy data can be addressed, or what improvements have been made compared to previous methods. Why could previous deep learning methods not handle noisy data while the current one can? Furthermore, noise in stock data is often caused by random Brownian motion, which the article does not analyze or explain theoretically.\n\n3. The article mentions the application of the Mamba model, but the method section does not clearly indicate where it is used for those unfamiliar with the Mamba model. Nor does it explain how the Mamba model contributes to stock prediction.\n\n4. The baselines are relatively weak, with only two methods specifically tailored for stock data.\n\n5. The article does not provide the code.\n\n\n[1] Yang, Linyi, et al. \"Numhtml: Numeric-oriented hierarchical transformer model for multi-task financial forecasting.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 10. 2022.\n\n[2] Liu, Shengzhong, et al. \"FOCAL: Contrastive learning for multimodal time-series sensing signals in factorized orthogonal latent space.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Cheng, Rui, and Qing Li. \"Modeling the momentum spillover effect for stock prediction via attribute-driven graph attention networks.\" Proceedings of the AAAI Conference on artificial intelligence. Vol. 35. No. 1. 2021."}},"nonreaders":[],"tmdate":1731427331770,"tcdate":1730603484203,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission868/Reviewer_Ap4Z"],"signatures":["ICLR.cc/2025/Conference/Submission868/Reviewer_Ap4Z"],"forum":"0x8wWloW2O","number":2,"license":"CC BY 4.0","cdate":1730603484203,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission868/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427331770,"domain":"ICLR.cc/2025/Conference","replyto":"0x8wWloW2O","id":"t3D39nmPmG","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["deep learning","time series"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Stock price prediction is a complex challenge due to the inherent volatility of financial markets and the influence of diverse factors such as macroeconomic conditions, capital flows, and market sentiment. Recent joint stock forecasting models focus on extracting temporal patterns from individual stock price series and combining them to model stock correlations. However, these models face two critical limitations: first, in long-term predictions, they retain both informative and excessive states, amplifying noise and increasing complexity; second, in short-term predictions, they prioritize market indices and technical indicators, neglecting the real-time influence of market sentiment, which can drive price movements independent of traditional indicators. While state space models (SSMs) like Mamba improve efficiency and capture long-distance relationships, they still underperform compared to Transformer-based models.\nTo address these challenges, we propose OracleMamba, a novel framework that integrates a dynamic market-guided module for short-term forecasting and a SelectiveMamba module for long-term forecasting. The dynamic market-guided module fuses objective market data and subjective sentiment analysis to enhance short-term prediction accuracy. The SelectiveMamba module efficiently captures both spectral and temporal features using a 3D scan mechanism, which extracts and filters key signals from the time-series data. By integrating spectral features to identify market rhythms and temporal features to track price movements over time, the SelectiveMamba module reduces noise and preserves critical information for long-term forecasts. This framework significantly improves both model efficiency and accuracy, outperforming existing approaches across real-world stock prediction tasks."},"_bibtex":{"value":"@misc{\nwu2025oraclemamba,\ntitle={OracleMamba: A Dynamic Market-Guided and Time State Selection Framework for Robust Stock Prediction},\nauthor={Song-Li Wu},\nyear={2025},\nurl={https://openreview.net/forum?id=0x8wWloW2O}\n}"},"title":{"value":"OracleMamba: A Dynamic Market-Guided and Time State Selection Framework for Robust Stock Prediction"},"pdf":{"value":"/pdf/8b67f605220c50dda3f4af8b04c6306f848bed41.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"wu|oraclemamba_a_dynamic_marketguided_and_time_state_selection_framework_for_robust_stock_prediction"},"authorids":{"value":["~Song-Li_Wu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Song-Li Wu"]}},"version":2},{"content":{"summary":{"value":"The paper proposes a novel hierarchical architecture, Hi-Mamba, designed to improve image super-resolution using a structured state space model approach. The architecture introduces the Hierarchical Mamba Block, consisting of Local SSM  and Region SSM components to achieve multi-scale visual context modeling with single-direction scanning. Additionally, a Direction Alternation Hierarchical Mamba Group is used to model spatial relationships by alternating scan directions. The proposed method aims to address the high computation cost associated with multi-direction scanning in existing Mamba-based models. Results on several SR benchmarks show improvements in PSNR and computational efficiency."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. How does the directional alternation in DA-HMG specifically enhance spatial relationship modeling, and have other combinations of scan directions been tested to assess their impact on performance? A more detailed explanation or visualization of how these directional alternations contribute to improved spatial modeling would be helpful.\n\n2. Could the authors provide a more detailed comparison of Hi-Mamba’s computational trade-offs, particularly in memory usage and inference latency, relative to other state-of-the-art methods like MambaIR and Transformer-based models?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The Hi-Mamba architecture introduces a hierarchical structure and a new directional alternation technique, which is a creative approach to enhancing Mamba's performance in SR tasks.\n\n2. The paper addresses high computation costs problem of SSM-based SR models due to multi-directional scanning, and provides an alternative that maintains performance with less computational demand."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. While the proposed changes are effective, they appear as relatively incremental modifications over existing SSM-based models, particularly MambaIR. The hierarchical structure and directional alternation are valuable but do not constitute a fundamentally new approach, which may limit the overall impact.\n\n2. The performance improvement of Hi-Mamba compared to other state-of-the-art models, such as MambaIR and Transformer-based SR models, is relatively small. While Hi-Mamba demonstrates slight gains in PSNR and SSIM across various benchmarks, the marginal advantages may not be compelling enough to justify its adoption over other established models, particularly in applications where computational efficiency is a critical factor.\n\n3. The paper lacks a thorough analysis of the complexity trade-offs introduced by the hierarchical structure and directional alternation. The reported reduction in computational complexity is promising, but additional comparisons or ablations on how the design changes impact efficiency, memory usage, and latency would strengthen the paper.\n\n4. While Hi-Mamba shows slight improvements over MambaIR and Transformer-based methods, the paper could be more comprehensive in its comparison against a broader range of state-of-the-art SR methods. Including additional evaluation metrics or visual comparisons on challenging datasets would provide a clearer picture of Hi-Mamba's effectiveness in real-world scenarios."}},"nonreaders":[],"tmdate":1731427888308,"tcdate":1730482550523,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3583/Reviewer_fUus"],"signatures":["ICLR.cc/2025/Conference/Submission3583/Reviewer_fUus"],"forum":"iPYwddLhXR","number":1,"license":"CC BY 4.0","cdate":1730482550523,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3583/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427888308,"domain":"ICLR.cc/2025/Conference","replyto":"iPYwddLhXR","id":"PQ3jRh2PhG","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Image Super-Resolution","Mamba","Structured State Space Models"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSM), such as Mamba, have shown strong representation ability in modeling long-range dependency with linear complexity, achieving successful applications from high-level to low-level vision tasks. However, SSM's sequential nature necessitates multiple scans in different directions to compensate for the loss of spatial dependency when unfolding the image into a 1D sequence. This multi-direction scanning strategy significantly increases the computation overhead and is unbearable for high-resolution image processing. To address this problem, we propose a novel Hierarchical Mamba network, namely, Hi-Mamba, for image super-resolution (SR).  Hi-Mamba consists of two key designs: (1) The Hierarchical Mamba Block (HMB) assembled by a Local SSM (L-SSM) and a Region SSM (R-SSM) both with the single-direction scanning, aggregates multi-scale representations to enhance the context modeling ability. (2) The Direction Alternation  Hierarchical Mamba Group (DA-HMG) allocates the isomeric single-direction scanning into cascading HMBs to enrich the spatial relationship modeling.\nExtensive experiments demonstrate the superiority of Hi-Mamba across five benchmark datasets for efficient SR. For example, Hi-Mamba achieves a significant PSNR improvement of 0.29 dB on Manga109 for $\\times3$ SR, compared to the strong lightweight MambaIR."},"_bibtex":{"value":"@misc{\nqiao2024himamba,\ntitle={Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution},\nauthor={Junbo Qiao and Jincheng Liao and Wei Li and Yulun Zhang and Yong Guo and Yi Wen and Zhangxizi Qiu and Jiao Xie and Jie Hu and Shaohui Lin},\nyear={2024},\nurl={https://openreview.net/forum?id=iPYwddLhXR}\n}"},"title":{"value":"Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution"},"pdf":{"value":"/pdf/e75e168c76e93b40ac8e29dd9015c5e7d72edc8b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"qiao|himamba_hierarchical_mamba_for_efficient_image_superresolution"},"authorids":{"value":["~Junbo_Qiao1","~Jincheng_Liao1","~Wei_Li60","~Yulun_Zhang1","~Yong_Guo1","~Yi_Wen3","~Zhangxizi_Qiu2","~Jiao_Xie1","~Jie_Hu8","~Shaohui_Lin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Junbo Qiao","Jincheng Liao","Wei Li","Yulun Zhang","Yong Guo","Yi Wen","Zhangxizi Qiu","Jiao Xie","Jie Hu","Shaohui Lin"]}},"version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"venue":{"value":"AdaptFM Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"pdf":{"value":"/pdf/afc446abfd6118b779943514344d2a153aba8753.pdf"},"keywords":{"value":["Foundation Models","Mamba","State Space Models","Adaptive Inference","Efficient Inference"]},"venueid":{"value":"ICML.cc/2026/Workshop/AdaptFM"},"paperhash":{"value":"gwak|recovering_selectivity_with_lti_state_space_operators_for_portable_longcontext_inference"},"authorids":{"value":["~Minseon_Gwak1","~N._Benjamin_Erichson1","~PooGyeon_Park1"]},"abstract":{"value":"Mamba has established selective state space models as powerful building blocks for long-context foundation models by replacing linear time-invariant (LTI) recurrences with input-dependent selective recurrences. However, efficient training of such recurrences often relies on specialized fused kernels, limiting portability across diverse hardware. We propose SelectLTI, a simple but effective block that augments an LTI recurrence with input-dependent modulation to recover selective behavior. The proposed block equips any LTI state-space layer with sigmoid-gated modulators, making it trainable and deployable without specialized fused kernels. We evaluate SelectLTI on Selective Copying, a task designed to isolate selective-memory behavior in sequence models. SelectLTI matches or exceeds S6, the selective state-space operator of Mamba, showing that selective behavior can be recovered from an LTI recurrence augmented with lightweight modulation using only standard primitives. We position SelectLTI as a portable building block for efficient long-context foundation models deployed across diverse compute environments."},"_bibtex":{"value":"@inproceedings{\ngwak2026recovering,\ntitle={Recovering Selectivity with {LTI} State Space Operators for Portable Long-Context Inference},\nauthor={Minseon Gwak and N. Benjamin Erichson and PooGyeon Park},\nbooktitle={AdaptFM: Resource-Adaptive Foundation Model Inference},\nyear={2026},\nurl={https://openreview.net/forum?id=sUzfGWAzOy}\n}"},"title":{"value":"Recovering Selectivity with LTI State Space Operators for Portable Long-Context Inference"},"authors":{"value":["Minseon Gwak","N. Benjamin Erichson","PooGyeon Park"]}},"tmdate":1780552888619,"pdate":1780353776095,"tcdate":1777673415436,"writers":["ICML.cc/2026/Workshop/AdaptFM","ICML.cc/2026/Workshop/AdaptFM/Submission47/Authors"],"signatures":["ICML.cc/2026/Workshop/AdaptFM/Submission47/Authors"],"forum":"sUzfGWAzOy","license":"CC BY 4.0","number":47,"cdate":1777673415436,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/AdaptFM/-/Submission","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Release","ICML.cc/2026/Workshop/AdaptFM/Submission47/-/Camera_Ready_Revision","ICML.cc/2026/Workshop/AdaptFM/-/Edit"],"mdate":1780552888619,"odate":1780353776095,"domain":"ICML.cc/2026/Workshop/AdaptFM","id":"sUzfGWAzOy","version":2},{"content":{"summary":{"value":"This paper studies the in-context learning of linear regressions with Mamba-2 and a hybrid model. The authors begin by showing the existence of a one-layer Mamba-2 model that is simulating one-step of gradient descent (GD) in its forward pass. The authors further compare the errors of Mamba-2 models and one-step GD in experiments. Then they study the hybrid layer of a Mamba-2 layer followed by a linear self-attention layer, claiming that a single hybrid layer could implement multiple steps of stochastic gradient descent (SGD) in-context. Finally the authors evalute hybrid models of different depths and compare them with SGD."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"1. In the proof to Theorem 5.1, at line 364 why is $\\|B\\|\\leq \\epsilon$? At line 749, the constructed $B$ seems to be identity.\n2. Line 372, is there a specific reason of choosing $t=l \\text{ mod } n$?\n3. In the proof to Theorem 5.2, by definition of operator $T$, the increment $\\Delta M$ in eq(9) should be a degree-$(2K)$ function of $x$ because $T$ is in degree-$2$ of $x$. Here $K$ is the number of GD. Then by equation (11) $Q$ should be in degree-$(2K-2)$ of $x$, and we have the weight matrix $B=Q$. Does this suggest that for this specific construction, any $K>1$ should cause the weight matrix depending on the input context?\n4. What is the learning rate for GD in Fig 1? Why does the error curve of GD keep flat while increasing the number of in-context examples?\n5. What is the number of steps of SGD in Fig 3?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"The authors uses the duality between Mamba-2 and causal linear self-attention. Hence the construction of one-layer Mamba-2 performing GD in-context is convincing."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The section (Sec 5.1) of hybrid model is confusing and questionable. The first theorem of Sec 5.1 claims a one-layer hybrid model could approximate one step of SGD. The statement of the theorem seems inaccurate and the proof appears to be unreliable to me. The authors are taking the length of context to infinity but this is not mentioned in the theorem. The proof relies on that the learned weights matrix (B) is small, which does not seem to be true by the construction. The second theorem surprisingly states that a single hybrid layer could perform multiple steps of SGD. However, from the proof, the constructed weights matrix seems to depend on the input context, which makes the theorem incorrect. \n\n2. The experiments do not really justify the theory and some hyperparameters are missing. In Fig 1, the performance of one-layer Mamba is much better than one step of GD, and the learning rate of GD is not reported. The error curve of GD is a bit confusing since the error does not decrease as the number of in-context example increases.  \n\nSee questions for details."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931364110,"tcdate":1762203098945,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19444/Reviewer_eSv3"],"signatures":["ICLR.cc/2026/Conference/Submission19444/Reviewer_eSv3"],"forum":"lhvdbCiiE6","number":3,"license":"CC BY 4.0","cdate":1762203098945,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19444/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931364110,"domain":"ICLR.cc/2026/Conference","replyto":"lhvdbCiiE6","id":"4jbFpFUR81","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mamba-2","Stochastic Gradient Descent","In-context Learning"]},"primary_area":{"value":"interpretability and explainable AI"},"abstract":{"value":"State space models (SSMs) have gained popularity as an alternative to Transformers by mitigating the quadratic computational cost associated with self-attention. However, despite their widespread adoption, the theoretical principles underlying their ability to perform in-context learning (ICL) remain poorly understood. In this work, we theoretically analyze the widely used Mamba-2 model (Dao et al. 2024) and demonstrate that a single-layer Mamba-2 can simulate one step of gradient descent, while a hybrid architecture combining Mamba with a Transformer (Mamba $\\circ$ TF) can perform mini-batch stochastic gradient descent. Our experimental results support these theoretical findings."},"_bibtex":{"value":"@misc{\nwang2026incontext,\ntitle={In-Context Stochastic Gradient Descent with Hybrid Mamba-2 and Linear Self-Attention Model},\nauthor={Zhijie Wang and Shuai Li},\nyear={2026},\nurl={https://openreview.net/forum?id=lhvdbCiiE6}\n}"},"title":{"value":"In-Context Stochastic Gradient Descent with Hybrid Mamba-2 and Linear Self-Attention Model"},"pdf":{"value":"/pdf/b29d3c388b906a2dc7dd59de9cc6bccbba82a3f8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wang|incontext_stochastic_gradient_descent_with_hybrid_mamba2_and_linear_selfattention_model"},"authorids":{"value":["~Zhijie_Wang7","~Shuai_Li3"]},"authors":{"value":["Zhijie Wang","Shuai Li"]}},"version":2},{"content":{"summary":{"value":"This paper targets the problem of long-range feature interactions for point cloud detection. It proposes a window-based 3D backbone based on linear group RNN and sparse convolution. In contrast to existing Transformers methods, this work increases the group size by leveraging the linear complexity of recent Mamba and RWKV. The paper also introduces the 3D spatial feature descriptor to capture 3D local spatial information and 3D voxel generation strategy to address the sparsity of point clouds. Experiments demonstrate the efficacy of this proposed method."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please see the weaknesses section. Additionally,  \n$\\bullet$\tPlease provide a comparison with HEDNet in Waymo, including latency, accuracy, and parameters.  \n$\\bullet$\tIt would be better to illustrate visualization results to support the claim LION can model long-range dependencies.  \n$\\bullet$\tWhat is the performance of replacing the LION Layer with DSVT block (use the setting in the original paper) in the LION Block? Moreover, please provide a comparison of the latency between the two variants."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"$\\bullet$  The problem studied in this paper is important, as the long-range relationship is critical in point cloud detection.\n\n$\\bullet$  LION-Mamba achieves strong performance with low GFLOPs on widely used outdoor datasets, including the Waymo Open Dataset and nuScene.\n\n$\\bullet$  The proposed 3D spatial feature descriptor and voxel generation are simple and easy to follow."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"$\\bullet$  [Novelty] My main concern is the overall limited technical contribution. The paper does not show significant differences from previous works. Additionally, it lacks a thorough discussion comparing it with existing approaches.  \n$\\qquad$(1) Model Structure. The proposed LION block uses the same encoder-decoder structure as SED block in HEDNet [1]. Furthermore, the 3D spatial feature descriptor is identical to SSR block, and the voxel merging and expanding operations are similar to RS conv. Besides, the LION layer has the same structure with DSVT block. This paper appears to merely integrate the DSVT block into SED block and replace Transformers with linear RNNs.  \n$\\qquad$(2) Window Partitions. The equal-size window partition along X/Y-axis has been widely adopted in voxel-based detectors, such as Flatformer and DSVT.   \n$\\qquad$(3) Voxel Generation: The way to “distinguishing foreground voxels without supervision” has been widely used in point cloud detection, such as SPSS-Conv [2] and VoxelNeXt [3].   \n$\\bullet$  [Latency] While there is some analysis of computation cost (GFLOPs), it lacks a comparison of latency with state-of-the-art algorithms, such as HEDNet and DSVT. For 3D object detection in outdoor scenarios like autonomous driving, the real-time application makes the latency more critical. Besides, in Figure 1, the comparison with LION-Mamba-L is missing.  \n$\\bullet$  In line 42-44, the authors claim LION can support thousands of voxel features to establish long-range relationships. However, in Line 153-157, this paper proposes the spatial information loss issue and need additional local sparse convolution to address. In my point of view, a sequence of thousands based on window partitions must include the voxel and all its neighbors. This inconsistency raises concerns about the robustness and effectiveness of LION.  \n$\\bullet$ The linear RNN, such as Mamba, is a unidirectional model. Is it reasonable to use two single-layer RNNs in a LION Layer? Will the bidirectional Mamba, such as Vision Mamba, enhance performance?  \n$\\bullet$  [Motivation] The X/Y-axis window partition is proposed to address the feature interactions problem with group size limitation. The motivation to use X/Y-axis window partition is not clear, given the large group size.  \n$\\bullet$  [Motivation] The motivation for choosing only four different offsets in voxel generation with auto-regressive property is unclear. A voxel has many neighbors in 3D space, so why did the author choose only these four offsets? Further explanation is needed.  \n\n[1] HEDNet: A hierarchical encoder-decoder network for 3d object detection in point clouds. NIPS’23  \n[2] Spatial Pruned Sparse Convolution for Efficient 3D Object Detection. NIPS’22   \n[3] VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking. CVPR’23"},"limitations":{"value":"Limitations have been included"}},"nonreaders":[],"tmdate":1730878664613,"tcdate":1720794343915,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission886/Reviewer_wvps"],"signatures":["NeurIPS.cc/2024/Conference/Submission886/Reviewer_wvps"],"forum":"5tGkAcY7uV","number":3,"license":"CC BY 4.0","cdate":1720794343915,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission886/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878664613,"domain":"NeurIPS.cc/2024/Conference","replyto":"5tGkAcY7uV","id":"OliWipJYUo","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["3D Object Detection","Linear RNN"]},"primary_area":{"value":"machine_vision"},"abstract":{"value":"The benefit of transformers in large-scale 3D point cloud perception tasks, such as 3D object detection, is limited by their quadratic computation cost when modeling long-range relationships. In contrast, linear RNNs have low computational complexity and are suitable for long-range modeling. Toward this goal, we propose a simple and effective window-based framework built on Linear group RNN (i.e., perform linear RNN for grouped features) for accurate 3D object detection, called LION. The key property is to allow sufficient feature interaction in a much larger group than transformer-based methods. However, effectively applying linear group RNN to 3D object detection in highly sparse point clouds is not trivial due to its limitation in handling spatial modeling. To tackle this problem, we simply introduce a 3D spatial feature descriptor and integrate it into the linear group RNN operators to enhance their spatial features rather than blindly increasing the number of scanning orders for voxel features. To further address the challenge in highly sparse point clouds, we propose a 3D voxel generation strategy to densify foreground features thanks to linear group RNN as a natural property of auto-regressive models. \nExtensive experiments verify the effectiveness of the proposed components and the generalization of our LION on different linear group RNN operators including Mamba, RWKV, and RetNet. Furthermore, it is worth mentioning that our LION-Mamba achieves state-of-the-art on Waymo, nuScenes, Argoverse V2, and ONCE datasets. Last but not least, our method supports kinds of advanced linear RNN operators (e.g., RetNet, RWKV, Mamba, xLSTM and TTT) on small but popular KITTI dataset for a quick experience with our linear RNN-based framework."},"_bibtex":{"value":"@inproceedings{\nliu2024lion,\ntitle={{LION}: Linear Group {RNN} for 3D Object Detection in Point Clouds},\nauthor={Zhe Liu and Jinghua Hou and Xinyu Wang and Xiaoqing Ye and Jingdong Wang and Hengshuang Zhao and Xiang Bai},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=5tGkAcY7uV}\n}"},"title":{"value":"LION: Linear Group RNN for 3D Object Detection in Point Clouds"},"pdf":{"value":"/pdf/4328f471f60186bd07dea76178ce8929385cb6e4.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"liu|lion_linear_group_rnn_for_3d_object_detection_in_point_clouds"},"authorids":{"value":["~Zhe_Liu12","~Jinghua_Hou1","~Xinyu_Wang7","~Xiaoqing_Ye1","~Jingdong_Wang1","~Hengshuang_Zhao2","~Xiang_Bai1"]},"authors":{"value":["Zhe Liu","Jinghua Hou","Xinyu Wang","Xiaoqing Ye","Jingdong Wang","Hengshuang Zhao","Xiang Bai"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"li|msfmamba_motionaware_state_fusion_mamba_for_efficient_microgesture_recognition"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Deng_Li:","https://dblp.org/search/pid/api?q=author:Jun_Shao:","~Bohao_Xing2","https://dblp.org/search/pid/api?q=author:Rong_Gao:","https://dblp.org/search/pid/api?q=author:Bihan_Wen:","~Heikki_Kälviäinen2","https://dblp.org/search/pid/api?q=author:Xin_Liu:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2510.10478"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2510-10478,\n  publtype={informal},\n  author={Deng Li and Jun Shao and Bohao Xing and Rong Gao and Bihan Wen and Heikki Kälviäinen and Xin Liu},\n  title={MSF-Mamba: Motion-aware State Fusion Mamba for Efficient Micro-Gesture Recognition},\n  year={2025},\n  month={October},\n  cdate={1759276800000},\n  journal={CoRR},\n  volume={abs/2510.10478},\n  url={https://doi.org/10.48550/arXiv.2510.10478}\n}\n"},"abstract":{"value":"Micro-gesture recognition (MGR) targets the identification of subtle and fine-grained human motions and requires accurate modeling of both long-range and local spatiotemporal dependencies. While CNNs are effective at capturing local patterns, they struggle with long-range dependencies due to their limited receptive fields. Transformer-based models address this limitation through self-attention mechanisms but suffer from high computational costs. Recently, Mamba has shown promise as an efficient model, leveraging state space models (SSMs) to enable linear-time processing However, directly applying the vanilla Mamba to MGR may not be optimal. This is because Mamba processes inputs as 1D sequences, with state updates relying solely on the previous state, and thus lacks the ability to model local spatiotemporal dependencies. In addition, previous methods lack a design of motion-awareness, which is crucial in MGR. To overcome these limitations, we propose motion-aware state fusion mamba (MSF-Mamba), which enhances Mamba with local spatiotemporal modeling by fusing local contextual neighboring states. Our design introduces a motion-aware state fusion module based on central frame difference (CFD). Furthermore, a multiscale version named MSF-Mamba+ has been proposed. Specifically, MSF-Mamba supports multiscale motion-aware state fusion, as well as an adaptive scale weighting module that dynamically weighs the fused states across different scales. These enhancements explicitly address the limitations of vanilla Mamba by enabling motion-aware local spatiotemporal modeling, allowing MSF-Mamba and MSF-Mamba to effectively capture subtle motion cues for MGR. Experiments on two public MGR datasets demonstrate that even the lightweight version, namely, MSF-Mamba, achieves SoTA performance, outperforming existing CNN-, Transformer-, and SSM-based models while maintaining high efficiency."},"title":{"value":"MSF-Mamba: Motion-aware State Fusion Mamba for Efficient Micro-Gesture Recognition"},"authors":{"value":["Deng Li","Jun Shao","Bohao Xing","Rong Gao","Bihan Wen","Heikki Kälviäinen","Xin Liu"]}},"tmdate":1769328235469,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2510-10478"],"tcdate":1762941929642,"writers":["~"],"signatures":["~Bohao_Xing2"],"forum":"LtLORVcGFV","license":"CC BY-SA 4.0","number":693625,"cdate":1759276800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769328235469,"domain":"DBLP.org","id":"LtLORVcGFV","version":2},{"content":{"summary":{"value":"This paper proposes a new hybrid architecture that combines the recurrent formulation of state-space models with selective attention layers (SALs). The key component introduced, SAL, identifies tokens that have long-context dependencies, refines their features and augments their representations with standard attention. This aims to increase performance on tasks that require long-context memory without incurring the quadratic cost in standard Transformers. The evaluation performed on various tasks  and extrapolation shows superior performance compared to standard (Transformer++), efficient (Mamba) and hybrid (Jamba) models.  In addition, performance on recall-intensive tasks and model sizes up to 1.3B are also encouraging."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. What is the computational benefit for different attention budgets and usage in different layers compared to full attention in terms of memory and latency? It would be useful if the authors provide some empirical evidence to quantify the benefits of SALs. \n2. Could the authors include a few additional baselines in the datasets under study? I'd suggest to report scores in a table from prior work with efficient or hybrid architectures on the same datasets (e..g RecurrentGemma).\n 3. The comparison to a baseline that uses only sliding window attention with the same window size as Tapain is missing. Could the authors report scores for this baseline across the tasks used in the experiment section? This would help to better understand the impact of selective attention."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Existing hybrid architectures that combine state-space models with standard attention typically are applied to a small subset of layers for all tokens. The idea of applying attention only to a specific subset of positions dynamically through selective attention is novel and provides a more efficient way to augment recurrent models with attention.  \n2. The proposed model outperforms consistently two strong baselines that represent efficient and hybrid models, namely Mamba and Jamba, on general and recall-intensive tasks. At the same time, it outperforms standard attention models represented by Transformer++ or is behind by a moderate margin (~10% relative in recall-intensive tasks). \n3. Apart from the quality, the proposed model has superior extrapolation capabilities up to 1M tokens and lower latency with increasing context size compared to the aforementioned baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Even though the goal of selective attention is to improve efficiency, the experimental section does not quantify the computational benefits in terms of memory and latency compared to full attention or different budgets in practice. I'd suggest extending the experiment in Figure 5 to include memory use and training/inference times. \n2. The comparison to previous efficient and hybrid models has limited coverage as it included only two baseline models and model sizes up to 1.3B. This reduces the potential impact of the main findings. To strengthen the claims regarding scaling, I'd suggest adding a larger model to reach the 7B mark and including a table with results compared to other recent efficient or hybrid architectures such as RecurrentGemma.  \n3. The experiment scope could benefit from recent general evaluation benchmarks for LLMs (MMLU, HELM, BBH), and instruction tuning or preference optimization experiments, with higher priority on the general evaluation.  The effect of different hyper-parameters such as sliding window size from 64 up to maximum context  length in a controlled experiment would also be useful."}},"nonreaders":[],"tmdate":1731429282484,"tcdate":1730706394883,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9385/Reviewer_BRJK"],"signatures":["ICLR.cc/2025/Conference/Submission9385/Reviewer_BRJK"],"forum":"9DnKZbOr4r","number":4,"license":"CC BY 4.0","cdate":1730706394883,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9385/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429282484,"domain":"ICLR.cc/2025/Conference","replyto":"9DnKZbOr4r","id":"EwWyOTSKnK","forumContent":{"TLDR":{"value":"We introduce Taipan, a novel architecture that combines State Space Models with Selective Attention Layers, achieving superior performance in memory-intensive tasks while enabling efficient language modeling for sequences up to 1 million tokens."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Efficient Language Model","Model Architecture","Long-context Language Model","In-context Retrieval","Hybrid Architecture","Linear Complexity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long sequences due to quadratic computational complexity in training and linearly scaling memory costs during inference. Recent State Space Models (SSMs) such as Mamba offer alternatives with constant memory usage, but they underperform in tasks requiring extensive in-context retrieval. We introduce Taipan, a novel hybrid architecture that combines Mamba-2 with Selective Attention Layers (SALs). These SALs identify tokens requiring long-range interactions, remove less important features, and then augment their representations using the attention module. This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. By constraining the attention budget, Taipan extends accurate predictions to context lengths of up to 1 million tokens while preserving computational efficiency. Our experiments demonstrate Taipan's superior performance across various scales and tasks, offering a promising solution for efficient long-context language modeling."},"_bibtex":{"value":"@misc{\nnguyen2025taipan,\ntitle={Taipan: Efficient and Expressive State Space Language Models with Selective Attention},\nauthor={Chien Van Nguyen and Huy Huu Nguyen and Thang M. Pham and Ruiyi Zhang and Hanieh Deilamsalehy and Puneet Mathur and Ryan A. Rossi and Trung Bui and Viet Dac Lai and Franck Dernoncourt and Thien Huu Nguyen},\nyear={2025},\nurl={https://openreview.net/forum?id=9DnKZbOr4r}\n}"},"title":{"value":"Taipan: Efficient and Expressive State Space Language Models with Selective Attention"},"pdf":{"value":"/pdf/4c569111617737e5532042f859ac730be0ac18ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"nguyen|taipan_efficient_and_expressive_state_space_language_models_with_selective_attention"},"authorids":{"value":["~Chien_Van_Nguyen1","~Huy_Huu_Nguyen1","~Thang_M._Pham1","~Ruiyi_Zhang3","~Hanieh_Deilamsalehy1","~Puneet_Mathur2","~Ryan_A._Rossi2","~Trung_Bui1","~Viet_Dac_Lai1","~Franck_Dernoncourt1","~Thien_Huu_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chien Van Nguyen","Huy Huu Nguyen","Thang M. Pham","Ruiyi Zhang","Hanieh Deilamsalehy","Puneet Mathur","Ryan A. Rossi","Trung Bui","Viet Dac Lai","Franck Dernoncourt","Thien Huu Nguyen"]}},"version":2},{"content":{"venue":{"value":"ISBI 2025"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/10980665/10980666/10980814.pdf"},"venueid":{"value":"dblp.org/conf/ISBI/2025"},"paperhash":{"value":"sun|r2genmamba_a_selective_state_space_model_for_radiology_report_generation"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Yongheng_Sun:","https://dblp.org/search/pid/api?q=author:Yueh_Z._Lee:","https://dblp.org/search/pid/api?q=author:Genevieve_A._Woodard:","https://dblp.org/search/pid/api?q=author:Hongtu_Zhu:","https://dblp.org/search/pid/api?q=author:Chunfeng_Lian:","~Mingxia_Liu1"]},"html":{"value":"https://doi.org/10.1109/ISBI60581.2025.10980814"},"_bibtex":{"value":"@inproceedings{DBLP:conf/isbi/SunLWZL025,\n  author={Yongheng Sun and Yueh Z. Lee and Genevieve A. Woodard and Hongtu Zhu and Chunfeng Lian and Mingxia Liu},\n  title={R2Gen-Mamba: A Selective State Space Model for Radiology Report Generation},\n  year={2025},\n  cdate={1735689600000},\n  pages={1-4},\n  url={https://doi.org/10.1109/ISBI60581.2025.10980814},\n  booktitle={ISBI},\n  crossref={conf/isbi/2025}\n}\n"},"abstract":{"value":"Radiology report generation is crucial in medical imaging, but the manual annotation process by physicians is time-consuming and labor-intensive, necessitating the development of automatic report generation methods. Existing research predominantly utilizes Transformers to generate radiology reports, which can be computationally intensive, limiting their use in real applications. In this work, we present R2Gen-Mamba, a novel automatic radiology report generation method that leverages the efficient sequence processing of the Mamba with the contextual benefits of Transformer architectures. Due to lower computational complexity of Mamba, R2Gen-Mamba not only enhances training and inference efficiency but also produces high-quality reports. Experimental results on two benchmark datasets with more than 210,000 radiograph-report pairs demonstrate the effectiveness of R2Gen-Mamba regarding report quality and computational efficiency compared with several state-of-the-art methods. The source code can be accessed online."},"title":{"value":"R2Gen-Mamba: A Selective State Space Model for Radiology Report Generation"},"authors":{"value":["Yongheng Sun","Yueh Z. Lee","Genevieve A. Woodard","Hongtu Zhu","Chunfeng Lian","Mingxia Liu"]}},"tmdate":1754405458072,"pdate":1735689600000,"externalIds":["dblp:conf/isbi/SunLWZL025"],"tcdate":1754405420253,"writers":["~"],"signatures":["~Mingxia_Liu1"],"forum":"D8aHSxkThf","license":"CC BY-SA 4.0","number":614948,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1754405458072,"domain":"DBLP.org","id":"D8aHSxkThf","version":2},{"content":{"comment":{"value":"### Weakness 3\n**We have clearly stated our motivation in our abstract and introduction.** Let us re-clarify it step-by-step:\n\n1. Recent methods for dynamic graph representation learning such as DyGFormer and CTAN have found that modeling long-term temporal information is necessary and contributes greatly. So we follow this finding and wish to propose a model that can achieve that. This corresponds to line 43-53 of our submission.\n\n2. The current state-of-the-art DyGFormer incorporates long-term temporal information by modeling very long historical node interaction sequences. Due to the high complexity of the Transformer in sequence modeling, DyGFormer requires much more computational resources as the sequence length increases. This poses challenge in efficiently modeling very long node interaction sequences (mentioned as the first challenge in line 52-53.)\n**Due to SSMs' low complexity in sequence modeling, we choose Mamba SSM as an alternative to improve efficiency in long sequence modeling (line 57-60).**\n\n3. Another challenge rises when we model long node interaction sequences. Long sequences introduces more information, causing burden to the models in distinguishing useful temporal information from redundant parts (line 53-56). Note that the claim \"long-term temporal information is useful in dynamic graph modeling\" does not contradict the claim \"we need to select the important part and discard the redundant information\". To address this, we propose to use the learned temporal patterns to help select information. We have used an example to illustrate our motivation for leveraging temporal patterns to achieve this (line 61-69). **Temporal patterns can help to prioritize information selection from relevant node interactions. The relevance of interactions is represented with weights ($\\beta_\\theta$ in Eq. 7) which are derived based on temporal patterns.**\n\n### Weakness 4\nPlease note that **only 7 datasets from DyGLib are continuous-time dynamic graphs (CTDGs), and we have considered all of them**. CTDGs are completely different from discrete-time dynamic graphs (DTDGs). We have explained their differences in line 35-42 and have explicitly specified that **we only consider CTDG modeling** in our title, abstract and everywhere else.\n\n**We have also mentioned and given explanation why we do not consider historical negative sampling evaluation in inductive setting in line 327-332 and Appendix D.** We wish you could pay attention to our contents more carefully.\n\n### Weakness 5\nThank you for the suggestion, we will include more detailed complexity analysis in our appendix.\n\n### Weakness 6\nWe appreciate and respect your opinion. But we have to emphasize that from our perspective, ICLR should not be a platform where only the papers delivering substantial theoretical contributions are valued. We admit that our submission is a more technical paper rather than a theoretical one. Our contributions have been clearly outlined in line 70-77. We believe these contributions are enough to make our work appear at ICLR and they also meet the interest of the community. We hope you could understand.\n\n[1] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n\n[2] Zhu, Lianghui, et al. \"Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model.\" Forty-first International Conference on Machine Learning."},"title":{"value":"Response to Reviewer zXNQ: Part B"}},"tmdate":1731935674912,"tcdate":1731930038372,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5146/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5146/Authors"],"forum":"f3gCs2a4ZD","number":8,"license":"CC BY 4.0","cdate":1731930038372,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5146/-/Official_Comment"],"mdate":1731935674912,"domain":"ICLR.cc/2025/Conference","replyto":"yfcv6Fk76j","id":"fAcM0wGXqF","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["dynamic graph","state space model"]},"supplementary_material":{"value":"/attachment/2dc1a3502a1d53b635f3c846cad8277b987ac60b.zip"},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Learning useful representations for continuous-time dynamic graphs (CTDGs) is challenging, due to the concurrent need to span long node interaction histories and grasp nuanced temporal details. In particular, two problems emerge: (1) Encoding longer histories requires more computational resources, making it crucial for CTDG models to maintain low computational complexity to ensure efficiency; (2) Meanwhile, more powerful models are needed to identify and select the most critical temporal information within the extended context provided by longer histories. To address these problems, we propose a CTDG representation learning model named DyGMamba, originating from the popular Mamba state space model (SSM). DyGMamba first leverages a node-level SSM to encode the sequence of historical node interactions. Another time-level SSM is then employed to exploit the temporal patterns hidden in the historical graph, where its output is used to dynamically select the critical information from the interaction history. We validate DyGMamba experimentally on the dynamic link prediction task. The results show that our model achieves state-of-the-art in most cases. DyGMamba also maintains high efficiency in terms of computational resources, making it possible to capture long temporal dependencies with a limited computation budget."},"_bibtex":{"value":"@misc{\nding2025dygmamba,\ntitle={Dy{GM}amba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models},\nauthor={Zifeng Ding and Yifeng Li and Yuan He and Antonio Norelli and Jingcheng Wu and Volker Tresp and Yunpu Ma and Michael M. Bronstein},\nyear={2025},\nurl={https://openreview.net/forum?id=f3gCs2a4ZD}\n}"},"title":{"value":"DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models"},"pdf":{"value":"/pdf/0921c754d017d250e0d2f4b59a779766e47b30a6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ding|dygmamba_efficiently_modeling_longterm_temporal_dependency_on_continuoustime_dynamic_graphs_with_state_space_models"},"authorids":{"value":["~Zifeng_Ding1","~Yifeng_Li4","~Yuan_He5","~Antonio_Norelli2","~Jingcheng_Wu3","~Volker_Tresp1","~Yunpu_Ma1","~Michael_M._Bronstein1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zifeng Ding","Yifeng Li","Yuan He","Antonio Norelli","Jingcheng Wu","Volker Tresp","Yunpu Ma","Michael M. Bronstein"]}},"version":2},{"content":{"summary":{"value":"This paper investigates recurrent architectures for large action models that enable efficient inference. It proposes the Large Recurrent Action Model (LRAM), featuring an xLSTM core, which achieves linear-time inference complexity and supports natural extrapolation to longer sequence lengths. Experiments conducted on 432 tasks across six domains demonstrate that LRAM performs competitively with Transformers, offering advantages in both speed and performance."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. Why is only xLSTM mentioned as the core of LRAM in the abstract and introduction? The authors also indicate that both xLSTM and Mamba provide parallelization benefits during training similar to the Transformer architecture while enabling faster inference.\n2. How is the shared action head used across all environments, given that each environment may have a different output dimension and that continuous actions require discretization?\n3. In Figure 2(b), should the performance of different environments be weighted according to the number of tasks in each environment to prevent any single environment from disproportionately influencing the results? For example, only in ProcGen, xLSTM [1:0] appears to show a distinct advantage.\n4. Why did the authors choose to place the results of fine-tuning in Section 4.3 in the appendix, considering that there is still space to present these results in the main paper?\n5. I am interested in the training details for multi-task training. With 432 tasks, could the learning curve be affected by conflicting gradients inherent to multi-task learning?\n6. In Section 4.4, the authors investigate inference time within a simulation environment. However, during inference, it is common to set $B=1$, where the latency of DT and xLSTM is comparable. A notable difference in inference time only appears when $B=16$. Given this, is the use of xLSTM truly necessary?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper is well-written and easy-to-read.\n2. The study compiles 894 million transitions and conducts experiments across 432 tasks in six domains, demonstrating a substantial research effort."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. There is a lack of analysis regarding certain experimental observations. Specifically, an explanation for the lower performance of the Decision Transformer (DT) compared to recurrent-based agents would be valuable, especially since DT has more parameters, which, according to scaling laws, would be expected to yield higher performance.\n2. Additionally, I am interested in understanding why xLSTM outperforms Mamba, as both architectures are designed for linear-time inference complexity. In Line 300, what is meant by \"This is an important advantage of xLSTM, as LRAM agents can strongly benefit from more data and consequently larger models\"?\n3. It would enhance clarity to make key findings more prominent, as in DeMa [1], which also investigates the benefits of integrating Mamba within sequence modeling frameworks.\n\n[1] Yang Dai, et al. Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning? 2024"}},"nonreaders":[],"tmdate":1732243341336,"tcdate":1730203342605,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11354/Reviewer_uzTK"],"signatures":["ICLR.cc/2025/Conference/Submission11354/Reviewer_uzTK"],"forum":"T1pUS4GZZq","number":1,"license":"CC BY 4.0","cdate":1730203342605,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11354/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732243341336,"domain":"ICLR.cc/2025/Conference","replyto":"T1pUS4GZZq","id":"GTMQzbMsxE","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["reinforcement learning","rnn","xlstm","mamba","multi-task","robotics"]},"supplementary_material":{"value":"/attachment/4b1723f92b663e03eabe18d64308fbfc67cab4a1.zip"},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"In recent years, there has been a trend in the field of Reinforcement Learning (RL) towards large action models trained offline on large-scale datasets via sequence modeling. Existing models are primarily based on the Transformer architecture, which results in powerful agents. However, due to slow inference times, Transformer-based approaches are impractical for real-time applications, such as robotics. Recently, modern recurrent architectures, such as xLSTM and Mamba, have been proposed that exhibit parallelization benefits during training similar to the Transformer architecture while offering fast inference. In this work, we study the aptitude of these modern recurrent architectures for large action models. Consequently, we propose a Large Recurrent Action Model (LRAM) with an xLSTM at its core that comes with linear-time inference complexity and natural sequence length extrapolation abilities. Experiments on 432 tasks from 6 domains show that LRAM compares favorably to Transformers in terms of performance and speed."},"_bibtex":{"value":"@misc{\nschmied2025a,\ntitle={A Large Recurrent Action Model: x{LSTM} enables Fast Inference for Robotics Tasks},\nauthor={Thomas Schmied and Thomas Adler and Vihang Prakash Patil and Maximilian Beck and Korbinian P{\\\"o}ppel and Johannes Brandstetter and G{\\\"u}nter Klambauer and Razvan Pascanu and Sepp Hochreiter},\nyear={2025},\nurl={https://openreview.net/forum?id=T1pUS4GZZq}\n}"},"title":{"value":"A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks"},"pdf":{"value":"/pdf/93f7f62e07ca5bd972621079ce7349afd666bf0d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"schmied|a_large_recurrent_action_model_xlstm_enables_fast_inference_for_robotics_tasks"},"authorids":{"value":["~Thomas_Schmied1","~Thomas_Adler1","~Vihang_Prakash_Patil1","~Maximilian_Beck1","~Korbinian_Pöppel1","~Johannes_Brandstetter1","~Günter_Klambauer1","~Razvan_Pascanu1","~Sepp_Hochreiter1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Thomas Schmied","Thomas Adler","Vihang Prakash Patil","Maximilian Beck","Korbinian Pöppel","Johannes Brandstetter","Günter Klambauer","Razvan Pascanu","Sepp Hochreiter"]}},"version":2},{"content":{"summary":{"value":"This paper explores parameter-efficient tuning methods such as the prominent methods like Adapter, LoRA, Prompt-tuning, and Partial tuning for the Mamba architecture. The authors highlight that these methods have not been studied in Mambas compared to the extensive research conducted on Transformers. The primary focus of this paper is on the Vision Mamba (Vim) architecture, with a minor portion addressing language models with Mamba compared with the Transformer counterparts Pythia. The study compares Vim and Vision Transformer models using PEFT methods, which address some key architectural and methodological choices like layer selection for LoRA and prompt placement in Prompt-tuning, for example. The authors present a two-step PEFT search method that hybridizes various PEFT options to offer a near-optimal solution (in terms of performance). Finally, the authors present a new PEFT method called Additional-scan, which increases the state dimension of the SSM, and an additional initialization method for the state matrix A is also presented. The experimental results employing Vim-S pre-trained on ImageNet-1K to demonstrate the effectiveness of the diverse PEFT options; Mamba-language models compared with Pythias are also presented."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- Could the authors provide additional results using Vim-Base or larger-scale architectures?\n- How was the data utilized in the hybrid search method? Was a held-out set created for the searches?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The paper is well-written and easy to follow.\n- Searching for diverse PEFT options for Mamba is informative and would be a valuable contribution for those who employ Mamba architectures. \n- Studying extensive PEFT options based on a baseline yields convincing results."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"#### #  Main concerns\n- The authors' claim in line 22 that PEFT is more effective for Mamba than for Transformers lacks adequate rationale and supporting evidence. There is no clear intuition as to why Mamba would benefit more from PEFT; the experiments appear biased due to an unfair comparison - specifically, Transformers were tested with only a limited range of PEFT options, while Mamba underwent more comprehensive testing (as illustrated in Table 1).\n- The authors primarily tested Vision Mamba rather than the broader Mamba architecture, yet they frequently used \"Mamba\" as an abbreviation for Vision Mamba. This reviewer is concerned this may be misleading, especially as the result table for the language models offers only limited experiments. The authors should clarify their terminology to ensure readers understand that Vision Mamba is the specific variant.\n- The authors presented a hybrid PEFT approach as an optimal-like solution, which is identified through their proposed search method. However, its conclusion does not specify a practical, generalizable option. This reviewer feels It's important to note that the effectiveness of the identified hybrid approach may not be universal, and the search process itself would be resource-intensive for a new architecture.\n\n#### # Missing/wrong details in preliminary\n- Section 2.1, titled *Mamba*, should provide foundational knowledge about Mamba as a preliminary, but it mainly presents on S4 or early SSMs. Since Mamba has slightly distinct notations and a signature architecture, the authors should revise this section to include more comprehensive details about Mamba.\n- As the experiments involve the Vision Mamba architecture, the reviewer suggests including the original architecture, which has subtle design differences (e.g., the definition of B).\n- The discretization presented in Eq.(2) is not the zero-order hold discretization as implied but rather the bilinear one. The authors should correct this detail.\n\n####  # Some concerns about experiments\n- The experiments used a limited set of baseline models. Although the PEFT options explored were diverse and met the experimental standards of a scientific study, only the Vim-S baseline is used (since the PEFT methods are more likely to be applied to larger architectures, Vim-S may not be a proper choice as a baseline), where this reviewer feels the main comparison focuses on Vision Mambas versus Vision Transformers.\n- Vision Transformers are not thoroughly explored compared to the fully examined Mamba models. For instance, ViT-S in Table 1 has only a few options compared to Mamba's more comprehensive use of PEFTs.\n- Some experimental comparisons lacked control:\n  - In language models, for instance, Pythia 1.4B and the counterpart Mamba 1.4B had significantly different baseline performances (44.8 vs. 53.0), skewing the reported improvements, as higher starting accuracy inflates gains.\n\n#### # Minors\n- There is a related work that covers a similar topic, including PEFTs in Mamba, which suggests that this paper is not the first to study PEFT for the Mamba architecture as mentioned in lines 71-74. This reviewer understands that the authors could not include the related work due to its publication date being after this submission. However, it is suggested to compare this study with that work in future revisions to provide context and emphasize differences.\n  - Parameter-Efficient Fine-Tuning of State Space Models, NeurIPS 2024 workshop (https://arxiv.org/abs/2410.09016)\n\n- Typo in Table 1: Linear 'Proving'"}},"nonreaders":[],"tmdate":1732982473328,"tcdate":1730652448145,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6536/Reviewer_Czw3"],"signatures":["ICLR.cc/2025/Conference/Submission6536/Reviewer_Czw3"],"forum":"UAKnJMIBwf","number":2,"license":"CC BY 4.0","cdate":1730652448145,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6536/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732982473328,"domain":"ICLR.cc/2025/Conference","replyto":"UAKnJMIBwf","id":"6tD04aQKDL","forumContent":{"TLDR":{"value":"We explore and propose parameter-efficient fine-tuning for Mamba."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["parameter-efficient fine-tuning","PEFT","Mamba","State Space Model","SSM"]},"primary_area":{"value":"transfer learning, meta learning, and lifelong learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"An ecosystem of Transformer-based models has been established by building large models with extensive data. \nParameter-efficient fine-tuning (PEFT) is a crucial technology for deploying these models to downstream tasks with minimal cost while achieving effective performance. Recently, Mamba, a State Space Model (SSM)-based model, has attracted attention as a potential alternative to Transformers. While many large-scale Mamba-based models have been proposed, efficiently adapting pre-trained Mamba-based models to downstream tasks remains unexplored.\nIn this paper, we conduct an exploratory analysis of PEFT methods for Mamba. We investigate the effectiveness of existing PEFT methods for Transformers when applied to Mamba. We also modify these methods to better align with the Mamba architecture. Additionally, we propose new Mamba-specific PEFT methods that leverage the distinctive structure of Mamba. Our experiments indicate that PEFT performs more effectively for Mamba than Transformers. Lastly, we demonstrate how to effectively combine multiple PEFT methods and provide a framework that outperforms previous works. To ensure reproducibility, we will release the code after publication."},"_bibtex":{"value":"@inproceedings{\nyoshimura2025mambapeft,\ntitle={Mamba{PEFT}: Exploring Parameter-Efficient Fine-Tuning for Mamba},\nauthor={Masakazu Yoshimura and Teruaki Hayashi and Yota Maeda},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=UAKnJMIBwf}\n}"},"title":{"value":"MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba"},"pdf":{"value":"/pdf/bb5dec1d9daadbada09960ed1cd34a6349f495dc.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"yoshimura|mambapeft_exploring_parameterefficient_finetuning_for_mamba"},"authorids":{"value":["~Masakazu_Yoshimura1","~Teruaki_Hayashi1","~Yota_Maeda1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Masakazu Yoshimura","Teruaki Hayashi","Yota Maeda"]}},"version":2},{"content":{"summary":{"value":"The paper proposes ConvT3, a convolutional state-space model (ConvSSM) that realizes a 3×3 state kernel while keeping linear-time parallel scans for long sequences. The key idea is to structure the state kernel as (i) a diagonalizable SSM matrix over hidden channels and (ii) a tridiagonal Toeplitz (TT) tensor over spatial axes. This structure enables efficient training while capturing richer spatial dynamics than prior 1×1 state kernels (e.g., ConvS5). On Moving-MNIST, ConvT3 achieves state-of-the-art results across most metrics; on PDEBench, it attains the best accuracy with efficiency close to ConvS5."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"See the weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"+ The paper shows how to move from 1×1 to 3×3 state kernels in ConvSSMs without losing linear-time scans, via a structured (diagonalizable) state tensor and tridiagonal Toeplitz (TT) formulation. \n\n+ Training stability: The paper uses a Hurwitz condition-based parameterization (constraining eigenvalues to have negative real parts) and a positive eigen-tensor construction to keep dynamics stable during training. This is validated by the training loss curve (Figure 3).\n\n+ Ablations on kernel size and minimal parameterization: MiniT3 outperformed ConvS5 despite the minimal increase in parameters, and the kernel-size ablations show the 3×3 state kernel A is the main driver of gains, not B/C alone."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Limited benchmarks: Moving-MNIST and PDEBench tasks show the benefits, but they are relatively synthetic. Adding natural video datasets or harder physics targets (e.g., Navier–Stokes) would strengthen the paper. \n\n- Compute and throughput details: The paper states linear-time scans and that efficiency is close to ConvS5, but the actual comparisons are limited. Per-component speed and throughput are missing. Also, efficiency measurement across sequence lengths and resolutions would make the efficiency claims stronger.\n\n- Ablation: The kernel-size studies of A, B, and C are helpful, but additional ablations such as 3×3 vs 5×5 state kernels would be beneficial."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919679048,"tcdate":1761782999234,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7602/Reviewer_pYFy"],"signatures":["ICLR.cc/2026/Conference/Submission7602/Reviewer_pYFy"],"forum":"w7csRoB5CO","number":3,"license":"CC BY 4.0","cdate":1761782999234,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7602/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919679048,"domain":"ICLR.cc/2026/Conference","replyto":"w7csRoB5CO","id":"YuQTALqSK5","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We propose ConvT3, a ConvSSM with extended state kernels structured by tridiagonal Toeplitz tensors, enabling efficient linear-time training, stable parameterization, and state-of-the-art performance in video and PDE modeling."},"keywords":{"value":["Spatiotemporal modeling","Video modeling","Physical system modeling","Tridiagonal Toeplitz tensor","Long-range sequence modeling"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Modeling long spatiotemporal sequences requires capturing both complex spatial correlations and temporal dependencies.\nConvolutional State Space Models (ConvSSMs) have been proposed to incorporate spatial modeling in State Space Models (SSMs) using the convolution of tensor-valued states and kernels. \nYet, existing implementations remain limited to $1\\times 1$ state kernels for computational feasibility, which limits the modeling capacity of ConvSSMs.\nWe introduce a novel spatiotemporal model, ConvT3 (ConvSSM using Tridiagonal Toeplitz Tensors), designed to equivalently realize ConvSSMs with extended $3\\times 3$ state kernels.\nConvT3 structures a state kernel for its corresponding tensor to be composed as a structured SSM matrix on hidden state dimensions and a constrained tridiagonal Toeplitz tensor on spatial dimensions.\nWe show that the structured tensor can be diagonalized, which enables efficient parallel training while leveraging $3\\times 3$ state convolutions.\nWe demonstrate that ConvT3 effectively embeds rich spatial and temporal information into the dynamics of tensor-valued states, achieving state-of-the-art performance on most metrics in long-range video generation and physical system modeling."},"_bibtex":{"value":"@inproceedings{\nhong2026convt,\ntitle={ConvT3: Structured State Kernels for Convolutional State Space Models},\nauthor={Jaeyoung Hong and Yun Young Choi and Joohwan Ko and Minseon Gwak},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=w7csRoB5CO}\n}"},"title":{"value":"ConvT3: Structured State Kernels for Convolutional State Space Models"},"pdf":{"value":"/pdf/18bdc61267dc0738df96918b9221dbf5154886a9.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"hong|convt3_structured_state_kernels_for_convolutional_state_space_models"},"authorids":{"value":["~Jaeyoung_Hong1","~Yun_Young_Choi1","~Joohwan_Ko2","~Minseon_Gwak1"]},"authors":{"value":["Jaeyoung Hong","Yun Young Choi","Joohwan Ko","Minseon Gwak"]}},"version":2},{"content":{"venue":{"value":"TIME 2004"},"pdf":{"value":"https://ieeexplore.ieee.org/iel5/9184/29135/01314430.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2004"},"paperhash":{"value":"giordano|onthefly_automata_construction_for_dynamic_linear_time_temporal_logic"},"authorids":{"value":["~Laura_Giordano1","https://dblp.org/search/pid/api?q=author:Alberto_Martelli:"]},"html":{"value":"https://doi.org/10.1109/TIME.2004.1314430"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/GiordanoM04,\n  author={Laura Giordano and Alberto Martelli},\n  title={On-the-Fly Automata Construction for Dynamic Linear Time Temporal Logic},\n  year={2004},\n  cdate={1072915200000},\n  pages={133-139},\n  url={https://doi.org/10.1109/TIME.2004.1314430},\n  booktitle={TIME},\n  crossref={conf/time/2004}\n}\n"},"abstract":{"value":"We present a tableau-based algorithm for obtaining a Buchi automaton from a formula in dynamic linear time temporal logic (DLTL), a logic which extends LTL by indexing the until operator with regular programs. The construction of the states of the automaton is similar to the standard construction for LTL, but a different technique must be used to verify the fulfillment of until formulas. The resulting automaton is a Buchi automaton rather than a generalized one. The construction can be done on-the-fly, while checking for the emptiness of the automaton."},"title":{"value":"On-the-Fly Automata Construction for Dynamic Linear Time Temporal Logic"},"authors":{"value":["Laura Giordano","Alberto Martelli"]}},"tmdate":1723849700386,"pdate":1072915200000,"tcdate":1723849593977,"writers":["~"],"signatures":["~Laura_Giordano1"],"forum":"NgCsKoB1s1","license":"CC BY-SA 4.0","number":69381,"cdate":1072915200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1723849700386,"domain":"DBLP.org","id":"NgCsKoB1s1","version":2},{"content":{"summary":{"value":"This paper proposes a hybrid architecture 3DSMT (Hybrid Spiking Mamba-Transformer), designed to achieve efficient and accurate point cloud analysis. The model combines spiking neural dynamics, state-space modeling (Mamba), and the Transformer attention mechanism, achieving a strong balance between performance and energy efficiency.\n\n● 3DSMT is composed of three core modules:Spiking Local Offset Attention (SLOA): captures local geometric features;\n\n●Spiking Mamba Block (SMB): performs global feature integration with linear complexity;\n\n●Spiking Position Encoding (SPE): encodes the spatial and temporal structure of unordered point clouds.\n\n Experiments on ModelNet40, ScanObjectNN, and ShapeNetPart demonstrate that 3DSMT achieves state-of-the-art (SOTA) performance among Spiking Neural Network (SNN) models."},"soundness":{"value":4},"confidence":{"value":5},"questions":{"value":"1.What is the processing order in the Spiking Mamba Block? Is it executed sequentially like traditional SSMs, or in parallel within each timestep? Given that Mamba inherently depends on token order, how does the model maintain global permutation invariance?\n\n2.The paper claims that 3DSMT achieves linear complexity, yet the architecture still includes Transformer modules, whose attention mechanism typically exhibits quadratic complexity with respect to the number of tokens. Why, then, is the complexity described as linear in the **abstract**? This point is particularly important and should be explicitly discussed, as it directly affects the validity of the complexity claim in both the abstract and the main text.\n\nAlthough there are some open issues and limitations, the paper demonstrates solid innovation, completeness, and clarity. I believe it represents a high-quality and meaningful contribution to the field."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1.The model elegantly integrates spiking neural networks, the Mamba dynamic mechanism, and the Transformer structure, effectively balancing accuracy and energy efficiency.\n\n2.The experimental validation is extensive and comprehensive: 3DSMT performs consistently well across both classification and segmentation tasks.\n\n3.The feature visualization is well presented and convincingly demonstrates the model’s representational capability.\n\n4.The ablation studies systematically evaluate the effects of timestep and data augmentation strategies, providing good empirical support."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.The novelty is somewhat limited: compared with Spiking Point Mamba (SPM) or Spiking Transformer, 3DSMT appears more as an engineering-level integration rather than a fundamentally new learning paradigm.\n\n2.The paper lacks discussion on real-time performance: latency and inference speed are not analyzed, which are crucial for SNN and neuromorphic applications.\n\n3.Scalability to large-scale point clouds is not validated — experiments are limited to inputs of 1K–2K points, and it remains unclear how the method performs on significantly larger point sets."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919465096,"tcdate":1761548689206,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7340/Reviewer_hWC7"],"signatures":["ICLR.cc/2026/Conference/Submission7340/Reviewer_hWC7"],"forum":"KkoS6y0pHP","number":1,"license":"CC BY 4.0","cdate":1761548689206,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7340/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919465096,"domain":"ICLR.cc/2026/Conference","replyto":"KkoS6y0pHP","id":"KmtfmnXvbC","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Point Cloud Analysis","Spiking neural network","Spiking Local Offset Attention","Spiking Mamba Block"]},"supplementary_material":{"value":"/attachment/214ec3d80d4c3f1fe2568ee1d4d94d9ef5be47e4.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"The sparse unordered structure of point clouds causes unnecessary computation and energy consumption in deep models. \nConventionally, the Transformer architecture is leveraged to model global relationships in point clouds, however, its quadratic complexity restricts scalability. Although the Mamba architecture enables efficient global modeling with linear complexity, it lacks natural adaptability to unordered point clouds. \nSpiking Neural Network (SNN) is an energy-efficient alternative to Artificial Neural Network (ANN), offering an ultra low-power event-driven paradigm. \nThe inherent sparsity and event-driven characteristics of SNN are highly compatible with the sparse distribution of point clouds. To balance efficiency and performance, we propose a hybrid spiking Mamba-Transformer (3DSMT) model for point cloud analysis. 3DSMT integrates a Spiking Local Offset Attention module to efficiently capture fine-grained local geometric features with a spiking Mamba block designed for unordered point clouds to achieve global feature integration with linear complexity. Experiments show that 3DSMT achieves state-of-the-art performance among SNN-based methods in shape classification, few-shot classification, and part segmentation tasks, significantly reducing computational energy consumption while also outperforming numerous ANN-based models.\nOur source code is in supplementary material and will be made publicly available"},"_bibtex":{"value":"@inproceedings{\nzhouzhiming2026dsmt,\ntitle={3{DSMT}: A Hybrid Spiking Mamba-Transformer for Point Cloud Analysis},\nauthor={Zhiming Zhou and Yong He and Qiaoyun Wu and Chaoxu Mu and Ajmal Saeed Mian},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KkoS6y0pHP}\n}"},"title":{"value":"3DSMT: A Hybrid Spiking Mamba-Transformer for Point Cloud Analysis"},"pdf":{"value":"/pdf/a0f1fd034e71e283c9a5d2fb0360581aed949914.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhou|3dsmt_a_hybrid_spiking_mambatransformer_for_point_cloud_analysis"},"authorids":{"value":["~Zhiming_Zhou4","~Yong_He2","~Qiaoyun_Wu1","~Chaoxu_Mu1","~Ajmal_Saeed_Mian1"]},"authors":{"value":["Zhiming Zhou","Yong He","Qiaoyun Wu","Chaoxu Mu","Ajmal Saeed Mian"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a clustering based content based scanning method and a global prompt based non causal modeling approach to address the issues of content independent scanning order and causal characteristics in the Mamba structured learned image codec. Ultimately, while preserving the original linear complexity of Mamba, this approach is effective, ,  reducing BD-rate by 15.91%, 21.34%, and 17.58% on the Kodak, Tecnick, and CLIC datasets over VTM-21.0."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"- I can understand that the method proposed in this paper can improve the memory consumption of Mamba based image codecs, but further analysis is needed to determine the source of the model's performance improvement, including comparisons with existing Mamba-based methods such as MambaIC, and the benefits of visualizing/analyzing the proposed non causal approach."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- Good RD performance while maintaining low complexity\n- Relieve the high memory usage problem of prior Mamba based learned image codec"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The methods proposed for both clustering partitioning and global prior modulation are common in prior \"content-adaptive\" studies, such as Rounting Transformer [1], even in the field of learned image compression, there are similar jobs available [2].\n- The main contribution of this paper is to improve the Mamba module in the Mamba-based learned image codec. However, it seems that there are many other differences compared to the previous works, e.g., MambaVC and MambaIC, including the entropy models, etc. The paper lacks detailed performance analysis.\n- Regarding the difference in receptive fields, according to Appendix Fig.16, it seems that this global receptive field and the so-called content aware features are also present in MambaIC, What is the actural difference?\n- The paper lacks a performance comparison with the latest methods listed, e.g., [3].\n\nRef:\n\n[1] Roy A, Saffar M, Vaswani A, et al. Efficient content-based sparse attention with routing transformers[J]. Transactions of the Association for Computational Linguistics, 2021, 9: 53-68.\n\n[2] Zhang Y, Duan Z, Lu M, et al. Another way to the top: Exploit contextual clustering in learned image coding[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(8): 9377-9386.\n\n[3] Li Y, Zhang H, Li L, et al. Learned image compression with hierarchical progressive context modeling[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025: 18834-18843."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915649030,"tcdate":1761795889598,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission966/Reviewer_GLne"],"signatures":["ICLR.cc/2026/Conference/Submission966/Reviewer_GLne"],"forum":"WwDNiisZQm","number":3,"license":"CC BY 4.0","cdate":1761795889598,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission966/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915649030,"domain":"ICLR.cc/2026/Conference","replyto":"WwDNiisZQm","id":"r8oKkyjC5V","forumContent":{"TLDR":{"value":"We propose Content-Aware Mamba for learned image compression, outperforming previous Mamba-based LIC models."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Learned Image Compression"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Recent learned image compression (LIC) leverages Mamba-style state-space models (SSMs) for global receptive fields with linear complexity. However, the standard Mamba adopts content-agnostic, predefined raster (or multi-directional) scans under strict causality. This rigidity hinders its ability to effectively eliminate redundancy between tokens that are content-correlated but spatially distant.\nWe introduce Content-Aware Mamba (CAM), an SSM that dynamically adapts its processing to the image content.\nSpecifically, CAM overcomes prior limitations with two novel mechanisms. First, it replaces the rigid scan with a content-adaptive token permutation strategy to prioritize interactions between content-similar tokens regardless of their location. Second, it overcomes the sequential dependency by injecting sample-specific global priors into the state-space model, which effectively mitigates the strict causality without multi-directional scans.\nThese innovations enable CAM to better capture global redundancy while preserving computational efficiency. Our Content-Aware Mamba-based LIC model (CMIC) achieves state-of-the-art rate-distortion performance, surpassing VTM-21.0 by 15.91\\%, 21.34\\%, and 17.58\\% in BD-rate on the Kodak, Tecnick, and CLIC datasets, respectively. Code will be released at https://github.com/UnoC-727/CMIC."},"_bibtex":{"value":"@inproceedings{\nchen2026contentaware,\ntitle={Content-Aware Mamba for Learned Image Compression},\nauthor={Yunuo Chen and Zezheng Lyu and Bing He and Hongwei Hu and Qi Wang and Yuan Tian and Li Song and Wenjun Zhang and Guo Lu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=WwDNiisZQm}\n}"},"title":{"value":"Content-Aware Mamba for Learned Image Compression"},"pdf":{"value":"/pdf/b79a0bc7117c87f6d25f05efc32124e42e00bb2f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"chen|contentaware_mamba_for_learned_image_compression"},"authorids":{"value":["~Yunuo_Chen1","~Zezheng_Lyu1","~Bing_He5","~Hongwei_Hu3","~Qi_Wang45","~Yuan_Tian5","~Li_Song3","~Wenjun_Zhang3","~Guo_Lu2"]},"authors":{"value":["Yunuo Chen","Zezheng Lyu","Bing He","Hongwei Hu","Qi Wang","Yuan Tian","Li Song","Wenjun Zhang","Guo Lu"]}},"version":2},{"content":{"summary":{"value":"The paper proposes Sparse State Expansion (SSE) for linear attention. Two ideas drive the method: (i) row‑sparse updates that treat state updates as a classification problem and write only to top‑k rows via a softmax head, and (ii) state expansion into N shared‑parameter partitions chosen by a write–read gate, so capacity (state size) scales without growing parameter count. Efficient masked/varlen implementations are provided. Empirically, SSE and its hybrid variant are tested on language modeling, retrieval, and math reasoning."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":2},"strengths":{"value":"1. Modeling state updates as information classification is well argued and operationalized \n2. Decoupling the model’s parameters and state's parameters is important direction for improving linear transformers."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. There is no convincing explanation on why SSE is only effective for linear attention but not deltanet.\n2. Baselines seem to be cherry picked since they are neither the most powerful nor fundamentally relevant sequence models. \n3. The paper uses hard partition selection (top‑k) and softmax row selection, but the gradient treatment for the discrete top‑k isn’t described. Can you please clarify how gradients flow through Eqs. 7–9 and whether any implementation tricks are needed for stability."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762920679334,"tcdate":1761893411107,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8934/Reviewer_Jg4D"],"signatures":["ICLR.cc/2026/Conference/Submission8934/Reviewer_Jg4D"],"forum":"R6DrJ4tnGV","number":3,"license":"CC BY 4.0","cdate":1761893411107,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8934/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762920679334,"domain":"ICLR.cc/2026/Conference","replyto":"R6DrJ4tnGV","id":"YFw52mwlxX","forumContent":{"TLDR":{"value":"SSE is a partitioned state expansion method under a row-sparse update framework for linear attention, improving retrieval and reasoning."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Linear Attention","Language Model"]},"supplementary_material":{"value":"/attachment/6e3475d7248015327464c0fe2192bc1fb8d0142d.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The Transformer architecture, despite its widespread success, struggles with long-context scenarios due to quadratic computation and linear memory growth. While various linear attention variants mitigate these efficiency constraints by compressing context into fixed-size states, they often degrade performance in tasks such as in-context retrieval and reasoning. To address this limitation and achieve more effective context compression, we propose two key innovations. First, we introduce a row-sparse update formulation for linear attention by conceptualizing state updating as information categorization. This enables sparse state updates via softmax-based top-$k$ row selection, thereby extending receptive fields and reducing information interference. Second, we present Sparse State Expansion (SSE) within the sparse framework, which expands the contextual state into multiple partitions, effectively decoupling parameter size from state capacity while maintaining the sparse row-selection paradigm. Supported by efficient parallelized implementations, our design achieves highly discriminative state representations. We extensively validate SSE in both pure linear and hybrid (SSE-H) architectures across language modeling, in-context retrieval, and mathematical reasoning benchmarks. SSE demonstrates strong retrieval performance and scales favorably with state size. Moreover, after reinforcement learning (RL) training, our 2B SSE-H model achieves state-of-the-art mathematical reasoning performance among small reasoning models, scoring 64.5 on AIME24 and 50.2 on AIME25, significantly outperforming similarly sized open-source Transformers. These results highlight SSE as a promising and efficient architecture for long-context modeling."},"_bibtex":{"value":"@inproceedings{\npan2026scaling,\ntitle={Scaling Linear Attention Capacity with Sparse State Expansion},\nauthor={Yuqi Pan and Yongqi An and Zheng Li and Yuhong Chou and Rui-Jie Zhu and Xiaohui Wang and Mingxuan Wang and Jinqiao Wang and Guoqi Li},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=R6DrJ4tnGV}\n}"},"title":{"value":"Scaling Linear Attention Capacity with Sparse State Expansion"},"pdf":{"value":"/pdf/73d9d62767b69cd0649b8caaa9d65c613f858782.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"pan|scaling_linear_attention_capacity_with_sparse_state_expansion"},"authorids":{"value":["~Yuqi_Pan2","~Yongqi_An1","~Zheng_Li48","~Yuhong_Chou1","~Rui-Jie_Zhu2","~Xiaohui_Wang4","~Mingxuan_Wang1","~Jinqiao_Wang1","~Guoqi_Li1"]},"authors":{"value":["Yuqi Pan","Yongqi An","Zheng Li","Yuhong Chou","Rui-Jie Zhu","Xiaohui Wang","Mingxuan Wang","Jinqiao Wang","Guoqi Li"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Linear Attention","Recurrent Memory","Multi-State"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Self-attention plays a key role in long-sequence modeling, but its quadratic time complexity fundamentally limits the scaling of Large Language Models. Recently, Multi-State Linear Attention (MSLA) mechanisms have been proposed to make sequence modeling more efficient by splitting the context into segments and compressing each in its own memory state. MSLA aggregates the information extracted from each memory state using only the mean alignment between a query and each segment's keys, ignoring how the alignment is distributed within the segment. In order to model the relevance of a segment to a query more comprehensively and\nincorporate the variance of the alignment in the aggregation, we propose MERLIN ($\\textbf{M}$om$\\textbf{E}$nt $\\textbf{R}$outing for multi-state $\\textbf{LIN}$ear attention), a method that tracks the first and second moments of each segment's keys. Using the moments of keys, a query can recover both the mean and the variance of its alignment with each segment to perform a more content-aware aggregation of per-state readouts. Experiments on multi-state variants of Gated DeltaNet-1.3B and Mamba-2-780M show that MERLIN consistently outperforms existing aggregation methods on various long-context tasks. In particular, MERLIN outperforms baselines on RULER needle-in-a-haystack and long-context QA (+8.3\\% for Gated DeltaNet-1.3B and +7.0\\% for Mamba-2-780M) and diverse subtasks in LongBench (+13.5\\% and +12.0\\%). It even outperforms baselines that consume 4x more memory. MERLIN provides a simple and general routing mechanism for improving long-context capabilities of MSLA models. Code will be available upon acceptance."},"_bibtex":{"value":"@inproceedings{\nanonymous2026multistate,\ntitle={Multi-State Linear Attention with Generalized Second-Order Routing},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=DmPoIxgMdr},\nnote={under review}\n}"},"title":{"value":"Multi-State Linear Attention with Generalized Second-Order Routing"},"pdf":{"value":"/pdf/0390efd8e26ed6a20305cfb9c4fb9502644b79b9.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230675766,"tcdate":1789593098627,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission26953/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission26953/Authors"],"forum":"DmPoIxgMdr","license":"CC BY 4.0","number":26953,"cdate":1789593098627,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission26953/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791230675766,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"DmPoIxgMdr","version":2},{"content":{"comment":{"value":"Thank you for engaging in the discussion. \n\nBelow we further clarify the motivation and the additional experimental results.\n\n&nbsp;\n\n---\n## 1. Motivation of Mamba for CD.\n\ntl;dr: While Reviewer vPYA has expressed doubt, ***\"using Mamba to capture CDs\"*** has been a widely accepted concept in prior/concurrent works [C,D,E,F].\n\nWe believe ***\"using Mamba to capture CDs\"*** is a natural extension, based on two key points: 1) Mamba [A] has emerged as a promising alternative to Transformer, and 2) capturing CDs has been shown to be essential in prior works [B,C]; more importantly, ***prior/concurrent works already have widely explored using Mamba to capture CDs [C,D,E,F]***, yielding ***competitive results compared to Transformers***. However, given that Mamba is a sequence-based model, appropriate adjustments are required, where we pose our contribution.\n\nNote that we also confirm if Mamba is useful for capturing time dependencies (as a replacement of \"MLP for TD\" in **Figure 2**) in **Table 11**. This experiment demonstrates that simple MLPs are more effective for TD, consistent with the observation in prior works in this area.\n\n&nbsp;\n\n[A] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv (2024)\n\n[B] Liu et al. \"iTransformer: Inverted transformers are effective for time series forecasting.\" ICLR (2024)\n\n[C] Wang et al. \"Is mamba effective for time series forecasting?.\" arXiv (2024)\n\n[D] Liang et al. \"Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting.\" arXiv (2024)\n\n[E] Weng et al. \"Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting.\" arXiv (2024)\n\n[F] Behrouz, A., Santacatterina, M., & Zabih, R. “Mambamixer: Efficient selective state space models with dual token and channel selection\" arXiv (2024)\n\n&nbsp;\n\n---\n## 2. Additional Experiments\n\nIn response to the reviewer's previous request, we have conducted experiments in **Appendix L**. \nWe chose **four datasets with varying numbers of channels** ( ETTm1 (7), PEMS04 (180), ECL (321), and Traffic (862)) based on prior works [A,B] (i.e., not arbitrarily chosen). Regarding the newly mentioned ETTh1 dataset, *we initially considered it redundant, as it belongs to the same ETT series as ETTm1*, which we had already tested. \n\nNonetheless, to address the additional request, we performed ***additional experiments on ETTh1*** as well, and the results are presented below. Note that we did not perform the pretraining task (CCM) for SOR-Mamba in order to ensure a fair comparison, as it could potentially lead to further performance gains. Additionally, we used $L=H=96$, as in previous works [A,B] and **Appendix L**, and performed a search over $D \\in \\\\{128, 256, 512\\\\}$ to find the optimal configuration for both methods. The results below demonstrate **superior performance compared to S-Mamba across all values of $L$**.\n\n| L | 48 | 96 | 192 | 336 | 720 |\n|----------|----------|----------|----------|----------|----------| \n| a) S-Mamba | .391 | .385 | .390 | .397 | .397 |\n| b) SOR-Mamba | **.379** | **.379** | **.381** | **.389** | **.386** |\n| a-b) MSE Imp. | .012  | .006 | .009 | .008 | .011 |\n\n[A] Liu, Yong, et al. \"iTransformer: Inverted transformers are effective for time series forecasting.\" ICLR (2024)\n\n[B] Wang, Zihan, et al. \"Is mamba effective for time series forecasting?.\" arXiv (2024)\n\n\n&nbsp;\n\nWe hope this addresses your concerns on the motivation and the additional experiment. \n\nIf you have any further questions, please feel free to ask; I would appreciate it."}},"tmdate":1732713314399,"tcdate":1732713314399,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission69/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission69/Authors"],"forum":"fyl82vAale","number":25,"license":"CC BY 4.0","cdate":1732713314399,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission69/-/Official_Comment"],"mdate":1732713314399,"domain":"ICLR.cc/2025/Conference","replyto":"zaroJMugDh","id":"0o7TJMprFk","forumContent":{"TLDR":{"value":"We propose SOR-Mamba, a time series forecasting method that uses Mamba with regularization to address the sequential order bias in capturing channel dependencies."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","State-space Model","Time Series"]},"supplementary_material":{"value":"/attachment/a3e6bf94e3ea1b0642f98af3f08c4590c0c5d03e.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data.\nHowever, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.\nExtensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios."},"_bibtex":{"value":"@misc{\nlee2025sequential,\ntitle={Sequential Order-Robust Mamba for Time Series Forecasting},\nauthor={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\nyear={2025},\nurl={https://openreview.net/forum?id=fyl82vAale}\n}"},"title":{"value":"Sequential Order-Robust Mamba for Time Series Forecasting"},"pdf":{"value":"/pdf/2172851f730b05caa6144b837024da19c84263c0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"lee|sequential_orderrobust_mamba_for_time_series_forecasting"},"authorids":{"value":["~Seunghan_Lee1","~Juri_Hong1","~Kibok_Lee1","~Taeyoung_Park1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seunghan Lee","Juri Hong","Kibok Lee","Taeyoung Park"]}},"version":2},{"content":{"summary":{"value":"This paper addresses the problem of the lack of modeling the hierarchical structure of input sequences in transformers. In response, a new attention mechanism, called stack attention, is proposed. The idea is to frame the sequence modeling problem as a task of running a PDA on the input vectors; that is, each time we take a vector as input, we update the stack of the PDA and produce a new vector as output. In this way, the sequence of input vectors is represented as a sequence of output vectors. The entire process is based on differentiable states and operations. Thus, this attention model operates in the same manner as those in sequence modeling, making it easy to incorporate the model into transformers. However, as a side effect, stack attention introduces recurrence into modeling, thus preventing training from being parallelized across the sequence. The stack attention models are tested on synthetic data generated by context-free grammars. Experimental results show that nondeterministic stack attention models surpass standard transformers and achieve better results than a strong nondeterministic stack RNN baseline. The stack attention models also show promising results on small-scale machine translation and language modeling tasks."},"presentation":{"value":"3 good"},"contribution":{"value":"3 good"},"soundness":{"value":"3 good"},"strengths":{"value":"I like this work! Given that language structure is not explicitly modeled in current Transformer models, this work opens a door to a new approach to considering hierarchical patterns in modeling languages. The design of the model is simple and elegant. The experiments support the claims well."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I have no major concerns, but a few comments.\n\nThe state of the stack attention models at a given timestep depends on its past state. This means the models share similar drawbacks and merits with recurrent models like RNNs. Compared to the self-attention used in standard Transformers, stack attention is slower for training because it processes one token at a time, rather than parallelizing the encoding process over the entire sequence. The author states in the appendix that this could be improved using the parallel prefix sum method, but no details are presented.\n\nA related problem is that the experiments here are small-scale. While it's fine to test the models on synthetic data for CFL tasks, the results on language modeling and machine translation aren't comparable to those in other papers. I understand that computational cost is a concern. However, to demonstrate the superiority of stack attention, it's necessary to compare it with previously reported results under the same setup.\n\nI’m not quite satisfied that the models are motivated by handling hierarchical structure behind languages but there is no discussion on what structure is captured. A simple way to examine this is to design probing tasks to see how much syntax is modeled in stack attention and to see how the learned syntax differs from human-annotated syntax. Unsupervised learning of syntactic structures can offer new insights into modeling natural languages.\"\n\nThere have been previous studies on extending standard attention models to hierarchical models, such as hierarchical attention and selective attention. These should be considered baselines for comparison, either in related work or experiments."},"confidence":{"value":"5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully."},"questions":{"value":"N/A"},"rating":{"value":"8: accept, good paper"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636744474,"tcdate":1698591466746,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6568/Reviewer_bkRf"],"signatures":["ICLR.cc/2024/Conference/Submission6568/Reviewer_bkRf"],"forum":"XVhm3X8Fum","number":1,"license":"CC BY 4.0","cdate":1698591466746,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6568/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636744474,"domain":"ICLR.cc/2024/Conference","replyto":"XVhm3X8Fum","id":"arLLsW5sCB","forumContent":{"TLDR":{"value":"We propose new attention operators that use stacks as a latent model of syntax, improving transformer language modeling on context-free languages and natural language."},"venue":{"value":"ICLR 2024 spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["transformer","attention","context-free languages","pushdown automata","formal languages","language modeling","machine translation"]},"supplementary_material":{"value":"/attachment/e1831149abdf78111ed7c5f362c2bd0628013755.zip"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Attention, specifically scaled dot-product attention, has proven effective for natural language, but it does not have a mechanism for handling hierarchical patterns of arbitrary nesting depth, which limits its ability to recognize certain syntactic structures. To address this shortcoming, we propose stack attention: an attention operator that incorporates stacks, inspired by their theoretical connections to context-free languages (CFLs). We show that stack attention is analogous to standard attention, but with a latent model of syntax that requires no syntactic supervision. We propose two variants: one related to deterministic pushdown automata (PDAs) and one based on nondeterministic PDAs, which allows transformers to recognize arbitrary CFLs. We show that transformers with stack attention are very effective at learning CFLs that standard transformers struggle on, achieving strong results on a CFL with theoretically maximal parsing difficulty. We also show that stack attention is more effective at natural language modeling under a constrained parameter budget, and we include results on machine translation."},"_bibtex":{"value":"@inproceedings{\ndusell2024stack,\ntitle={Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns},\nauthor={Brian DuSell and David Chiang},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=XVhm3X8Fum}\n}"},"title":{"value":"Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns"},"pdf":{"value":"/pdf/1725d2a5bfd546b5acdabab5eb5d281caf92d1e4.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference"},"paperhash":{"value":"dusell|stack_attention_improving_the_ability_of_transformers_to_model_hierarchical_patterns"},"authorids":{"value":["~Brian_DuSell1","~David_Chiang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Brian DuSell","David Chiang"]}},"version":2},{"content":{"summary":{"value":"This work proposes a hybrid Mamba network for few-shot segmentation, including a support recapped Mamba to periodically recap the support features when scanning query and a query intercepted Mamba to forbid the mutual interactions among query pixels."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"See weakness"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The idea of adapting Mamba for few-shot segmentation is good. The performance looks good."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The motivation of using mamba for FSS is unconvincing. The authors state that they use mamba for FSS is because that mamba's complexity is only linear. However, the proposed mamba-based method is used only for fuse support foreground (FG) into query features. There are also many light-weight feature fusing methods for FSS and show good performance and efficiency. The authors need to provide convincing reasons how the mamba works better than other feature fusing methods. Otherwise, just applying a new technology to FSS without deep insight is not acceptable.\n2. The experimental analysis is insufficient. First, the comparison experiments between the proposed mamba methods and other feature fusing methods are missing. I cannot know if the proposed method is better, on performance or efficiency. I mean the fair comparison on a same baseline method to compare with other methods, rather than just the Table 1 and Table 2. The overall performance comparison is unfair because the baseline methods are different. Second, the analysis is insufficient. The authors state that the proposed mamba methods solves the support forgetting and intra-class gap issue problems. But how these problems are solved? The Table 3 and Figure 5 just show that the proposed method achieves better performance. However, it is still unclear how the proposed method solve these problems.\n3. It also lacks the comparison with SAM-based FSS methods."},"limitations":{"value":"No."}},"nonreaders":[],"tmdate":1730878780012,"tcdate":1721292654576,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2498/Reviewer_dFqt"],"signatures":["NeurIPS.cc/2024/Conference/Submission2498/Reviewer_dFqt"],"forum":"Qe2BKeCEBC","number":4,"license":"CC BY 4.0","cdate":1721292654576,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2498/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878780012,"domain":"NeurIPS.cc/2024/Conference","replyto":"Qe2BKeCEBC","id":"GMO8cBk6AZ","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["few-shot segmentation","mamba","intra-sequence dependencies"]},"supplementary_material":{"value":"/attachment/c2e47c3a1b58104b90d7e4c87b1052435097d66e.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Many few-shot segmentation (FSS) methods use cross attention to fuse support foreground (FG) into query features, regardless of the quadratic complexity. A recent advance Mamba can also well capture intra-sequence dependencies, yet the complexity is only linear. Hence, we aim to devise a cross (attention-like) Mamba to capture inter-sequence dependencies for FSS. A simple idea is to scan on support features to selectively compress them into the hidden state, which is then used as the initial hidden state to sequentially scan query features. Nevertheless, it suffers from (1) support forgetting issue: query features will also gradually be compressed when scanning on them, so the support features in hidden state keep reducing, and many query pixels cannot fuse sufficient support features; (2) intra-class gap issue: query FG is essentially more similar to itself rather than to support FG, i.e., query may prefer not to fuse support features but their own ones from the hidden state, yet the success of FSS relies on the effective use of support information. To tackle them, we design a hybrid Mamba network (HMNet), including (1) a support recapped Mamba to periodically recap the support features when scanning query, so the hidden state can always contain rich support information; (2) a query intercepted Mamba to forbid the mutual interactions among query pixels, and encourage them to fuse more support features from the hidden state. Consequently, the support information is better utilized, leading to better performance. Extensive experiments have been conducted on two public benchmarks, showing the superiority of HMNet. The code is available at https://github.com/Sam1224/HMNet."},"_bibtex":{"value":"@inproceedings{\nxu2024hybrid,\ntitle={Hybrid Mamba for Few-Shot Segmentation},\nauthor={Qianxiong Xu and Xuanyi Liu and Lanyun Zhu and Guosheng Lin and Cheng Long and Ziyue Li and Rui Zhao},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=Qe2BKeCEBC}\n}"},"title":{"value":"Hybrid Mamba for Few-Shot Segmentation"},"pdf":{"value":"/pdf/7fca1561230feeb691e2dbccd7f1ef8f79dfa5ec.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"xu|hybrid_mamba_for_fewshot_segmentation"},"authorids":{"value":["~Qianxiong_Xu1","~Xuanyi_Liu3","~Lanyun_Zhu1","~Guosheng_Lin1","~Cheng_Long1","~Ziyue_Li2","~Rui_Zhao6"]},"authors":{"value":["Qianxiong Xu","Xuanyi Liu","Lanyun Zhu","Guosheng Lin","Cheng Long","Ziyue Li","Rui Zhao"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a new MAE like pre-training method designed specifically for vision mamba models. The method was inspired from the difference in AR and MAE pre-training methods applied to transformers and vims, and proposed design choices, such as the scan order, masking strategies, masking ratio, etc. as well as a hybrid transformer mamba architecture. MAP shows a good gain compared with existing MAP models without a pre-training stage."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Related to weakness #2, does the method work also on videos. Probably yes, given the existence of video MAE methods, then in this case, are we able to support more frames than what was possible with ViT or hierarchical or hybrid transformer models?\n2. Related to weakness #2, similar to videos, would object detection or segmentation, as classic testbed of MAE methods, benefit from the mamba design and use larger image resolution? The same might even apply to MLLMs, but these may be out of scope for this paper."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper explored comprehensively various design choices in the MAE design and did find that the changes were necessary for mamba training performance.\n2. Results were showcased extensively on image classification tasks and 3D point cloud tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Quite limited performance gain compared with large ViT models, even with the hybrid MMMT design, e.g. 85.9 original MAE baseline vs. the strongest 86.2 result with also a much lower throughput.\n2. The original promise of mamba was on the long sequence or long context modeling, but the design in this paper does not fully demonstrate this advantage. See the question section for more discussion here.\n3. If I am understanding correctly, the goal of enabling mamba pre-training was to extend the method to longer sequences. (If this is not the case, then within the ViT-supported sequence length, we could perfectly use ViT-L with various MAE variants but not MAP). In this case, some of the design choices, to some extent, were not compatible with the original mamba design, e.g. the transformer layers in the hybrid design and the transformer decoder layers. These may not work on long sequences and may prevent the context scaling as follow up."}},"nonreaders":[],"tmdate":1731427595355,"tcdate":1730952221613,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2428/Reviewer_3np1"],"signatures":["ICLR.cc/2025/Conference/Submission2428/Reviewer_3np1"],"forum":"hBVywPrw6d","number":4,"license":"CC BY 4.0","cdate":1730952221613,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2428/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427595355,"domain":"ICLR.cc/2025/Conference","replyto":"hBVywPrw6d","id":"hvXnRhFg1P","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"TLDR":{"value":"We propose a new pre-training strategy tailored for the Mamba-Transformer backbone."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Hybrid Vision Backbone","Masked Autoregressive","Self-supervised Learning"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has achieved significant advantages in long-context modeling and autoregressive tasks, but its scalability with large parameters remains a major limitation in vision applications. pretraining is a widely used strategy to enhance backbone model performance. Although the success of Masked Autoencoder in Transformer pretraining is well recognized, it does not significantly improve Mamba's visual learning performance. We found that using the correct autoregressive pretraining can significantly boost the performance of the Mamba architecture. Based on this analysis, we propose Masked Autoregressive Pretraining(MAP) to pretrain a hybrid Mamba-Transformer vision backbone network. This strategy combines the strengths of both MAE and Autoregressive pretraining, improving the performance of Mamba and Transformer modules within a unified paradigm. Experimental results show that both the pure Mamba architecture and the hybrid Mamba-Transformer vision backbone network pretrained with MAP significantly outperform other pretraining strategies, achieving state-of-the-art performance. We validate the effectiveness of the method on both 2D and 3D datasets and provide detailed ablation studies to support the design choices for each component."},"_bibtex":{"value":"@misc{\nliu2024map,\ntitle={{MAP}: Unleashing Hybrid Mamba-Transformer Vision Backbone{\\textquoteright}s Potential with Masked Autoregressive Pretraining},\nauthor={Yunze Liu and Li Yi},\nyear={2024},\nurl={https://openreview.net/forum?id=hBVywPrw6d}\n}"},"title":{"value":"MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone’s Potential with Masked Autoregressive Pretraining"},"pdf":{"value":"/pdf/10243949177ada96d6426f1e3d852f7611be953b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"liu|map_unleashing_hybrid_mambatransformer_vision_backbones_potential_with_masked_autoregressive_pretraining"},"authorids":{"value":["~Yunze_Liu2","~Li_Yi2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yunze Liu","Li Yi"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a Active Noise Cancellation (ANC) network. It represents the first attempt to actively cancel general audio signals with deep learning. The paper introduces a segmentation strategy for separate modeling and incorporates Mamba as a sequence modeling module. Additionally, a new training strategy is proposed during training to generate more fitting anti-signals."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"see in weaknesses"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"This paper is the first to consider actively canceling general audio signals to address the issues inherent in Active Noise Cancellation. It incorporates a frequency band segmentation strategy in its design to better focus on the characteristics of each frequency band and introduces Mamba for sequence modeling. In addition to structural innovations, the paper proposes a Near Optimal Anti-Signal Optimization strategy to solve the problem of different paths producing different frequencies."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The first issue is an expression problem at the beginning of Section 4.1, which should state \"Table 1 presents the NMSE for ANC algorithms...\" instead of \"Table 4 presents the NMSE for ANC algorithms...\". The second issue is that the baseline models used for comparison are relatively old and do not strongly demonstrate that this model has reached the state-of-the-art (SOTA). The third issue is that the proposed optimization strategy first finds y' based on the given reference signal x, and then finds the final target y based on y'. Essentially, it is still based on the reference signal for the loss function. Therefore, the significance of this approach and why it leads to performance improvements need to be clarified."}},"nonreaders":[],"tmdate":1731427338940,"tcdate":1730341855838,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission906/Reviewer_mDwo"],"signatures":["ICLR.cc/2025/Conference/Submission906/Reviewer_mDwo"],"forum":"LwLaFpJpfM","number":1,"license":"CC BY 4.0","cdate":1730341855838,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission906/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427338940,"domain":"ICLR.cc/2025/Conference","replyto":"LwLaFpJpfM","id":"vdoA1pt4Xk","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"TLDR":{"value":"Active Audio Cancellation with Mamba architecture"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Active Noise Cancellation","Audio","Speech","Mamba"]},"supplementary_material":{"value":"/attachment/0138f0d4514c738def15a0818d0adc00f8c95537.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"A novel deep learning approach for Active Audio Cancellation (AAC) is presented, which surpasses traditional Active Noise Cancellation (ANC) by effectively canceling any audio signal, regardless of its spectral content. We propose, for the first time, a deep learning approach to AAC using a novel multi-band Mamba architecture. This architecture partitions input audio into multiple frequency bands, allowing for precise anti-signal generation and enhanced phase alignment across frequencies, thereby improving overall cancellation performance. Additionally, we introduce an optimization-driven loss function that provides near-optimal supervisory signals for anti-signal generation. Our experimental results demonstrate substantial improvements over existing methods, achieving up to 7.2dB gain in ANC scenarios and up to 6.2dB improvement in AAC for voice audio signals, outperforming existing methods."},"_bibtex":{"value":"@misc{\nmishaly2025active,\ntitle={Active Audio Cancellation with Multi-Band Mamba Network},\nauthor={Yehuda Mishaly and Lior Wolf and Eliya Nachmani},\nyear={2025},\nurl={https://openreview.net/forum?id=LwLaFpJpfM}\n}"},"title":{"value":"Active Audio Cancellation with Multi-Band Mamba Network"},"pdf":{"value":"/pdf/6a9b8ed0e8febd6fbfe31dfeada416ba176c34be.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"mishaly|active_audio_cancellation_with_multiband_mamba_network"},"authorids":{"value":["~Yehuda_Mishaly1","~Lior_Wolf1","~Eliya_Nachmani1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yehuda Mishaly","Lior Wolf","Eliya Nachmani"]}},"version":2},{"content":{"summary":{"value":"This paper introduces an algorithmic framework, namely \"*Prefix-Scannable Models*\" (PSMs), to characterize and generalize sequence models that support both parallelizable training and efficient sequential inference. The authors term this property \"Sequential-Parallel Duality\" (SPD) and formalize it with the notation SPD(T(n), m(n)), capturing the training compute $T(n)$ and inference memory $m(n)$.\nThe authors first unify a large class of recent efficient models (e.g., Mamba, GLA, DeltaNet, etc.) under this definition. They show that these models are all instances of an associative prefix scan based on a shared \"affine state update\". Because their aggregation operator is associative, they can achieve SPD(n, 1) compared to the SPD($n^2, n$) of Transformers. Afterwards, to generalize the framework to non-associative aggregation operators (such as softmax attention), the classic Blelloch parallel prefix scan is used for training, which fixes a specific binary-tree parenthesization. The crucial insight is proposing an online \"binary counter\" algorithm for inference (Alg. 2) that exactly reproduces this same parenthesization, guaranteeing training-inference consistency. This generalization creates a new class of models, including the proposed *Transformer-PSM*, that achieve SPD(n, log n) with $O(1)$ amortized inference time. Experiments show that Transformer-PSM achieves very good length generalization on state-tracking tasks, and at the same time matching the performance of a full Transformer on associative recall and language modelling."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. How does Transformer-PSM compare to recent hybrids which alternate Transformer and SSM layers? Do such models fall in the SPD framework too? What are the hypothesized trade-offs in terms of expressivity and parallelization between these two different ways of combining Transformers and efficient recurrences?\n\n2. You successfully parallelize a non-linear operator by restricting its dependencies to a fixed binary-tree structure. Concurrent work like DEER and ELK provides an algorithmic alternative for parallelizing general non-linear RNNs via fixed-point iteration. Could you compare these two fundamental approaches?\n\n3. I would be interested to see more results on other state-tracking tasks such as the ones used in Grazzi et al. 2025. Can the authors provide some results on those tasks, e.g. parity, modular arithmetic w/ and w/o brackets? Additionally, can you also evaluate Mamba2 and DeltaProduct with negative eigenvalues in the state-transition matrix in your experiments?\n\n**-- References --**\n\nSiems et al. Deltaproduct: Improving state-tracking in linear rnns via householder products. In NeurIPS 2025.\n\nGrazzi et al. Unlocking state-tracking in linear rnns through negative eigenvalues. In ICLR 2025.\n\nGus et al. Log-linear attention. Preprint arXiv:2506.04761, 2025.\n\nAI21 Labs. Jamba: A Hybrid Transformer-Mamba Language Model. arXiv 2024\n\nGlorioso et al. Zamba: A Compact 7B SSM Hybrid Model. arXiv, 2024\n\nLim et al. Parallelizing non-linear sequential models over the sequence length. In ICLR 2024\n\nGonzalez et al. Towards Scalable and Stable Parallelization of Nonlinear RNNs. In NeurIPS 2024"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":4},"strengths":{"value":"1. **Methodological and technical contributions.** I think that the clean, algorithmic lens it provides for understanding and unifying the recent \"zoo\" of efficient sequence models is a significant conceptual contribution to the field. Formalizing the SPD(T, m) property, unifying Linear RNNs, SSMs and Transformers, and the transition from associative to non-associative operators is a non-trivial and useful generalization. Technically, using the online binary counter (Alg. 2) to exactly match the Blelloch scan's fixed parenthesization is another instance of applying a classic algorithm to solve a modern deep learning problem. \n\n2. **Empirical evaluations**. The proposed Transformer-PSM model shows very good length generalization result on the $S_5$ state-tracking task (Figure 3), up to sequences >6x longer than the longest training sequence. Furthermore, the MQAR result in Figure 4 show that the Tranformer-PSM can match the recall power of a full Transformer, while the WikiText-103 experiment (Figure 5) shows the trade-off between chunk size and perplexity on language modelling, validating the model's ability to achieve RNN-like efficiency and Transformer-like expressivity on these tasks.\n\n3. **Clarity and Structure**: The paper is well-written, and the use of formal definitions (Def 2.5 for SPD, Def 3.1 for PSM) and algorithms (Alg 1, 2, 3, 4) make the core ideas succinct and easy to follow."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Limited scale**: The experiments are on small-to-medium datasets (WikiText-103) with relatively small models. While the synthetic results are interesting, it remains to be verified how the architectural benefits and trade-offs (especially $O(\\log n)$ memory) of Transformer-PSM scale to state-of-the-art compute with billions of parameters.\n\n2. **Missing baselines and other related work**: While the theoretical unification is broad, the baselines used in the empirical comparisons are narrow, focusing almost exclusively on GPT-2 and Mamba. I think the paper would greatly benefit if the authors would also compare to several highly relevant lines of concurrent work such as:\n   - *Transformer-SSM Hybrids*: Models like Jamba, Zamba, etc., which interleave Transformer and SSM layers.\n   - *Log-Linear Attention*: The paper cites \"Log-linear attention\" (Guo et al., 2025) as concurrent work that \"fits this view\" but provides no empirical comparison, even though it appears to be a direct competitor in the SPD(n, log n) class.\n   - *Parallel Non-linear RNNs*: Other approaches to parallelizing non-linear recurrences, such as those based on fixed-point iteration (e.g., DEER and ELK), offer a completely different algorithmic approach to the same problem (parallelizing non-linearity) and should be discussed or benchmarked.\n   - *Linear RNNs with enhanced expressivity*: Methods such as DeltaProduct, that have been shown to work great on state-tracking tasks too."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924583989,"tcdate":1762263059016,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14117/Reviewer_8wg9"],"signatures":["ICLR.cc/2026/Conference/Submission14117/Reviewer_8wg9"],"forum":"tuLF84azND","number":4,"license":"CC BY 4.0","cdate":1762263059016,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14117/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924583989,"domain":"ICLR.cc/2026/Conference","replyto":"tuLF84azND","id":"NgfHfKDafW","forumContent":{"TLDR":{"value":"Propose new architecture Transformer-PSM that generalizes state space models to use softmax attention"},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["state space models","linear RNNs","linear transformers","sequence modeling"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Modern neural sequence models are designed to meet the dual mandate of parallelizable training and fast sequential inference. Recent developments have given rise to various models, such as Gated Linear Attention (GLA) and Mamba, that achieve such ``sequential-parallel duality.'' This raises a natural question: can we characterize the full class of neural sequence models that support near-constant-time parallel evaluation and linear-time, constant-space sequential inference? We begin by describing a broad class of such models -- state space models -- as those whose state updates can be computed using the classic parallel prefix scan algorithm with a custom associative aggregation operator. We then define a more general class, Prefix-Scannable Models (PSMs), by relaxing the state aggregation operator to allow arbitrary (potentially non-associative) functions such as softmax attention. This generalization unifies many existing architectures, including element-wise RNNs (e.g., Mamba) and linear transformers (e.g., GLA, Mamba2, mLSTM), while also introducing new models with softmax-like operators that achieve O(1) amortized compute per token and log(N) memory for sequence length N. We empirically evaluate such models on illustrative small-scale language modeling and canonical synthetic tasks, including state tracking and associative recall. Empirically, we find that PSMs retain the expressivity of transformer-based architectures while matching the inference efficiency of state space models -- in some cases exhibiting better length generalization than either."},"_bibtex":{"value":"@inproceedings{\nyau2026sequential,\ntitle={Sequential Parallel Duality in Prefix Scannable Models},\nauthor={Morris Yau and Sharut Gupta and Valerie Engelmayer and Kazuki Irie and Stefanie Jegelka and Jacob Andreas},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=tuLF84azND}\n}"},"title":{"value":"Sequential Parallel Duality in Prefix Scannable Models"},"pdf":{"value":"/pdf/f5eb77bbc97daf92bd3d76db45182ed7855ed21b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"yau|sequential_parallel_duality_in_prefix_scannable_models"},"authorids":{"value":["~Morris_Yau3","~Sharut_Gupta1","~Valerie_Engelmayer1","~Kazuki_Irie1","~Stefanie_Jegelka3","~Jacob_Andreas1"]},"authors":{"value":["Morris Yau","Sharut Gupta","Valerie Engelmayer","Kazuki Irie","Stefanie Jegelka","Jacob Andreas"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2605.22031v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"fang|somamba_stateownership_mamba_for_unrolled_mri_reconstruction"},"html":{"value":"https://doi.org/10.48550/arXiv.2605.22031"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2605-22031,\n  publtype={informal},\n  author={Pengcheng Fang and Hongli Chen and Fangfang Tang and Feng Liu and Xiaohao Cai and Shanshan Shan},\n  title={SO-Mamba: State-Ownership Mamba for Unrolled MRI Reconstruction},\n  year={2026},\n  month={May},\n  cdate={1777593600000},\n  journal={CoRR},\n  volume={abs/2605.22031},\n  url={https://doi.org/10.48550/arXiv.2605.22031}\n}\n"},"abstract":{"value":"Accelerated MRI reconstruction requires recovering missing details while preserving anatomically coherent structures across large spatial regions. State-space models such as Mamba provide efficient long-range modeling, making them attractive learned regularizers for unrolled reconstruction. However, in a data-consistency-coupled unrolled solver, different stages operate on different reconstruction iterates, where the resident carrier should preserve coherent reconstruction content across stages while stage-dependent non-resident evidence is tied to the current update. Treating these roles uniformly can place persistent resident-carrier evidence and update-dependent non-resident evidence into the same recurrent content route. We therefore propose SO-Mamba, a state-ownership Mamba regularizer that assigns reconstruction evidence within each Mamba stage to recurrent residency, state-interface access, and non-state output correction. SO-Mamba implements this ownership rule with a State-Ownership Router (SOR), which constructs a resident carrier for recurrent content and routes non-resident evidence to affine modulation of the B/C state interfaces and an output correction outlet. The resident carrier supplies the Mamba content route, while the non-resident evidence stream adapts the state interfaces and contributes through the output outlet without entering the recurrent content route. We further introduce a two-level outer-band leakage diagnostic that separates hidden-state storage from readout expression by measuring outer-band energy in the selective-scan state trajectory and the post-scan Mamba readout. Experiments on five public MRI reconstruction benchmarks spanning diverse anatomies, sampling patterns, and coil configurations show that SO-Mamba consistently improves over CNN-, Transformer-, and Mamba-based baselines with competitive computational efficiency."},"title":{"value":"SO-Mamba: State-Ownership Mamba for Unrolled MRI Reconstruction"},"authors":{"value":[{"fullname":"Pengcheng Fang","username":"~Pengcheng_Fang1"},{"fullname":"Hongli Chen","username":""},{"fullname":"Fangfang Tang","username":""},{"fullname":"Feng Liu","username":""},{"fullname":"Xiaohao Cai","username":""},{"fullname":"Shanshan Shan","username":""}]}},"tmdate":1783893369339,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2605-22031"],"tcdate":1783893355083,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Pengcheng_Fang1"],"forum":"pLIuCmmU7I","license":"CC BY-SA 4.0","number":51667,"cdate":1777593600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1783893369339,"domain":"OpenReview.net/Public_Article","id":"pLIuCmmU7I","version":2},{"content":{"summary":{"value":"The work deals with the challenges posed by data corruption in offline RL and proposes a novel approach RDT. The primary contribution is the application of sequence modeling, specifically through a decision transformer, which has shown greater robustness against corrupted data compared to conventional RL methods that rely on temporal difference learning. The RDT model incorporates three key techniques—embedding dropout, Gaussian weighted learning, and iterative data correction—to enhance robustness. Empirical results indicate that RDT achieves superior performance under various corruption scenarios and provides resilience to both training-time data corruption and testing-time observation perturbations."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Could the authors provide more clarity on the choice of dropout probabilities for different embeddings (states, actions, rewards) and its sensitivity to these values? How were these values chosen in the experiments?\n\nThe paper states that actions and rewards are corrected iteratively but leaves out state correction. What challenges or limitations prevented state correction? Would the authors consider incorporating state correction as future work, and how would it theoretically affect robustness?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- Novel Method: The use of sequence modeling as a robust alternative to traditional temporal-difference-based methods for offline RL under corrupted data conditions is a significant and novel contribution.\n- The paper introduces a robust architecture combining embedding dropout, Gaussian weighted learning, and iterative data correction, addressing different aspects of the corruption challenge.\n- The experiments are extensive, covering several benchmarks and a variety of corruption scenarios, demonstrating consistent outperformance of RDT over baselines.\n- The study highlights the potential of sequence modeling to handle real-world data corruption in offline RL, an area with increasing practical relevance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"-The datasets used are predominantly standard RL benchmarks. Incorporating more complex, real-world datasets could provide stronger support for the robustness claims in real-world applications.\n\n- While the paper corrects actions and rewards through iterative data correction, it leaves out the correction of states, which might impact robustness in tasks with high state complexity.\n\n- The study lacks a detailed comparison with other robust sequence modeling approaches outside of decision transformers, such as those incorporating value functions or other regularization techniques."}},"nonreaders":[],"tmdate":1731428040316,"tcdate":1730763354066,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission7573/Reviewer_4tSr"],"signatures":["ICLR.cc/2025/Conference/Submission7573/Reviewer_4tSr"],"forum":"phAlw3JPms","number":4,"license":"CC BY 4.0","cdate":1730763354066,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission7573/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428040316,"domain":"ICLR.cc/2025/Conference","replyto":"phAlw3JPms","id":"JMJdJvouKH","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Offline Reinforcement Learning","Data Corruption","Robust Reinforcement Learning"]},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Learning policy from offline datasets through offline reinforcement learning (RL) holds promise for scaling data-driven decision-making while avoiding unsafe and costly online interactions. However, real-world data collected from sensors or humans often contains noise and errors, posing a significant challenge for existing offline RL methods, particularly when the real-world data is limited. Our study reveals that prior research focusing on adapting predominant offline RL methods based on temporal difference learning still falls short under data corruption when the dataset is limited. In contrast, we discover that vanilla sequence modeling methods, such as Decision Transformer, exhibit robustness against data corruption, even without specialized modifications. To unlock the full potential of sequence modeling, we propose **R**obust **D**ecision **T**ransformer (**RDT**) by incorporating three simple yet effective robust techniques: embedding dropout to improve the model's robustness against erroneous inputs, Gaussian weighted learning to mitigate the effects of corrupted labels, and iterative data correction to eliminate corrupted data from the source. \nExtensive experiments on MuJoCo, Kitchen, and Adroit tasks demonstrate RDT's superior performance under various data corruption scenarios compared to prior methods. Furthermore, RDT exhibits remarkable robustness in a more challenging setting that combines training-time data corruption with test-time observation perturbations. These results highlight the potential of sequence modeling for learning from noisy or corrupted offline datasets, thereby promoting the reliable application of offline RL in real-world scenarios.\nOur code is available at https://github.com/jiawei415/RobustDecisionTransformer。"},"_bibtex":{"value":"@inproceedings{\nxu2025tackling,\ntitle={Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling},\nauthor={Jiawei Xu and Rui Yang and Shuang Qiu and Feng Luo and Meng Fang and Baoxiang Wang and Lei Han},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=phAlw3JPms}\n}"},"title":{"value":"Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling"},"pdf":{"value":"/pdf/ff035218e38fcc7a0bae34cdec894007dca6e550.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"xu|tackling_data_corruption_in_offline_reinforcement_learning_via_sequence_modeling"},"authorids":{"value":["~Jiawei_Xu1","~Rui_Yang8","~Shuang_Qiu2","~Feng_Luo4","~Meng_Fang1","~Baoxiang_Wang1","~Lei_Han1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jiawei Xu","Rui Yang","Shuang Qiu","Feng Luo","Meng Fang","Baoxiang Wang","Lei Han"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a tree scanning algorithm for state space models specifically for Mamba. The naive and fixed scan patterns like raster or local scans commonly used for vision tasks do not consider the topological structure of 2D image input. The proposed algorithm generates a minimum spanning tree which can help Mamba to model the semantic information of the input. The paper further introduces a dynamic programming procedure to avoid the quadratic complexity of the tree scanning algorithm. The experiments on various vision and language modeling tasks show that the tree topology scan helps improve accuracy."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. In eq 5, Ω is the index set of all vertices. As $S(E_{ij})$ aggregates the connected vertices, some vertices will be visited many times during the recurrent scan in Mamba. It seems like redundant computations. Could authors either empirically or theoretically justify why this is needed? \n\n2. In language understanding, the tree scanning algorithm is applied during finetuning. Intuitively, this will change the causality of tokens trained during pretraining. Is there any specific reason the algorithm is not used during pre-training?\n\n3. How is Eq 5 (state aggregation process) derived from Eq 4 (SSM state computation)? Could authors provide more explanation about it? \n\n4. The semantic segmentation results seem marginal compared to other tasks, and this is the only dense prediction task. Can this be due to aggregation instead of processing all pixels by Mamba?"},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The motivation and the proposed scan algorithm make sense and are easy to understand. \n2. The proposed method can be useful not only for SSMs but also for other models that require sequential scans. \n3. As the proposed scan already generates a structure of the input based on the relevance of tokens, the sequence learning process can potentially be minimal. \n4. Using the proposed method, GrootV outperforms many recent baselines for multiple tasks (image classification/segmentation, object detection, and language understanding)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Regarding the root setting, the authors show that the root setting to all vertices outperforms the ones with only the first or the last. However, it increases the traverse time (1 vertex vs all vertices), and the accuracy improvement using all vertex is marginal (only 0.4% on Imagenet-1K). I understand that the dynamic programming procedure improves the speed but still increases by the sequence length $L$. I am still not fully convinced how effective this is. Can authors comment on it? \n\n3. Related to point 2, there are no speed comparisons, especially with different scanning strategies, root settings, and the use of the dynamic programming procedure in practice.\n\n1. Some details are missing in the paper. \n    1. Figure 1 is not explained in the paper. What exactly is the parameter generator?  Is it the initialization stage of the state, input, and output matrices (A, B, C, and D), or is the projection of these parameters the same as in Mamba? In the tree state space model part, is the main difference using TSA instead of eq 4 compared to the original Mamba? \n    2. In Figure 4, how is the specific position defined? Is it based on the root vertex setting? \n    3. The cross scan used in Table 4 is not described."},"limitations":{"value":"The limitation is discussed and isreasonable."}},"nonreaders":[],"tmdate":1730878928224,"tcdate":1720937751863,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission4467/Reviewer_xmLw"],"signatures":["NeurIPS.cc/2024/Conference/Submission4467/Reviewer_xmLw"],"forum":"W8rFsaKr4m","number":3,"license":"CC BY 4.0","cdate":1720937751863,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission4467/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878928224,"domain":"NeurIPS.cc/2024/Conference","replyto":"W8rFsaKr4m","id":"cJbjQhuGRd","forumContent":{"venue":{"value":"NeurIPS 2024 spotlight"},"keywords":{"value":["State Space Model","Vision Recognition","Language Understanding"]},"supplementary_material":{"value":"/attachment/6e40433747a167e805c5b0a45c13991a8732ac20.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"The state space models, employing recursively propagated features, demonstrate strong representation capabilities comparable to Transformer models and superior efficiency.\nHowever, constrained by the inherent geometric constraints of sequences, it still falls short in modeling long-range dependencies.\nTo address this issue, we propose the MambaTree network, which first dynamically generates a tree topology based on spatial relationships and input features.\nThen, feature propagation is performed based on this graph, thereby breaking the original sequence constraints to achieve stronger representation capabilities.\nAdditionally, we introduce a linear complexity dynamic programming algorithm to enhance long-range interactions without increasing computational cost.\nMambaTree is a versatile multimodal framework that can be applied to both visual and textual tasks.\nExtensive experiments demonstrate that our method significantly outperforms existing structured state space models on image classification, object detection and segmentation.\nBesides, by fine-tuning large language models, our approach achieves consistent improvements in multiple textual tasks at minor training cost."},"_bibtex":{"value":"@inproceedings{\nxiao2024mambatree,\ntitle={MambaTree: Tree Topology is All You Need in State Space Model},\nauthor={Yicheng Xiao and Lin Song and Shaoli Huang and Jiangshan Wang and Siyu Song and Yixiao Ge and Xiu Li and Ying Shan},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=W8rFsaKr4m}\n}"},"title":{"value":"MambaTree: Tree Topology is All You Need in State Space Model"},"pdf":{"value":"/pdf/ecf846fd239301d12bd43f4cb289ebe072572a80.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"xiao|mambatree_tree_topology_is_all_you_need_in_state_space_model"},"authorids":{"value":["~Yicheng_Xiao1","~Lin_Song2","~Shaoli_Huang2","~Jiangshan_Wang2","~Siyu_Song1","~Yixiao_Ge2","~Xiu_Li1","~Ying_Shan2"]},"authors":{"value":["Yicheng Xiao","Lin Song","Shaoli Huang","Jiangshan Wang","Siyu Song","Yixiao Ge","Xiu Li","Ying Shan"]}},"version":2},{"content":{"summary":{"value":"This paper presents the Trajectory Generative Pre-trained Transformer (TrajGPT), a Transformer model designed to handle irregularly sampled time series, particularly within the healthcare domain. TrajGPT introduces a novel attention mechanism, Selective Recurrent Attention (SRA). Inspired by discretized ordinary differential equations (ODEs), the model achieves interpolation and extrapolation in irregular time series, excelling in trajectory prediction tasks. Experimental results show outstanding performance in zero-shot learning scenarios, underscoring TrajGPT’s capability to capture complex health dynamics from incomplete data."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"- Do you believe this method could be easily extended to the context of multivariate time series?\n\n- If expanded to multivariate time series, could it be combined with methods like GNNs to extract improved spatio-temporal representations?\n\n- Despite the relationship, it remains unclear what the main advantages or potential inductive bias would be in using this method over Mamba blocks. Could you clarify this further? I would be happy to increase my score if the authors could expand on this information to clarify this important issue."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":4},"strengths":{"value":"- Comprehensive literature review.\n\n- Thorough evaluation across different scenarios, with comparisons to multiple models and diverse datasets. The inclusion of interpretability through added figures is beneficial, and the ablation study is particularly valuable for assessing the contribution of each model component.\n\n- The method is highly compelling, supported by a robust and thorough mathematical explanation.\n\n- Novelty of the Attention Mechanism: The introduction of the Selective Recurrent Attention (SRA) module, which uses data-dependent decay, is an innovative contribution.\n\n- Interpretation of TrajGPT as a Discretized ODE: Modeling TrajGPT as a discretized ODE is a valuable approach, allowing the model to capture continuous dynamics in irregular time series and perform time-specific inference, facilitating both interpolation and extrapolation in temporal data.\n\n- Throughout the paper, I noted the similarities to State Space Models, which led me to find Appendix C particularly valuable."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"There are few weaknesses in this paper:\n\n- Since the relationship with State Space Models (SSMs) is discussed, it might have been interesting to include a method using Mamba to compare results with this type of model.\n\n- Including a comparison with methods such as Multi-Gaussian Processes, which have been previously used for this type of approach, could strengthen the evaluation [1].\n\n- Ultimately, as I understand it, this method has only been applied to univariate time series; it would be beneficial to explore its extension to multivariate time series in future work.\n\n[1] Moor, M., Horn, M., Rieck, B., Roqueiro, D., & Borgwardt, K. (2019, October). Early recognition of sepsis with Gaussian process temporal convolutional networks and dynamic time warping. In Machine learning for healthcare conference (pp. 2-26). PMLR."}},"nonreaders":[],"tmdate":1732532766545,"tcdate":1730310104978,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12647/Reviewer_yGWz"],"signatures":["ICLR.cc/2025/Conference/Submission12647/Reviewer_yGWz"],"forum":"OYT7yZfBFw","number":1,"license":"CC BY 4.0","cdate":1730310104978,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12647/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732532766545,"domain":"ICLR.cc/2025/Conference","replyto":"OYT7yZfBFw","id":"FzE9dtCeny","forumContent":{"TLDR":{"value":"We propose TrajGPT for irregular time-series representation learning and health trajectory analysis"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["GPT","Representation learning","Linear attention","ODE","Irregularly-sample time series","zero-shot learning","trajectory analysis"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"In many domains, such as healthcare, time-series data is often irregularly sampled with varying intervals between observations. This poses challenges for classical time-series models that require equally spaced data. To address this, we propose a novel time-series Transformer called **Trajectory Generative Pre-trained Transformer (TrajGPT)**. TrajGPT employs a novel Selective Recurrent Attention (SRA) mechanism, which utilizes a data-dependent decay to adaptively filter out irrelevant past information based on contexts. By interpreting TrajGPT as discretized ordinary differential equations (ODEs), it effectively captures the underlying continuous dynamics and enables time-specific inference for forecasting arbitrary target timesteps. Experimental results demonstrate that TrajGPT excels in trajectory forecasting, drug usage prediction, and phenotype classification without requiring task-specific fine-tuning. By evolving the learned continuous dynamics,  TrajGPT can interpolate and extrapolate disease risk trajectories from partially-observed time series. The visualization of predicted health trajectories shows that TrajGPT forecasts unseen diseases based on the history of clinically relevant phenotypes (i.e., contexts)."},"_bibtex":{"value":"@misc{\nsong2025trajgpt,\ntitle={Traj{GPT}: Irregular Time-Series Representation Learning for Health Trajectory Analysis},\nauthor={Ziyang Song and Qincheng Lu and He Zhu and David L. Buckeridge and Yue Li},\nyear={2025},\nurl={https://openreview.net/forum?id=OYT7yZfBFw}\n}"},"title":{"value":"TrajGPT: Irregular Time-Series Representation Learning for Health Trajectory Analysis"},"pdf":{"value":"/pdf/bdf50797891a25ace3348c1c76fa15558a63566c.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"song|trajgpt_irregular_timeseries_representation_learning_for_health_trajectory_analysis"},"authorids":{"value":["~Ziyang_Song4","~Qincheng_Lu1","~He_Zhu5","~David_L._Buckeridge1","~Yue_Li15"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Ziyang Song","Qincheng Lu","He Zhu","David L. Buckeridge","Yue Li"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.11244v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"choi|spotmamba_learning_longrange_dependency_on_spatiotemporal_graphs_with_selective_state_spaces"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jinhyeok_Choi:","https://dblp.org/search/pid/api?q=author:Heehyeon_Kim:","https://dblp.org/search/pid/api?q=author:Minhyeong_An:","~Joyce_Jiyoung_Whang1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.11244"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-11244,\n  publtype={informal},\n  author={Jinhyeok Choi and Heehyeon Kim and Minhyeong An and Joyce Jiyoung Whang},\n  title={SpoT-Mamba: Learning Long-Range Dependency on Spatio-Temporal Graphs with Selective State Spaces},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.11244},\n  url={https://doi.org/10.48550/arXiv.2406.11244}\n}\n"},"abstract":{"value":"Spatio-temporal graph (STG) forecasting is a critical task with extensive applications in the real world, including traffic and weather forecasting. Although several recent methods have been proposed to model complex dynamics in STGs, addressing long-range spatio-temporal dependencies remains a significant challenge, leading to limited performance gains. Inspired by a recently proposed state space model named Mamba, which has shown remarkable capability of capturing long-range dependency, we propose a new STG forecasting framework named SpoT-Mamba. SpoT-Mamba generates node embeddings by scanning various node-specific walk sequences. Based on the node embeddings, it conducts temporal scans to capture long-range spatio-temporal dependencies. Experimental results on the real-world traffic forecasting dataset demonstrate the effectiveness of SpoT-Mamba."},"title":{"value":"SpoT-Mamba: Learning Long-Range Dependency on Spatio-Temporal Graphs with Selective State Spaces"},"authors":{"value":["Jinhyeok Choi","Heehyeon Kim","Minhyeong An","Joyce Jiyoung Whang"]}},"tmdate":1739940934749,"pdate":1704067200000,"tcdate":1739940931869,"writers":["~"],"signatures":["~Joyce_Jiyoung_Whang2"],"forum":"CRBzZr5sTb","license":"CC BY-SA 4.0","number":326582,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1739940934749,"domain":"DBLP.org","id":"CRBzZr5sTb","version":2},{"content":{"summary":{"value":"This paper introduces M-VAR, a new autoregressive image model based on VAR. The core idea is to decouple VAR into intra-scale modeling and inter-scale modeling. For intra-scale modeling, softmax attention is used, while mamba is used for inter-scale modeling. On ImageNet 256x256 and ImageNet 512x512, M-VAR achieves better efficiency/FID than VAR."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. What's the setting for speed comparison (hardware, inference engine, batch size, etc)? In addition to relative speedup ratios, adding measured latency/throughput in the tables will be better.\n2. Why M-VAR can deliver better FID than VAR? I can see that M-VAR has advantages over VAR from the efficiency perspective. But, from the model capacity perspective, I do not see clear advantages."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. This paper is well-written. The motivation is clear and reasonable. The proposed method is also presented clearly. \n2. On ImageNet 256x256 and 512x512, M-VAR demonstrates better results than VAR.\n3. Ablation study shows intra-scale attention + mamba works better than global attention on VAR."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Technical contribution is limited. Replacing global attention with hybrid model architectures has been extensively explored in the AI community. A big concern of such designs is that they may not preserve advantages after scaling up and applying them to real-world use cases (e.g., text-to-image generation). Given that this work only has ImageNet results, the value of the current manuscript is limited for the community. \n2. It is unclear why M-VAR can deliver better FID than VAR. From the model capacity perspective, global attention should have a stronger/similar capacity than intra-scale attention and mamba.  \n3. Current design choices seem quite random, lacking detailed ablation studies. For example, there are many different choices for intra-scale modeling and inter-scale modeling (RWKV, linear attention, etc). Is there any insight on why choosing the current design?\n4. According to the ImageNet experiments, the improvements look a bit incremental."}},"nonreaders":[],"tmdate":1731427517042,"tcdate":1730660619658,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1959/Reviewer_qR12"],"signatures":["ICLR.cc/2025/Conference/Submission1959/Reviewer_qR12"],"forum":"BYoN2c0o6M","number":1,"license":"CC BY 4.0","cdate":1730660619658,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1959/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427517042,"domain":"ICLR.cc/2025/Conference","replyto":"BYoN2c0o6M","id":"HW1vQJ4us3","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Scale-wise Autoregressive Model"]},"primary_area":{"value":"generative models"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"There exists recent work in computer vision, named VAR, that proposes a new autoregressive paradigm for image generation. Diverging from the vanilla next-token prediction, VAR structurally reformulates the image generation into a coarse to fine next-scale prediction. \nIn this paper, we show that this scale-wise autoregressive framework can be effectively decoupled into \\textit{intra-scale modeling}, which captures local spatial dependencies within each scale, and \\textit{inter-scale modeling}, which models cross-scale relationships progressively from coarse-to-fine scales.\nThis decoupling structure allows to rebuild VAR in a more computationally efficient manner. Specifically, for intra-scale modeling --- crucial for generating high-fidelity images --- we retain the original bidirectional self-attention design to ensure comprehensive modeling; for inter-scale modeling, which semantically connects different scales but is computationally intensive, we apply linear-complexity mechanisms like Mamba to substantially reduce computational overhead. \nWe term this new framework M-VAR. Extensive experiments demonstrate that our method outperforms existing models in both image quality and generation speed. For example, our 1.5B model, with fewer parameters and faster inference speed, outperforms the largest VAR-d32-2B. Moreover, our largest model M-VAR-d32 impressively registers 1.78 FID on ImageNet 256$\\times$256 and outperforms the prior-art autoregressive models LlamaGen/VAR by 0.4/0.19 and popular diffusion models LDM/DiT by 1.82/0.49, respectively."},"_bibtex":{"value":"@misc{\nren2024mvar,\ntitle={M-{VAR}: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation},\nauthor={Sucheng Ren and Yaodong Yu and Nataniel Ruiz and Feng Wang and Alan Yuille and Cihang Xie},\nyear={2024},\nurl={https://openreview.net/forum?id=BYoN2c0o6M}\n}"},"title":{"value":"M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation"},"pdf":{"value":"/pdf/8835246a72db2706a543e2db5e5821a0a7a9965d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"ren|mvar_decoupled_scalewise_autoregressive_modeling_for_highquality_image_generation"},"authorids":{"value":["~Sucheng_Ren1","~Yaodong_Yu4","~Nataniel_Ruiz1","~Feng_Wang14","~Alan_Yuille1","~Cihang_Xie3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Sucheng Ren","Yaodong Yu","Nataniel Ruiz","Feng Wang","Alan Yuille","Cihang Xie"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2510.11017v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"feng|highresolution_spatiotemporal_modeling_with_globallocal_state_space_models_for_videobased_human_pose_estimation"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Runyang_Feng:","https://dblp.org/search/pid/api?q=author:Hyung_Jin_Chang:","https://dblp.org/search/pid/api?q=author:Tze_Ho_Elden_Tse:","https://dblp.org/search/pid/api?q=author:Boeun_Kim:","~Yi_Chang4","https://dblp.org/search/pid/api?q=author:Yixing_Gao:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2510.11017"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2510-11017,\n  publtype={informal},\n  author={Runyang Feng and Hyung Jin Chang and Tze Ho Elden Tse and Boeun Kim and Yi Chang and Yixing Gao},\n  title={High-Resolution Spatiotemporal Modeling with Global-Local State Space Models for Video-Based Human Pose Estimation},\n  year={2025},\n  month={October},\n  cdate={1759276800000},\n  journal={CoRR},\n  volume={abs/2510.11017},\n  url={https://doi.org/10.48550/arXiv.2510.11017}\n}\n"},"abstract":{"value":"Modeling high-resolution spatiotemporal representations, including both global dynamic contexts (e.g., holistic human motion tendencies) and local motion details (e.g., high-frequency changes of keypoints), is essential for video-based human pose estimation (VHPE). Current state-of-the-art methods typically unify spatiotemporal learning within a single type of modeling structure (convolution or attention-based blocks), which inherently have difficulties in balancing global and local dynamic modeling and may bias the network to one of them, leading to suboptimal performance. Moreover, existing VHPE models suffer from quadratic complexity when capturing global dependencies, limiting their applicability especially for high-resolution sequences. Recently, the state space models (known as Mamba) have demonstrated significant potential in modeling long-range contexts with linear complexity; however, they are restricted to 1D sequential data. In this paper, we present a novel framework that extends Mamba from two aspects to separately learn global and local high-resolution spatiotemporal representations for VHPE. Specifically, we first propose a Global Spatiotemporal Mamba, which performs 6D selective space-time scan and spatial- and temporal-modulated scan merging to efficiently extract global representations from high-resolution sequences. We further introduce a windowed space-time scan-based Local Refinement Mamba to enhance the high-frequency details of localized keypoint motions. Extensive experiments on four benchmark datasets demonstrate that the proposed model outperforms state-of-the-art VHPE approaches while achieving better computational trade-offs."},"title":{"value":"High-Resolution Spatiotemporal Modeling with Global-Local State Space Models for Video-Based Human Pose Estimation"},"authors":{"value":["Runyang Feng","Hyung Jin Chang","Tze Ho Elden Tse","Boeun Kim","Yi Chang","Yixing Gao"]}},"tmdate":1768542539733,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2510-11017"],"tcdate":1768542532834,"writers":["~"],"signatures":["~Yi_Chang4"],"forum":"ZAEiwcsYQU","license":"CC BY-SA 4.0","number":767723,"cdate":1759276800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1768542539733,"domain":"DBLP.org","id":"ZAEiwcsYQU","version":2},{"content":{"summary":{"value":"This paper introduces PRLS-RFF, a framework designed to improve the robustness of Radio Frequency Fingerprint Identification (RFFI) against domain shifts (such as changes in channel, receiver distance, and time) and open-world scenarios. The authors propose a dual-stream backbone architecture that utilizes Selective State-Space Models (Mamba) to process and fuse time-domain (I/Q) and time-frequency (Continuous Wavelet Transform - CWT) representations. To train this backbone, the paper employs a self-supervised pretraining approach using a multi-view non-contrastive objective. Crucially, the views are generated using \"physics-consistent\" augmentations tailored to RF signals, including Additive White Gaussian Noise (AWGN), phase distortion, Carrier Frequency Offset (CFO), and I/Q imbalance. The goal is to learn representations that are invariant to these specific perturbations while retaining device-intrinsic signatures.\n\nEmpirical evaluations on a UAV RF fingerprint dataset demonstrate that PRLS-RFF outperforms recent baselines in cross-domain identification (varying distances), few-shot adaptation, and open-world recognition tasks."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please see weakness for details."},"rating":{"value":2},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"I appreciate the authors' effort to apply new architectures like Mamba and self-supervised learning to the field of RF Fingerprinting: While dual-stream / dual-encoder architectures and self-supervised learning are established in other fields, their specific application here—coupling Mamba-based long-context modeling with RF-specific physical augmentations for fingerprinting—is new."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The main weaknesses of the paper are lack of clarity and lack of motivation. A lot of things are presumed, which ultimately weakens the scientific contribution of this paper.\n\n- Architecture design in Section 3.3 and 3.4 seems ad-hoc and not well-justified in the paper. The motivation seems weak. Maybe I missed them, but there seems to be no dedicated ablation / exploration experiments on design choices of the main architecture in Figure 1 and 2 - to name a few, why Mamba block not other blocks (Transformer, CNN, RNN, etc)? how are # parameters decided for each architecture blocks?\n\n- Mamba block in Figure 1 is not clearly introduced in the paper.\n\n- Why the self-supervised loss is \"non-contrastive\"? I wonder if there is a contrastive counterpart and how it performs.\n\n- The dataset used and the evaluation metric (i.e. top1 accuracy) are not clearly introduced in the paper.\n\n- The baselines in Table one are not clearly described. For example CNN-LSTM is super unclear - there could be numerous ways of designing the architecture and altering the design choices like # parameters each layer, training loss - what we are exactly comparing against here?\n\n- typo: line 366 \"about 1 2 points\" should be \"about 1-2 points\""}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918167468,"tcdate":1762150023911,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5634/Reviewer_j9Bp"],"signatures":["ICLR.cc/2026/Conference/Submission5634/Reviewer_j9Bp"],"forum":"N5gNhO2OPB","number":4,"license":"CC BY 4.0","cdate":1762150023911,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5634/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918167468,"domain":"ICLR.cc/2026/Conference","replyto":"N5gNhO2OPB","id":"YNh4dBfVEx","forumContent":{"TLDR":{"value":"Physics-consistent multi-view self-supervised contrastive pretraining for a Mamba dual-stream RF model, learning hardware-invariant fingerprints and improving cross-domain classification & open-world detection with fewer labels."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Selective State-Space Models (Mamba);Radio-Frequency Fingerprinting (RFF);Cross-Domain Generalization; Open-World Device Recognition; Wireless Security"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Under domain shifts and open-world conditions, reliably re-identifying radio frequency (RF) devices is essential for wireless security. As a hardware-rooted physical-layer signature, the RF fingerprint has been widely used for device re-identification. However, supervised RF fingerprint identification (RFFI) models often overfit acquisition artifacts and rely on extensive supervision, leading to sharp cross-domain performance drops and weak open-world behavior. To address these limitations, we introduce PRLS-RFF, which targets physically consistent RF fingerprint representations. We design a dual-stream Mamba-based backbone with physics-consistent, multi-view perturbations to encode representation-level invariances. Additionally, to better capture the structure of RF fingerprints (RFFs) across transient and steady-state regimes, the backbone fuses time-domain and time--frequency features using efficient long-context modeling. As a result, the learned representations are domain-robust, which can support reliable open-set recognition. To validate its robustness, we conduct extensive experiments on several public datasets and observe the performance surpassing state-of-the-art models in both cross-domain identification and open-set recognition tasks."},"_bibtex":{"value":"@misc{\nzhang2026prlsrff,\ntitle={{PRLS}-{RFF}: Physically Consistent Representation Learning with Self-Supervised Pretraining for {RF} Fingerprinting},\nauthor={Tao Zhang and Siya Mi and Yu Zhang and Zihua Wang and chunlin wen and Xiu-Shen Wei},\nyear={2026},\nurl={https://openreview.net/forum?id=N5gNhO2OPB}\n}"},"title":{"value":"PRLS-RFF: Physically Consistent Representation Learning with Self-Supervised Pretraining for RF Fingerprinting"},"pdf":{"value":"/pdf/61f1ddb3a683159bfdc1cadc6df2970af594f6a7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|prlsrff_physically_consistent_representation_learning_with_selfsupervised_pretraining_for_rf_fingerprinting"},"authorids":{"value":["~Tao_Zhang74","~Siya_Mi1","~Yu_Zhang21","~Zihua_Wang1","~chunlin_wen1","~Xiu-Shen_Wei1"]},"authors":{"value":["Tao Zhang","Siya Mi","Yu Zhang","Zihua Wang","chunlin wen","Xiu-Shen Wei"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.02615v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"fu|lamambadiff_lineartime_highfidelity_diffusion_models_based_on_local_attention_and_mamba"},"authorids":{"value":["~Yunxiang_Fu1","https://dblp.org/search/pid/api?q=author:Chaoqi_Chen:","https://dblp.org/search/pid/api?q=author:Yizhou_Yu:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.02615"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-02615,\n  publtype={informal},\n  author={Yunxiang Fu and Chaoqi Chen and Yizhou Yu},\n  title={LaMamba-Diff: Linear-Time High-Fidelity Diffusion Models Based on Local Attention and Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.02615},\n  url={https://doi.org/10.48550/arXiv.2408.02615}\n}\n"},"abstract":{"value":"Recent Transformer-based diffusion models have shown remarkable performance, largely attributed to the ability of the self-attention mechanism to accurately capture both global and local contexts by computing all-pair interactions among input tokens. However, their quadratic complexity poses significant computational challenges for long-sequence inputs. Conversely, a recent state space model called Mamba offers linear complexity by compressing a filtered global context into a hidden state. Despite its efficiency, compression inevitably leads to information loss of fine-grained local dependencies among tokens, which are crucial for effective visual generative modeling. Motivated by these observations, we introduce Local Attentional Mamba (LaMamba) blocks that combine the strengths of self-attention and Mamba, capturing both global contexts and local details with linear complexity. Leveraging the efficient U-Net architecture, our model exhibits exceptional scalability and surpasses the performance of DiT across various model scales on ImageNet at 256x256 resolution, all while utilizing substantially fewer GFLOPs and a comparable number of parameters. Compared to state-of-the-art diffusion models on ImageNet 256x256 and 512x512, our largest model presents notable advantages, such as a reduction of up to 62% GFLOPs compared to DiT-XL/2, while achieving superior performance with comparable or fewer parameters. Our code is available at https://github.com/yunxiangfu2001/LaMamba-Diff."},"title":{"value":"LaMamba-Diff: Linear-Time High-Fidelity Diffusion Models Based on Local Attention and Mamba"},"authors":{"value":["Yunxiang Fu","Chaoqi Chen","Yizhou Yu"]}},"tmdate":1731137640307,"pdate":1704067200000,"tcdate":1731137636894,"writers":["~"],"signatures":["~Yunxiang_Fu1"],"forum":"xCaBeUciMs","license":"CC BY-SA 4.0","number":176315,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1731137640307,"domain":"DBLP.org","id":"xCaBeUciMs","version":2},{"content":{"summary":{"value":"This work proposes a novel method for continual offline reinforcement learning called VQ-CD that combines vector quantization with diffusion for continual learning.\nThe vector quantization part enables training of a unified state and action space such that VQ-CD  can be trained jointly on environments that provide different state/action spaces.\nThe policy is based on a diffusion model and weight masking that relies on task information.\nThe weight masks are constructed in a manner that they can be readily merged into a single model after training.\nVQ-CD exhibits strong performance on Ant-dir, CW10, and D4RL environments with different state/action spaces compared to baselines."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- $w$ in line 193 has not been defined, what is it?\n- Is there any intuition about why the constraint in eq. (2) is important?\n- The constraint in eq. (2) could easily be incorporated in the loss function, why is clipping used?\n- What are \"nonsignificant masks\" (line 352)\n- What is the source of the data for CW10 and MuJoCo Ant-dir?\n- How is it possible that VQ-CD is better than Multitask on two of the D4RL tasks?\n- Why would the gaussian constraint for VAE hurt space alignment?\n- How does reconstruction look for VAE-CD in Table 2? Only VQ-CD and AE-CD are shown there.\n- Why does multitask baseline performance decrease with VQ state/action spaces (Figure 11)?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"The idea of leveraging vector quantization to learn a unified state/action space is strong and enables joint training on spaces of different dimensionality.\n\nEmpirical evidence seems strong and the method is compared against plenty of baselines.\n\nAblation studies highlight the importance of combining VQ with CD."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Significance of results:**\n\nWhile I appreciate the number of experiments conducted and baselines the authors trained, I am not entirely certain the compared methods are state of the art.\nOne particular baseline that comes to mind is L2M [1], which constructs unified state and action spaces  (similar to what QSA does) and incorporates a task matching mechanism to train separate weights for each task (similar to SWA).\nTherefore, L2M should be added as a baseline. Moreover, the unified state/action space of L2M can be used as an additional ablation study to the VQ approach, as it is simpler and does not require training.\n\n\n**Limitations:**\n\nThe authors do not elaborate on limitations, however there are notable ones that should be mentioned:\n- Training needs to be conducted on each task separately, resulting in separate model checkpoints with different masked weights, this scales linearly with the number of tasks\n- VQ-CD relies on task information, other methods [1] can be used in a task-agnostic manner\n\n**Presentation:**\n\nFirst, the title overclaims the contribution of the paper as it does not \"solve\" the field, please consider renaming.\nPlease also rephrase \"any CL tas sequence settings\", to something like \"CL with different state/action spaces\", as \"any CL task\" is ill-defined.\nFigure 1 only shows the VQ-CD training pipeline, but line 465 talks about \"return-based action generation\" which is never explained in the methodology.\nIn fact, it is never explained whether the diffusion part constitutes the policy and how the inverse dynamics model is actually used.\nCan the authors elaborate in more detail about what is the policy and how the inverse dynamics model is used?\nFigure 2 is never referred to in the text.\nCan the authors adjust Figure 4 so that the task boundaries are visible? As of now it is not clear at which steps the task is switched as there are now clear differences in the learning curves.\nLine 423 sounds like state and action padding is being used for VQ-CD, but I assume it is only used for baselines, is this correct?\nLine 526 sounds like one action vector is being decomposed into several latent vectors, however VQ only quantizes into a single latent.\nIt is not clear what the labels in Figure 7 indicate, can the authors elaborate and make it more explicit?\n\n**References:**\n\n[1] Learning to Modulate pre-trained Models in RL, Schmied et al., NeurIPS 2023"}},"nonreaders":[],"tmdate":1732537581367,"tcdate":1730386992043,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5556/Reviewer_jUqW"],"signatures":["ICLR.cc/2025/Conference/Submission5556/Reviewer_jUqW"],"forum":"Hr3TBaZl4S","number":2,"license":"CC BY 4.0","cdate":1730386992043,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5556/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732537581367,"domain":"ICLR.cc/2025/Conference","replyto":"Hr3TBaZl4S","id":"GD4rL6cZAc","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We propose to solve continual offline RL with various state and action spaces by adopting selective weights activation on the quantized alignment spaces."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["continual offline reinforcement learning"]},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based continual learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space. Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 16 baselines, our method reaches the SOTA performance."},"_bibtex":{"value":"@misc{\nhu2025solving,\ntitle={Solving Continual Offline {RL} through Selective Weights Activation on Aligned Spaces},\nauthor={Jifeng Hu and Sili Huang and Li Shen and Zhejian Yang and Shengchao Hu and Shisong Tang and Hechang Chen and Yi Chang and Dacheng Tao and Lichao Sun},\nyear={2025},\nurl={https://openreview.net/forum?id=Hr3TBaZl4S}\n}"},"title":{"value":"Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces"},"pdf":{"value":"/pdf/19ade0bc8e9c98b3410f06b22f7f2a308a2ed7fe.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"hu|solving_continual_offline_rl_through_selective_weights_activation_on_aligned_spaces"},"authorids":{"value":["~Jifeng_Hu1","~Sili_Huang1","~Li_Shen1","~Zhejian_Yang1","~Shengchao_Hu1","~Shisong_Tang1","~Hechang_Chen2","~Yi_Chang4","~Dacheng_Tao1","~Lichao_Sun1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jifeng Hu","Sili Huang","Li Shen","Zhejian Yang","Shengchao Hu","Shisong Tang","Hechang Chen","Yi Chang","Dacheng Tao","Lichao Sun"]}},"version":2},{"content":{"summary":{"value":"The paper proposes a new class of selective state space models (SSMs) for sequence modeling that achieves Transformer-quality performance while scaling linearly in sequence length. The paper addresses the key problem in SSMs for selecting data by selecting particular inputs. The paper presents a hardware-aware algorithm that computes the model recurrently with a scan instead of convolution, avoiding materializing the expanded state to reduce memory usage. This results in faster computation than previous methods.\n\nThe paper simplifies prior deep sequence model architectures into a homogeneous architecture which is called as Mamba, incorporating the selective SSMs. Mamba enjoys fast inference, linear scaling, and improved performance on long sequences.\n\nIn the results the authors show that  Mamba achieves state of the art on synthetic tasks, audio/genomics modeling, and language modeling and outperforms Transformers of the same size on language modeling in both pretraining and downstream tasks.\n\nThe results suggest selective SSMs and the Mamba architecture could be a strong candidate for a general sequence model backbone for foundation models across modalities. The paper demonstrates the potential for linear-time models to match or exceed the performance of quadratic Transformers."},"presentation":{"value":"3 good"},"contribution":{"value":"3 good"},"soundness":{"value":"3 good"},"strengths":{"value":"+ A key limitation of prior SSMs is the inability to efficiently select data in an input-dependent manner. The paper introduces a key mechanism by parameterizing the SSM parameters based on the input, allowing the model to filter out irrelevant information and remember relevant information indefinitely.\n+ The results as compared to Pythia, and Transforms on many benchmarks are impressive."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The model still has a quadratic memory requirement during training like Transformers."},"confidence":{"value":"2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked."},"questions":{"value":"1) Have you evaluated scaling behavior beyond 1.4B parameters? How does it compare to Transformers at 10B scales?\n\n2) The input selection mechanism introduces additional hyper parameters. How sensitive are the results to hyperparameters like the projection rank?"},"rating":{"value":"6: marginally above the acceptance threshold"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636386944,"tcdate":1699073663018,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission4202/Reviewer_iEaX"],"signatures":["ICLR.cc/2024/Conference/Submission4202/Reviewer_iEaX"],"forum":"AL1fq05o7H","number":3,"license":"CC BY 4.0","cdate":1699073663018,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission4202/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636386944,"domain":"ICLR.cc/2024/Conference","replyto":"AL1fq05o7H","id":"bq8XeiBhwY","forumContent":{"venue":{"value":"Submitted to ICLR 2024"},"TLDR":{"value":"We introduce a selection mechanism into state space models, leading to state-of-the-art results on general sequence modeling including language."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Sequence model","language model","state space model","RNN","SSM","S4","Mamba"]},"primary_area":{"value":"representation learning for computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language.  We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to *selectively* propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (**Mamba**). Mamba enjoys fast inference (5$\\times$ higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-1.4B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation."},"_bibtex":{"value":"@misc{\ngu2024mamba,\ntitle={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},\nauthor={Albert Gu and Tri Dao},\nyear={2024},\nurl={https://openreview.net/forum?id=AL1fq05o7H}\n}"},"title":{"value":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces"},"pdf":{"value":"/pdf/27144ad57139872b668e5c522eeed631b23f5dce.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"gu|mamba_lineartime_sequence_modeling_with_selective_state_spaces"},"authorids":{"value":["~Albert_Gu1","~Tri_Dao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Albert Gu","Tri Dao"]}},"version":2},{"content":{"venue":{"value":"AAAI 2025"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/33098/35253"},"venueid":{"value":"dblp.org/conf/AAAI/2025"},"paperhash":{"value":"zhang|point_cloud_mamba_point_cloud_learning_via_state_space_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Tao_Zhang_0042:","~Haobo_Yuan1","https://dblp.org/search/pid/api?q=author:Lu_Qi:","https://dblp.org/search/pid/api?q=author:Jiangning_Zhang:","~Qianyu_Zhou1","https://dblp.org/search/pid/api?q=author:Shunping_Ji:","https://dblp.org/search/pid/api?q=author:Shuicheng_Yan:","~Xiangtai_Li1"]},"html":{"value":"https://doi.org/10.1609/aaai.v39i10.33098"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/ZhangYQZ0JYL25,\n  author={Tao Zhang and Haobo Yuan and Lu Qi and Jiangning Zhang and Qianyu Zhou and Shunping Ji and Shuicheng Yan and Xiangtai Li},\n  title={Point Cloud Mamba: Point Cloud Learning via State Space Model},\n  year={2025},\n  cdate={1735689600000},\n  pages={10121-10130},\n  url={https://doi.org/10.1609/aaai.v39i10.33098},\n  booktitle={AAAI},\n  crossref={conf/aaai/2025}\n}\n"},"abstract":{"value":"Recently, state space models have exhibited strong global modeling capabilities and linear computational complexity in contrast to transformers. This research focuses on applying such architecture to more efficiently and effectively model point cloud data globally with linear computational complexity. In particular, for the first time, we demonstrate that Mamba-based point cloud methods can outperform previous methods based on transformer or multi-layer perceptrons (MLPs). To enable Mamba to process 3-D point cloud data more effectively, we propose a novel Consistent Traverse Serialization method to convert point clouds into 1-D point sequences while ensuring that neighboring points in the sequence are also spatially adjacent. Consistent Traverse Serialization yields six variants by permuting the order of x, y, and z coordinates, and the synergistic use of these variants aids Mamba in comprehensively observing point cloud data. Furthermore, to assist Mamba in handling point sequences with different orders more effectively, we introduce point prompts to inform Mamba of the sequence’s arrangement rules. Finally, we propose positional encoding based on spatial coordinate mapping to inject positional information into point cloud sequences more effectively. Point Cloud Mamba surpasses the state-of-the-art (SOTA) point-based method PointNeXt and achieves new SOTA performance on the ScanObjectNN, ModelNet40, ShapeNetPart, and S3DIS datasets. It is worth mentioning that when using a more powerful local feature extraction module, our PCM achieves 79.6 mIoU on S3DIS, significantly surpassing the previous SOTA models, DeLA and PTv3, by 5.5 mIoU and 4.9 mIoU, respectively."},"title":{"value":"Point Cloud Mamba: Point Cloud Learning via State Space Model"},"authors":{"value":["Tao Zhang","Haobo Yuan","Lu Qi","Jiangning Zhang","Qianyu Zhou","Shunping Ji","Shuicheng Yan","Xiangtai Li"]}},"tmdate":1747732914776,"pdate":1735689600000,"tcdate":1747129113080,"writers":["~"],"signatures":["~Haobo_Yuan1"],"forum":"XBvpMIP6PH","license":"CC BY-SA 4.0","number":443968,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747732914776,"domain":"DBLP.org","id":"XBvpMIP6PH","version":2},{"content":{"venue":{"value":"IEEE Transactions on Multimedia"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/6046/4456689/11415389.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"li|msfmamba_motionaware_state_fusion_mamba_for_efficient_microgesture_recognition"},"html":{"value":"https://doi.org/10.1109/TMM.2026.3668511"},"abstract":{"value":"Micro-gesture recognition (MGR) targets the identification of subtle and fine-grained human motions and requires accurate modeling of both long-range and local spatiotemporal dependencies. While convolutional neural networks (CNNs) are effective at capturing local patterns, they struggle with long-range dependencies due to their limited receptive fields. Transformer-based models address this limitation through self-attention mechanisms but suffer from high computational costs. Recently, Mamba has shown promise as an efficient model, leveraging state space models (SSMs) to enable linear-time processing. However, directly applying the vanilla Mamba to MGR may not be optimal. This is because Mamba processes inputs as 1D sequences, with state updates relying solely on the previous state, and thus lacks the ability to model local spatiotemporal dependencies. In addition, previous methods lack a design of motion-awareness, which is crucial in MGR. To overcome these limitations, we propose motion-aware state fusion mamba (MSF-Mamba), which enhances Mamba with local spatiotemporal modeling by fusing local contextual neighboring states. Our design introduces a motion-aware state fusion module based on central frame difference (CFD). Furthermore, a multiscale version named MSF-Mamba$^{+}$ has been proposed. Specifically, MSF-Mamba$^{+}$ supports multiscale motion-aware state fusion, as well as an adaptive scale weighting module that dynamically weighs the fused states across different scales. These enhancements explicitly address the limitations of vanilla Mamba by enabling motion-aware local spatiotemporal modeling, allowing MSF-Mamba and MSF-Mamba$^{+}$ to effectively capture subtle motion cues for MGR. Experiments on two public MGR datasets (i.e., SMG and iMiGUE) demonstrate that even the lightweight version, namely, MSF-Mamba, achieves state-of-the-art performance, outperforming existing CNN-, Transformer-, and SSM-based models while maintaining high efficiency. For example, MSF-Mamba improves Top-1 accuracy by +2.2% and +1.5% over VideoMamba on SMG and iMiGUE, respectively. MSF-Mamba$^{+}$ outperforms VideoMamba on SMG and iMiGUE, achieving Top-1 accuracy improvements of 2.9% and 3.0%, respectively. The code will be accessible on https://github.com/Leedeng/MSF-Mamba"},"title":{"value":"MSF-Mamba: Motion-aware State Fusion Mamba for Efficient Micro-Gesture Recognition"},"authors":{"value":[{"fullname":"Deng Li"},{"fullname":"Jun Shao"},{"fullname":"Bohao Xing"},{"fullname":"Rong Gao"},{"fullname":"Bihan Wen"},{"fullname":"Heikki Kälviäinen","username":"~Heikki_Kälviäinen2"},{"fullname":"Xin Liu"}]}},"tmdate":1789089822401,"pdate":1767225600000,"externalIds":["doi:10.1109/tmm.2026.3668511"],"tcdate":1775023951880,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Heikki_Kälviäinen2"],"forum":"uLZRbSC8cL","license":"CC BY-SA 4.0","number":60631,"cdate":1772225882693,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/-/Edit"],"mdate":1789089822401,"domain":"OpenReview.net/Public_Article","id":"uLZRbSC8cL","version":2},{"content":{"summary":{"value":"Latests long-context recurrent models such as Mamba2 have demonstrated performance similar to transformer models, even in large scale scenarios, while being more efficient (no quadratic complexity of attention).\nHowever, they need to be retrained from scratch, which could slow down the adoption of their architecture—especially with so many pre-trained transformer models already available.\nMoreover, for non-text modalities such as speech, large scale Mamba 2 models are simply not available yet.\nHence authors propose leveraging off-the-shelf pre-trained (transformer) models and converting them into the target (mamba 2) model with linear complexity.\nMore specifically, they explore the possibility of converting an existing pre-trained transformer into a linear-complexity model for a specific downstream task; approach proposed is called Cross-Architecture Layerwise Distillation (CALD) and combines parameter transfer and distillation.\nMultiple scenarios are explored, including converting RoBERTa to Linformer for NLP tasks, Pythia to Mamba for language modeling, and Wav2Vec2 to Mamba2 for speech tasks; converted models show minimal to no performance loss compared to standard transformers while being of linear (instead of quadratic) complexity."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"Q: the conversion/distillation is made for a particular downstream task, would it be possible to make it for the pre-trained - general purpose - model ?\n\nQ: can we further speed-up inference through quantization (with added benefits)?"},"rating":{"value":8},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":4},"strengths":{"value":"-new approach to address model conversion/distillation from transformer (quadratic) architectures to mamba 2 (linear) architectures\n\n-not only applied to written language but also to speech with convincing results (3 types of tasks overall: NPL, LM and Speech)"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"-the obtained model is no longer a general purpose model (converted for a specific downstream task)\n\n-some inference speed evaluations would have been a plus (we know that obtained models with linear complexity should be faster though)"}},"nonreaders":[],"tmdate":1731427593370,"tcdate":1729698124437,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2415/Reviewer_VgsV"],"signatures":["ICLR.cc/2025/Conference/Submission2415/Reviewer_VgsV"],"forum":"90Db4RUBc7","number":1,"license":"CC BY 4.0","cdate":1729698124437,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2415/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427593370,"domain":"ICLR.cc/2025/Conference","replyto":"90Db4RUBc7","id":"4Fyn58M9X9","forumContent":{"TLDR":{"value":"Fine-tuning and converting pretrained speech and language transformer models into task-specific efficient/linear-complexity models without re-pretraining"},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["pretrained models","efficient attention","uptraining","speech processing"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Architectures such as Linformer and Mamba have recently emerged as competitive linear time replacements for transformers. However, corresponding large pretrained models are often unavailable, especially in non-text domains. To remedy this, we present a Cross-Architecture Layerwise Distillation (CALD) approach that jointly converts a transformer model to a linear time substitute and fine-tunes it to a target task. We also compare several means to guide the fine-tuning to optimally retain the desired inference capability from the original model. The methods differ in their use of the target model and the trajectory of the parameters. In a series of empirical studies on language processing, language modeling, and speech processing, we show that CALD can effectively recover the result of the original model, and that the guiding strategy contributes to the result. Some reasons for the variation are suggested."},"_bibtex":{"value":"@inproceedings{\nhe2025joint,\ntitle={Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity},\nauthor={Mutian He and Philip N. Garner},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=90Db4RUBc7}\n}"},"title":{"value":"Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity"},"pdf":{"value":"/pdf/a9cc8edf71023451b13c38f586f7e6cc4ad8d7cf.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"he|joint_finetuning_and_conversion_of_pretrained_speech_and_language_models_towards_linear_complexity"},"authorids":{"value":["~Mutian_He1","~Philip_N._Garner1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Mutian He","Philip N. Garner"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their feedback, and particularly regarding the novelty and motivation of the framework.\n \n---\n \n> If the A, B decomposition is, in the authors' own words, \"simply a tidy reparameterization,\" then the paper's theoretical contribution has to stand on the structured patterns and graph analysis alone\n \nThe framework's simplicity does not take away its novelty: while we agree similar forms have appeared in the SSM and linear recurrence literature, we have not seen any previous work come up with an expression as concise, clean and interpretable.\n \nMoreover, as reviewer MDcW pointed out, our framework is the first to include *softmax* attention naturally. Linear attention methods only support it by resorting to an infinite-dimensional kernel, which is not possible to use in practice. This is a fundamental difference with previous works.\n \n \n---\n \n> The fact that the framework is \"only truly meaningful\" for attention and SSMs is a problem, because the higher-order recurrence direction ends up without a solid connection to architectures people actually use.\n \nWe targeted attention and linear recurrences precisely because these are architectures people actually use in LLMs. On one side we have attention, along with its variants such as local attention, dilated attention, BigBird, and Longformer. On the other side stand linear recurrences such as SSMs (e.g. S4, Mamba, Mamba-2) and linear attention variants (e.g. linear attention, RetNet, GLA).\n \nOur goal is to interpolate between both behaviors by playing with the pattern of the recurrence: order 1 recovers linear recurrences (linear complexity), while infinite order is comparable to attention (quadratic complexity). We acknowledge that the framework has not been thought to encompass other families of token-mixing layers.\n \n---\n \n> I also still do not see enough evidence that shortest path and congestion correspond to expressivity in any meaningful sense beyond the synthetic tasks, which were more or less designed to match these definitions.\n \nWe agree that performance on the synthetic tasks does not necessarily translate to performance on real language modeling tasks. Still, such tasks are regularly used in the literature to validate architectural innovations and theoretical insights. For instance, the performance discrepancy between SSMs and attention on the associative recall task has motivated the Mamba architecture, which introduced a selective gating mechanism to target this bottleneck [1, 2]. Similarly, multi-hop recall has been extensively studied to understand the limitations of current architectures [3, 4]. In particular, it has been linked to entity tracking [4] and reasoning [5], which are key capabilities for LLMs.\n \n---\n \nWe will make sure the framework's novelty and differences to the literature are stated more clearly in the final version of the paper, and hope this answers the reviewer's concerns.\n \n---\n \n[1] Gu, Albert and Tri Dao. *\"Mamba: Linear-Time Sequence Modeling with Selective State Spaces.\"* ArXiv abs/2312.00752 (2023).\n \n[2] Dao, Tri and Gu, Albert. *\"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality\"*. ICML (2024).\n \n[3] Wang et al. *\"Learning Compositional Functions with Transformers from Easy-to-Hard Data\"*. PMLR (2025).\n \n[4] Fagnou et al. *\"Chain and Causal Attention for Efficient Entity Tracking\"*. EMNLP (2024).\n \n[5] Mavi et al. *\"Multi-hop Question Answering\"*. Foundations and Trends in Information Retrieval (2024)."}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782370255983,"tcdate":1775453696624,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission34554/Authors"],"signatures":["ICML.cc/2026/Conference/Submission34554/Authors"],"forum":"mcJwXaOYrL","number":1,"license":"CC BY 4.0","cdate":1775453696624,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission34554/Official_Review1/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782370255983,"domain":"ICML.cc/2026/Conference","replyto":"U99MxQRyUq","id":"qL9ESTVc5R","forumContent":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["token mixing","linear recurrence"]},"_bibtex":{"value":"@inproceedings{\nfagnou2026trading,\ntitle={Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing},\nauthor={Erwan Fagnou and Paul Caillon and Blaise Delattre and Alexandre Allauzen},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=mcJwXaOYrL}\n}"},"title":{"value":"Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing"},"paperhash":{"value":"fagnou|trading_complexity_for_expressivity_through_structured_generalized_linear_token_mixing"},"originally_submitted_PDF":{"value":"/pdf/a37b6d5f4627232d78fb3a1eb2cb0d622c4c1b67.pdf"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Token mixing layers play a key role in how language models can learn\n  and generate long-range dependencies. Their efficiency relies on the\n  necessary trade-off between decoding speed and the memory\n  requirements, along with the cache size. Considering causal generation, this paper explores new trade-offs\n  thanks to a  unified framework which separates two crucial\n  features: (i) the direct influence of inputs on\n  outputs in one generation step; (ii) the recurrent propagation of\n  information through past outputs.\n  \n  This framework encompasses major architectures such as attention and\n  state-space models, but also generalizes the recurrence equations by\n  allowing each state to depend on multiple past states rather than\n  only the immediate predecessor. By introducing structure, we design\n  new recurrence patterns that provably achieve the desired\n  complexity, while providing theoretical insights on their\n  expressivity -- trading runtime for expressivity in a principled\n  way. Empirical validation is performed on synthetic tasks, along\n  with language modeling.  Together, these results provide a\n  unified toolkit for the understanding and design of efficient and\n  expressive token mixers across model families."},"pdf":{"value":"/pdf/5eaaac55fe438feef819e6f4d04d4b6d276b0850.pdf"},"lay_summary":{"value":"Modern AI systems process text by combining information from many previous words. Current methods face an important trade-off: some are powerful but computationally expensive, while others are faster but struggle to remember long-range information. This limitation becomes especially important as language models are asked to handle increasingly long documents and conversations.\n\nIn this work, we introduce a general framework that helps explain and compare different ways AI models mix information across text. Our framework shows that many existing architectures can be understood through the same underlying principles, while also enabling new designs that balance efficiency and expressiveness in different ways.\n\nUsing this framework, we develop several new token-mixing strategies that reduce computational costs while preserving the ability to capture long-range dependencies. We also provide theoretical tools to analyze how well these methods can transmit and store information over time.\n\nExperiments on synthetic memory tasks and language modeling demonstrate that carefully structured models can achieve much of the performance of expensive attention mechanisms at a fraction of the computational cost. These results offer new directions for designing faster and more scalable language models."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Erwan_Fagnou1","~Paul_Caillon1","~Blaise_Delattre1","~Alexandre_Allauzen1"]},"authors":{"value":["Erwan Fagnou","Paul Caillon","Blaise Delattre","Alexandre Allauzen"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.18135v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"sun|r2genmamba_a_selective_state_space_model_for_radiology_report_generation"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Yongheng_Sun:","https://dblp.org/search/pid/api?q=author:Yueh_Z._Lee:","https://dblp.org/search/pid/api?q=author:Genevieve_A._Woodard:","https://dblp.org/search/pid/api?q=author:Hongtu_Zhu:","https://dblp.org/search/pid/api?q=author:Chunfeng_Lian:","~Mingxia_Liu1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.18135"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-18135,\n  publtype={informal},\n  author={Yongheng Sun and Yueh Z. Lee and Genevieve A. Woodard and Hongtu Zhu and Chunfeng Lian and Mingxia Liu},\n  title={R2Gen-Mamba: A Selective State Space Model for Radiology Report Generation},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.18135},\n  url={https://doi.org/10.48550/arXiv.2410.18135}\n}\n"},"abstract":{"value":"Radiology report generation is crucial in medical imaging,but the manual annotation process by physicians is time-consuming and labor-intensive, necessitating the develop-ment of automatic report generation methods. Existingresearch predominantly utilizes Transformers to generateradiology reports, which can be computationally intensive,limiting their use in real applications. In this work, we presentR2Gen-Mamba, a novel automatic radiology report genera-tion method that leverages the efficient sequence processingof the Mamba with the contextual benefits of Transformerarchitectures. Due to lower computational complexity ofMamba, R2Gen-Mamba not only enhances training and in-ference efficiency but also produces high-quality reports.Experimental results on two benchmark datasets with morethan 210,000 X-ray image-report pairs demonstrate the ef-fectiveness of R2Gen-Mamba regarding report quality andcomputational efficiency compared with several state-of-the-art methods. The source code can be accessed online."},"title":{"value":"R2Gen-Mamba: A Selective State Space Model for Radiology Report Generation"},"authors":{"value":["Yongheng Sun","Yueh Z. Lee","Genevieve A. Woodard","Hongtu Zhu","Chunfeng Lian","Mingxia Liu"]}},"tmdate":1754405441053,"pdate":1704067200000,"externalIds":["dblp:journals/corr/abs-2410-18135"],"tcdate":1754405420206,"writers":["~"],"signatures":["~Mingxia_Liu1"],"forum":"NjOG1aUVLE","license":"CC BY-SA 4.0","number":614945,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1754405441053,"domain":"DBLP.org","id":"NjOG1aUVLE","version":2},{"content":{"comment":{"value":"## Responses to weaknesses:\n\n***Weakness 1**: First, the papers' main methods are both not new but borrow from other domains without many new contributions/modifications. For example, the low-rank compression with tensor-train is not new. Also the sparse-grid method is also a typical method in the PDE domain.*\n\n**Response**: Thanks you for your concerns on our work’s contributions. \n\n- We completely agree that sparse-grid method and tensor-train decomposition methods have already been reported in the math community. Both of them are widely used in high-dimensional function approximation [R1] and uncertainty quantification [R2] of PDE-described problems. However, neither of them have been used in BP-free training.\n- Our algorithmic contributions are two fold: (1). we apply sparse-grid to achieve BP-free loss evaluation in PINN, (2). we apply tensor-train to perform variance-reduction in zeroth-order (ZO) training. These two combined allows us to train a real-size PINN on photonic hardware for the first time, advancing the state-of-the-art in photonic neural network training.\n- With full respect to the reviewer, we would like to remark that many novel ideas in engineering communities (including machine learning) come from the application of existing theories developed in the math community. This kind of research is regarded as novel and often produce high-impact research results. For instance, Neural ODE [R3] uses the well-known ODE model to replace ResNet and makes the model infinitely deep; the recent MAMBA [R4] applies state-space model in dynamic system to solve long-sequence modeling problems and achieved state-of-the-art performance in NLP.\n\n[R1] Griebel, Michael. *Sparse grids and related approximation schemes for higher dimensional problems*. SFB 611, 2005.\n\n[R2] Bigoni, Daniele, Allan P. Engsig-Karup, and Youssef M. Marzouk. \"Spectral tensor-train decomposition.\" SIAM Journal on Scientific Computing 38.4 (2016): A2405-A2439.\n\n[R3] Chen, Ricky TQ, et al. \"Neural ordinary differential equations.\" *Advances in neural information processing systems* 31 (2018).\n\n[R4] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" *arXiv preprint arXiv:2312.00752* (2023)."},"title":{"value":"Response to Reviewer kjo5 (Part I)"}},"tmdate":1732128566578,"tcdate":1732128566578,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13290/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13290/Authors"],"forum":"XLDaepymR5","number":3,"license":"CC BY 4.0","cdate":1732128566578,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13290/-/Official_Comment"],"mdate":1732128566578,"domain":"ICLR.cc/2025/Conference","replyto":"DVbLCKUDgl","id":"A2DaMHelmW","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Optical Neural Networks","Physics-Informed Neural Networks","On-Chip Learning","Scalability","Hardware-Software Co-Design"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Physics-informed neural networks (PINNs) have shown promise in solving partial differential equations (PDEs), with growing interest in their energy-efficient, real-time training on edge devices. Photonic computing offers a potential solution due to its high operation speed.\nHowever, the lack of photonic memory and the large footprint of current photonic devices prevent training realistic-size PINNs on photonic chips. This paper proposes a completely back-propagation-free (BP-free) and highly salable framework to enable training real-size PINNs on silicon photonics platforms. Our approach involves three key innovations: (1) a sparse-grid Stein derivative estimator to avoid the BP in the loss evaluation of a PINN, (2) a dimension-reduced zeroth-order optimization via tensor-train decomposition to achieve better scalability and convergence in BP-free training, and (3) a scalable on-chip photonic PINN training accelerator design using photonic tensor cores. We validate the performance of our numerical methods in both low- and high-dimensional PDE benchmarks. Through circuit simulation based on real device parameters, we further demonstrate the significant performance benefit (e.g., real-time training, huge chip area reduction) of our photonic accelerator. Our framework addresses the fundamental challenges of photonic AI and will enable real-time training of real-size PINNs on photonic chips."},"_bibtex":{"value":"@misc{\nzhao2025scalable,\ntitle={Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networks},\nauthor={Yequan Zhao and Xinling Yu and Xian Xiao and Zhixiong Chen and Ziyue Liu and Geza Kurczveil and Raymond G Beausoleil and Sijia Liu and Zheng Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=XLDaepymR5}\n}"},"title":{"value":"Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networks"},"pdf":{"value":"/pdf/7df917a10dd58d5a7cc3effefaa8f017774ee981.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhao|scalable_backpropagationfree_training_of_optical_physicsinformed_neural_networks"},"authorids":{"value":["~Yequan_Zhao1","~Xinling_Yu1","~Xian_Xiao1","~Zhixiong_Chen2","~Ziyue_Liu1","~Geza_Kurczveil1","~Raymond_G_Beausoleil1","~Sijia_Liu1","~Zheng_Zhang2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yequan Zhao","Xinling Yu","Xian Xiao","Zhixiong Chen","Ziyue Liu","Geza Kurczveil","Raymond G Beausoleil","Sijia Liu","Zheng Zhang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2409.04356v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"wang|serpmamba_advancing_highresolution_retinal_vessel_segmentation_with_selective_statespace_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Hongqiu_Wang:","https://dblp.org/search/pid/api?q=author:Yixian_Chen:","https://dblp.org/search/pid/api?q=author:Wu_Chen:","https://dblp.org/search/pid/api?q=author:Huihui_Xu:","https://dblp.org/search/pid/api?q=author:Haoyu_Zhao:","https://dblp.org/search/pid/api?q=author:Bin_Sheng_0001:","https://dblp.org/search/pid/api?q=author:Huazhu_Fu:","~Guang_Yang1","https://dblp.org/search/pid/api?q=author:Lei_Zhu_0003:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2409.04356"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2409-04356,\n  publtype={informal},\n  author={Hongqiu Wang and Yixian Chen and Wu Chen and Huihui Xu and Haoyu Zhao and Bin Sheng and Huazhu Fu and Guang Yang and Lei Zhu},\n  title={Serp-Mamba: Advancing High-Resolution Retinal Vessel Segmentation with Selective State-Space Model},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2409.04356},\n  url={https://doi.org/10.48550/arXiv.2409.04356}\n}\n"},"abstract":{"value":"Ultra-Wide-Field Scanning Laser Ophthalmoscopy (UWF-SLO) images capture high-resolution views of the retina with typically 200 spanning degrees. Accurate segmentation of vessels in UWF-SLO images is essential for detecting and diagnosing fundus disease. Recent studies have revealed that the selective State Space Model (SSM) in Mamba performs well in modeling long-range dependencies, which is crucial for capturing the continuity of elongated vessel structures. Inspired by this, we propose the first Serpentine Mamba (Serp-Mamba) network to address this challenging task. Specifically, we recognize the intricate, varied, and delicate nature of the tubular structure of vessels. Furthermore, the high-resolution of UWF-SLO images exacerbates the imbalance between the vessel and background categories. Based on the above observations, we first devise a Serpentine Interwoven Adaptive (SIA) scan mechanism, which scans UWF-SLO images along curved vessel structures in a snake-like crawling manner. This approach, consistent with vascular texture transformations, ensures the effective and continuous capture of curved vascular structure features. Second, we propose an Ambiguity-Driven Dual Recalibration (ADDR) module to address the category imbalance problem intensified by high-resolution images. Our ADDR module delineates pixels by two learnable thresholds and refines ambiguous pixels through a dual-driven strategy, thereby accurately distinguishing vessels and background regions. Experiment results on three datasets demonstrate the superior performance of our Serp-Mamba on high-resolution vessel segmentation. We also conduct a series of ablation studies to verify the impact of our designs. Our code shall be released upon publication of this work."},"title":{"value":"Serp-Mamba: Advancing High-Resolution Retinal Vessel Segmentation with Selective State-Space Model"},"authors":{"value":["Hongqiu Wang","Yixian Chen","Wu Chen","Huihui Xu","Haoyu Zhao","Bin Sheng","Huazhu Fu","Guang Yang","Lei Zhu"]}},"tmdate":1731493236695,"pdate":1704067200000,"tcdate":1731490221571,"writers":["~"],"signatures":["~Guang_Yang1"],"forum":"9OGlEkQZAd","license":"CC BY-SA 4.0","number":220962,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1731493236695,"domain":"DBLP.org","id":"9OGlEkQZAd","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2402.19047v4"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"cirone|theoretical_foundations_of_deep_selective_statespace_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Nicola_Muca_Cirone:","~Antonio_Orvieto3","~Benjamin_Walker1","https://dblp.org/search/pid/api?q=author:Cristopher_Salvi:","https://dblp.org/search/pid/api?q=author:Terry_J._Lyons:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2402.19047"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2402-19047,\n  publtype={informal},\n  author={Nicola Muca Cirone and Antonio Orvieto and Benjamin Walker and Cristopher Salvi and Terry J. Lyons},\n  title={Theoretical Foundations of Deep Selective State-Space Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2402.19047},\n  url={https://doi.org/10.48550/arXiv.2402.19047}\n}\n"},"abstract":{"value":"Structured state-space models (SSMs) such as S4, stemming from the seminal work of Gu et al., are gaining popularity as effective approaches for modeling sequential data. Deep SSMs demonstrate outstanding performance across a diverse set of domains, at a reduced training and inference cost compared to attention-based transformers. Recent developments show that if the linear recurrence powering SSMs allows for multiplicative interactions between inputs and hidden states (e.g. GateLoop, Mamba, GLA), then the resulting architecture can surpass in both in accuracy and efficiency attention-powered foundation models trained on text, at scales of billion parameters. In this paper, we give theoretical grounding to this recent finding using tools from Rough Path Theory: we show that when random linear recurrences are equipped with simple input-controlled transitions (selectivity mechanism), then the hidden state is provably a low-dimensional projection of a powerful mathematical object called the signature of the input -- capturing non-linear interactions between tokens at distinct timescales. Our theory not only motivates the success of modern selective state-space models such as Mamba but also provides a solid framework to understand the expressive power of future SSM variants."},"title":{"value":"Theoretical Foundations of Deep Selective State-Space Models"},"authors":{"value":["Nicola Muca Cirone","Antonio Orvieto","Benjamin Walker","Cristopher Salvi","Terry J. Lyons"]}},"tmdate":1754034342793,"pdate":1704067200000,"tcdate":1737827337917,"writers":["~"],"signatures":["~Benjamin_Walker1"],"forum":"fN3YnZAOQC","license":"CC BY-SA 4.0","number":277828,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1754034342793,"domain":"DBLP.org","id":"fN3YnZAOQC","version":2},{"content":{"comment":{"value":"We appreciate your recognition of our work and your positive feedback. We are also grateful for your rating, which leans towards accepting our submission. We provide responses to your follow-up questions below.\n\n> In Theorem 4.1, epsilon is still not defined in the main text\n\nThanks for pointing this out. We have updated our manuscript and included the notation in the main body of the paper.\n\n\n\n> Can you please discuss how realistic is this assumption. In the real case, A would be a matrix containing model parameters so we cannot assert that it would be so nicely behaved. \n\n\nMamba models utilize matrices A, B, C, and D, as defined in the original papers [15, 16]. Notably, we observe that HiPPO [16] exhibits similar characteristics to the exponential series, as illustrated in Figure 5 of [16].\n\nTo address the reviewer's concern, we provide an empirical error analysis in Section A.2, where we instantiate the A and B matrices using the HiPPO-LegT and HiPPO-LegS formulations described in [16]. As shown in Figure 5, Section A.2, the errors remain bounded within the context of a discretized high-dimensional SSM.\n\n[15] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n\n[16] Gu, Albert, et al. \"Hippo: Recurrent memory with optimal polynomial projections.\" Advances in neural information processing systems 33 (2020): 1474-1487."}},"tmdate":1732250199603,"tcdate":1732249997713,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11122/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission11122/Authors"],"forum":"mnna9LUg7P","number":13,"license":"CC BY 4.0","cdate":1732249997713,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11122/-/Official_Comment"],"mdate":1732250199603,"domain":"ICLR.cc/2025/Conference","replyto":"xcxBvshcwv","id":"RBcNR7MGBu","forumContent":{"TLDR":{"value":"We propose a 8-bit static quantization method for selective state space models."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["State Space Models","Model quantization"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSMs) have emerged as an appealing alternative to Transformers for large language models, achieving state-of-the-art accuracy with constant memory complexity which allows for holding longer context lengths than attention-based networks. The superior computational efficiency of SSMs in long sequence modeling positions them favorably over Transformers in many scenarios. However, improving the efficiency of SSMs on request-intensive cloud-serving and resource-limited edge applications is still a formidable task. SSM quantization is a possible solution to this problem, making SSMs more suitable for wide deployment, while still maintaining their accuracy. Quantization is a common technique to reduce the model size and to utilize the low bit-width acceleration features on modern computing units, yet existing quantization techniques are poorly suited for SSMs. Most notably, SSMs have highly sensitive feature maps within the selective scan mechanism (i.e., linear recurrence) and massive outliers in the output activations which are not present in the output of token-mixing in the self-attention modules. To address this issue, we propose a static 8-bit per-tensor SSM quantization method which suppresses the maximum values of the input activations to the selective SSM for finer quantization precision and quantizes the output activations in an outlier-free space with Hadamard transform. Our 8-bit weight-activation quantized Mamba 2.8B SSM benefits from hardware acceleration and achieves a 1.72 $\\times$ lower generation latency on an Nvidia Orin Nano 8G, with only a 0.9\\% drop in average accuracy on zero-shot tasks. When quantizing Jamba, a 52B parameter SSM-style language model, we observe only a $1\\%$  drop in accuracy, demonstrating that our SSM quantization method is both effective and scalable for large language models, which require appropriate compression techniques for deployment. The experiments demonstrate the effectiveness and practical applicability of our approach for deploying SSM-based models of all sizes on both cloud and edge platforms."},"_bibtex":{"value":"@inproceedings{\nchiang2025quamba,\ntitle={Quamba: A Post-Training Quantization Recipe for Selective State Space Models},\nauthor={Hung-Yueh Chiang and Chi-Chih Chang and Natalia Frumkin and Kai-Chiang Wu and Diana Marculescu},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=mnna9LUg7P}\n}"},"title":{"value":"Quamba: A Post-Training Quantization Recipe for Selective State Space Models"},"pdf":{"value":"/pdf/0c4ea9cb96dd9e16a33f3931fb1a1347d00b5817.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"chiang|quamba_a_posttraining_quantization_recipe_for_selective_state_space_models"},"authorids":{"value":["~Hung-Yueh_Chiang1","~Chi-Chih_Chang1","~Natalia_Frumkin1","~Kai-Chiang_Wu1","~Diana_Marculescu4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hung-Yueh Chiang","Chi-Chih Chang","Natalia Frumkin","Kai-Chiang Wu","Diana Marculescu"]}},"version":2},{"content":{"summary":{"value":"The paper presents a novel approach to TCP using a Counterfactual Mamba model with Covariate-based Decorrelation towards Selective Parameters (Mamba-CDSP). It addresses the limitations of current methods, particularly those based on LSTM and Transformer architectures, in capturing long-sequence interactions effectively and efficiently. The proposed Mamba-CDSP aims to mitigate confounding bias by decorrelating current treatments from historical covariates and outcomes. The authors provide empirical experiment over both synthetic and real-world datasets, demonstrating that Mamba-CDSP significantly improves prediction performance and running efficiency compared to existing methods."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"See weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The motivation of this paper is well claimed.\n2. The writing of this paper is good and easy to follow. The preliminaries part is particularly clear.\n3. The experiment part is comprehensive, consisting of synthetic and real-world datasets."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The novelty and technical contribution of this paper are not enough. This paper claims to address the deconfounding and over-balance issues in the TCP task, but these issues are not first proposed by this paper. Below, I will further explain my opinion.\n2. The authors claim that they are the first to consider the step-by-step deconfounding in the TCP task. However, to my knowledge, there are several existing works that achieve a similar goal, such as [1] and [2]. The authors should illustrate the difference between them.\n3. Why would the linear correlation between $a_t$ and $h_{t-1}$ lead to the over-balancing issue while the non-linear correlation would not, as you remark in line 292? Can you provide a rational analysis regarding this method?\n4. This method seems to simply replace the transformer with the Mamba model by slightly tailoring the backbone, which lacks technical contribution.\n5. The sizes of the datasets used in the experiments are quite small; for example, there are only 5,000 patients in the MIMIC-III real-world dataset, which cannot adequately reveal the efficiency of replacing the Transformer with the Mamba model.\n\n\n\n[1] Estimating Counterfactual Treatment Outcomes over Time through Adversarially Balanced Representations. ICLR 2020.\n\n[2] Counterfactual Generative Models for Time-Varying Treatments. KDD 2024."}},"nonreaders":[],"tmdate":1733204494931,"tcdate":1730468640347,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission7314/Reviewer_1NGR"],"signatures":["ICLR.cc/2025/Conference/Submission7314/Reviewer_1NGR"],"forum":"yheQRc5xWB","number":2,"license":"CC BY 4.0","cdate":1730468640347,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission7314/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733204494931,"domain":"ICLR.cc/2025/Conference","replyto":"yheQRc5xWB","id":"yNggOWj2Uq","forumContent":{"TLDR":{"value":"This paper presents a novel method for counterfactual prediction over time series with state-space models."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series; State-space Models; Treatment Effect Estimation"]},"primary_area":{"value":"causal reasoning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Time-varying counterfactual prediction (TCP) from observational data supports the answer of when and how to assign multiple sequential treatments, yielding importance in various applications. Despite the progress achieved by recent advances, e.g., LSTM or Transformer based causal approaches, their capability of capturing interactions in long sequences remains to be improved in both prediction performance and running efficiency. In parallel with the development of TCP, the success of the state-space models (SSMs) has achieved remarkable progress toward long-sequence modeling with saved running time. Consequently, studying how Mamba simultaneously benefits the effectiveness and efficiency of TCP  becomes a compelling research direction. In this paper, we propose to exploit advantages of the SSMs to tackle the TCP task, by introducing a counterfactual Mamba model with Covariate-based Decorrelation towards Selective Parameters (Mamba-CDSP). Motivated by the over-balancing problem in TCP of the direct covariate balancing methods, we propose to de-correlate between the current treatment and the representation of historical covariates, treatments, and outcomes, which can mitigate the confounding bias while preserve more covariate information. In addition, we show that the overall de-correlation in TCP is equivalent to regularizing the selective parameters of Mamba over each time step, which leads our approach to be effective and lightweight. We conducted extensive experiments on both synthetic and real-world datasets, demonstrating that Mamba-CDSP not only outperforms baselines by a large margin, but also exhibits prominent running efficiency."},"_bibtex":{"value":"@inproceedings{\nwang2025effective,\ntitle={Effective and Efficient Time-Varying Counterfactual Prediction with State-Space Models},\nauthor={Haotian Wang and Haoxuan Li and Hao Zou and Haoang Chi and Long Lan and Wanrong Huang and Wenjing Yang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=yheQRc5xWB}\n}"},"title":{"value":"Effective and Efficient Time-Varying Counterfactual Prediction with State-Space Models"},"pdf":{"value":"/pdf/6543649eb8bf53c6ca75c251a0032d3e63a00c35.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wang|effective_and_efficient_timevarying_counterfactual_prediction_with_statespace_models"},"authorids":{"value":["~Haotian_Wang2","~Haoxuan_Li6","~Hao_Zou1","~Haoang_Chi1","~Long_Lan2","~Wanrong_Huang1","~Wenjing_Yang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Haotian Wang","Haoxuan Li","Hao Zou","Haoang Chi","Long Lan","Wanrong Huang","Wenjing Yang"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a new method called MambaMatcher, which applies selective state space models (SSMs) to semantic matching tasks and combines Mamba's linear complexity algorithm to efficiently process 4D correlation tensors. Compared with existing methods, MambaMatcher has improved the accuracy of keypoint localization without sacrificing computational efficiency, demonstrating significant performance improvements."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. How does MambaMatcher balance the integration of feature-metric and correlation-metric methods? \n2. Are there scenarios where this integration could present limitations or potential drawbacks?\n3. Is there a significant difference in performance when processing images with different levels of complexity or size?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. This paper introduces the first use of selective state-space models (SSMs) to efficiently model high-dimensional inter-image correlations.\n2. This paper proposes a similarity-aware selective scan mechanism improves accuracy and efficiency in refining high-resolution correlations.\n3. MambaMatcher integrates feature-metric and correlation-metric methods, achieving state-of-the-art performance with lower computational costs."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The specific description of the Correlation Aggregation via Similarity aware Selective Scan section is not clear enough. \n2. There are some grammar errors in the paper ( e.g., Fig.4 Ground-truth correspondence and page 3, line 132).\n3. Some formulas have commas or periods after them, while others do not. There is no uniformity in format ( e.g., Eq.10 and Eq.11)."}},"nonreaders":[],"tmdate":1731428304838,"tcdate":1730081995747,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5072/Reviewer_aTzv"],"signatures":["ICLR.cc/2025/Conference/Submission5072/Reviewer_aTzv"],"forum":"GmE8ovvXaJ","number":1,"license":"CC BY 4.0","cdate":1730081995747,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5072/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428304838,"domain":"ICLR.cc/2025/Conference","replyto":"GmE8ovvXaJ","id":"CxeFttmTbF","forumContent":{"TLDR":{"value":"We introduce MambaMatcher, which formulates Mamba layers to process the correlation map between image pairs to establish accurate semantic correspondences with improved performance and efficiency."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["semantic correspondence","state-space models","correlation aggregation","Mamba","feature aggregation","Similarity-aware Selective Scan"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Establishing semantic correspondences between images is a fundamental yet challenging task in computer vision. Traditional feature-metric methods enhance visual features but may miss complex inter-image relationships, while recent correlation-metric approaches attempt to model these relationships but are hindered by high computational costs due to processing 4D correlation maps. We introduce MambaMatcher, a novel method that overcomes these limitations by efficiently modeling high-dimensional correlations using selective state-space models (SSMs), treating multi-level correlation scores as states. By implementing a similarity-aware selective scan mechanism adapted from Mamba’s linear-complexity algorithm, MambaMatcher refines the 4D correlation tensor effectively without compromising feature map resolution or receptive field. Experiments on standard semantic correspondence benchmarks demonstrate that MambaMatcher achieves state-of-the-art performance without relying on large input images or computationally expensive diffusion-based feature extractors, effectively capturing rich inter-image correlations while maintaining computational efficiency."},"_bibtex":{"value":"@misc{\nkim2025selective,\ntitle={Selective State-Space Modeling of Correlation Maps for Semantic Correspondence},\nauthor={Seungwook Kim and Minsu Cho},\nyear={2025},\nurl={https://openreview.net/forum?id=GmE8ovvXaJ}\n}"},"title":{"value":"Selective State-Space Modeling of Correlation Maps for Semantic Correspondence"},"pdf":{"value":"/pdf/904fad05fa77d7d08d5defc1a4f7abaade7973d6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"kim|selective_statespace_modeling_of_correlation_maps_for_semantic_correspondence"},"authorids":{"value":["~Seungwook_Kim2","~Minsu_Cho1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seungwook Kim","Minsu Cho"]}},"version":2},{"content":{"summary":{"value":"This paper addresses the look-back Window Limitation problem in Long-term Time Series Forecasting (LTSF). Current Transformer-based models suffer from the look-back window limitation, and as the look-back window increases, computational complexity grows and may include redundant information that interferes with learning. The paper proposes two modules: the Information-Bottleneck-Filter module, which improves performance by preventing information redundancy in LTSF, and the Hybrid-Transformer-Mamba (HTM) module, which enhances long-range modeling performance by applying Mamba to patched sequences. The model created by combining these two modules with the PatchTST model showed good performance in LTSF tasks with a long look-back window of 1024."},"soundness":{"value":1},"confidence":{"value":3},"questions":{"value":"See Weaknesses"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"* Demonstrates the necessity of models with longer look-back windows and shows the limitations of existing transformers.\n* Conducted experiments using input sequences of length up to 1024.\n* Combined the Mamba-Transformer hybrid with the information bottleneck approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The overall completeness is significantly lacking.\n  - There is no explanation for Figure 2 in both the caption and main text, and the caption was written carelessly.\n  - While Lookback Window Limitation is a key concept appearing from the title, there is no definition or explanation for it, and the cited papers do not address this concept either.\n  - Revise the artifact in Figure 3(a).\n  - Add explanations for each axis in the graphs (Figure 3,5), and when abbreviations are used, write out the full terms in the caption (Figure 6).\n  - Instead of mixing the terms Ettm and ETTm, standardize the usage to ETTm.\n\n* Since this paper's main component is the Mamba-Transformer hybrid, it should: 1. Include recently proposed Mamba-Transformer hybrid models in the related work section 2. Compare performance with these models in the main table. e.g. *SST: Multi-Scale Hybrid Mamba-Transformer Experts for Long-Short Range Time Series Forecasting Can Mamba Learn How to Learn?*, *A Comparative Study on In-Context Learning Tasks*\n\n* Please revise the disorganized notations and the inappropriate figures in the Methods section, and include sufficient explanations about the model components, even if in the appendix.\n\n* There is no explanation about how many seeds were used for repeated experiments. The outlier values raise suspicions that no repeated experiments were conducted at all, reducing the reliability of the results.\n\n* In Table 1, PIH's performance appears to largely come from PatchTST's excellence. Please provide a full table that compares basemodel and basemodel+IBF+HTM, which is briefly introduced in section 4.2.\n\n* Recent time series forecasting papers generally use more public datasets. Please add datasets such as ECL, Exchange, Solar-Energy, and PEMS.\n\n* Looking at Figure 5, your methodology doesn't appear to effectively filter redundant information or efficiently utilize longer inputs as the look-back period increases. More experiments on additional datasets and sufficient explanations of the results seem necessary.\n\n* Is your methodology also applicable to the iTransformer model?\n\nWhile this paper attempts to tackle an important problem in time-series forecasting and has some interesting aspects, in my opinion, it lacks distinctiveness compared to existing transformer-mamba hybrids, and the paper shows insufficient overall completeness and ability to demonstrate its results."}},"nonreaders":[],"tmdate":1731428335983,"tcdate":1730168659649,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6771/Reviewer_3NWY"],"signatures":["ICLR.cc/2025/Conference/Submission6771/Reviewer_3NWY"],"forum":"hVpAjJPfgZ","number":1,"license":"CC BY 4.0","cdate":1730168659649,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6771/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428335983,"domain":"ICLR.cc/2025/Conference","replyto":"hVpAjJPfgZ","id":"QQDg2QA5vD","forumContent":{"TLDR":{"value":"We adopts a Transformer-Mamba hybrid architecture and information bottleneck for extreme long lookback windows in LSTF."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["long-term time series forcasting","Mamba","Information Bottleneck"]},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Long-term time series forecasting (LTSF) aims to predict future trends based on historical data. While longer lookback windows theoretically provide more comprehensive insights, current Transformer-based models face the Lookback Window Limitation (LWL). On one hand, longer windows introduce redundant information, which can hinder model learning. On the other hand, Transformers tend to overfit temporal noise rather than extract meaningful temporal information when dealing with longer sequences, compounded by their quadratic complexity. In this paper, we aim to overcome LWL, enabling models to leverage more historical information for improved performance. Specifically, to mitigate information redundancy, we introduce the Information Bottleneck Filter (IBF), which applies information bottleneck theory to extract essential subsequences from the input. Additionally, to address the limitations of the Transformer architecture in handling long sequences, we propose the Hybrid-Transformer-Mamba (HTM), which combines the linear complexity and long-range modeling capabilities of Mamba with the Transformer's strength in modeling short sequences. We integrate these two model-agnostic modules into various existing methods and conduct experiments on seven datasets. The results demonstrate that incorporating these modules effectively overcomes the lookback window limitations. Notably, by combining them with the Patch strategy, we design the PIH (\\textbf{P}atch-\\textbf{I}BF-\\textbf{H}TM), successfully extending the window length to 1024—a significantly larger window than previously achieved—and achieving state-of-the-art results, highlighting the potential of exploring even longer windows."},"_bibtex":{"value":"@misc{\nzhang2024overcoming,\ntitle={Overcoming Lookback Window Limitations: Exploring Longer Windows in Long-Term Time Series Forecasting},\nauthor={Jiahui Zhang and Wenjie Du and Zhengyang Zhou and Qihe Huang and Xuqiang Li and Yang Wang},\nyear={2024},\nurl={https://openreview.net/forum?id=hVpAjJPfgZ}\n}"},"title":{"value":"Overcoming Lookback Window Limitations: Exploring Longer Windows in Long-Term Time Series Forecasting"},"pdf":{"value":"/pdf/d81ea206a0e4871a9de5d22719240efde5117469.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhang|overcoming_lookback_window_limitations_exploring_longer_windows_in_longterm_time_series_forecasting"},"authorids":{"value":["~Jiahui_Zhang7","~Wenjie_Du2","~Zhengyang_Zhou1","~Qihe_Huang2","~Xuqiang_Li1","~Yang_Wang32"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jiahui Zhang","Wenjie Du","Zhengyang Zhou","Qihe Huang","Xuqiang Li","Yang Wang"]}},"version":2},{"content":{"venue":{"value":"TIME 2000"},"pdf":{"value":"https://ieeexplore.ieee.org/iel5/6915/18603/00856586.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2000"},"paperhash":{"value":"combi|modeling_temporal_aspects_of_visual_and_textual_objects_in_multimedia_databases"},"authorids":{"value":["~Carlo_Combi1"]},"html":{"value":"https://doi.org/10.1109/TIME.2000.856586"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/Combi00,\n  author={Carlo Combi},\n  title={Modeling Temporal Aspects of Visual and Textual Objects in Multimedia Databases},\n  year={2000},\n  cdate={946684800000},\n  pages={59-68},\n  url={https://doi.org/10.1109/TIME.2000.856586},\n  booktitle={TIME},\n  crossref={conf/time/2000}\n}\n"},"abstract":{"value":"We consider the problem of modeling temporal aspects of visual and related textual entities in multimedia databases, within the framework of an object-oriented temporal data model. We distinguish two different time aspects in multimedia visual data: intrinsic time, i.e., the time managed according to the sequence of frames composing a visual data item, and extrinsic time, i.e., the (valid) time of the real world represented by the visual data item. We also propose a three-layered approach for composing temporal visual data. Relationships between visual data and related textual data are then considered: we propose a taxonomy for temporal textural observations on visual data, based on their temporal properties. In relation to textural temporal data and visual data, we consider the existing contraints along their valid times, possibly given different granularities and with indeterminacy."},"title":{"value":"Modeling Temporal Aspects of Visual and Textual Objects in Multimedia Databases"},"authors":{"value":["Carlo Combi"]}},"tmdate":1747652538459,"pdate":946684800000,"tcdate":1747652519239,"writers":["~"],"signatures":["~Carlo_Combi1"],"forum":"R2yjzycIXz","license":"CC BY-SA 4.0","number":526609,"cdate":946684800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747652538459,"domain":"DBLP.org","id":"R2yjzycIXz","version":2},{"content":{"summary":{"value":"This paper proposed a new network for image deraining by introducing the state space models, or more specifically, the recently proposed Mamba structure, in the Fourier spaces. By virtue of the Mamba, the frequency correlations can be better captured and utilized in the Fourier domain, such that the better deraining performance can be expected. Experiments on various deraining datasets have been conducted to demonstrate the effectiveness of the proposed method."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- In Section 4.1, the authors mentioned the \"progressive training strategy\". What does it mean? And why is it called \"progressive\"?\n\n- There are several writing issues that I can find:\n1) Line 75-76: \"...build correlation the correlation among...\";\n2) The rightmost of Line 73: \".arrangement\";\n3) According to the formatting of other parts of this manuscript, the \"Fourier transformation.\" in Section 3.1 should not take the whole line."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- The application of Mamba to the Fourier domain is new and worthy to explore.\n\n- The experimental results are promising"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The motivation should be strengthened. Currently, it seems the work is mostly about applying Mamba in the Fourier domain, though this task itself could be non-trivial. However, it is unclear why Mamba is more effective than other architectures for processing the Fourier frequencies. Visualizing the intermediate features might be useful.\n\n- As reviewed by the authors, another work (Zhen et al., 2024) considers Mamba in the frequency domain but with wavelet transformation. It is better to discuss more about wavelet transformation and Fourier transformation in dealing with frequencies. Besides, as shown in Table 1, the performance of (Zhen et al., 2024) is close to the proposed method on several datasets, but the comparison with this method on SPA-Data is missing.\n\n- The quantitative comparison is conducted with respect to PSNR and SSIM. However, as more and more researchers realized, these two metrics can be often inconsistent with human perceptions and thus some newly developed metrics that can better reflect human perceptions should be considered."}},"nonreaders":[],"tmdate":1733104292347,"tcdate":1730475090281,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8897/Reviewer_xRKZ"],"signatures":["ICLR.cc/2025/Conference/Submission8897/Reviewer_xRKZ"],"forum":"WNPrfGpcu6","number":2,"license":"CC BY 4.0","cdate":1730475090281,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8897/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733104292347,"domain":"ICLR.cc/2025/Conference","replyto":"WNPrfGpcu6","id":"FzNTefy6nt","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Fourier transform","mamba","image deraining"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Image deraining aims to remove rain streaks from rainy images and restore clear backgrounds. Currently, some research that employs the Fourier transform has proved to be effective for image deraining, due to it acting as an effective frequency prior for capturing rain streaks. However, despite there exists dependency of low frequency and high frequency in images, these Fourier-based methods rarely exploit the correlation of different frequencies for conjuncting their learning procedures, limiting the full utilization of frequency information for image deraining. Alternatively, the recently emerged Mamba technique depicts its effectiveness and efficiency for modeling correlation in various domains (e.g., spatial, temporal), and we argue that introducing Mamba into its unexplored Fourier spaces to correlate different frequencies would help improve image deraining. This motivates us to propose a new framework termed FourierMamba, which performs image deraining with Mamba in the Fourier space. Owing to the unique arrangement of frequency orders in Fourier space, the core of FourierMamba lies in the scanning encoding of different frequencies, where the low-high frequency order formats exhibit differently in the spatial dimension  (unarranged in axis) and channel dimension (arranged in axis). Therefore, we design FourierMamba that correlates Fourier space information in the spatial and channel dimensions with distinct designs. Specifically, in the spatial dimension Fourier space, we introduce the zigzag coding to scan the frequencies to rearrange the orders from low to high frequencies,  thereby orderly correlating the connections between frequencies; in the channel dimension  Fourier space with arranged orders of frequencies in axis, we can directly use Mamba to perform frequency correlation and improve the channel information representation. Extensive experiments reveal that our method outperforms state-of-the-art methods both qualitatively and quantitatively."},"_bibtex":{"value":"@misc{\nli2025fouriermamba,\ntitle={FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining},\nauthor={Dong Li and Yidi Liu and Xueyang Fu and Jie Huang and Senyan Xu and Qi Zhu and Zheng-Jun Zha},\nyear={2025},\nurl={https://openreview.net/forum?id=WNPrfGpcu6}\n}"},"title":{"value":"FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining"},"pdf":{"value":"/pdf/6fa74f1e4ee9892097a7165abc478fc4869b265d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"li|fouriermamba_fourier_learning_integration_with_state_space_models_for_image_deraining"},"authorids":{"value":["~Dong_Li15","~Yidi_Liu3","~Xueyang_Fu1","~Jie_Huang4","~Senyan_Xu1","~Qi_Zhu9","~Zheng-Jun_Zha2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Dong Li","Yidi Liu","Xueyang Fu","Jie Huang","Senyan Xu","Qi Zhu","Zheng-Jun Zha"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a motion Mamba diffusion model to estimate the corresponding full-body posture based on sparse tracking. This model utilizes diffusion models for motion generation and state space models for sequence modeling. This model generates motion through a motion Mamba block, which includes a bidirectional time Mamba module (TMM) to recognize temporal dependencies between frames and a spatial Mamba module (SMM) to enhance single-frame motion features."},"suitability":{"value":2},"strengths":{"value":"1. This paper addresses the problem of sequence modeling and motion generation.\n2. This paper can generate realistic full-body movements."},"confidence":{"value":2},"rating":{"value":4},"limitations":{"value":"1. What is the difference between the proposed Mamba diffusion model and the general diffusion model? Is it simply introducing Mamba into the diffusion model?\n2. What are the specific details of the proposed motion Mamba model?\n3. The expression of SMM in line 516 of 4.2 is incorrect.\n4. In the experimental section, which method did the data in lines 671 and 674 decrease compared to? It is not evident from the table data that the data described in the paper has been reduced. Where is the ambiguity of the data in lines 677 and 680 reflected?"}},"nonreaders":[],"tmdate":1721657237149,"tcdate":1716118074004,"writers":["acmmm.org/ACMMM/2024/Conference","acmmm.org/ACMMM/2024/Conference/Submission2195/Reviewer_F2pB"],"signatures":["acmmm.org/ACMMM/2024/Conference/Submission2195/Reviewer_F2pB"],"forum":"dA6yat7UNM","number":1,"license":"CC BY 4.0","cdate":1716118074004,"readers":["everyone"],"invitations":["acmmm.org/ACMMM/2024/Conference/Submission2195/-/Official_Review","acmmm.org/ACMMM/2024/Conference/-/Edit","acmmm.org/ACMMM/2024/Conference/Submission2195/Official_Review1/-/Final_Rating"],"mdate":1721657237149,"domain":"acmmm.org/ACMMM/2024/Conference","replyto":"dA6yat7UNM","id":"HFGcnWw23X","forumContent":{"venue":{"value":"MM2024 Oral"},"abstract":{"value":"In the domain of generative multimedia and interactive experiences, generating realistic and accurate full-body poses from sparse tracking is crucial for many real-world applications, while achieving sequence modeling and efficient motion generation remains challenging. Recently, state space models (SSMs) with efficient hardware-aware designs (i.e., Mamba) have shown great potential for sequence modeling, particularly in temporal contexts. However, processing motion data is still challenging for SSMs. Specifically, the sparsity of input conditions makes motion generation an ill-posed problem. Moreover, the complex structure of the human body further complicates this task. To address these issues, we present Motion Mamba Diffusion (MMD), a novel conditional diffusion model, which effectively utilizes the sequence modeling capability of SSMs and the robust generation ability of diffusion models to track full-body poses accurately. In particular, we design a bidirectional Temporal Mamba Module (TMM) to model motion sequence. Additionally, a Spatial Mamba Module (SMM) is further proposed for feature enhancement within a single frame. Extensive experiments on the large motion capture dataset (AMASS) demonstrate that our proposed approach outperforms the latest methods in terms of accuracy and smoothness and achieves new state-of-the-art performance. Moreover, our approach runs in real-time, making it ideal for employment in practical applications. The source code will be made public upon acceptance of this paper."},"relevance_to_conference":{"value":"We propose Motion Mamba Diffusion,  a novel conditional generation framework that combines the robust generation capabilities of diffusion models with the sequence modeling proficiency of state space models. This innovative approach enables the effective and precise capture and generation of realistic human movements from sparse tracking, thereby greatly enhancing the user experience in interactive multimedia applications. Furthermore, the high-quality human body movements produced by Motion Mamba Diffusion have the potential to enrich artistic and cultural expressions through digital artworks and interactive multimedia installations. In summary, Motion Mamba Diffusion represents a significant advancement in human-computer interaction, multimedia applications, and the intersection of art and culture, aligning closely with various themes of ACM multimedia."},"_bibtex":{"value":"@inproceedings{\ndong2024realistic,\ntitle={Realistic Full-Body Motion Generation from Sparse Tracking with State Space Model},\nauthor={Kun Dong and Jian Xue and Zehai Niu and Xing Lan and Ke Lv and Qingyuan Liu and Xiaoyu Qin},\nbooktitle={ACM Multimedia 2024},\nyear={2024},\nurl={https://openreview.net/forum?id=dA6yat7UNM}\n}"},"title":{"value":"Realistic Full-Body Motion Generation from Sparse Tracking with State Space Model"},"secondary_subject_area":{"value":["[Generation] Generative Multimedia","[Experience] Multimedia Applications","[Experience] Art and Culture"]},"pdf":{"value":"/pdf/4c38630b5b420d223f14e40ed6afcba140e8662a.pdf"},"venueid":{"value":"acmmm.org/ACMMM/2024/Conference"},"paperhash":{"value":"dong|realistic_fullbody_motion_generation_from_sparse_tracking_with_state_space_model"},"primary_subject_area":{"value":"[Experience] Interactions and Quality of Experience"},"authorids":{"value":["~Kun_Dong4","~Jian_Xue2","~Zehai_Niu1","~Xing_Lan1","~Ke_Lv2","~Qingyuan_Liu2","~Xiaoyu_Qin2"]},"authors":{"value":["Kun Dong","Jian Xue","Zehai Niu","Xing Lan","Ke Lv","Qingyuan Liu","Xiaoyu Qin"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"We constrain Mamba's selective recurrence so that its Lipschitz constant and hidden-state norm admit explicit bounds, and state precisely what those bounds do and do not certify against hidden-state poisoning."},"keywords":{"value":["Selective state-space models","Hidden-state poisoning","LipMamba","Lipschitz constant","Spectral normalisation","State-retention bound","PAC-Bayesian bound","Adversarial robustness"]},"primary_area":{"value":"alignment, fairness, safety, privacy, and societal considerations"},"abstract":{"value":"Selective state-space models (SSMs) such as Mamba are efficient linear-time alternatives to attention, but their input-dependent discretisation creates an architecture-specific attack surface: short adversarial token sequences can drive the step size Δt toward saturation so that ¯At = exp(ΔtA) → 0, erasing prior context (hidden-state poisoning, HiSPA). Existing defences are detectors that offer\nno analytical guarantee. We introduce LipMamba, a selective-SSM parameterisation with spectrally-normalised projections, an eigenvalue-bounded state matrix and a discretisation step clamped to [Δmin,Δmax], and prove three results with complete proofs: (i) an explicit Lipschitz constant for the selective (input-dependent) scan on the bounded-input domain, which identifies the lower clamp\nΔmin > 0 as the condition that keeps the state bounded and thereby makes precise the error explosion previously observed in input-dependent SSMs; (ii) a state retention bound giving an analytically computed trigger length ℓ⋆ below which no trigger can reduce the hidden-state norm below a chosen fraction of its pre-trigger value, which removes the ¯At → 0 mechanism of HiSPA; and (iii) a PAC-Bayesian bound on adversarial risk whose gap term is governed by the same constant. We are explicit about scope: the analytical global constant grows geometrically with depth and yields a vacuous global certificate, so per-input radii are reported under a local Lipschitz estimate and are empirical; the retention bound constrains norm, not content, and for the constraint values used here it certifies triggers of one token at 50% retention and of up to six tokens against near-complete erasure. On RoBench-25 and HarmBench, LipMamba-370M raises poisoned accuracy under HiSPA at trigger length 24 from 23.4% (Mamba) to 85.3%, retains 80.4% accuracy at empirical local-Lipschitz radius 0.18, and adds 3.5–4.9% perplexity overhead over the public Mamba checkpoints on WikiText-103. Ablations attribute\nthe gain chiefly to spectral normalisation of the selective projections and to step clamping."},"_bibtex":{"value":"@inproceedings{\nanonymous2026lipschitzconstrained,\ntitle={Lipschitz-Constrained Selective State-Space Models with State-Retention and {PAC}-Bayesian Bounds Against Hidden-State Poisoning},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=kYf3aopGAD},\nnote={under review}\n}"},"title":{"value":"Lipschitz-Constrained Selective State-Space Models with State-Retention and PAC-Bayesian Bounds Against Hidden-State Poisoning"},"pdf":{"value":"/pdf/4a91ff4de91f20c20858466166f08dea1afaf1d8.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791234164113,"tcdate":1789765041849,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission48776/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission48776/Authors"],"forum":"kYf3aopGAD","license":"CC BY 4.0","number":48776,"cdate":1789765041849,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission48776/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791234164113,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"kYf3aopGAD","version":2},{"content":{"TLDR":{"value":"SpatialVLA-Mamba unifies spatial grounding, efficient Mamba decoding, and intrinsic CoT-RL to achieve precise, robust, and efficient robotic control in simulation."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Vision-Language-Action Models","Spatial Reasoning","State-Space Models (Mamba)","Reinforcement Learning","Robotics and Embodied AI"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Recent progress in vision-language-action (VLA) models has enabled robots to follow natural language instructions across diverse manipulation tasks. However, existing approaches struggle with three persistent challenges: limited spatial grounding, which hampers centimeter-level precision; inefficiency and instability in long-horizon execution due to transformer-based decoders; and brittleness under distribution shift, where minor visual or linguistic variations can cause failure. We present SpatialVLA-Mamba, a framework that addresses these challenges through three innovations. First, a spatial-aware encoder augments RGB features with depth and geometric primitives, providing explicit metric grounding. Second, a Mamba-based state-space decoder replaces transformers, offering linear-time complexity and stable long-sequence modeling for extended action horizons. Third, a Chain-of-Thought Reinforcement Learning (CoT-RL) loop introduces intrinsic self-refinement: the policy generates textual outcome summaries of candidate trajectories, evaluates them with CLIPScore against the goal instruction, and updates itself via PPO without reliance on external language models. Experiments in Webots show that SpatialVLA-Mamba reduces spatial error by over 35\\% relative to strong baselines, improves unseen-task success to 67.3\\%, and achieves higher robustness to sensor noise and linguistic paraphrasing, while requiring less GPU memory and runtime. These results highlight the importance of combining spatial grounding, efficient sequence modeling, and intrinsic reasoning for reliable embodied control, pointing toward embodied foundation models that are accurate, efficient, and self-correcting."},"_bibtex":{"value":"@misc{\nli2026spatialvlamamba,\ntitle={Spatial{VLA}-Mamba: Efficient State-Space Models with Self-Refinement for Spatially-Grounded Robotic Control},\nauthor={Sifan Li and Hao Tang},\nyear={2026},\nurl={https://openreview.net/forum?id=sTn4EqE49A}\n}"},"title":{"value":"SpatialVLA-Mamba: Efficient State-Space Models with Self-Refinement for Spatially-Grounded Robotic Control"},"pdf":{"value":"/pdf/791e8f3a5f47b17ff7571aae8c62a960e29cf3f0.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"li|spatialvlamamba_efficient_statespace_models_with_selfrefinement_for_spatiallygrounded_robotic_control"},"authorids":{"value":["~Sifan_Li1","~Hao_Tang6"]},"authors":{"value":["Sifan Li","Hao Tang"]}},"tmdate":1768977704377,"tcdate":1758286743823,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18350/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission18350/Authors"],"forum":"sTn4EqE49A","license":"CC BY 4.0","number":18350,"cdate":1758286743823,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission18350/-/Full_Submission","ICLR.cc/2026/Conference/-/Withdrawn_Submission"],"mdate":1768977704377,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"sTn4EqE49A","version":2},{"content":{"summary":{"value":"A method for multi-variate probabilistic time series forecasting is presented which combines the methods for temporal dynamics modeling using deep Koopman model with Neural Processes. The model consists of an encoder computing the hidden state h_t, a learned Koopman operator 'A' predicting the future hidden states and a decoder used to predict the final values using the hidden state h_t. In addition a neural process is used to model the latent dynamics S of the time series, which is used as an input to the Koopman operator to compute the future states."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- What is the utility of the neural process latent variable S at a high level? Does it help model the correlations between the different time series? Can it be thought of as an encoding of each time series which also model the inter-correlations between the time series?\n- It seems that S is an s-dimensional latent variable meaning that it encodes the whole time series in a low dimensional subspace. Many papers have modeled time series as state space models involving low-dimensional latent decomposition. What is the advantage of the proposed approach over these approaches.\nExample papers modeling time series into low dimensional sub-spaces:\n  - Wang, Yuyang, et al. \"Deep factors for forecasting.\" International conference on machine learning. PMLR, 2019.\n  - Rangapuram, Syama Sundar, et al. \"Deep state space models for time series forecasting.\" Advances in neural information processing systems 31 (2018).\n  - Paria, Biswajit, et al. \"Hierarchically regularized deep forecasting.\" arXiv preprint arXiv:2106.07630 (2021).\n- What is the utility of the delay embedding dimension k?\n- Do context and target sets refer to training and testing time periods? can the context and target sets be interspersed, meaning can this method be used to predict missing values of irregularly sampled time series?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"Strengths\n- This paper proposes a new way for probabilistic modeling of multi-variate time series by representing the time series into a low dimensional space using neural processes and using them to model the temporal dynamics of the time series.\n- The idea presented in the paper is an intuitive approach for multi-variate forecasting. Modeling the latent dynamics of the time series is an important aspect that is missed by most state-of-the-art modeling approaches. As such, the method presented in this paper is interesting to the time series community.\n- The paper presents extensive experimental evaluation and ablation studies demonstrating the effectiveness of the approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Weaknesses\n- The description of the model is quite cryptic and not written in a clear manner. See questions below for parts that are unclear.\n- While the paper describes everything using mathematical equations, the utility or advantage of such equations/models is unclear. How do each of the components of the model help with a better forecast? (i am looking for a high level motivation for each of the components of the model)\n- Several works have introduced Koopman theory into time series modeling, however, there is no clear discussion of existing methods and their comparison with this work. The novelty of the paper is unclear without a clear distinction with existing work.\n- Other probabilistic forecasting methods modeling latent factors in time series have not been discussed of compared (for example deep state space models)."}},"nonreaders":[],"tmdate":1731429316822,"tcdate":1730686947306,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9644/Reviewer_VVQE"],"signatures":["ICLR.cc/2025/Conference/Submission9644/Reviewer_VVQE"],"forum":"5oSUgTzs8Y","number":4,"license":"CC BY 4.0","cdate":1730686947306,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9644/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429316822,"domain":"ICLR.cc/2025/Conference","replyto":"5oSUgTzs8Y","id":"bnVRvrWXRk","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Probabilistic time series prediction; Neural Process; Deep Koopman model"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The probabilistic forecasting of time series is a well-recognized challenge, particularly in disentangling correlations among interacting time series and addressing the complexities of distribution modeling. By treating time series as temporal dynamics, we introduce **KooNPro**, a novel probabilistic time series forecasting model that combines variance-aware deep **Koo**pman model with **N**eural **Pro**cess. KooNPro introduces a variance-aware continuous spectrum using Gaussian distributions to capture complex temporal dynamics with improved stability. It further integrates the Neural Process to capture fine dynamics, enabling enhanced dynamics capture and prediction. Extensive experiments on nine real-world datasets demonstrate that KooNPro consistently outperforms state-of-the-art baselines. Ablation studies highlight the importance of the Neural Process component and explore the impact of key hyperparameters. Overall, KooNPro presents a promising novel approach for probabilistic time series forecasting."},"_bibtex":{"value":"@inproceedings{\nzheng2025koonpro,\ntitle={Koo{NP}ro: A Variance-Aware Koopman Probabilistic Model Enhanced by Neural Process for Time Series Forecasting},\nauthor={Ronghua Zheng and Hanru Bai and Weiyang Ding},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=5oSUgTzs8Y}\n}"},"title":{"value":"KooNPro: A Variance-Aware Koopman Probabilistic Model Enhanced by Neural Process for Time Series Forecasting"},"pdf":{"value":"/pdf/78b89237a756e38304347883a7bef22cb77ff44a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"zheng|koonpro_a_varianceaware_koopman_probabilistic_model_enhanced_by_neural_process_for_time_series_forecasting"},"authorids":{"value":["~Ronghua_Zheng1","~Hanru_Bai2","~Weiyang_Ding1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Ronghua Zheng","Hanru Bai","Weiyang Ding"]}},"version":2},{"content":{"summary":{"value":"This paper proposes MambaMatch, a new method building on the efficacy of Mamba networks for two-view correspondences. \nThe rationale behind this approach is to selectively mine information from true correspondences - relying on the selectivity of Mamba networks - and to dispose the potentially erroneous information from false correspondences.\nInstead of simply relying on Mamba networks, the authors proposes to improve the channel-awareness to alleviate the weak cross-channel awareness in Mamba networks. \nThe authors also propose to capture reasonable local context of correspondences for improved pruning, via the Local-Context Enhancement Module (LCEM). \nState-of-the-art experimental results on the tasks of relative pose estimation and visual localization demonstrates the strength of MambaMatch, while incurring lower FLOPs and latency (i.e., more efficient) compared to existing methods."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Some questions I have regarding the paper - \n\n1. In equation 16, what does $\\mathcal{L}$ refer to?\n2. What does it mean that the key points are normalized “with image size [OR] camera intrinsic”?\n3. How were the FLOPs measured? Many open-source FLOPs-measuring libraries fail to capture the metrics correctly, especially in the presence of hardware optimizations in xformers or mamba. Can you provide per-module FLOPs calculation of the given method?\n4. What are the GPU vRAM usage across the methods being compared?\n5. It seems that when using Matcher such as SP+SG, the performance is much higher compared to using SIFT+NN. How do they differ in terms of efficiency?\n\nWhile I believe the idea of using Mamba networks for the task of two-view correspondence is interesting, and the performances are satisfactory, I find the weaknesses stated above strongly outweigh the strengths of the paper."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- This paper presents an original idea of integrating Mamba networks for the task of establishing two-view correspondence. \n- The authors identify that vanilla Mamba networks lack channel awareness, and integrate a channel-aware module to boost the performance of MambaMatch.\n- MambaMatch demonstrates state-of-the-art performances on relative pose estimation and visual localization.\n- Additional experimental analysis, such as the generalization experiments and the ablation experiments, provide additional insights into the performances and abilities of MambaMatch."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"## Misleading / Inaccurate presentation / theory\n\nWhile the overall idea and direction of research is original and interesting, I find that the way it is presented in the paper is severely misleading in multiple aspects.\n\n**1. In Figure 1**, MLP-based filters are shown to output all correspondence weights as 'yellow', which means that all correspondences are seen as vague. I understand that the authors were trying to illustrate that \"MLPs treat all input equally (L053)\", as MLP weights are static and are fixed regardless of the input at inference. However, this does not mean that all inlier probabilities are predicted to the same value, as static weights still result in different results for different inputs.\n\nAlso, while it is understandable that the authors tried to differentiate attention-based filters with Mamba-based filters by illustrating the Mamba-based filters can 'reject' outliers by removing all the red(-ish) lines, Mamba networks do not inherently 'reject' outliers. If a certain type of thresholding was applied to reject correspondences with low inlier ratios, the same can be done for Attention-based methods and MLP-based methods. Please correct me if I have wrongly understood the figure and the authors' intentions. \n\n**2. Resuming on the aspect of 'outlier rejection'**, the authors mention similar ideas throughout the paper, e.g., \"dispose of the potentially interfering information of false correspondences (L016)\", \" enabling models to focus on or disregard specific inputs (L066)\", or \"We specifically engineer a new outlier filter based on Mamba that can more effectively filter out mismatches by discarding particular inputs (L095)\". \n\nThese propositions would be acceptable, if not for their contrast to how attention is described - \"this soft attention is allocated across all data throughout, leading to attention leakage and the retention of redundant context (L057)\", \"..learning an attention map...suffer from attention leakage... (L123)\". Assuming that the authors are using the term 'attention leakage' to address situations where false correspondence still receive 'at least a little' attention, this inherently happens in Mamba networks as well. The term 'selectivity' in Mamba networks refer to the idea of having its parameters ($\\overline{B}, \\overline{C}, \\Delta$) dependent on the input - and for less important inputs, it would ideally mean that the step size $\\Delta$ is smaller.  While this facilitates effective compression of context, 'smaller' is not really 'zero' - and I think this 'small' can also be regarded as 'attention leakage'.\n\nTherefore, to the questions considered in the paper (L061) :: (i) attention mechanisms also selectively treat inliers and outliers, and allocate attention appropriately to different points, as effective as Mamba mechanisms do. (ii) Attention mechanisms also 'fully capture the information'. But yes, context compression happens in Mamba networks but not in vanilla attention networks. The manuscript should not overstate what Mamba networks can do, and not understate what attention networks can do. \n\n**3. Misleading explanation regarding the efficiency and functioning of Mamba.**\n\nL68: \"...and employs hardware scanning methods. This presents a potential avenue for addressing (ii)\", where (ii) is regarding context compression. Hardware-aware algorithms in Mamba address the efficiency, not the compression of context.\n\nL175, equation 3 explanation: While it is true that the discretized SSM can be represented as convolution, that is NOT the case for Mamba. With the integration of selectivity, the parameters of Mamba networks become input-dependent, and they cannot be represented as convolution (static fixed kernels) anymore. Therefore, the statement \"This characteristic endows Mamba with the inherent capability for parallel computation\" is definitely flawed. The authors make the same error in section 4.1, where they mention \"This enables the network to inherit Mamba’s characteristics, allowing it to focus on the inputs from inliers while disregarding outlier information\" (i.e., input dependence) and proceeds to represent the network through a convolution operation (i.e., input independence). This is not only misleading, but this makes me suspect that the authors do not have a solid understanding of the Mamba networks. \n\n## Paper is overall hard to follow, with vague propositions at times\n\nL25: \"...our approach outperforms existing state-of-the-art methods on several visual tasks...\" - the proposed method is for two-view correspondence, and this statement is too vague / overstating the achievements of MambaMatch.\n\nIt is mentioned throughout the paper that Mamba networks \"lack channel awareness (L80)\", \"falls short in channel information acquisition (L150)\", and \"underperforms in cross-channel information access(257)\". Apart from these three sentences sounding ambiguous when put together, the paper lacks context about **why this is so**. They just include a reference (Guo et al., 2024) - and when reading the referenced paper, it is mentioned that \"channel attention is used to reduce channel redundancy caused by the excessive hidden\nstate number\", which is validated by a Channel Activation Visualization (Figure 3(b) of Guo et al., 2024). The authors should at least provide more context, or provide similar visualizations. In particular, since the ablation results in Table 6 shows that simply applying C.A without Mamba results in worse results, at least 3 channel activation visualizations of (a) with LCEM only (b) with LCEM and Mamba only, and (c) with LCEM and CAMF should be provided to sufficiently evidence this claim. \n\nThe method section is not in order, and do not fully explain the overall pipeline of the paper. From Figure 2, it seems that the initial correspondence set is passed through Position-Context Initialization, then the MambaMatch layers which contain modules in the order of Local Context Enhancement, Channel-Aware Mamba Filter, and Cluster Sequence Block. These are just mentioned in the overview, but in sections 4.1 and 4.2, only the Channel-Aware Mamba Filter and Local Context Enhancement Modules are discussed. This makes the method very difficult to follow. \n\nAlso, the inlier predictor seems like it should be placed outside of the MambaMatch layers which is repeated 4 times, since the inlier predictor outputs tensors with channel 1 (1-D), while MambaMatch layers accept tensors with channel dimensions of 128 (128-D).\n\n\"...while the second pipeline is discussed appropriately in the analysis. (L133)\" should guide the readers specifically to Table 4, where it shows compatibility with Matchers. Otherwise, it is hard to follow what the authors are referring to. \n\n$\\mathcal{M}$ in Equations 10 and 11 should all be differentiated, unless the same MLP is being used for all $\\mathcal{M}$.\n\n## Less major issues, including errors in writing.\n\n- Lack of visualization of \"MambaMatch focusing on or discarding particular inputs\" along the 4 layers of MambaMatch layers. The qualitative results in the paper are mainly visualization of the matches after pruning.\n- How is the pruning done? How are erroneous correspondences filtered out? Is there any sort of thresholding involved?\n- Regarding Table 6, are all settings in the Table trained following the same settings as MambaMatch? This is just a wary question, I find it weird that the performance increase is so low for (ii) and (iii). Or were they just carried out from weights of (iv)?\n- In Table 6, there should be a comparison when even LCEM is not used.\n\n\n### Writing mistakes: \n- 'Correspondence' capitalized in L107 and L126\n- ZOH is explained twice in L162 and L169, but it is actually only carried out in the latter. \n- Typo in caption of Figure 3: Channel -> Channel\n- Skin to -> Akin to (L417)"}},"nonreaders":[],"tmdate":1731428644305,"tcdate":1730700871881,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6894/Reviewer_BQWp"],"signatures":["ICLR.cc/2025/Conference/Submission6894/Reviewer_BQWp"],"forum":"YuJdtpPV4n","number":3,"license":"CC BY 4.0","cdate":1730700871881,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6894/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428644305,"domain":"ICLR.cc/2025/Conference","replyto":"YuJdtpPV4n","id":"ooq0D82jGw","forumContent":{"TLDR":{"value":"Correspondences Learning with Selective State Spaces"},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Correspondence learning","image matching","feature matching"]},"supplementary_material":{"value":"/attachment/2d09b8e2610b05f0cb43cedcc37982b32e359fa9.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Two-view correspondence learning aims to discern true and false correspondences between image pairs by recognizing their underlying different information. Previous methods either treat the information equally or fail to discard the superfluous information of false correspondences, tending to be invalid in practical scenarios. Therefore, inspired by Mamba's inherent competence of selectivity, we propose MambaMatch as a Mamba-based correspondence filter to selectively mine information from true correspondences and to dispose of the potentially interfering information of false correspondences. Specifically, the selection is achieved by adaptively adjusting model parameters in a high-dimensional latent space, which also avoids attention leakage and implements context compression, ensuring the precise and efficient exploitation of pertinent information. Meanwhile, channel awareness is tailored to serve as a complementary aspect of comprehensive information acquisition. Moreover, we design a novel local-context enhancement module to capture reasonable local context that is crucial for correspondence pruning. Extensive experiments demonstrate that our approach outperforms existing state-of-the-art methods on several visual tasks while saving time and space costs."},"_bibtex":{"value":"@misc{\nfang2024mambamatch,\ntitle={MambaMatch: Learning Two-View Correspondences with Selective State Spaces},\nauthor={Xiang Fang and Shihua Zhang and Jiayi Ma},\nyear={2024},\nurl={https://openreview.net/forum?id=YuJdtpPV4n}\n}"},"title":{"value":"MambaMatch: Learning Two-View Correspondences with Selective State Spaces"},"pdf":{"value":"/pdf/63484318bfc95ae154ff8a779ed5675dcfd84a4f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"fang|mambamatch_learning_twoview_correspondences_with_selective_state_spaces"},"authorids":{"value":["~Xiang_Fang2","~Shihua_Zhang2","~Jiayi_Ma2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xiang Fang","Shihua Zhang","Jiayi Ma"]}},"version":2},{"content":{"TLDR":{"value":"We establish a single-trajectory analogue of PAC learnability for a class of stable selective state-space models using a Ho–Kalman-based algorithm that allows temporally correlated inputs."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Selective State-Space Models","PAC learnability","Single-Trajectory learning","Time series"]},"primary_area":{"value":"learning theory"},"abstract":{"value":"Selective State-Space Models (SSMs) are dynamical systems where the state-transition, input and readout maps may depend on the current input. This mechanism is central to the celebrated Mamba architecture for long-range tasks, but it is also present in more classical model classes used in control engineering and time-series predictions, such as Linear Parameter-Varying (LPV) systems. In this paper we study learnability from a single time-series for a certain class of selective SSMs of which the state-transition, input and readout maps are affine functions of suitable features of the current inputs. We propose a concrete learning algorithm and a finite-sample probabilistic $\\widetilde O(1/N^{\\zeta})$, $\\zeta \\leq 1/2$ PAC bound on the parameter-estimation (and hence prediction error) for this learning algorithm. The algorithm itself is an extension of the Ho--Kalman realization-based learning algorithm for linear systems, and it relies on computing the so-called Markov parameters of the SSMs. Our results show that the proposed class of selective SSMs satisfies the single-trajectory analogue of PAC learnability, i.e., it is possible to learn a model with high-enough probability and high-enough accuracy, given a sufficient number of data points."},"_bibtex":{"value":"@inproceedings{\nanonymous2026pac,\ntitle={{PAC} Learnability of Stable Selective State-Space Models from a Single Time Series},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=QRmvAEdAw4},\nnote={under review}\n}"},"title":{"value":"PAC Learnability of Stable Selective State-Space Models from a Single Time Series"},"pdf":{"value":"/pdf/03557757ce99f2d4274004fedbc2e140d6763cd0.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231807261,"tcdate":1789676277111,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission34336/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission34336/Authors"],"forum":"QRmvAEdAw4","license":"CC BY 4.0","number":34336,"cdate":1789676277111,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission34336/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791231807261,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"QRmvAEdAw4","version":2},{"content":{"summary":{"value":"This paper proposes MambaMatch, an outlier removal approach of image matching. MambaMatch is constructed with the prevalent Mamba layer to capture contextual information in an efficient manner. Specifically, Channel-Aware Mamba Filter (CAMF) is designed by combining the Mamba layer with a channel attention structure. Moreover, Local Context Enhancement (LCEM) is introduced to improve the ability to capture local context. Experimental results demonstrate the superiority of MambaMatch when combined with the SIFT keypoints."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please provide more discussions and experimental results to address the above weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Mamba is a potential architecture to achieve a good balance between the accuracy and efficiency of outlier removal. This paper provides a reasonable implementation and initially validates its effectiveness. \n2. The organization and presentation of this paper are clear."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**1. The explanation of the first motivation is confusing.**\n\nIn Line 57, the authors discuss the problem of the existing attention-based methods with the statement: “However, this soft attention is allocated across all data throughout, leading to attention leakage and the retention of redundant context”. Then, in Line 87, the superiority of the proposed Mamba is introduced as: “our Mamba-based approach not only focuses on potentially correct matches with higher weights but also discards false matches, avoiding attention leakage and achieving context compression.\n\nHowever, the selection mechanism of Mamba cannot guarantee to avoid attention leakage according to the equations (2), (3), (4), (5). The “attention weights” of Mamba are still soft rather than hard. It is confusing why Mamba can avoid the attention leakage problem appearing in the existing attention-based methods like Transformer.\n\n**2. The comparison experiment is not sufficient.**\n\nExperimental results in Table 1 demonstrate the superiority of MambaMatch when combined with the SIFT keypoints. Meanwhile, the improvement provided by MambaMatch is smaller when it is combined with some advanced Matchers as shown in Table 4. Therefore, it is unclear whether MambaMatch can outperform the existing outlier removal approaches when combined with the advanced Matchers.\n\n**3. Some expressions are unclear.**\n\n3.1. In Figure 1, the authors state that “The transition of the yellow hue from closer to red to closer to green signifies a progression from lower to higher weights.” However, the false matches fully disappear in Figure 1 (d), without the color closer to red or closer to green. What are the scores provided by “Mamba-based Filters” for the false matches?\n\n3.2. In Line 212, the authors state that “Drawing on the works of Zhang et al. (2019) and Liu & Yang (2023), we combine the Cluster-Sequence Block in our framework without delving into intricate specifics.” The role or function of the Cluster-Sequence Block in MambaMatch should be briefly introduced, even if its technical detail is unnecessary."}},"nonreaders":[],"tmdate":1731428644370,"tcdate":1730728857901,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6894/Reviewer_vJjC"],"signatures":["ICLR.cc/2025/Conference/Submission6894/Reviewer_vJjC"],"forum":"YuJdtpPV4n","number":4,"license":"CC BY 4.0","cdate":1730728857901,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6894/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428644370,"domain":"ICLR.cc/2025/Conference","replyto":"YuJdtpPV4n","id":"kchFe18S7G","forumContent":{"TLDR":{"value":"Correspondences Learning with Selective State Spaces"},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Correspondence learning","image matching","feature matching"]},"supplementary_material":{"value":"/attachment/2d09b8e2610b05f0cb43cedcc37982b32e359fa9.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Two-view correspondence learning aims to discern true and false correspondences between image pairs by recognizing their underlying different information. Previous methods either treat the information equally or fail to discard the superfluous information of false correspondences, tending to be invalid in practical scenarios. Therefore, inspired by Mamba's inherent competence of selectivity, we propose MambaMatch as a Mamba-based correspondence filter to selectively mine information from true correspondences and to dispose of the potentially interfering information of false correspondences. Specifically, the selection is achieved by adaptively adjusting model parameters in a high-dimensional latent space, which also avoids attention leakage and implements context compression, ensuring the precise and efficient exploitation of pertinent information. Meanwhile, channel awareness is tailored to serve as a complementary aspect of comprehensive information acquisition. Moreover, we design a novel local-context enhancement module to capture reasonable local context that is crucial for correspondence pruning. Extensive experiments demonstrate that our approach outperforms existing state-of-the-art methods on several visual tasks while saving time and space costs."},"_bibtex":{"value":"@misc{\nfang2024mambamatch,\ntitle={MambaMatch: Learning Two-View Correspondences with Selective State Spaces},\nauthor={Xiang Fang and Shihua Zhang and Jiayi Ma},\nyear={2024},\nurl={https://openreview.net/forum?id=YuJdtpPV4n}\n}"},"title":{"value":"MambaMatch: Learning Two-View Correspondences with Selective State Spaces"},"pdf":{"value":"/pdf/63484318bfc95ae154ff8a779ed5675dcfd84a4f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"fang|mambamatch_learning_twoview_correspondences_with_selective_state_spaces"},"authorids":{"value":["~Xiang_Fang2","~Shihua_Zhang2","~Jiayi_Ma2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xiang Fang","Shihua Zhang","Jiayi Ma"]}},"version":2},{"content":{"summary":{"value":"The paper proposes a novel method for image deraining, called FreqMamba, which leverages frequency analysis and the Mamba model to achieve state-of-the-art results. The method consists of three branches: spatial Mamba, frequency band Mamba, and Fourier global modeling, which work together to capture both local and global degradation patterns. The authors conduct extensive experiments to validate the effectiveness of their approach, comparing it to several state-of-the-art methods on various datasets. The results show that FreqMamba outperforms the other methods both visually and quantitatively, achieving better spectral fidelity and fewer residues."},"suitability":{"value":2},"strengths":{"value":"Integrating frequency analysis with spatial modeling via Mamba is a novel approach for image deraining. Exploiting the complementary strengths of frequency and spatial domains could lead to improved global and local degradation modeling.\nThe paper provides a sound theoretical foundation, describing frequency analysis concepts like Fourier transform, wavelet transform, and state space models clearly. The proposed three-branch architecture (Fourier Modeling, Spatial Mamba, Frequency Band Mamba) is well-motivated."},"confidence":{"value":3},"rating":{"value":5},"limitations":{"value":"While the integration of different components is novel, the individual components (Fourier transform, wavelet transform, Mamba) are well-established techniques.\nThe proposed architecture appears to be computationally expensive due to the multiple branches and transformations involved, which could limit its practical applications."}},"nonreaders":[],"tmdate":1721657148188,"tcdate":1716128234967,"writers":["acmmm.org/ACMMM/2024/Conference","acmmm.org/ACMMM/2024/Conference/Submission657/Reviewer_G9GE"],"signatures":["acmmm.org/ACMMM/2024/Conference/Submission657/Reviewer_G9GE"],"forum":"jCswanMk5e","number":2,"license":"CC BY 4.0","cdate":1716128234967,"readers":["everyone"],"invitations":["acmmm.org/ACMMM/2024/Conference/Submission657/-/Official_Review","acmmm.org/ACMMM/2024/Conference/-/Edit","acmmm.org/ACMMM/2024/Conference/Submission657/Official_Review2/-/Final_Rating"],"mdate":1721657148188,"domain":"acmmm.org/ACMMM/2024/Conference","replyto":"jCswanMk5e","id":"aQTMX1PQsB","forumContent":{"venue":{"value":"MM2024 Poster"},"supplementary_material":{"value":"/attachment/ab60cdfea22c7241d0a5a32406039674fa827113.zip"},"abstract":{"value":"Images corrupted by rain streaks often lose vital frequency information for perception, and image deraining aims to solve this issue which relies on global and local degradation modeling.\nRecent studies have witnessed the effectiveness and efficiency of Mamba for perceiving global and local information based on its exploiting local correlation among patches, however, rarely attempts have been explored to extend it with frequency analysis for image deraining, limiting its ability to perceive global degradation that is relevant to frequency modeling (e.g. Fourier transform).\nIn this paper, we propose FreqMamba, an effective and efficient paradigm that leverages the complementary between Mamba and frequency analysis for image deraining. The core of our method lies in extending Mamba with frequency analysis from two perspectives: extending it with frequency-band for exploiting frequency correlation, and connecting it with Fourier transform for global degradation modeling.\nSpecifically, FreqMamba introduces complementary triple interaction structures including spatial Mamba, frequency band Mamba, and Fourier global modeling. Frequency band Mamba decomposes the image into sub-bands of different frequencies to allow 2D scanning from the frequency dimension. Furthermore, leveraging Mamba's unique data-dependent properties, we use rainy images at different scales to provide degradation priors to the network, thereby facilitating efficient training. Extensive experiments show that our method outperforms state-of-the-art methods both visually and quantitatively."},"relevance_to_conference":{"value":"From the perspective of global modeling, this paper examines the limitations of Fourier transform and Mamba in image deraining tasks, and establishes a seamless relationship between them by introducing the frequency dimension Mamba . We propose FreqMamba, a general and efficient paradigm that leverages the unique and complementary capabilities of frequency analysis and Mamba for image deraining."},"_bibtex":{"value":"@inproceedings{\nzou2024freqmamba,\ntitle={FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining},\nauthor={Zhen Zou and Hu Yu and Jie Huang and Feng Zhao},\nbooktitle={ACM Multimedia 2024},\nyear={2024},\nurl={https://openreview.net/forum?id=jCswanMk5e}\n}"},"title":{"value":"FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining"},"pdf":{"value":"/pdf/7f294a61f0eaab9b96f6233d76bccaa158d7cf74.pdf"},"venueid":{"value":"acmmm.org/ACMMM/2024/Conference"},"paperhash":{"value":"zou|freqmamba_viewing_mamba_from_a_frequency_perspective_for_image_deraining"},"primary_subject_area":{"value":"[Content] Vision and Language"},"authorids":{"value":["~Zhen_Zou1","~Hu_Yu2","~Jie_Huang4","~Feng_Zhao6"]},"authors":{"value":["Zhen Zou","Hu Yu","Jie Huang","Feng Zhao"]}},"version":2},{"content":{"comment":{"value":"> [W1] Since Mamba itself is **designed for “TD”**, the motivation for **applying it to “CD”** is not well justified. \n\nWe agree with the reviewer that **Mamba can be thought to be designed for TD**, as it is an **”SSM”-based architecture** that processes inputs sequentially. However, we believe that ***its essence does not lie in temporal modeling itself***, but rather in its ability to **efficiently replace** the **(quadratic-complexity) attention mechanism** with a **(linear-complexity) selective-scan mechanism** for modeling token dependencies. \n\n\nThis property allows Mamba to be effectively extended ***beyond temporal domains***  as an **alternative to Transformer**,  and has been successfully applied to various **non-sequential** tasks—such as tabular data [A, B] and channel dependency modeling in TS [C–J]—where its efficiency and dependency-capturing capability remain advantageous. In particular, both prior studies [J] and our results (**Table 7**) consistently show that ***Mamba achieves stronger performance when applied to CD than TD***, reinforcing its effectiveness beyond TD.\n\n\nFurthermore, this point is also **acknowledged in our paper**:\n\n\n- 1. **[L47–50]** As noted in L47–50, we explicitly state:  “~ Mamba is an SSM-based model designed for sequential inputs, making it more natural to capture TD rather than CD. Nonetheless, we focus on Mamba capturing CD instead of TD, following recent works ~ due to their superior performance.”\n- 2. **[Table 4]** Table 4 shows that various methods leverage Mamba **beyond TD** to effectively capture CD.\n\n\nA detailed explanation of why applying a **sequential architecture** to a **non-sequential domain** is largely a **superficial difference**, and that the true essence of Mamba lies in replacing the ***”quadratic-complexity” attention*** of Transformers with a ***”linear-complexity” selective-scan mechanism***, can be found in our response to **[Q1]** below.\n\n\nPlease feel free to reach out with any further questions regarding this point!\n\n\n&nbsp;\n\n- [A] Thielmann, Anton Frederik, et al. \"Mambular: A sequential model for tabular deep learning.\" arXiv 2024.\n- [B] Ahamed, Md Atik, and Qiang Cheng. \"MambaTab: A plug-and-play model for learning tabular data.\" 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 2024.\n- [C] Aobo Liang, Xingguo Jiang, Yan Sun, and Chang Lu. Bi-mamba+: Bidirectional mamba for time series forecasting. arXiv 2024.\n- [D] ​​Zixuan Weng, Jindong Han, Wenzhao Jiang, and Hao Liu. Simplified mamba with disentangled dependency encoding for long-term time series forecasting. arXiv 2024.\n- [E] Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv 2024.\n- [F] Md Atik Ahamed and Qiang Cheng. Timemachine: A time series is worth 4 mambas for long-term forecasting. In ECAI, 2024.\n- [G] Xiuding Cai, Yaoyao Zhu, Xueyao Wang, and Yu Yao. Mambats: Improved selective state space models for long-term time series forecasting. arXiv 2024.\n- [H] Shusen Ma, Yu Kang, Peng Bai, and Yun-Bo Zhao. Fmamba: Mamba based on fast-attention for multivariate time-series forecasting. arXiv 2024.\n- [I] Sijie Xiong, Shuqing Liu, Cheng Tang, Fumiya Okubo, Haoling Xiong, and Atsushi Shimada. Attention mamba: Time series modeling with adaptive pooling acceleration and receptive field enhancements. arXiv 2025.\n- [J] Zihan Wang, Fanheng Kong, Shi Feng, Ming Wang, Xiaocui Yang, Han Zhao, Daling Wang, and Yifei Zhang. Is mamba effective for time series forecasting? Neurocomputing 2025."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763688519913,"tcdate":1763688519913,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23977/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission23977/Authors"],"forum":"N6MIqE11eF","number":1,"license":"CC BY 4.0","cdate":1763688519913,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23977/-/Official_Comment"],"mdate":1763688519913,"domain":"ICLR.cc/2026/Conference","replyto":"8ht4NKW9aq","id":"v9tYVUb0bp","forumContent":{"TLDR":{"value":"We propose FSMamba, a time series forecasting method that uses Mamba with regularization strategy to address the sequential order bias in capturing channel dependencies."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Time Series Forecasting","Mamba"]},"supplementary_material":{"value":"/attachment/d0aef3ef8995051541105dd0e448b12055805dee.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Mamba is a sequential model that has recently emerged as a promising alternative to Transformers, offering near-linear complexity.\nHowever, although channels in time series (TS) data generally lack a sequential order, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this, prior works have adopted bidirectional Mamba to scan channels in both forward and reverse orders. In this paper, we show that unidirectional Mamba can effectively replace the bidirectional Mamba with simple strategies. To this end, we propose FSMamba, a TS forecasting method employing a unidirectional Mamba that incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order. Furthermore, we introduce channel similarity modeling, a pretraining task to preserve similarities between channels from the data space to the latent space to enhance the ability to capture CD. Extensive experiments demonstrate the efficacy of our method, achieving state-of-the-art performance on diverse datasets."},"_bibtex":{"value":"@misc{\nlee2026is,\ntitle={Is Bidirectionality Necessary in Mamba for Time Series Forecasting?},\nauthor={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\nyear={2026},\nurl={https://openreview.net/forum?id=N6MIqE11eF}\n}"},"title":{"value":"Is Bidirectionality Necessary in Mamba for Time Series Forecasting?"},"pdf":{"value":"/pdf/bf428f2b7b338e6732b17e2ed317a213028a6cfe.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"lee|is_bidirectionality_necessary_in_mamba_for_time_series_forecasting"},"authorids":{"value":["~Seunghan_Lee1","~Juri_Hong1","~Kibok_Lee1","~Taeyoung_Park1"]},"authors":{"value":["Seunghan Lee","Juri Hong","Kibok Lee","Taeyoung Park"]}},"version":2},{"content":{"venue":{"value":"CVPR 2026"},"abstract":{"value":"The rapid advances in deep learning have significantly enhanced the accuracy of multimodal 3D human pose estimation (HPE). However, the state-of-the-art (SOTA) HPE pipelines still rely on Transformers, whose quadratic complexity makes real-time processing for long sequences impractical. Mamba addresses this issue through selective state-space modeling, enabling efficient sequence processing without sacrificing representational power. Nevertheless, it struggles to capture complex spatial dependencies in multimodal settings. To bridge this gap, we propose VIMCAN, a hybrid architecture that combines the efficient sequence modeling of Mamba with the spatial reasoning of Cross-Attention, and performs robust visual–inertial fusion and human pose estimation between RGB keypoints and wearable IMU data. By leveraging Mamba’s dynamic parameterization for temporal modeling and Attention for spatial dependency extraction, VIMCAN achieves superior accuracy, with mean per-joint position errors (MPJPE) of 17.2 mm on TotalCapture and 45.3 mm on 3DPW. VIMCAN outperforms prior Transformer-based and other SOTA approaches while supporting real-time inference at over 60 frames per second on consumer-grade hardware. The source code will be available on GitHub."},"_bibtex":{"value":"@inproceedings{\nyang2026vimcan,\ntitle={{VIMCAN}: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network},\nauthor={Zepeng Yang and Junxuan Bai and Hao Li and Ju Dai and Junjun Pan and Yongfeng Yin and Bin Li},\nbooktitle={Conference on Computer Vision and Pattern Recognition 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=2K9YSZX1UP}\n}"},"title":{"value":"VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network"},"pdf":{"value":"https://openaccess.thecvf.com/content/CVPR2026/papers/Yang_VIMCAN_Visual-Inertial_3D_Human_Pose_Estimation_with_Hybrid_Mamba-Cross-Attention_Network_CVPR_2026_paper.pdf"},"venueid":{"value":"thecvf.com/CVPR/2026/Conference"},"paperhash":{"value":"yang|vimcan_visualinertial_3d_human_pose_estimation_with_hybrid_mambacrossattention_network"},"authorids":{"value":["~Zepeng_Yang3","~Junxuan_Bai1","~Hao_Li90","~Ju_Dai3","~Junjun_Pan1","~Yongfeng_Yin1","~Bin_Li14"]},"authors":{"value":["Zepeng Yang","Junxuan Bai","Hao Li","Ju Dai","Junjun Pan","Yongfeng Yin","Bin Li"]}},"tmdate":1789656644063,"pdate":1789656438284,"tcdate":1765220577534,"writers":["thecvf.com/CVPR/2026/Conference","thecvf.com/CVPR/2026/Conference/Submission32357/Authors"],"signatures":["thecvf.com/CVPR/2026/Conference/Submission32357/Authors"],"forum":"2K9YSZX1UP","license":"CC BY 4.0","number":32357,"cdate":1765220577534,"readers":["everyone"],"invitations":["thecvf.com/CVPR/2026/Conference/-/Submission","thecvf.com/CVPR/2026/Conference/Submission32357/-/Full_Submission","thecvf.com/CVPR/2026/Conference/-/Post_Submission","thecvf.com/CVPR/2026/Conference/Submission32357/-/Supplementary_Material","thecvf.com/CVPR/2026/Conference/-/Edit","thecvf.com/CVPR/2026/Conference/-/Compute_Flag"],"mdate":1789656644063,"odate":1789656438284,"domain":"thecvf.com/CVPR/2026/Conference","id":"2K9YSZX1UP","version":2},{"content":{"summary":{"value":"The paper introduces **ProtMamba**, a novel protein language model that is homology-aware but alignment-free, addressing the limitations of traditional multiple sequence alignments (MSAs) in protein modeling. ProtMamba is built on the Mamba architecture, which enables it to handle very long sequences by efficiently processing concatenated homologous protein sequences. The model is trained using a hybrid of autoregressive modeling and Fill-in-the-Middle (FIM) objectives, making it highly versatile for tasks like protein sequence generation and mutational fitness prediction. ProtMamba demonstrates competitive performance on benchmarks like ProteinGym, outperforming similar-sized models in terms of efficiency and predictive accuracy. Additionally, the model excels in sequence generation tasks, producing novel sequences with structural properties comparable to natural proteins."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. How do the authors compute the total number of tokens and FLOPs for training? Can you provide more details on the implementation, such as whether you use any packages for FLOP calculations or how you approximate them?\n\n2. Why did you choose to use a Poisson distribution for masking instead of other distributions?\n\n3. In Figure 12, why does the model, in some DMS experiments, perform better without retrieval (MSA/homology sequences) than with MSA? This is especially surprising given that the model was trained over long contexts of homologous sequences. For instance, in VRPI_BPT7_Tsuboyama_2023_2WNM, there is a significant difference. Could you provide some explanations for this discrepancy?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- ProtMamba is Homology-aware yet alignment-free\n- Mamba architecture adaptation efficiently handle long contexts\n- Explores hybrid training scheme for pLMs\n- ProtMamba strong results on ProteinGym for mutational fitness prediction\n- ProtMamba can generate reasonable sequences given homology context/sequences"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Why use position encoding for ProtMamba? Given the recurrent nature of Mamba, positional information should theoretically be learned implicitly, which is why the original Mamba model does not employ explicit position encodings. The authors claim this is a significant modification, yet they fail to provide any experiments or ablation studies to demonstrate how this change improves model performance. I would like to see a comparison of with positional encoding (PE) vs. without PE, and additionally, a comparison of standard PE (additive) vs. the concatenation method used in ProtMamba. Authors should provide more details about PE implementation and comparison.\n\n2. Although the motivation for using a long-context language model like Mamba is compelling, the paper does not benchmark a vanilla transformer at any context length, which is a significant weakness. Without this comparison, it is difficult to argue whether ProtMamba is the optimal architecture for this task in terms of performance. While it is clear that state space models (SSMs) like Mamba will likely outperform transformers in terms of efficiency, the lack of performance benchmarks makes it hard to assess if ProtMamba achieves the best results. \n\n3. Regarding the ProteinGym benchmark, when incorporating MSA or homology sequences, ProtMamba does not outperform MSA-based models. This challenges the fundamental premise of the paper that a homology-aware, MSA-free model should perform better and eliminate the need for MSAs. The lack of superior performance compared to MSA-based models suggests that ProtMamba's approach may not fully leverage homology information as effectively as MSA based models."}},"nonreaders":[],"tmdate":1731428771142,"tcdate":1730000290740,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6891/Reviewer_biCn"],"signatures":["ICLR.cc/2025/Conference/Submission6891/Reviewer_biCn"],"forum":"BMfHO2lXGe","number":2,"license":"CC BY 4.0","cdate":1730000290740,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6891/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428771142,"domain":"ICLR.cc/2025/Conference","replyto":"BMfHO2lXGe","id":"V3HOLu3SK4","forumContent":{"TLDR":{"value":"ProtMamba is a protein state space model that can handle very long context and enables autoregressive generation and fitness prediction conditioned on a set of homologous sequences."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["proteins","protein sequence","protein language model","computational biology","generative model","protein engineering","protein fitness prediction","protein design"]},"supplementary_material":{"value":"/attachment/dae9053c91a055be470f2724557660f77a9bb545.zip"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Protein design has important implications for drug discovery, personalized medicine, and biotechnology. Models based on multiple sequence alignments efficiently capture the evolutionary information in homologous protein sequences, but multiple sequence alignment construction is imperfect. We present ProtMamba, a homology-aware but alignment-free protein language model based on the Mamba architecture. In contrast with attention-based models, ProtMamba efficiently handles very long context, comprising hundreds of protein sequences. We train ProtMamba on a large dataset of concatenated homologous sequences, using two GPUs. We combine autoregressive modeling and masked language modeling through a fill-in-the-middle training objective. This makes the model adapted to various protein design applications. We demonstrate ProtMamba's usefulness for the generation of novel sequences and for fitness prediction. ProtMamba reaches competitive performance with other protein language models despite its smaller size, which sheds light on the importance of long-context conditioning."},"_bibtex":{"value":"@misc{\nsgarbossa2025protmamba,\ntitle={ProtMamba: a homology-aware but alignment-free protein state space model},\nauthor={Damiano Sgarbossa and Cyril Malbranke and Anne-Florence Bitbol},\nyear={2025},\nurl={https://openreview.net/forum?id=BMfHO2lXGe}\n}"},"title":{"value":"ProtMamba: a homology-aware but alignment-free protein state space model"},"pdf":{"value":"/pdf/b0e6d7eb3e3ff7a0d5496fa74b52528e6ef7a954.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sgarbossa|protmamba_a_homologyaware_but_alignmentfree_protein_state_space_model"},"authorids":{"value":["~Damiano_Sgarbossa1","~Cyril_Malbranke1","~Anne-Florence_Bitbol1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Damiano Sgarbossa","Cyril Malbranke","Anne-Florence Bitbol"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.00762v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"zhang|point_cloud_mamba_point_cloud_learning_via_state_space_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Tao_Zhang:","~Xiangtai_Li1","~Haobo_Yuan1","https://dblp.org/search/pid/api?q=author:Shunping_Ji:","https://dblp.org/search/pid/api?q=author:Shuicheng_Yan:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.00762"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-00762,\n  publtype={informal},\n  author={Tao Zhang and Xiangtai Li and Haobo Yuan and Shunping Ji and Shuicheng Yan},\n  title={Point Cloud Mamba: Point Cloud Learning via State Space Model},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.00762},\n  url={https://doi.org/10.48550/arXiv.2403.00762}\n}\n"},"abstract":{"value":"In this work, for the first time, we demonstrate that Mamba-based point cloud methods can outperform point-based methods. Mamba exhibits strong global modeling capabilities and linear computational complexity, making it highly attractive for point cloud analysis. To enable more effective processing of 3-D point cloud data by Mamba, we propose a novel Consistent Traverse Serialization to convert point clouds into 1-D point sequences while ensuring that neighboring points in the sequence are also spatially adjacent. Consistent Traverse Serialization yields six variants by permuting the order of x, y, and z coordinates, and the synergistic use of these variants aids Mamba in comprehensively observing point cloud data. Furthermore, to assist Mamba in handling point sequences with different orders more effectively, we introduce point prompts to inform Mamba of the sequence's arrangement rules. Finally, we propose positional encoding based on spatial coordinate mapping to inject positional information into point cloud sequences better. Based on these improvements, we construct a point cloud network named Point Cloud Mamba, which combines local and global modeling. Point Cloud Mamba surpasses the SOTA point-based method PointNeXt and achieves new SOTA performance on the ScanObjectNN, ModelNet40, and ShapeNetPart datasets."},"title":{"value":"Point Cloud Mamba: Point Cloud Learning via State Space Model"},"authors":{"value":["Tao Zhang","Xiangtai Li","Haobo Yuan","Shunping Ji","Shuicheng Yan"]}},"tmdate":1715777713999,"pdate":1704067200000,"tcdate":1714208338362,"writers":["~"],"signatures":["~Xiangtai_Li1"],"forum":"m4sODCx36R","license":"CC BY-SA 4.0","number":4613,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1715777713999,"domain":"DBLP.org","id":"m4sODCx36R","version":2},{"content":{"summary":{"value":"The paper introduces the concept of matrix mixer and sequence alignment for explaining recent sequence models including Transformer, linear transformer, and Mamba. It also proposes a quasiseparable matrix mixer (Hydra) as an alternative to the bidirectional SSM model. The experiments show that the quasiseparable matrix structure performs better than others including low rank (linear attention), attention, and dense. Also, Hydra outperforms other naive bidirectional approaches and some baselines for masked language modeling and image classification."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Is there any speed benefit with the quasiseparable bidirectional model compared to the naive bidirectional approaches? It would be good to include some speed comparisons in Table 3."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The idea of designing a structure matrix for a bidirectional case is reasonable and effective while the implementation is simple.  \n2. The explanation of different matrix mixers, their relation to other methods, and the advantages/disadvantages are clear. This is very informative. \n3. The ablation study shows how different matrix mixers perform and the benefit of quasiseparable matrix structure."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The presentation of the paper needs to be improved. \n    1. It contains unnecessary details. The main contributions of the paper are matrix mixer, sequence alignment, and Hydra, but the focus diverges throughout the paper, especially in Section 2. Sections 2.3 and 2.4 can be moved as a side note after introducing Hydra.\n    2. The purpose of introducing Cauchy and Vandermonde matrix mixers is unclear. How can this be useful or helpful in some ways? It's not clearly explained or shown in the experiments. \n\nOverall, the current representation makes it difficult for the readers to understand the core contributions. \n\n2. The experiments shown in the paper are limited. The main comparison table (Table 4) does not include any recent transformers or mamba-based models. Also, the experiments include one example of non-causal language modeling (masked language modeling) and image classification as applications for Hydra (bidirectional model settings). Could authors show at least one more example application that Hydra can be useful and better than other comparable models?"},"limitations":{"value":"The limitations are discussed and are reasonable."}},"nonreaders":[],"tmdate":1730878827010,"tcdate":1720936363922,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission3122/Reviewer_onHY"],"signatures":["NeurIPS.cc/2024/Conference/Submission3122/Reviewer_onHY"],"forum":"preo49P1VY","number":3,"license":"CC BY 4.0","cdate":1720936363922,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission3122/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878827010,"domain":"NeurIPS.cc/2024/Conference","replyto":"preo49P1VY","id":"eKWfV10hCC","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Deep learning","sequence models","state space models","mamba"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"A wide array of sequence models are built on a framework modeled after Transformers, comprising alternating sequence mixer and channel mixer layers.  This paper studies a unifying *matrix mixer* view of sequence mixers that can be conceptualized as a linear map on the input sequence. This framework encompasses a broad range of well-known sequence models, including the self-attention of Transformers as well as recent strong alternatives such as structured state space models (SSMs), and allows understanding downstream characteristics such as efficiency and expressivity through properties of their structured matrix class. We identify a key axis of matrix parameterizations termed *sequence alignment*, which increases the flexibility and performance of matrix mixers, providing insights into the strong performance of Transformers and recent SSMs such as Mamba. Furthermore, the matrix mixer framework offers a systematic approach to developing sequence mixers with desired properties, allowing us to develop several new sub-quadratic sequence models. In particular, we propose a natural bidirectional extension of the Mamba model (**Hydra**), parameterized as a *quasiseparable matrix mixer*, which demonstrates superior performance over other sequence models including Transformers on non-causal tasks. As a drop-in replacement for attention layers, \\name outperforms BERT by 0.8 points on the GLUE benchmark and ViT by 2% Top-1 accuracy on ImageNet."},"_bibtex":{"value":"@inproceedings{\nhwang2024hydra,\ntitle={Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers},\nauthor={Sukjun Hwang and Aakash Lahoti and Ratish Puduppully and Tri Dao and Albert Gu},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=preo49P1VY}\n}"},"title":{"value":"Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers"},"pdf":{"value":"/pdf/975e174ed724f910ffb63bad51c2578e145b4c50.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"hwang|hydra_bidirectional_state_space_models_through_generalized_matrix_mixers"},"authorids":{"value":["~Sukjun_Hwang1","~Aakash_Lahoti1","~Ratish_Puduppully1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Sukjun Hwang","Aakash Lahoti","Ratish Puduppully","Tri Dao","Albert Gu"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Mamba","State Space Model","Long-term Dependencies"]},"supplementary_material":{"value":"/attachment/e45ab80b2ed006f687bedefcd18e8faee2e7b216.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"State space models (SSMs) have attracted attention for their memory and computational efficiency in modeling long sequences. In particular, Mamba enables modeling of time-varying dynamics through the selective SSM (S6), an input-dependent mechanism, while preserving parallel computation.\nDespite its empirical success, compared to input-independent SSMs such as S4 and S5, Mamba may underperform on long-sequence tasks that require preserving information regardless of subsequent inputs.\nHere we analyze how input variability affects long-term dependencies in the S6 recurrence.\nWe show that larger input variance can make it difficult to preserve previous information over long sequences, suggesting that the discretized state matrix should contain values close to $1$ at initialization.\nWe further show that under the standard Mamba initialization, increasing the state size shifts the discretized state matrix distribution toward values close to $0$, without increasing the proportion of values close to $1$.\nThese findings lead to a simple initialization approach based on the Gumbel-Softmax trick, which addresses the issues identified in our analysis by modifying the initialization distribution without any architectural changes.\nWe empirically validate the derived initialization properties on synthetic and LRA tasks, showing that the proposed initialization improves performance on tasks that require preserving prior information over long sequences regardless of subsequent inputs, while maintaining comparable performance on context-dependent tasks. \nThese results help explain why input variability can degrade long-term dependencies in selective SSMs."},"_bibtex":{"value":"@inproceedings{\nanonymous2026effects,\ntitle={Effects of Input Variability on Long-Term Dependencies in Selective State Space Models},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=1mANJJuebs},\nnote={under review}\n}"},"title":{"value":"Effects of Input Variability on Long-Term Dependencies in Selective State Space Models"},"pdf":{"value":"/pdf/ca966bf2241d87cfc838a5da2213229b899c0678.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791233364747,"tcdate":1789739497480,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission43562/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission43562/Authors"],"forum":"1mANJJuebs","license":"CC BY 4.0","number":43562,"cdate":1789739497480,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission43562/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791233364747,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"1mANJJuebs","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2509.23779v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"jiang|trained_mamba_emulates_online_gradient_descent_in_incontext_linear_regression"},"authorids":{"value":["~Jiarui_Jiang1","https://dblp.org/search/pid/api?q=author:Wei_Huang_0034:","~Miao_Zhang4","https://dblp.org/search/pid/api?q=author:Taiji_Suzuki:",""]},"html":{"value":"https://doi.org/10.48550/arXiv.2509.23779"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2509-23779,\n  publtype={informal},\n  author={Jiarui Jiang and Wei Huang and Miao Zhang and Taiji Suzuki and Liqiang Nie},\n  title={Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression},\n  year={2025},\n  month={September},\n  cdate={1756684800000},\n  journal={CoRR},\n  volume={abs/2509.23779},\n  url={https://doi.org/10.48550/arXiv.2509.23779}\n}\n"},"abstract":{"value":"State-space models (SSMs), particularly Mamba, emerge as an efficient Transformer alternative with linear complexity for long-sequence modeling. Recent empirical works demonstrate Mamba's in-context learning (ICL) capabilities competitive with Transformers, a critical capacity for large foundation models. However, theoretical understanding of Mamba's ICL remains limited, restricting deeper insights into its underlying mechanisms. Even fundamental tasks such as linear regression ICL, widely studied as a standard theoretical benchmark for Transformers, have not been thoroughly analyzed in the context of Mamba. To address this gap, we study the training dynamics of Mamba on the linear regression ICL task. By developing novel techniques tackling non-convex optimization with gradient descent related to Mamba's structure, we establish an exponential convergence rate to ICL solution, and derive a loss bound that is comparable to Transformer's. Importantly, our results reveal that Mamba can perform a variant of \\textit{online gradient descent} to learn the latent function in context. This mechanism is different from that of Transformer, which is typically understood to achieve ICL through gradient descent emulation. The theoretical results are verified by experimental simulation."},"title":{"value":"Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression"},"authors":{"value":["Jiarui Jiang","Wei Huang","Miao Zhang","Taiji Suzuki","Liqiang Nie"]}},"tmdate":1784598737536,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2509-23779"],"tcdate":1768440124310,"writers":["~"],"signatures":["~Liqiang_Nie2"],"forum":"WxzJGXtzQZ","license":"CC BY-SA 4.0","number":753441,"cdate":1756684800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784598737536,"domain":"DBLP.org","id":"WxzJGXtzQZ","version":2},{"content":{"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["state space model","Mamba","in-context learning","training dynamics"]},"supplementary_material":{"value":"/attachment/a3a77917b54ee06a2a2681ffd5e6c91a5e35108b.zip"},"primary_area":{"value":"theory"},"abstract":{"value":"State-space models (SSMs), particularly Mamba, emerge as an efficient Transformer alternative with linear complexity for long-sequence modeling. Recent empirical works demonstrate Mamba's in-context learning (ICL) capabilities competitive with Transformers, a critical capacity for large foundation models. However, theoretical understanding of Mamba’s ICL remains limited, restricting deeper insights into its underlying mechanisms. Even fundamental tasks such as linear regression ICL, widely studied as a standard theoretical benchmark for Transformers, have not been thoroughly analyzed in the context of Mamba. To address this gap, we study the training dynamics of Mamba on the linear regression ICL task. By developing novel techniques tackling non-convex optimization with gradient descent related to Mamba's structure, we establish an exponential convergence rate to ICL solution, and derive a loss bound that is comparable to Transformer's. Importantly, our results reveal that Mamba can perform a variant of \\textit{online gradient descent} to learn the latent function in context. This mechanism is different from that of Transformer, which is typically understood to achieve ICL through gradient descent emulation. The theoretical results are verified by experimental simulation."},"_bibtex":{"value":"@inproceedings{\njiang2025trained,\ntitle={Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression},\nauthor={Jiarui Jiang and Wei Huang and Miao Zhang and Taiji Suzuki and Liqiang Nie},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=JxkZyedsdS}\n}"},"title":{"value":"Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression"},"pdf":{"value":"/pdf/3972218fd08b9aa6ea3b67a36b4e696d6a265ca6.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"jiang|trained_mamba_emulates_online_gradient_descent_in_incontext_linear_regression"},"authorids":{"value":["~Jiarui_Jiang1","~Wei_Huang6","~Miao_Zhang4","~Taiji_Suzuki1","~Liqiang_Nie2"]},"authors":{"value":["Jiarui Jiang","Wei Huang","Miao Zhang","Taiji Suzuki","Liqiang Nie"]}},"tmdate":1783627358184,"pdate":1758217075274,"tcdate":1746878438050,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission16014/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission16014/Authors"],"forum":"JxkZyedsdS","license":"CC BY 4.0","number":16014,"cdate":1746878438050,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission16014/-/Full_Submission","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission16014/-/Camera_Ready_Revision"],"mdate":1783627358184,"odate":1761704926340,"domain":"NeurIPS.cc/2025/Conference","id":"JxkZyedsdS","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.10854v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"liu|mambads_nearsurface_meteorological_field_downscaling_with_topography_constrained_selective_state_space_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zili_Liu:","https://dblp.org/search/pid/api?q=author:Hao_Chen_0045:","~Lei_Bai2","https://dblp.org/search/pid/api?q=author:Wenyuan_Li_0002:","https://dblp.org/search/pid/api?q=author:Wanli_Ouyang:","https://dblp.org/search/pid/api?q=author:Zhengxia_Zou:","~Zhenwei_Shi2"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.10854"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-10854,\n  publtype={informal},\n  author={Zili Liu and Hao Chen and Lei Bai and Wenyuan Li and Wanli Ouyang and Zhengxia Zou and Zhenwei Shi},\n  title={MambaDS: Near-Surface Meteorological Field Downscaling with Topography Constrained Selective State Space Modeling},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.10854},\n  url={https://doi.org/10.48550/arXiv.2408.10854}\n}\n"},"abstract":{"value":"In an era of frequent extreme weather and global warming, obtaining precise, fine-grained near-surface weather forecasts is increasingly essential for human activities. Downscaling (DS), a crucial task in meteorological forecasting, enables the reconstruction of high-resolution meteorological states for target regions from global-scale forecast results. Previous downscaling methods, inspired by CNN and Transformer-based super-resolution models, lacked tailored designs for meteorology and encountered structural limitations. Notably, they failed to efficiently integrate topography, a crucial prior in the downscaling process. In this paper, we address these limitations by pioneering the selective state space model into the meteorological field downscaling and propose a novel model called MambaDS. This model enhances the utilization of multivariable correlations and topography information, unique challenges in the downscaling process while retaining the advantages of Mamba in long-range dependency modeling and linear computational complexity. Through extensive experiments in both China mainland and the continental United States (CONUS), we validated that our proposed MambaDS achieves state-of-the-art results in three different types of meteorological field downscaling settings. We will release the code subsequently."},"title":{"value":"MambaDS: Near-Surface Meteorological Field Downscaling with Topography Constrained Selective State Space Modeling"},"authors":{"value":["Zili Liu","Hao Chen","Lei Bai","Wenyuan Li","Wanli Ouyang","Zhengxia Zou","Zhenwei Shi"]}},"tmdate":1731634956740,"pdate":1704067200000,"tcdate":1731373413363,"writers":["~"],"signatures":["~Zhenwei_Shi2"],"forum":"4mfFkjuWZi","license":"CC BY-SA 4.0","number":182409,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1731634956740,"domain":"DBLP.org","id":"4mfFkjuWZi","version":2},{"content":{"summary":{"value":"This paper primarily discusses the similarities and differences between the Mamba model and the Linear Attention Transformer, and conducts an in-depth analysis of the key factors contributing to Mamba's success in visual tasks. The paper elaborates on six major design differences of Mamba compared to the Linear Attention Transformer: input gate, forget gate, shortcut connection, attention-free normalization, single-head attention, and modified block design. For each of these designs, the author meticulously analyzes their advantages and disadvantages, and evaluates their applicability in visual tasks.\n\nThrough experimental research, the paper emphasizes that the forget gate and block design are the core contributors to Mamba's success. Based on these findings, the author proposes a model called Mamba-Like Linear Attention (MLLA), which integrates the advantages of the forget gate and block design into the linear attention framework. Experimental results show that MLLA outperforms various visual Mamba models in image classification and high-resolution dense prediction tasks, while maintaining parallel computing capabilities and fast inference speed."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. How many GPUs are used to perform this research, including the training cost and ablation experiments cost?\n2. Why Fig. 1 is placed on page 2 but cited on page 4?\n3. What are the results of Mamba + Swin?"},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper provides a new perspective for understanding Mamba's success by thoroughly analyzing the similarities and differences between the Mamba model and the Linear Attention Transformer. By rephrasing formulas, the paper considers Mamba as a variant of the Linear Attention Transformer and clearly points out Mamba's six design features, including input gate, forget gate, shortcut connection, attention-free normalization, single-head attention, and modified block design.\n\n2. Each design aspect is meticulously analyzed, and its advantages and disadvantages in visual tasks are evaluated. Experimental results verify that the forget gate and block design significantly contribute to Mamba's performance improvement, while other designs either have minimal marginal contributions or may potentially harm model performance.\n\n3.  The Mamba-Like Linear Attention (MLLA) model is proposed, combining the advantages of the forget gate and block design in linear attention. The MLLA model outperforms existing Mamba models in image classification and high-resolution dense prediction tasks while maintaining computational parallelization and high-speed inference capabilities."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"There are some minor issues:\n\n1. In the final MLLA model, there is no SSM but a forget gate. The forget gate was originally proposed in RNN and LSTM. Thus calling it a Mamba-like model seems to be improper.\n\n2. Previous \"Demystify\" papers are not cited in the paper, e.g., [a, b, c]. The final architecture MLLA follows the macro structure of Swin and [a,b,c] all show that the macro structure is very important but the local blocks are not so important. The paper should build a connection with the previous \"Demystify\" papers and tell us what are the most effective parts?\n\n[a] On the Connection between Local Attention and Dynamic Depth-wise Convolution, ICLR 2022\n[b] What Makes for Hierarchical Vision Transformer? DOI: 10.1109/TPAMI.2023.3282019\n[c] Demystify Transformers & Convolutions in Modern Image Deep Networks, arXiv:2211.05781"},"limitations":{"value":"Nope."}},"nonreaders":[],"tmdate":1730878967845,"tcdate":1720882405780,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission4979/Reviewer_QFdu"],"signatures":["NeurIPS.cc/2024/Conference/Submission4979/Reviewer_QFdu"],"forum":"LvJ1R88KAk","number":4,"license":"CC BY 4.0","cdate":1720882405780,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission4979/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878967845,"domain":"NeurIPS.cc/2024/Conference","replyto":"LvJ1R88KAk","id":"6EQc6OVDVz","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","Linear Attention Transformer"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba’s success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba’s success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-\nInspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https://github.com/LeapLabTHU/MLLA."},"_bibtex":{"value":"@inproceedings{\nhan2024demystify,\ntitle={Demystify Mamba in Vision: A Linear Attention Perspective},\nauthor={Dongchen Han and Ziyi Wang and Zhuofan Xia and Yizeng Han and Yifan Pu and Chunjiang Ge and Jun Song and Shiji Song and Bo Zheng and Gao Huang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=LvJ1R88KAk}\n}"},"title":{"value":"Demystify Mamba in Vision: A Linear Attention Perspective"},"pdf":{"value":"/pdf/419f5b34c2a7aefa14ac7e991c160ab502bd73f5.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"han|demystify_mamba_in_vision_a_linear_attention_perspective"},"authorids":{"value":["~Dongchen_Han1","~Ziyi_Wang21","~Zhuofan_Xia2","~Yizeng_Han1","~Yifan_Pu1","~Chunjiang_Ge1","~Jun_Song5","~Shiji_Song1","~Bo_Zheng5","~Gao_Huang1"]},"authors":{"value":["Dongchen Han","Ziyi Wang","Zhuofan Xia","Yizeng Han","Yifan Pu","Chunjiang Ge","Jun Song","Shiji Song","Bo Zheng","Gao Huang"]}},"version":2},{"content":{"TLDR":{"value":"We introduce MambaSL, a minimally redesigned single-layer Mamba that achieves state-of-the-art accuracy on the UEA30 benchmark, with reproducible evaluation covering all baselines."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["time series classification","single-layer Mamba","modular selective SSM","multi-head adaptive pooling","skip connection"]},"supplementary_material":{"value":"/attachment/dfc093493a30882d173d5eedf1107e0c42d4d9fc.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Despite recent advances in state space models (SSMs) such as Mamba across various sequence domains, research on their standalone capacity for time series classification (TSC) has remained limited. \nWe propose MambaSL, a framework that minimally redesigns the selective SSM and projection layers of a single-layer Mamba, guided by four TSC-specific hypotheses.\nTo address benchmarking limitations—restricted configurations, partial University of East Anglia (UEA) dataset coverage, and insufficiently reproducible setups—we re-evaluate 20 strong baselines across all 30 UEA datasets under a unified protocol. \nAs a result, MambaSL achieves state-of-the-art performance with statistically significant average improvements, while ensuring reproducibility via public checkpoints for all evaluated models.\nTogether with visualizations, these results demonstrate the potential of Mamba-based architectures as a TSC backbone."},"_bibtex":{"value":"@inproceedings{\njung2026mambasl,\ntitle={Mamba{SL}: Exploring Single-Layer Mamba for Time Series Classification},\nauthor={Yoo-Min Jung and Leekyung Kim},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=YDl4vqQqGP}\n}"},"title":{"value":"MambaSL: Exploring Single-Layer Mamba for Time Series Classification"},"pdf":{"value":"/pdf/36a22e78ae778d70b99d6fc145e8d5a9be7e0c1d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jung|mambasl_exploring_singlelayer_mamba_for_time_series_classification"},"authorids":{"value":["~Yoo-Min_Jung1","~Leekyung_Kim1"]},"authors":{"value":["Yoo-Min Jung","Leekyung Kim"]}},"tmdate":1777660838533,"pdate":1769436061955,"tcdate":1758193895922,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11231/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission11231/Authors"],"forum":"YDl4vqQqGP","license":"CC BY 4.0","number":11231,"cdate":1758193895922,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission11231/-/Full_Submission","ICLR.cc/2026/Conference/Submission11231/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission11231/-/Camera_Ready_Revision"],"mdate":1777660838533,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"YDl4vqQqGP","version":2},{"content":{"TLDR":{"value":"We propose Causal Mamba, an efficient framework for longitudinal counterfactual estimation that combines a multi-stream Mamba backbone with multi-task representation balancing to model heterogeneous temporal dynamics while mitigating over-balancing."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Causal Inference","Longitudinal Treatment Effect Estimation","Mamba","Representation Learning","Multi-Task Learning"]},"primary_area":{"value":"causal reasoning"},"abstract":{"value":"Accurately and efficiently estimating counterfactual outcomes over time from observational data is essential\nin various applications. However, existing methods suffer from computational inefficiency and over-balancing, limiting their practical use.\nIn this paper, we propose Causal Mamba (CM),\na novel end-to-end framework for efficient counterfactual estimation over time.\nIts GCSI-Mamba backbone introduces Gated Cross-Sequence Interaction (GCSI) and follows an interact-then-propagate paradigm, \nfirst modeling dependencies among treatment, covariate, and outcome streams \nand then using stream specific Mamba branches to capture their respective temporal dynamics, \nthereby preserving stream heterogeneity while retaining linear complexity with respect to sequence length.\nThe Multi-Task Collaboration (MTC) module further coordinates outcome prediction, treatment domain confusion, \nand covariate reconstruction to reduce treatment specific bias while preserving relevant information for counterfactual prediction, \navoiding information loss caused by over-balancing.\nExperimental results on synthetic and real-world datasets demonstrate\nthat CM not only outperforms baselines but also achieves greater runtime efficiency."},"_bibtex":{"value":"@inproceedings{\nanonymous2026causal,\ntitle={Causal Mamba for Estimating Counterfactual Outcomes over Time},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=MI3qyTCJp5},\nnote={under review}\n}"},"title":{"value":"Causal Mamba for Estimating Counterfactual Outcomes over Time"},"pdf":{"value":"/pdf/e2eb85b9b5a307742514b5e1c1f6ff858db037f6.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230228893,"tcdate":1789545460818,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission24041/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission24041/Authors"],"forum":"MI3qyTCJp5","license":"CC BY 4.0","number":24041,"cdate":1789545460818,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission24041/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791230228893,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"MI3qyTCJp5","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.06966v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"li|dygmamba_continuous_state_space_modeling_on_dynamic_graphs"},"authorids":{"value":["~Dongyuan_Li1","~Shiyin_Tan1","https://dblp.org/search/pid/api?q=author:Ying_Zhang_0065:","~Ming_Jin3","~Shirui_Pan1","https://dblp.org/search/pid/api?q=author:Manabu_Okumura:","~Renhe_Jiang1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.06966"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-06966,\n  publtype={informal},\n  author={Dongyuan Li and Shiyin Tan and Ying Zhang and Ming Jin and Shirui Pan and Manabu Okumura and Renhe Jiang},\n  title={DyG-Mamba: Continuous State Space Modeling on Dynamic Graphs},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.06966},\n  url={https://doi.org/10.48550/arXiv.2408.06966}\n}\n"},"abstract":{"value":"Dynamic graph learning aims to uncover evolutionary laws in real-world systems, enabling accurate social recommendation (link prediction) or early detection of cancer cells (classification). Inspired by the success of state space models, e.g., Mamba, for efficiently capturing long-term dependencies in language modeling, we propose DyG-Mamba, a new continuous state space model (SSM) for dynamic graph learning. Specifically, we first found that using inputs as control signals for SSM is not suitable for continuous-time dynamic network data with irregular sampling intervals, resulting in models being insensitive to time information and lacking generalization properties. Drawing inspiration from the Ebbinghaus forgetting curve, which suggests that memory of past events is strongly correlated with time intervals rather than specific details of the events themselves, we directly utilize irregular time spans as control signals for SSM to achieve significant robustness and generalization. Through exhaustive experiments on 12 datasets for dynamic link prediction and dynamic node classification tasks, we found that DyG-Mamba achieves state-of-the-art performance on most of the datasets, while also demonstrating significantly improved computation and memory efficiency."},"title":{"value":"DyG-Mamba: Continuous State Space Modeling on Dynamic Graphs"},"authors":{"value":["Dongyuan Li","Shiyin Tan","Ying Zhang","Ming Jin","Shirui Pan","Manabu Okumura","Renhe Jiang"]}},"tmdate":1768478347728,"pdate":1704067200000,"tcdate":1727499425198,"writers":["~"],"signatures":["~Renhe_Jiang1"],"forum":"GDWFNQY04C","license":"CC BY-SA 4.0","number":99050,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1768478347728,"domain":"DBLP.org","id":"GDWFNQY04C","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2507.01535v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"liu|trackingmim_efficient_mambainmamba_serialization_for_realtime_uav_object_tracking"},"authorids":{"value":["~Bingxi_Liu1","https://dblp.org/search/pid/api?q=author:Calvin_Chen:","https://dblp.org/search/pid/api?q=author:Junhao_Li:","https://dblp.org/search/pid/api?q=author:Guyang_Yu:","https://dblp.org/search/pid/api?q=author:Haoqian_Song:","https://dblp.org/search/pid/api?q=author:Xuchen_Liu:","https://dblp.org/search/pid/api?q=author:Jinqiang_Cui:","https://dblp.org/search/pid/api?q=author:Hong_Zhang_0013:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2507.01535"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2507-01535,\n  publtype={informal},\n  author={Bingxi Liu and Calvin Chen and Junhao Li and Guyang Yu and Haoqian Song and Xuchen Liu and Jinqiang Cui and Hong Zhang},\n  title={TrackingMiM: Efficient Mamba-in-Mamba Serialization for Real-time UAV Object Tracking},\n  year={2025},\n  month={July},\n  cdate={1751328000000},\n  journal={CoRR},\n  volume={abs/2507.01535},\n  url={https://doi.org/10.48550/arXiv.2507.01535}\n}\n"},"abstract":{"value":"The Vision Transformer (ViT) model has long struggled with the challenge of quadratic complexity, a limitation that becomes especially critical in unmanned aerial vehicle (UAV) tracking systems, where data must be processed in real time. In this study, we explore the recently proposed State-Space Model, Mamba, leveraging its computational efficiency and capability for long-sequence modeling to effectively process dense image sequences in tracking tasks. First, we highlight the issue of temporal inconsistency in existing Mamba-based methods, specifically the failure to account for temporal continuity in the Mamba scanning mechanism. Secondly, building upon this insight,we propose TrackingMiM, a Mamba-in-Mamba architecture, a minimal-computation burden model for handling image sequence of tracking problem. In our framework, the mamba scan is performed in a nested way while independently process temporal and spatial coherent patch tokens. While the template frame is encoded as query token and utilized for tracking in every scan. Extensive experiments conducted on five UAV tracking benchmarks confirm that the proposed TrackingMiM achieves state-of-the-art precision while offering noticeable higher speed in UAV tracking."},"title":{"value":"TrackingMiM: Efficient Mamba-in-Mamba Serialization for Real-time UAV Object Tracking"},"authors":{"value":["Bingxi Liu","Calvin Chen","Junhao Li","Guyang Yu","Haoqian Song","Xuchen Liu","Jinqiang Cui","Hong Zhang"]}},"tmdate":1769151461863,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2507-01535"],"tcdate":1769151455640,"writers":["~"],"signatures":["~Bingxi_Liu1"],"forum":"V4YfXVGaFM","license":"CC BY-SA 4.0","number":790860,"cdate":1751328000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769151461863,"domain":"DBLP.org","id":"V4YfXVGaFM","version":2},{"content":{"comment":{"value":"2. It's unclear why we should aggregate the data and thus lose information when the raw data is itself quite illustrative. \nThis allows us to show how our initialization changes the success rate of learning to copy. For the 24-layer Mambas in the figure, the success rate is not perfect (it's much higher for shallower models). It's unclear how we would display more than four dimensions in one plot (plotting the interaction of all four components is already somewhat cumbersome). If you think it would be helpful, we can plot averages and error bars over lower-opacity data points, but we do find it illustrative to actually present the raw data.\n\n3. We'll put the legend below the plot, this is a good point.\n\n4. The purpose of our paper was to address the copying problems noted by [0], and we think that an application to general language modeling/pretraining is outside the scope of our study, which was meant to be a scientific study of the capacity of Mamba/SSMs and a further study of the concept of mimetic initialization. \n\n5. Yes, we're using the short conv block before the SSM block. We just replaced the state space layer with linear attention. We'll clarify this in the paper. We can also add some more intuition to the paper, but the idea is just that Mamba is more expressive than vanilla linear attention: it has more parameters for gating and discretization. Vanilla linear attention has a fixed causal mask, while Mamba has a learnable one. Just because we initialize to be close to linear attention does not mean that the layer remains linear attention after training.\n\n6. We provide some GLA results above, but previous work on studying copying in SSMs focused on Mamba (and we in fact study two different Mamba architectures), and we were unable to attain good copying results using GLA alone. Most likely the particular parameterization of Mamba just happens to make learning this task easier, but there is undeniable similarity with GLA. Note that S6 is actually just the Mamba 1 state space layer, so we did study that. We believe Mamba 1 & 2 to be the most popular SSM architectures and much previous work has focused on them [0,2], most likely because it works particularly well [2] compared to others. Further, if an architecture is in fact the same \"class\" as Mamba 1/2, then our technique and the accompanying intuition extends very easily. We'll make this clearer in the paper.\n\n\n7. Quite a lot, see Figure 1: only one layer retains the linear attention prior in this example. \n\n8. The init has a large effect size in basically all regimes studied, and the difference is more a matter of which settings Mamba most struggles to copy/recall. For example, it's better for larger vocab sizes because Mamba especially struggles to copy in this setting and our init almost entirely fixes the issue (Fig 6). The most significant effect of our init is to allow Mamba 2's copying capacity to actually grow with the state size, which doesn't happen otherwise (Figure 7b). Without our init, generalization ability is quite limited. And this is for Mamba 2, which is already much better at copying than Mamba 1. For Mamba 1, our init makes all the difference in essentially all settings: without it, learning to copy is just extremely hard/impossible (e.g., see Fig 6, lefthand side).\n\n9. Yes, it's simply a different synthetic task than copying which has been studied by the Mamba 2 paper (Fig. 8) [1]. The copying ability of Mamba has been studied in particular by [0], and the MQAR ability by [2], which is largely why we study both. MQAR synthetic performance has been noted to be correlated with larger-scale performance [3]. It's a distinct task, but one of the main synthetic tasks that seems important in the literature.\n\n\n[0] Jelassi et al. \"Repeat After Me: Transformers are Better than State Space Models at Copying\" (2024).\n\n[1] Dao & Gu. \"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality\" (2024).\n\n[2] Arora et al. \"Simple linear attention language models balance the recall-throughput tradeoff\" (2024).\n\n[3] Poli et al. \"Mechanistic Design and Scaling of Hybrid Architectures\" (2024).\n\n[4] Gu & Dao. \"Mamba: Linear-Time Sequence Modeling with Selective State Spaces\" (2024)."},"title":{"value":"Response 3/3"}},"tmdate":1732209418985,"tcdate":1732209418985,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11529/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission11529/Authors"],"forum":"iVy7aRMb0K","number":11,"license":"CC BY 4.0","cdate":1732209418985,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11529/-/Official_Comment"],"mdate":1732209418985,"domain":"ICLR.cc/2025/Conference","replyto":"vodQJgVBbb","id":"KU7Y7foPOA","forumContent":{"TLDR":{"value":"Mamba learns to recall more easily when we initialize it to be more like linear attention"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","SSM","sequence models","language models","initialization","linear attention"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with respect to their input sequence length. But in practice, state space models have fairly large state sizes, and we conjecture that they should be able to perform much better at these tasks than previously reported. We investigate whether their poor copying and recall performance could be due in part to training difficulties rather than fundamental capacity constraints. Based on observations of their \"attention'' maps, we propose a structured initialization technique that allows state space layers to more readily mimic attention. Across a variety of architecture settings, our initialization makes it substantially easier for Mamba to learn to copy and do associative recall from scratch."},"_bibtex":{"value":"@misc{\ntrockman2025mimetic,\ntitle={Mimetic Initialization Helps State Space Models Learn to Recall},\nauthor={Asher Trockman and Hrayr Harutyunyan and J Zico Kolter and Sanjiv Kumar and Srinadh Bhojanapalli},\nyear={2025},\nurl={https://openreview.net/forum?id=iVy7aRMb0K}\n}"},"title":{"value":"Mimetic Initialization Helps State Space Models Learn to Recall"},"pdf":{"value":"/pdf/5b9624e012b1653fd84db25266eba2a30fcf0588.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"trockman|mimetic_initialization_helps_state_space_models_learn_to_recall"},"authorids":{"value":["~Asher_Trockman1","~Hrayr_Harutyunyan1","~J_Zico_Kolter1","~Sanjiv_Kumar1","~Srinadh_Bhojanapalli1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Asher Trockman","Hrayr Harutyunyan","J Zico Kolter","Sanjiv Kumar","Srinadh Bhojanapalli"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a hardware-efficient algorithm for training linear transformers with a delta update (DeltaNet; SMS21). This architecture has an attention formulation that prevents the direct application of chunk-wise parallel algorithms for computing its output. To address this issue, the authors introduce a re-parameterization of DeltaNet as a matrix-valued RNN whose recurrence is given by a generalized Householder transformation. This enables the use of WY representation which is memory efficient and eliminates the need to materialize the hidden state matrices. Experiments on synthetic benchmarks and language modeling tasks shows competitive performance compared to strong baselines (Mamba, GLA) and faster speed than the original Deltanet implementation."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- Q1: What is the effect of chunk size in the chunk-wise parallel algorithm for DeltaNet? Varying the chunk size $C$ and showing its effect in efficiency would be interesting to explore. \n- Q2: The chunk-level hidden states $S_{[t]}$'s are discarded to save memory. From Eq. 7, it seems that their computation depends on the previous hidden states $S_{[t-1]}$'s. Are these kept in memory for the re-computation in the backward pass?  \n- Q3: GLA with convolution performs worse than w/o convolution with the larger model size. Do you expect this to be the case for DeltaNet as well? It would be good to add this result if possible. \n\nMinor:\n- In Table 2, is the L1/L2 norm referring to the normalization of queries and keys? Please specify. \n- In Eq.1, why is this equation showing the state $S_{[t+1]}$ instead of $S_{[t]}$? The latter is used in Eq. 2. Same for Eq. 7.  \n- l172: stable -> stabilize\n- l213-214: we -> we follow\n- l321: vallina -> vanilla"},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"- The paper is well motivated and situated with respect to prior work. It provides sufficient background for linear transformers, demonstrates great scholarship in crediting prior work, and has a clear exposition of the proposed idea. In addition, it presents an informative overview that compares the formulations of recent linear transformers that highlights their differences.  \n- Proposes an efficient algorithm for training linear transformers with the delta update which is a competitive variant. The re-parameterization is non-obvious and leverages WY representation for Householder matrices in a novel way. Previously, this architecture could not be easily scaled to larger models and datasets with a recurrent formulation. In addition, it introduces two competitive hybrid methods based on DeltaNet that leverage local and global full attention. \n - Demonstrates the effectiveness of the proposed approach on two synthetic benchmarks and eleven language modeling and understanding tasks compared to strong baselines such as Mamba and GLA. The results are consistent, have a good coverage, and are important for the researchers working on efficient transformers. \n- The experiments are thorough and have convincing settings, namely all the variants are trained from scratch with the same configurations, there are ablations to justify the design choices, and the experimental reporting is very detailed."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- W1. In terms of scale, the model explores two different architectures of increasing size up to 1.3B parameters. Even though this size is considerable, it is still relatively small compared to the LLMs that are widely used such as Llama, Mistral (7B+ size). There is always the question of whether the quality is maintained with further model increase.\n- W2. The improved results compared to Mamba and GLA make use of additional architectural components: convolution and local/global attention, without them the results are comparable to the other models."},"limitations":{"value":"Yes, they have."}},"nonreaders":[],"tmdate":1730879511845,"tcdate":1720856438167,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission11970/Reviewer_eUyw"],"signatures":["NeurIPS.cc/2024/Conference/Submission11970/Reviewer_eUyw"],"forum":"y8Rm4VNRPH","number":3,"license":"CC BY 4.0","cdate":1720856438167,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission11970/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879511845,"domain":"NeurIPS.cc/2024/Conference","replyto":"y8Rm4VNRPH","id":"hQNYHP50sF","forumContent":{"TLDR":{"value":"DeltaNet, i.e., Linear Transformer with delta rule, suffers from slow recurrent training. This work develops a novel algorithm that enables the parallelization of training over sequence length, allowing for large-scale experiments"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["linear transformer","recurrent neural network","hardware-aware algorithm","state space model","sequence modeling","efficient training"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Transformers with linear attention (i.e., linear transformers) and state-space models have recently been suggested as a viable linear-time alternative to transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive variants of linear transformers which replace the additive update in linear transformers with the delta rule (DeltaNet) have been found to be more effective at associative recall, existing algorithms for training such models do not parallelize over sequence length and are thus inefficient to train on modern hardware. This work describes a hardware-efficient algorithm for training linear transformers with the delta rule, which exploits a memory-efficient representation for computing products of Householder matrices. This algorithm allows us to scale up DeltaNet to standard language modeling settings. We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines such as Mamba and GLA in terms of perplexity and zero-shot performance on downstream tasks. We also experiment with two hybrid models which combine DeltaNet layers with (1) sliding-window attention layers every other layer or (2) two global attention layers, and find that these hybrids outperform strong transformer baselines."},"_bibtex":{"value":"@inproceedings{\nyang2024parallelizing,\ntitle={Parallelizing Linear Transformers with the Delta Rule over Sequence Length},\nauthor={Songlin Yang and Bailin Wang and Yu Zhang and Yikang Shen and Yoon Kim},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=y8Rm4VNRPH}\n}"},"title":{"value":"Parallelizing Linear Transformers with the Delta Rule over Sequence Length"},"pdf":{"value":"/pdf/397e5724c60c7bb0691c6436dc8a56f5a0336f4f.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"yang|parallelizing_linear_transformers_with_the_delta_rule_over_sequence_length"},"authorids":{"value":["~Songlin_Yang1","~Bailin_Wang3","~Yu_Zhang36","~Yikang_Shen1","~Yoon_Kim1"]},"authors":{"value":["Songlin Yang","Bailin Wang","Yu Zhang","Yikang Shen","Yoon Kim"]}},"version":2},{"content":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["3D Object Detection","Mamba"]},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Transformer-based architectures have been proven successful in detecting 3D objects from point clouds. However, the quadratic complexity of the attention mechanism struggles to encode rich information as point cloud resolution increases. Recently, state space models (SSM) such as Mamba have gained great attention due to their linear complexity and long sequence modeling ability for language understanding. To exploit the potential of Mamba on 3D scene-level perception, for the first time, we propose 3DET-Mamba, which is a novel SSM-based model designed for indoor 3d object detection. Specifically, we divide the point cloud into different patches and use a lightweight yet effective Inner Mamba to capture local geometric information. To observe the scene from a global perspective, we introduce a novel Dual Mamba module that models the point cloud in terms of spatial distribution and continuity. Additionally, we design a Query-aware Mamba module that decodes context features into object sets under the guidance of learnable queries. Extensive experiments demonstrate that 3DET-Mamba surpasses previous 3DETR on indoor 3D detection benchmarks such as ScanNet, improving AP25/AP50 from 65.0\\%/47.0\\% to 70.4\\%/54.4\\%, respectively."},"_bibtex":{"value":"@inproceedings{\nli2024detmamba,\ntitle={3{DET}-Mamba: Causal Sequence Modelling for End-to-End 3D Object Detection},\nauthor={Mingsheng Li and Jiakang Yuan and Sijin Chen and Lin Zhang and Anyu Zhu and Xin Chen and Tao Chen},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=iOleSlC80F}\n}"},"title":{"value":"3DET-Mamba: Causal Sequence Modelling for End-to-End 3D Object Detection"},"pdf":{"value":"/pdf/e562d505611831135151fdb8395d9283e0291da6.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"li|3detmamba_causal_sequence_modelling_for_endtoend_3d_object_detection"},"authorids":{"value":["~Mingsheng_Li2","~Jiakang_Yuan1","~Sijin_Chen1","~Lin_Zhang15","~Anyu_Zhu1","~Xin_Chen16","~Tao_Chen6"]},"authors":{"value":["Mingsheng Li","Jiakang Yuan","Sijin Chen","Lin Zhang","Anyu Zhu","Xin Chen","Tao Chen"]}},"tmdate":1730873838506,"pdate":1727287630727,"tcdate":1713835726871,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission112/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission112/Authors"],"forum":"iOleSlC80F","license":"CC BY 4.0","number":112,"cdate":1713835726871,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/-/Submission","NeurIPS.cc/2024/Conference/-/Edit","NeurIPS.cc/2024/Conference/-/Post_Submission","NeurIPS.cc/2024/Conference/Submission112/-/Revision","NeurIPS.cc/2024/Conference/Submission112/-/Camera_Ready_Revision"],"mdate":1730873838506,"odate":1730873838493,"domain":"NeurIPS.cc/2024/Conference","id":"iOleSlC80F","version":2},{"content":{"venue":{"value":"NeurIPS 2024 poster"},"pdf":{"value":"/pdf/073e18769156aeb0e81bf89d7cad50c6a7431274.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"li|3detmamba_causal_sequence_modelling_for_endtoend_3d_object_detection"},"authorids":{"value":["~Mingsheng_Li2","~Jiakang_Yuan1","~Sijin_Chen1","~Lin_Zhang15","~Anyu_Zhu1","~Xin_Chen16","~Tao_Chen6"]},"abstract":{"value":"Transformer-based architectures have been proven successful in detecting 3D\nobjects from point clouds. However, the quadratic complexity of the attention\nmechanism struggles to encode rich information as point cloud resolution increases.\nRecently, state space models (SSM) such as Mamba have gained great attention\ndue to their linear complexity and long sequence modeling ability for language\nunderstanding. To exploit the potential of Mamba on 3D scene-level perception,\nfor the first time, we propose 3DET-Mamba, which is a novel SSM-based model\ndesigned for indoor 3D object detection. Specifically, we divide the point cloud\ninto different patches and use a lightweight yet effective Inner Mamba to capture\nlocal geometric information. To observe the scene from a global perspective,\nwe introduce a novel Dual Mamba module that models the point cloud in terms\nof spatial distribution and continuity. Additionally, we design a Query-aware\nMamba module that decodes context features into object sets under the guidance of\nlearnable queries. Extensive experiments demonstrate that 3DET-Mamba surpasses\nprevious 3DETR on indoor 3D detection benchmarks such as ScanNet, improving\nAP@0.25/AP@0.50 from 65.0%/47.0% to 70.4%/54.4%, respectively."},"title":{"value":"3DET-Mamba: Causal Sequence Modelling for End-to-End 3D Object Detection"},"authors":{"value":["Mingsheng Li","Jiakang Yuan","Sijin Chen","Lin Zhang","Anyu Zhu","Xin Chen","Tao Chen"]}},"tmdate":1730723667780,"pdate":1727287200000,"tcdate":1730723667780,"writers":["~Mingsheng_Li2","~Jiakang_Yuan1","~Sijin_Chen1","~Lin_Zhang15","~Anyu_Zhu1","~Xin_Chen16","~Tao_Chen6"],"signatures":["~Jiakang_Yuan1"],"forum":"RVVPxVuwae","license":"CC BY-NC 4.0","number":31105,"cdate":1730723667780,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1730723667780,"domain":"OpenReview.net/Archive","id":"RVVPxVuwae","version":2},{"content":{"summary":{"value":"This paper presents MM-UNet (Meta Mamba UNet), a U-shaped encoder–decoder framework that integrates State Space Models (SSMs)—specifically Mamba blocks—into a UNet-like architecture for 3D medical image segmentation. The authors claim that SSMs can model long-range dependencies with linear computational complexity, offering a more efficient alternative to Transformers. They further introduce a “meta-block” design that unifies CNN and Mamba components, allowing flexible hybrid configurations, and a bi-directional scan strategy to mitigate discontinuities when flattening 3D volumes. Experiments on AMOS22 and Synapse datasets demonstrate competitive Dice scores (91.0 % and 87.1 %, respectively), surpassing several CNN- and Mamba-based baselines."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Can the authors provide quantitative comparisons of computational cost (training/inference time, FLOPs, memory, parameters) versus nnUNet and U-Mamba?\n2. How does MM-UNet perform against finetuned SAM or Med-SAM models? Including these baselines would contextualize performance against current state-of-the-art.\n3. Can authors test on some non-CT datasets to show the method's robustness against modality?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The meta-block design elegantly unifies CNN-only, Mamba-only, and hybrid configurations within a single architecture.\n2. Results on two benchmarks include both quantitative and qualitative analyses, clearly showing how different block choices affect performance.\n3. The paper is well written, logically structured, and visually clear。"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The method mainly combines existing ideas from prior Mamba-UNet variants (e.g., U-Mamba, VM-UNet) without introducing new learning principles or theoretical insights.\n2. Despite emphasizing SSMs’ linear complexity, the paper omits runtime, FLOPs, memory usage, and parameter count comparisons. The claimed computational benefits remain unsubstantiated.\n3. Experiments are restricted to CT, whereas the title and framing suggest general “volumetric medical image segmentation.” Validation on additional modalities (e.g., MRI from AMOS) would strengthen generalizability claims. The improvements are also not consistent: Gains vary irregularly across organs (e.g., Gallbladder +5 % on AMOS22 vs –4 % on Synapse), suggesting limited robustness.\n4. Foundation-model-based approaches (e.g., finetuned SAM, Med-SAM, SAM 2) are omitted, even though they represent current strong baselines in volumetric segmentation."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925836126,"tcdate":1761965759972,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15562/Reviewer_iVKL"],"signatures":["ICLR.cc/2026/Conference/Submission15562/Reviewer_iVKL"],"forum":"VaXFVeFkKD","number":3,"license":"CC BY 4.0","cdate":1761965759972,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15562/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925836126,"domain":"ICLR.cc/2026/Conference","replyto":"VaXFVeFkKD","id":"7A4PV01uUe","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Mamba","Medical Image Segmentation"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"State Space Models (SSMs) have recently demonstrated outstanding performance in long-sequence modeling, particularly in natural language processing. However, their direct application to medical image segmentation poses several challenges. SSMs, originally designed for 1D sequences, struggle with 3D spatial structures in medical images due to discontinuities introduced by flattening. Additionally, SSMs have difficulty fitting high-variance data, which is common in medical imaging.\nIn this paper, we analyze the intrinsic limitations of SSMs in medical image segmentation and propose a unified U-shaped encoder-decoder architecture, Meta Mamba UNet (MM-UNet), designed to leverage the advantages of SSMs while mitigating their drawbacks. MM-UNet incorporates hybrid modules that integrate SSMs within residual connections, reducing variance and improving performance. Furthermore, we introduce a novel bi-directional scan order strategy to alleviate discontinuities when processing medical images.\nExtensive experiments on AMOS22 and Synapse datasets demonstrate the superiority of MM-UNet over state-of-the-art methods. MM-UNet achieves a Dice score of 91.0% on AMOS22, surpassing nnUNet by 1.7%, and a Dice score of 87.1% on Synapse. These results confirm the effectiveness of integrating SSMs in medical image segmentation through architectural design optimizations"},"_bibtex":{"value":"@misc{\nxie2025mmunet,\ntitle={{MM}-{UN}et: Meta Mamba {UN}et for Medical Image Segmentation},\nauthor={Bin Xie and Yan Yan and Gady Agam},\nyear={2025},\nurl={https://openreview.net/forum?id=VaXFVeFkKD}\n}"},"title":{"value":"MM-UNet: Meta Mamba UNet for Medical Image Segmentation"},"pdf":{"value":"/pdf/b8b3f780aa91bf99af14acbc238617b378c2d89f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xie|mmunet_meta_mamba_unet_for_medical_image_segmentation"},"authorids":{"value":["~Bin_Xie1","~Yan_Yan6","~Gady_Agam1"]},"authors":{"value":["Bin Xie","Yan Yan","Gady Agam"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.01590v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"ali|the_hidden_attention_of_mamba_models"},"authorids":{"value":["~Ameen_Ali1","~Itamar_Zimerman1","https://dblp.org/search/pid/api?q=author:Lior_Wolf:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.01590"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-01590,\n  publtype={informal},\n  author={Ameen Ali and Itamar Zimerman and Lior Wolf},\n  title={The Hidden Attention of Mamba Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.01590},\n  url={https://doi.org/10.48550/arXiv.2403.01590}\n}\n"},"abstract":{"value":"The Mamba layer offers an efficient selective state space model (SSM) that is highly effective in modeling multiple domains, including NLP, long-range sequence processing, and computer vision. Selective SSMs are viewed as dual models, in which one trains in parallel on the entire sequence via an IO-aware parallel scan, and deploys in an autoregressive manner. We add a third view and show that such models can be viewed as attention-driven models. This new perspective enables us to empirically and theoretically compare the underlying mechanisms to that of the self-attention layers in transformers and allows us to peer inside the inner workings of the Mamba model with explainability methods. Our code is publicly available."},"title":{"value":"The Hidden Attention of Mamba Models"},"authors":{"value":["Ameen Ali","Itamar Zimerman","Lior Wolf"]}},"tmdate":1727701239028,"pdate":1704067200000,"tcdate":1716412058656,"writers":["~"],"signatures":["~Itamar_Zimerman1"],"forum":"t0EkwZJKsp","license":"CC BY-SA 4.0","number":20077,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1727701239028,"domain":"DBLP.org","id":"t0EkwZJKsp","version":2},{"content":{"venue":{"value":"NeurIPS 2025 spotlight"},"TLDR":{"value":"Structured Linear Controlled Differential Equations: A unifying framework for sequence models with structured, input-dependent state-transition matrices"},"keywords":{"value":["Sequencing Modelling","Theoretical Deep Learning","Mathematical Foundations","Neural Controlled Differential Equations","State Tracking","Model Architecture","Continuous-Time Models"]},"supplementary_material":{"value":"/attachment/cc75bbad690658b3a2bd2d2db4c828a4967d6af8.zip"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"This work introduces Structured Linear Controlled Differential Equations (SLiCEs), a unifying framework for sequence models with structured, input-dependent state-transition matrices that retain the maximal expressivity of dense matrices whilst being cheaper to compute. The framework encompasses existing architectures, such as input-dependent block-diagonal linear recurrent neural networks and DeltaNet's diagonal-plus-low-rank structure, as well as two novel variants based on sparsity and the Walsh-Hadamard transform. We prove that, unlike the diagonal state-transition matrices of S4D and Mamba, SLiCEs employing block-diagonal, sparse, or Walsh-Hadamard matrices match the maximal expressivity of dense matrices. Empirically, SLiCEs solve the $A_5$ state-tracking benchmark with a single layer, achieve best-in-class length generalisation on regular language tasks among parallel-in-time models, and match the performance of log neural controlled differential equations on six multivariate time-series classification datasets while cutting the average time per training step by a factor of twenty."},"_bibtex":{"value":"@inproceedings{\nwalker2025structured,\ntitle={Structured Linear {CDE}s: Maximally Expressive and Parallel-in-Time Sequence Models},\nauthor={Benjamin Walker and Lingyi Yang and Nicola Muca Cirone and Cristopher Salvi and Terry Lyons},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=HKDyRDzy1E}\n}"},"title":{"value":"Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models"},"pdf":{"value":"/pdf/a62b8e2ab79a2187b58fdc494a9bd35bddc3da47.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"walker|structured_linear_cdes_maximally_expressive_and_parallelintime_sequence_models"},"authorids":{"value":["~Benjamin_Walker1","~Lingyi_Yang1","~Nicola_Muca_Cirone1","~Cristopher_Salvi1","~Terry_Lyons2"]},"authors":{"value":["Benjamin Walker","Lingyi Yang","Nicola Muca Cirone","Cristopher Salvi","Terry Lyons"]}},"tmdate":1783627360126,"pdate":1758217065881,"tcdate":1746873408986,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission15778/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission15778/Authors"],"forum":"HKDyRDzy1E","license":"CC BY 4.0","number":15778,"cdate":1746873408986,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission15778/-/Full_Submission","NeurIPS.cc/2025/Conference/Submission15778/-/Supplementary_Material","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission15778/-/Camera_Ready_Revision"],"mdate":1783627360126,"odate":1761704923179,"domain":"NeurIPS.cc/2025/Conference","id":"HKDyRDzy1E","version":2},{"content":{"summary":{"value":"This paper introduces LongMamba, a training-free techniques to enhance Mamba model's ability to handle long sequences beyond the training context lenght.  \n\nThe authors discovered that Mamba's hidden state channels can be categorized into two types: local channels that focus on nearby context, and global channels that attend to the entire input sequence. \nThe key limitation they identified is that global channels fails to be functional when processing sequences longer than the training length and they hypothesize the exponential decay is the root cause.   \n\nTo this end, they propsoe LongMamba that addresses this by enlarging receptive fields of global channels by filtering out the less important tokens while maintaining similar decay distributions as during training.\nThrough extensive experiments on tasks like Passkey Retrieval, Document Retrieval, Language Modeling, and LongBench, LongMamba shows promising result without additional training and outperform vanilla mamba and previous methods."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"(1) How does the LongMamba perform on the code categroy of LongBench?\n\n(2) Does the token filtering process bring extra compute or memory cost?\n\n(3) How does the \"local and global channels\" behave on larger Mamba model?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"- The finding about the categroies of channels of Mamba models(local and global) is interesting and meaningful, which could bring insight to the community for better understanding the Mamba model's behavior.\n\n- The experiment results are strong as it outperforms the vanilla Mamba model by large margin.\n\n- This paper is well-written and easy to follow, for example, they reveal the \"categroies of channels\" phenomena with sufficient analysis and visualization and figure 2 (the method overview) is tidy and clear"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper is generally good but I believe the experiments section can be improved by adding direct comparsion with other long-context models beyond Mamba variants.\nI find this is necessary because the core motivation of this work is to address the poor long-context ability of Mambas compared to the transformer-based model, as said in the introduction section. Therefore, it is useful to learn how much LongMamba helps bridge the performance gap. More specifical, the author may follow the [Mamba](https://arxiv.org/abs/2312.00752)'s main experiment setting and compare the LongMamba to varients of transformers (like [Pythia](https://arxiv.org/abs/2304.01373) ) or hypird model"}},"nonreaders":[],"tmdate":1731428409701,"tcdate":1730625775515,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13423/Reviewer_P417"],"signatures":["ICLR.cc/2025/Conference/Submission13423/Reviewer_P417"],"forum":"fMbLszVO1H","number":2,"license":"CC BY 4.0","cdate":1730625775515,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13423/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428409701,"domain":"ICLR.cc/2025/Conference","replyto":"fMbLszVO1H","id":"megbPX72oy","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Models","State Space Models","Long Context Understanding"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State space models (SSMs) have emerged as an efficient alternative to Transformer models for language modeling, offering linear computational complexity and constant memory usage as context length increases. However, despite their efficiency in handling long contexts, recent studies have shown that SSMs, such as Mamba models, generally underperform compared to Transformers in long-context understanding tasks. To address this significant shortfall and achieve both efficient and accurate long-context understanding, we propose LongMamba, a training-free technique that significantly enhances the long-context capabilities of Mamba models. LongMamba builds on our discovery that the hidden channels in Mamba can be categorized into local and global channels based on their receptive field lengths, with global channels primarily responsible for long-context capability. These global channels can become the key bottleneck as the input context lengthens. Specifically, when input lengths largely exceed the training sequence length, global channels exhibit limitations in adaptively extend their receptive fields, leading to Mamba’s poor long-context performance. The key idea of LongMamba is to mitigate the hidden state memory decay in these global channels by preventing the accumulation of unimportant tokens in their memory. This is achieved by first identifying critical tokens in the global channels and then applying token filtering to accumulate only those critical tokens. Through extensive benchmarking across synthetic and real-world long-context scenarios, LongMamba sets a new standard for Mamba’s long-context performance, significantly extending its operational range without requiring additional training. Our code is available at https://github.com/GATECH-EIC/LongMamba."},"_bibtex":{"value":"@inproceedings{\nye2025longmamba,\ntitle={LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement},\nauthor={Zhifan Ye and Kejing Xia and Yonggan Fu and Xin Dong and Jihoon Hong and Xiangchi Yuan and Shizhe Diao and Jan Kautz and Pavlo Molchanov and Yingyan Celine Lin},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=fMbLszVO1H}\n}"},"title":{"value":"LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement"},"pdf":{"value":"/pdf/066062c0673cfc50aeafa9e3c96d0cf7a33fc3ca.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"ye|longmamba_enhancing_mambas_longcontext_capabilities_via_trainingfree_receptive_field_enlargement"},"authorids":{"value":["~Zhifan_Ye1","~Kejing_Xia1","~Yonggan_Fu1","~Xin_Dong1","~Jihoon_Hong1","~Xiangchi_Yuan1","~Shizhe_Diao2","~Jan_Kautz1","~Pavlo_Molchanov1","~Yingyan_Celine_Lin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zhifan Ye","Kejing Xia","Yonggan Fu","Xin Dong","Jihoon Hong","Xiangchi Yuan","Shizhe Diao","Jan Kautz","Pavlo Molchanov","Yingyan Celine Lin"]}},"version":2},{"content":{"summary":{"value":"The paper introduces Selective Rotary Position Embedding (Selective RoPE), an input-dependent mechanism designed to generalize standard Rotary Position Embeddings (RoPE) by performing rotations at arbitrary, selective frequencies."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"Do you expect GLA  and Softmax Transformers to benefit equally from Selective RoPE?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"Interesting theoretical insights, such as:\n- Softmax attention implicitly applies a selective rotation, to encode relative positional information between tokens\n- Linear Transformers can be enhanced by using both forgetting via real decay and rotation via imaginary gate."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Very limited evaluation for language modeling.  Would be good to have at least RoPE as baseline (Table 3) and evaluate Selective RoPE in different settings (i.e context length)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762941770709,"tcdate":1761993354647,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21436/Reviewer_NYgW"],"signatures":["ICLR.cc/2026/Conference/Submission21436/Reviewer_NYgW"],"forum":"AQo1SEElNb","number":4,"license":"CC BY 4.0","cdate":1761993354647,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission21436/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762941770709,"domain":"ICLR.cc/2026/Conference","replyto":"AQo1SEElNb","id":"h6PRb8jVaZ","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce Selective RoPE, an input-dependent rotary embedding that enhances gated linear transformers."},"keywords":{"value":["RoPE","Linear Transformer","Attention","State Space Models","Forget Gate"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Positional information is essential for language modeling. Softmax Transformers with Rotary Position Embeddings (RoPE) encode it with fixed-angle rotations, while linear Transformers rely on input-dependent gates that only decay past key-value norms. We provide a theoretical argument for the necessity of a rotation and decay component in well-performing sequence models, and observe that the missing ingredient in linear models is precisely the rotation that softmax attention performs implicitly. We introduce Selective Rotary Position Embedding (*Selective RoPE*), an input-dependent, learnable rotary embedding that generalizes RoPE to arbitrary angles and composes seamlessly with decay gates. Equipping gated linear attention with *Selective RoPE* yields a complex-valued recurrent layer that can be implemented efficiently with the “RoPE trick”. On synthetic benchmarks (MQAR, copying, state tracking) and 370M-parameter language-model pre-training, the method improves recall, downstream accuracy, and expressivity while adding minimal architectural overhead. We open-source our implementation [here](https://github.com/timurcarstensen/selective-rope)."},"_bibtex":{"value":"@inproceedings{\nmovahedi2026selective,\ntitle={Selective Rotary Position Embedding},\nauthor={Sajad Movahedi and Timur Carstensen and Arshia Afzal and Frank Hutter and Antonio Orvieto and Volkan Cevher},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=AQo1SEElNb}\n}"},"title":{"value":"Selective Rotary Position Embedding"},"pdf":{"value":"/pdf/943324e41b7e6a418a5aab0f6ad1261b8e48ad8e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"movahedi|selective_rotary_position_embedding"},"authorids":{"value":["~Sajad_Movahedi1","~Timur_Carstensen1","~Arshia_Afzal1","~Frank_Hutter1","~Antonio_Orvieto3","~Volkan_Cevher1"]},"authors":{"value":["Sajad Movahedi","Timur Carstensen","Arshia Afzal","Frank Hutter","Antonio Orvieto","Volkan Cevher"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["State Space Model","SSM","Selective State Space Models","Federated Learning","Distributed Learning","Distributed Optimization","Convergence Analysis","Upper Bounds","Gradient Bounds","Smoothness Bounds","Optimization","Deep Learning","AI","Mamba2","Recurrent Stabilitym"]},"supplementary_material":{"value":"/attachment/b91ffb4cd99f3fbebc6f1f51b3ce71d739f52c0a.zip"},"primary_area":{"value":"optimization"},"abstract":{"value":"Modern state space models (SSMs), such as Mamba2, provide a compelling alternative to transformers by combining linear-time sequence modeling with recurrent state-space dynamics. However, the behavior of SSMs in distributed learning settings remains poorly understood. In particular, the existing standard federated learning methods are largely architecture-agnostic, and do not account for the stability, selectivity, and state-space parameterization that characterize modern selective SSMs. To address this, we derive architecture-aware gradient and smoothness bounds for single- and multi-layer selective SSMs, and convergence bounds for FedAvg and FedProx, characterizing how recurrent stability, input-dependent discretization, and state projection norms affect federated optimization. We then numerically validate the single-layer bounds on sequences generated by a teacher SSM, using a learner that follows the analyzed recurrence. We use this analysis to formulate expectations about the effects of local training and client heterogeneity, and examine these expectations by comparing nine federated learning algorithms on Mamba2 language modeling across six text domains. These experiments illustrate how SSM-specific bounds can provide a basis for interpreting the behavior of practical federated learning algorithms."},"_bibtex":{"value":"@inproceedings{\nanonymous2026distributed,\ntitle={Distributed Learning with Selective State Space Models: Architecture-Aware Convergence Analysis},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=ndNEB1UyID},\nnote={under review}\n}"},"title":{"value":"Distributed Learning with Selective State Space Models: Architecture-Aware Convergence Analysis"},"pdf":{"value":"/pdf/976a32f7c5f34fb9f0691871465e89bb6ef99039.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791226022498,"tcdate":1789059957660,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission13410/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission13410/Authors"],"forum":"ndNEB1UyID","license":"CC BY 4.0","number":13410,"cdate":1789059957660,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission13410/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791226022498,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"ndNEB1UyID","version":2},{"content":{"venue":{"value":"AAAI 2025"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/32531/34686"},"venueid":{"value":"dblp.org/conf/AAAI/2025"},"paperhash":{"value":"li|mambacad_state_space_model_for_3d_computeraided_design_generative_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xueyang_Li:","https://dblp.org/search/pid/api?q=author:Yunzhong_Lou:","https://dblp.org/search/pid/api?q=author:Yu_Song:","~Xiangdong_Zhou1"]},"html":{"value":"https://doi.org/10.1609/aaai.v39i5.32531"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/LiLSZ25,\n  author={Xueyang Li and Yunzhong Lou and Yu Song and Xiangdong Zhou},\n  title={Mamba-CAD: State Space Model for 3D Computer-Aided Design Generative Modeling},\n  year={2025},\n  cdate={1735689600000},\n  pages={5013-5021},\n  url={https://doi.org/10.1609/aaai.v39i5.32531},\n  booktitle={AAAI},\n  crossref={conf/aaai/2025}\n}\n"},"abstract":{"value":"Computer-Aided Design (CAD) generative modeling has a strong and long-term application in the industry. Recently, the parametric CAD sequence as the design logic of an object has been widely mined by sequence models. However, the industrial CAD models, especially in component objects, are fine-grained and complex, requiring a longer parametric CAD sequence to define. To address the problem, we introduce Mamba-CAD, a self-supervised generative modeling for complex CAD models in the industry, which can model on a longer parametric CAD sequence. Specifically, we first design an encoder-decoder framework based on a Mamba architecture and pair it with a CAD reconstruction task for pre-training to model the latent representation of CAD models; and then we utilize the learned representation to guide a generative adversarial network to produce the fake representation of CAD models, which would be finally recovered into parametric CAD sequences via the decoder of Mamba-CAD. To train Mamba-CAD, we further create a new dataset consisting of 77,078 CAD models with longer parametric CAD sequences. Comprehensive experiments are conducted to demonstrate the effectiveness of our model under various evaluation metrics, especially in the generation length of valid parametric CAD sequences."},"title":{"value":"Mamba-CAD: State Space Model for 3D Computer-Aided Design Generative Modeling"},"authors":{"value":["Xueyang Li","Yunzhong Lou","Yu Song","Xiangdong Zhou"]}},"tmdate":1747290192817,"pdate":1735689600000,"tcdate":1747290189128,"writers":["~"],"signatures":["~Xiangdong_Zhou1"],"forum":"nYKlISBzX0","license":"CC BY-SA 4.0","number":473878,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747290192817,"domain":"DBLP.org","id":"nYKlISBzX0","version":2},{"content":{"summary":{"value":"This work follows a long line of works towards efficient modeling of long sequences. Given a sequence $X$ of $n$ vectors of dimension $d$, authors propose to form an aggregate representation of context $X[<=t]$ for the vector $X_t$ by linearly projecting down $X[<=t]$ to $r$ dimensions and mean-pooling the representions to get a size-$r$ history $H_t$. Vector $H_t$ is incorporated into $X_t$ via attention (though there might be other ways of doing this e.g. concatenate + FF). In line with standard practice, authors also use local attention.\n\nAuthors compare the method to dense attention (and other baselines) on LM, text summarization, speech generation, point-cloud completion, forecasting tasks over a wide range of sequence lengths, and report modest gains on some of them and good 0-shot extrapolation to longer inputs."},"presentation":{"value":"4 excellent"},"contribution":{"value":"2 fair"},"soundness":{"value":"4 excellent"},"strengths":{"value":"1. compared to other relatively involved aggregation methods such as state spaces, the authors propose to use cummulative mean which compared to state space methods has a simpler and faster implementation based on cumsum.\n\n2. Experimental evaluation is comprehensive and honest, encompassing tasks over various modalities over a wide range of sequence lengths. Wherever possible authors evaluate 0-shot extrapolation to longer inputs and show that their method extrapolates better than the baselines (due to locality of attention and cummulative itself)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. On sequences of length upto 8k, the proposed method lags behind hardware-optimized attention in terms of both performance and throughput. For most practical applications this regime is highly relevant.\n\n2. cummulative sum/mean is a special case of state spaces [S4, DSS, S4D] (similarly exponential moving average MEGA [Ma et al ICLR 2023]). Several follow-up works also use local attention layers similar to what authors use. Hence the paper doesnt offer a new technqiue but as pointed out above it does study a special case of using only cumsum/cummean.\n\n3. Performance compared to learnt global convolutions (state spaces, etc) is significantly lower on Long Range Arena pointing to the importance of decayed convolutions as compared to only fixed convolutions such as cummulative sum/mean without a decay factor. This is also evident by the author's need to additionally use positions enbeddings which models based on state spaces dont require (e.g. GSS [Mehta et al ICLR 2023]). State space + local attention has been conistently shown to extrapolate to much longer inputs (e.g. GSS: PG19, train length 4096, eval length 65k) where the performance improves with longer inputs as the model is able to leverage more context.\n\n4. Positional embeddings - the authors use additive relative bias in attention (c.f. T5, Alibi) which is undesirable due to IO aspects and will significantly slow down FLASH implementation. Rotary embeddings instead induce multiplicative relative bias and are friendly to FLASH (one can chose not to use FLASH and use vanilla attention but there is no reason why other users will not leverage it.)"},"confidence":{"value":"5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully."},"questions":{"value":"Typo Section 3.2 : complexity is O(wn + rn*logn) during training as parallel prefix sum (cumsum) is nlogn. I do understand that log will probably not show up in runtime as other things will dominate.\n\n1. Please discuss the implementation of cummean. e.g. whether you're using FFT or else if you're using cumsum discuss how you ensure that the sums dont do outside the fp16 bounds (pre-normalize by the max, etc). e.g. for fp16 the largest possible value is 65k and so its not possible to post-divide by the index for input length >66k. If you used bfloat16 instead please mention."},"rating":{"value":"5: marginally below the acceptance threshold"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636459889,"tcdate":1698771622771,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission4776/Reviewer_uFEK"],"signatures":["ICLR.cc/2024/Conference/Submission4776/Reviewer_uFEK"],"forum":"Rn3qJGOitY","number":3,"license":"CC BY 4.0","cdate":1698771622771,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission4776/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636459889,"domain":"ICLR.cc/2024/Conference","replyto":"Rn3qJGOitY","id":"yYqgUlIKfd","forumContent":{"TLDR":{"value":"To address efficiency degradation problem, we propose an linear attention via orthogonal memory (LAVO) to achieve strong performance while maintaining linear complexity."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Efficient attention; Efficiency degradation; Unbounded language modeling"]},"primary_area":{"value":"representation learning for computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Efficient attentions have greatly improved the computational efficiency of Transformers. However, most existing linear attention mechanisms suffer from an efficiency degradation problem, leading to inefficiencies in causal language modeling and hindering their application in long-range language models. This problem is more pronounced under language modeling with unbounded contexts. In this paper, we propose Linear Attention Via Orthogonal memory (LAVO) to address these limitations, achieving strong performance while maintaining linear complexity. LAVO employs orthogonal decomposition to compress a context into a fixed-size orthogonal memory while effectively minimizing redundancy within the context. Given that orthogonal memory compresses global information, we further dissect the context to amplify fine-grained local information. Additionally, we embed the relative position encoding into LAVO to improve the extrapolation ability. Experimental results show that LAVO greatly improves the efficiency of the causal language model with the best extrapolation performance and outperforms other efficient baselines. Further, we endeavor to employ LAVO for unbounded language modeling and successfully scale the context length to 128K."},"_bibtex":{"value":"@misc{\nzhang2024linear,\ntitle={Linear Attention via Orthogonal Memory},\nauthor={Jun Zhang and Shuyang Jiang and Jiangtao Feng and Lin Zheng and Lingpeng Kong},\nyear={2024},\nurl={https://openreview.net/forum?id=Rn3qJGOitY}\n}"},"title":{"value":"Linear Attention via Orthogonal Memory"},"pdf":{"value":"/pdf/5007fdb61ef9d3d7576452b8513e42be79b1b737.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|linear_attention_via_orthogonal_memory"},"authorids":{"value":["~Jun_Zhang27","~Shuyang_Jiang2","~Jiangtao_Feng1","~Lin_Zheng1","~Lingpeng_Kong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jun Zhang","Shuyang Jiang","Jiangtao Feng","Lin Zheng","Lingpeng Kong"]}},"version":2},{"content":{"summary":{"value":"The paper proposes FutureFill, a fast generation method designed for convolutional sequence models, aiming to reduce generation time complexity from linear to square root relative to context length. FutureFill utilizes a prefill cache that is theoretically more efficient in both time and memory, requiring less storage than traditional convolutional and attention-based models. The authors validate their approach through theoretical analysis and experimental evaluation on synthetic generation tasks."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. How does FutureFill perform on standard language modeling or machine translation benchmarks compared to other optimized sequence models? Including such comparisons would significantly strengthen the paper’s claims about practical applicability. \n2. The paper briefly mentions the complexity of benchmarking FutureFill on accelerated hardware, but how feasible is it to implement this method in practice? Given the reliance on FFT, how does FutureFill perform on GPUs without optimized FFT support? \n3. The magin formatting of this paper is changed, which may violate the requirement of ICLR."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"1. FutureFill offers a novel approach to reducing the generation time for convolutional models from linear to square root relative to context length. This theoretical improvement addresses one of the core challenges of convolutional models, positioning FutureFill as a potentially valuable contribution to the field of efficient sequence generation. \n2. By using a prefill cache that scales with the number of generated tokens rather than the full context, FutureFill significantly reduces memory overhead. This is particularly valuable in applications where memory is a bottleneck, making the method more accessible for deployment in resource-constrained environments.\n3. FutureFill offers an alternative to attention-based models, which suffer from quadratic complexity, especially with longer sequences. By exploring a convolution-based approach, the paper contributes to the broader goal of overcoming the limitations of self-attention in handling long sequences."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper relies heavily on synthetic tasks for evaluation, which may not fully represent real-world applications. Without testing FutureFill on more practical benchmarks or large-scale NLP tasks, it’s difficult to assess its true effectiveness and scalability.\n2. While the authors emphasize efficiency improvements, they do not compare FutureFill against other advanced, optimized generation techniques. \n3. The authors focus mainly on theoretical comparisons with standard convolution and attention models. However, many recent techniques in sequence modeling—such as token pruning, cache compression, or sparse attention mechanisms—are entirely overlooked, which limits the relevance of FutureFill in the context of state-of-the-art sequence generation methods."}},"nonreaders":[],"tmdate":1731428225378,"tcdate":1730707456706,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8109/Reviewer_7b51"],"signatures":["ICLR.cc/2025/Conference/Submission8109/Reviewer_7b51"],"forum":"kTIKlSzPTL","number":4,"license":"CC BY 4.0","cdate":1730707456706,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8109/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428225378,"domain":"ICLR.cc/2025/Conference","replyto":"kTIKlSzPTL","id":"vPDGDuaTHI","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["sequence modeling","online learning","fast inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We address the challenge of efficient auto-regressive generation in sequence prediction models by introducing FutureFill—a method for fast generation that applies to any sequence prediction algorithm based on convolutional operators. Our approach reduces the generation time requirement from linear to square root relative to the context length. Additionally, FutureFill requires a prefill cache sized only by the number of tokens generated, which is smaller than the cache requirements for standard convolutional and attention-based models. We validate our theoretical findings with experimental evidence demonstrating correctness and efficiency gains in a synthetic generation task."},"_bibtex":{"value":"@misc{\nagarwal2025futurefill,\ntitle={FutureFill: Fast Generation from Convolutional Sequence Models},\nauthor={Naman Agarwal and Xinyi Chen and Evan Dogariu and Vladimir Feinberg and Daniel Suo and Peter Bartlett and Elad Hazan},\nyear={2025},\nurl={https://openreview.net/forum?id=kTIKlSzPTL}\n}"},"title":{"value":"FutureFill: Fast Generation from Convolutional Sequence Models"},"pdf":{"value":"/pdf/8bfc7ce1f567262ee709496baef5d76155a06f7c.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"agarwal|futurefill_fast_generation_from_convolutional_sequence_models"},"authorids":{"value":["~Naman_Agarwal1","~Xinyi_Chen1","~Evan_Dogariu1","~Vladimir_Feinberg2","~Daniel_Suo1","~Peter_Bartlett1","~Elad_Hazan1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Naman Agarwal","Xinyi Chen","Evan Dogariu","Vladimir Feinberg","Daniel Suo","Peter Bartlett","Elad Hazan"]}},"version":2},{"content":{"venue":{"value":"DEMO 2026 Poster"},"pdf":{"value":"/pdf/8ee7dc00248048500f1c5fa3642fae06525bca80.pdf"},"keywords":{"value":["Offline Reinforcement Learning","Mamba"]},"venueid":{"value":"ICML.cc/2026/Workshop/DEMO"},"paperhash":{"value":"cao|mamba_as_decision_maker_exploring_multiscale_sequence_modeling_in_offline_reinforcement_learning"},"authorids":{"value":["~Jiahang_Cao1","~Qiang_Zhang10","~Ziqing_Wang2","~Jingkai_SUN1","~Jiaxu_Wang1","~Hao_Cheng15","~Yecheng_Shao1","~Wen_Zhao2","~Gang_Han1","~Yijie_Guo3","~Renjing_Xu1"]},"abstract":{"value":"Sequential modeling has demonstrated remarkable capabilities in offline robotics reinforcement learning (RL), with Decision Transformer (DT) being one of the most notable representatives, achieving significant success. \n    However, RL trajectories possess unique properties to be distinguished from the conventional sequence (e.g., text or audio): (1) local correlation, where the next states in RL are theoretically determined solely by current states and actions based on the Markov Decision Process (MDP), and (2) global correlation, where each step's features are related to long-term historical information due to the time-continuous nature of trajectories.\n    In this paper, we propose a novel action sequence predictor, named Mamba Decision Maker (MambaDM), where Mamba is expected to be a promising alternative for sequence modeling paradigms, owing to its efficient modeling of multi-scale dependencies. \n    In particular, we introduce a novel mixer module that proficiently extracts and integrates both global and local features of the input sequence, effectively capturing interrelationships in RL datasets. Extensive experiments demonstrate that MambaDM achieves superior performance in D4RL and Robomimic datasets, including locomotion, manipulation, maze, and game tasks. Furthermore, we empirically investigate the scaling laws of MambaDM, finding that increasing model size does not bring performance improvement, but scaling the dataset amount by 2$\\times$ for MambaDM can obtain up to 33.7\\% score improvement on the Atari dataset.\n    This paper delves into the sequence modeling capabilities of MambaDM in the robotics RL domain, paving the way for future advancements in robust and efficient decision-making systems."},"_bibtex":{"value":"@inproceedings{\ncao2026mamba,\ntitle={Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in  Offline Reinforcement Learning},\nauthor={Jiahang Cao and Qiang Zhang and Ziqing Wang and Jingkai SUN and Jiaxu Wang and Hao Cheng and Yecheng Shao and Wen Zhao and Gang Han and Yijie Guo and Renjing Xu},\nbooktitle={Decision-Making from Offline Datasets to Online Adaptation: Black-Box Optimization to Reinforcement Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=kKjkhduxje}\n}"},"title":{"value":"Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in  Offline Reinforcement Learning"},"authors":{"value":["Jiahang Cao","Qiang Zhang","Ziqing Wang","Jingkai SUN","Jiaxu Wang","Hao Cheng","Yecheng Shao","Wen Zhao","Gang Han","Yijie Guo","Renjing Xu"]}},"tmdate":1783646459772,"pdate":1779747011353,"tcdate":1777993494659,"writers":["ICML.cc/2026/Workshop/DEMO","ICML.cc/2026/Workshop/DEMO/Submission35/Authors"],"signatures":["ICML.cc/2026/Workshop/DEMO/Submission35/Authors"],"forum":"kKjkhduxje","license":"CC BY 4.0","number":35,"cdate":1777993494659,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/DEMO/-/Submission","ICML.cc/2026/Workshop/DEMO/-/Post_Submission","ICML.cc/2026/Workshop/DEMO/-/Edit","ICML.cc/2026/Workshop/DEMO/Submission35/-/Camera_Ready_Submission"],"mdate":1783646459772,"odate":1779903468361,"domain":"ICML.cc/2026/Workshop/DEMO","id":"kKjkhduxje","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2405.15881v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"mo|scaling_diffusion_mamba_with_bidirectional_ssms_for_efficient_image_and_video_generation"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Shentong_Mo:","~Yapeng_Tian1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2405.15881"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2405-15881,\n  publtype={informal},\n  author={Shentong Mo and Yapeng Tian},\n  title={Scaling Diffusion Mamba with Bidirectional SSMs for Efficient Image and Video Generation},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2405.15881},\n  url={https://doi.org/10.48550/arXiv.2405.15881}\n}\n"},"abstract":{"value":"In recent developments, the Mamba architecture, known for its selective state space approach, has shown potential in the efficient modeling of long sequences. However, its application in image generation remains underexplored. Traditional diffusion transformers (DiT), which utilize self-attention blocks, are effective but their computational complexity scales quadratically with the input length, limiting their use for high-resolution images. To address this challenge, we introduce a novel diffusion architecture, Diffusion Mamba (DiM), which foregoes traditional attention mechanisms in favor of a scalable alternative. By harnessing the inherent efficiency of the Mamba architecture, DiM achieves rapid inference times and reduced computational load, maintaining linear complexity with respect to sequence length. Our architecture not only scales effectively but also outperforms existing diffusion transformers in both image and video generation tasks. The results affirm the scalability and efficiency of DiM, establishing a new benchmark for image and video generation techniques. This work advances the field of generative models and paves the way for further applications of scalable architectures."},"title":{"value":"Scaling Diffusion Mamba with Bidirectional SSMs for Efficient Image and Video Generation"},"authors":{"value":["Shentong Mo","Yapeng Tian"]}},"tmdate":1727457361606,"pdate":1704067200000,"tcdate":1727454164413,"writers":["~"],"signatures":["~Yapeng_Tian1"],"forum":"4lyziPx6Ua","license":"CC BY-SA 4.0","number":88268,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1727457361606,"domain":"DBLP.org","id":"4lyziPx6Ua","version":2},{"content":{"venue":{"value":"CoRR 2023"},"pdf":{"value":"http://arxiv.org/pdf/2303.14526v1"},"venueid":{"value":"dblp.org/journals/CORR/2023"},"paperhash":{"value":"wang|selective_structured_statespaces_for_longform_video_understanding"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jue_Wang:","https://dblp.org/search/pid/api?q=author:Wentao_Zhu:","~Pichao_Wang1","https://dblp.org/search/pid/api?q=author:Xiang_Yu:","https://dblp.org/search/pid/api?q=author:Linda_Liu:","https://dblp.org/search/pid/api?q=author:Mohamed_Omar:","https://dblp.org/search/pid/api?q=author:Raffay_Hamid:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2303.14526"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2303-14526,\n  publtype={informal},\n  author={Jue Wang and Wentao Zhu and Pichao Wang and Xiang Yu and Linda Liu and Mohamed Omar and Raffay Hamid},\n  title={Selective Structured State-Spaces for Long-Form Video Understanding},\n  year={2023},\n  cdate={1672531200000},\n  journal={CoRR},\n  volume={abs/2303.14526},\n  url={https://doi.org/10.48550/arXiv.2303.14526}\n}\n"},"abstract":{"value":"Effective modeling of complex spatiotemporal dependencies in long-form videos remains an open problem. The recently proposed Structured State-Space Sequence (S4) model with its linear complexity offers a promising direction in this space. However, we demonstrate that treating all image-tokens equally as done by S4 model can adversely affect its efficiency and accuracy. To address this limitation, we present a novel Selective S4 (i.e., S5) model that employs a lightweight mask generator to adaptively select informative image tokens resulting in more efficient and accurate modeling of long-term spatiotemporal dependencies in videos. Unlike previous mask-based token reduction methods used in transformers, our S5 model avoids the dense self-attention calculation by making use of the guidance of the momentum-updated S4 model. This enables our model to efficiently discard less informative tokens and adapt to various long-form video understanding tasks more effectively. However, as is the case for most token reduction methods, the informative image tokens could be dropped incorrectly. To improve the robustness and the temporal horizon of our model, we propose a novel long-short masked contrastive learning (LSMCL) approach that enables our model to predict longer temporal context using shorter input videos. We present extensive comparative results using three challenging long-form video understanding datasets (LVU, COIN and Breakfast), demonstrating that our approach consistently outperforms the previous state-of-the-art S4 model by up to 9.6% accuracy while reducing its memory footprint by 23%."},"title":{"value":"Selective Structured State-Spaces for Long-Form Video Understanding"},"authors":{"value":["Jue Wang","Wentao Zhu","Pichao Wang","Xiang Yu","Linda Liu","Mohamed Omar","Raffay Hamid"]}},"tmdate":1760550648362,"pdate":1672531200000,"externalIds":["dblp:journals/corr/abs-2303-14526"],"tcdate":1760550632475,"writers":["~"],"signatures":["~Pichao_WANG3"],"forum":"rTFnXrc79H","license":"CC BY-SA 4.0","number":645356,"cdate":1672531200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1760550648362,"domain":"DBLP.org","id":"rTFnXrc79H","version":2},{"content":{"venue":{"value":"CVPR 2023"},"pdf":{"value":"https://ieeexplore.ieee.org/iel7/10203037/10203050/10204675.pdf"},"venueid":{"value":"dblp.org/conf/CVPR/2023"},"paperhash":{"value":"wang|selective_structured_statespaces_for_longform_video_understanding"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jue_Wang:","https://dblp.org/search/pid/api?q=author:Wentao_Zhu:","~Pichao_Wang1","https://dblp.org/search/pid/api?q=author:Xiang_Yu:","https://dblp.org/search/pid/api?q=author:Linda_Liu:","https://dblp.org/search/pid/api?q=author:Mohamed_Omar:","~Raffay_Hamid1"]},"html":{"value":"https://doi.org/10.1109/CVPR52729.2023.00618"},"_bibtex":{"value":"@inproceedings{DBLP:conf/cvpr/WangZWYLOH23,\n  author={Jue Wang and Wentao Zhu and Pichao Wang and Xiang Yu and Linda Liu and Mohamed Omar and Raffay Hamid},\n  title={Selective Structured State-Spaces for Long-Form Video Understanding},\n  year={2023},\n  cdate={1672531200000},\n  pages={6387-6397},\n  url={https://doi.org/10.1109/CVPR52729.2023.00618},\n  booktitle={CVPR},\n  crossref={conf/cvpr/2023}\n}\n"},"abstract":{"value":"Effective modeling of complex spatiotemporal dependencies in long-form videos remains an open problem. The recently proposed Structured State-Space Sequence ($S4$) model with its linear complexity offers a promising direction in this space. However, we demonstrate that treating all imagetokens equally as done by $S4$ model can adversely affect its efficiency and accuracy. To address this limitation, we present a novel Selective $S4$ (i.e., $S5)$ model that employs a lightweight mask generator to adaptively select informative image tokens resulting in more efficient and accurate modeling of long-term spatiotemporal dependencies in videos. Unlike previous mask-based token reduction methods used in transformers, our $S5$ model avoids the dense self-attention calculation by making use of the guidance of the momentum-updated $S4$ model. This enables our model to efficiently discard less informative tokens and adapt to various long-form video understanding tasks more effectively. However, as is the case for most token reduction methods, the informative image tokens could be dropped incorrectly. To improve the robustness and the temporal horizon of our model, we propose a novel long-short masked contrastive learning (LSMCL) approach that enables our model to predict longer temporal context using shorter input videos. We present extensive comparative results using three challenging long-form video understanding datasets (LVU, COIN and Breakfast), demonstrating that our approach consistently outperforms the previous state-of-the-art S4 model by up to 9.6% accuracy while reducing its memory footprint by 23%."},"title":{"value":"Selective Structured State-Spaces for Long-Form Video Understanding"},"authors":{"value":["Jue Wang","Wentao Zhu","Pichao Wang","Xiang Yu","Linda Liu","Mohamed Omar","Raffay Hamid"]}},"tmdate":1760550632429,"pdate":1672531200000,"tcdate":1741256712058,"writers":["~"],"signatures":["~Raffay_Hamid1"],"forum":"tgsxwjpJVP","license":"CC BY-SA 4.0","number":361320,"cdate":1672531200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1760550632429,"domain":"DBLP.org","id":"tgsxwjpJVP","version":2},{"content":{"summary":{"value":"This paper observes the changes in the distribution of forget gate values in gate linear/nonlinear RNNs before and after training on two small-scale synthetic tasks, i.e., Copying and Selective Copying (Figures 1 & 2). Specifically, the distribution shifts from a relatively broad and uniform initialization to a state where most values cluster near the two extremes, 0 or 1. Based on this, the paper proposes using the Gumbel-Softmax trick to initialize the gate values to distributions closer to these two extremes. Experimental results on different gate initialization methods show that the proposed initialization approach can accelerate performance improvement and convergence speed for the copying/selective copying tasks (Figures 4 & 6). The paper also compares the language modeling perplexity loss of different gate initialization methods on the WikiText-103 dataset (Table 1), and conducts hyperparameter fine-tuning experiments for the proposed initialization method (Figure 8)."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"My questions have been presented in the WEAKNESSES section."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"(1) The authors observe that in the two small-scale synthetic tasks, Copying and Selective Copying, the forget gate values in gated linear/nonlinear RNN models tend to cluster near the extreme values of 0 and 1 after training, and this phenomenon is analyzed and explored to some extent.\n\n(2) This paper proposes a method for initializing the forget gates using the Gumbel-Softmax trick, which can accelerate both the performance improvement and convergence speed of the models on these two small-scale synthetic tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**(1) The core contribution of this paper may lack generalizability to real-world larger-scale (with parameter size not less than 1B) pre-training and multi-task reasoning scenarios.** \n\nThe gate value distributions observed only from the Copying/Selective Copying tasks may differ significantly from those in large-scale, general language modeling settings in real-world applications. As seen in Figures 1 and 2, there is a substantial proportion of forget gates close to 1 (corresponding to input gates close to 0) after training. I believe this is because in the copying/selective copying tasks, only the segments that need to be copied require memorization, while all other segments are treated as noise and can be largely forgotten. However, in practical applications, the model not only performs retrieval operations but also needs to form summarizing memories of factual information from a large number of segments. Therefore, the extreme situation where the forget gate is close to 1 and the input gate is close to 0 should not be as common as observed in copying/selective copying tasks.  \n\n**(2) Purely from the perspective of controlling gate value distribution, the initialization method proposed in this paper is not very meaningful in deep networks (e.g., 24 layers or more).** \n\nIn shallow networks (such as the 3-layer and 6-layer models discussed in this paper), the token mixing range that is achieved by gate values close to 0 or 1 could, in deeper networks, be equivalently modeled by progressively increasing the lower bound of the forget gate across layers. Mathematically, this approach (which was proposed in the HGRN paper[2]) should be able to fit the same effect.  \n\n**(3) The experimental evaluation is apparently not comprehensive enough.** \n\nRecent studies (for example, Table 1 in the DeltaNet paper [1]) have conducted comprehensive evaluations of model performance at scales such as 340M/1.3B, across more than a dozen commonsense reasoning and retrieval tasks. The persuasiveness of these results is much higher than the language modeling experiments in this paper, which are limited to Wikitext-103.  \n\nMinor weaknesses: \n\n(1) The phrase \"removing the hidden states\" may cause ambiguity; it might be better to use \"decay\" or \"forget\" instead.\n\nReferences:  \n[1] https://arxiv.org/pdf/2406.06484  \n[2] https://arxiv.org/abs/2311.04823"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926481092,"tcdate":1761923904441,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16349/Reviewer_eQho"],"signatures":["ICLR.cc/2026/Conference/Submission16349/Reviewer_eQho"],"forum":"yLA9l9hykz","number":3,"license":"CC BY 4.0","cdate":1761923904441,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16349/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926481092,"domain":"ICLR.cc/2026/Conference","replyto":"yLA9l9hykz","id":"Gh0gsjxsnC","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Recurrent Neural Network","Gate mechanism"]},"supplementary_material":{"value":"/attachment/eabf0830918567b555bcceb1949f51417a439308.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Linear Recurrent Neural Networks (RNNs) have attracted attention for their memory and computational efficiency.\nIn particular, gated linear RNNs enable nonlinear transformations through gating mechanisms while still maintaining linear time complexity by removing hidden states from them.\nHowever, the impact of the gate mechanisms and such removal of hidden states from them remains unexplored.\nHere we empirically investigate the impact of these gating mechanisms and find that gate values near zero or one highly depend on hidden states, leading to unintended distribution shifts of gate values when hidden states are removed in gated linear RNNs.\nBased on our findings, we propose an algorithm to mitigate the distribution shifts, which empirically improves performance on long-sequence modeling tasks."},"_bibtex":{"value":"@misc{\niwamori2026towards,\ntitle={Towards Understanding Gated Linear Recurrent Neural Networks},\nauthor={Toshiya Iwamori and Mahito Sugiyama},\nyear={2026},\nurl={https://openreview.net/forum?id=yLA9l9hykz}\n}"},"title":{"value":"Towards Understanding Gated Linear Recurrent Neural Networks"},"pdf":{"value":"/pdf/9b9edb2b974c5293bac768c2ba3eb824740d18c8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"iwamori|towards_understanding_gated_linear_recurrent_neural_networks"},"authorids":{"value":["~Toshiya_Iwamori1","~Mahito_Sugiyama1"]},"authors":{"value":["Toshiya Iwamori","Mahito Sugiyama"]}},"version":2},{"content":{"venue":{"value":"AAAI 2025"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/32885/35040"},"venueid":{"value":"dblp.org/conf/AAAI/2025"},"paperhash":{"value":"wang|mamba_yolo_a_simple_baseline_for_object_detection_with_state_space_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zeyu_Wang:","https://dblp.org/search/pid/api?q=author:Chen_Li_0025:","","https://dblp.org/search/pid/api?q=author:Xinzhong_Zhu:","https://dblp.org/search/pid/api?q=author:Hongbo_Li:"]},"html":{"value":"https://doi.org/10.1609/aaai.v39i8.32885"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/Wang0XZL25,\n  author={Zeyu Wang and Chen Li and Huiying Xu and Xinzhong Zhu and Hongbo Li},\n  title={Mamba YOLO: A Simple Baseline for Object Detection with State Space Model},\n  year={2025},\n  cdate={1735689600000},\n  pages={8205-8213},\n  url={https://doi.org/10.1609/aaai.v39i8.32885},\n  booktitle={AAAI},\n  crossref={conf/aaai/2025}\n}\n"},"abstract":{"value":"Driven by the rapid development of deep learning technology, the YOLO series has set a new benchmark for real-time object detectors. Additionally, transformer-based structures have emerged as the most powerful solution in the field, greatly extending the model's receptive field and achieving significant performance improvements. However, this improvement comes at a cost, as the quadratic complexity of the self-attentive mechanism increases the computational burden of the model. To address this problem, we introduce a simple yet effective baseline approach called Mamba YOLO. Our contributions are as follows: 1) We propose that the ODMamba backbone introduce a State Space Model (SSM) with linear complexity to address the quadratic complexity of self-attention. Unlike the other Transformer-base and SSM-base method, ODMamba is simple to train without pretraining. 2) For real-time requirement, we designed the macro structure of ODMamba, determined the optimal stage ratio and scaling size. 3) We design the RG Block that employs a multi-branch structure to model the channel dimensions, which addresses the possible limitations of SSM in sequence modeling, such as insufficient receptive fields and weak image localization. This design captures localized image dependencies more accurately and significantly. Extensive experiments on the publicly available COCO benchmark dataset show that Mamba YOLO achieves state-of-the-art performance compared to previous methods. Specifically, a tiny version of Mamba YOLO achieves a 7.5% improvement in mAP on a single 4090 GPU with an inference time of 1.5 ms."},"title":{"value":"Mamba YOLO: A Simple Baseline for Object Detection with State Space Model"},"authors":{"value":["Zeyu Wang","Chen Li","Huiying Xu","Xinzhong Zhu","Hongbo Li"]}},"tmdate":1768388483355,"pdate":1735689600000,"externalIds":["dblp:conf/aaai/Wang0XZL25"],"tcdate":1768387328887,"writers":["~"],"signatures":["~Huiying_Xu1"],"forum":"CxOm9IlJvu","license":"CC BY-SA 4.0","number":751316,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1768388483355,"domain":"DBLP.org","id":"CxOm9IlJvu","version":2},{"content":{"TLDR":{"value":"We introduce a tensor attention framework and propose log-linear attention, which expands beyond fixed-size hidden states to achieve log-linear complexity."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["subquadratic architecture","triton kernel","structured matrices"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The attention mechanism in Transformers is an important primitive for accurate and scalable sequence modeling. Its quadratic-compute and linear-memory complexity however remain significant bottlenecks. Linear attention and state-space models enable linear-time, constant-memory sequence modeling and can moreover be trained efficiently through matmul-rich parallelization across sequence length. However, at their core these models are still RNNs, and thus their use of a fixed-size hidden state to model the context is a fundamental limitation. This paper develops log-linear attention, an attention mechanism that balances linear attention's efficiency and the expressiveness of softmax attention. Log-linear attention replaces the fixed-size hidden state with a logarithmically growing set of hidden states. We show that with a particular growth function, log-linear attention admits a similarly matmul-rich parallel form whose compute cost is log-linear in sequence length. Log-linear attention is a general framework and can be applied on top of existing linear attention variants. As case studies, we instantiate log-linear variants of two recent architectures---Mamba-2 and Gated DeltaNet---and find they perform well compared to their linear-time variants."},"_bibtex":{"value":"@inproceedings{\nguo2026loglinear,\ntitle={Log-Linear Attention},\nauthor={Han Guo and Songlin Yang and Tarushii Goel and Eric P. Xing and Tri Dao and Yoon Kim},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=mOJgZWkXKW}\n}"},"title":{"value":"Log-Linear Attention"},"pdf":{"value":"/pdf/481db6dd1f1973193f58822e77e56402f31473e9.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"guo|loglinear_attention"},"authorids":{"value":["~Han_Guo1","~Songlin_Yang1","~Tarushii_Goel1","~Eric_P._Xing1","~Tri_Dao1","~Yoon_Kim1"]},"authors":{"value":["Han Guo","Songlin Yang","Tarushii Goel","Eric P. Xing","Tri Dao","Yoon Kim"]}},"tmdate":1775877345968,"pdate":1769436590385,"tcdate":1758327501462,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22191/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission22191/Authors"],"forum":"mOJgZWkXKW","license":"CC BY 4.0","number":22191,"cdate":1758327501462,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission22191/-/Full_Submission","ICLR.cc/2026/Conference/Submission22191/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission22191/-/Camera_Ready_Revision"],"mdate":1775877345968,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"mOJgZWkXKW","version":2},{"content":{"summary":{"value":"This paper proposes SF-Mamba, a novel visual encoder addressing limitations of existing Vision Transformers and visual Mamba models. ViTs suffer from quadratic complexity, while visual Mamba faces non-causal information flow constraints and inefficiency with short tokens. SF-Mamba introduces two core innovations: auxiliary patch swapping for bidirectional information flow under unidirectional scanning, and batch folding with periodic state reset to enhance GPU parallelism. Extensive experiments on image classification, object detection, and segmentation demonstrate superior accuracy-throughput trade-offs compared to SOTA baselines."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please refer to the weakness part."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. This paper targets critical pain points of visual Mamba (causality constraint, short-sequence inefficiency) with lightweight, non-intrusive solutions.\n2. Comprehensive validation across three fundamental vision tasks (classification, detection, segmentation) with consistent performance gains.\n3. Practical optimizations (e.g., adaptive $\\(B_1\\)$, Triton kernel for swapping) enhance real-world applicability, with code release planned."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The macro-architecture heavily relies on MambaVision’s hybrid (Mamba+Attention) design, lacking significant innovations in overall network structure.\n2. Ablation studies on auxiliary token initialization and discard timing are limited; deeper analysis of their impact on different tasks is needed.\n3. No discussion on generalization to ultra-high-resolution images or low-resource devices (e.g., edge GPUs), restricting scope insights."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917964610,"tcdate":1761807882736,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5237/Reviewer_V3Nn"],"signatures":["ICLR.cc/2026/Conference/Submission5237/Reviewer_V3Nn"],"forum":"X4KsowemNB","number":3,"license":"CC BY 4.0","cdate":1761807882736,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5237/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917964610,"domain":"ICLR.cc/2026/Conference","replyto":"X4KsowemNB","id":"623UxrM3Ux","forumContent":{"TLDR":{"value":"A high-performance, high-speed vision encoder based on improved information flow and GPU parallelism"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mamba","State space model","vision encoder"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"The realm of Mamba for vision has been advanced in recent years to strike for the alternatives of Vision Transformers (ViTs) that suffer from the quadratic complexity.\nWhile the recurrent scanning mechanism of Mamba offers computational efficiency, it inherently limits non-causal interactions between image patches.\nPrior works have attempted to address this limitation through various multi-scan strategies; however, these approaches suffer from inefficiencies due to suboptimal scan designs and frequent data rearrangement. Moreover, Mamba exhibits relatively slow computational speed under short token lengths, commonly used in visual tasks.\nIn pursuit of a truly efficient vision encoder, we rethink the scan operation for vision and the computational efficiency of Mamba.\nTo this end, we propose SF-Mamba, a novel visual Mamba with two key proposals: auxiliary patch swapping for encoding bidirectional information flow under an unidirectional scan and batch folding with periodic state reset for advanced GPU parallelism.\nExtensive experiments on image classification, object detection, and instance and semantic segmentation consistently demonstrate that our proposed SF-Mamba significantly outperforms state-of-the-art baselines while improving throughput across different model sizes.\nWe will release the source code after publication."},"_bibtex":{"value":"@misc{\nyoshimura2026sfmamba,\ntitle={{SF}-Mamba: Rethinking State Space Model for Vision},\nauthor={Masakazu Yoshimura and Teruaki Hayashi and Yuki Hoshino and Wei-Yao Wang and Takeshi Ohashi},\nyear={2026},\nurl={https://openreview.net/forum?id=X4KsowemNB}\n}"},"title":{"value":"SF-Mamba: Rethinking State Space Model for Vision"},"pdf":{"value":"/pdf/9683eabb7692d6dc64ba96b03dee4200513e80d4.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yoshimura|sfmamba_rethinking_state_space_model_for_vision"},"authorids":{"value":["~Masakazu_Yoshimura1","~Teruaki_Hayashi1","~Yuki_Hoshino1","~Wei-Yao_Wang1","~Takeshi_Ohashi1"]},"authors":{"value":["Masakazu Yoshimura","Teruaki Hayashi","Yuki Hoshino","Wei-Yao Wang","Takeshi Ohashi"]}},"version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Mamba","quadratic stability","Lyapunov","system identification","Lipschitz robustness"]},"_bibtex":{"value":"@inproceedings{\ncao2026a,\ntitle={A Control-Theoretic View of Mamba on Stability and Robustness},\nauthor={Liang Cao and Weide Liu and Zhuo Chen and Yan Qin},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=kLFm50q9kO}\n}"},"title":{"value":"A Control-Theoretic View of Mamba on Stability and Robustness"},"paperhash":{"value":"cao|a_controltheoretic_view_of_mamba_on_stability_and_robustness"},"originally_submitted_PDF":{"value":"/pdf/46968ec117239654b47837482d8849f8ff35f45f.pdf"},"primary_area":{"value":"general_machine_learning->sequential_network_and_time_series_modeling"},"abstract":{"value":"Selective State Space Models (SSMs) such as Mamba have emerged as efficient alternatives to Transformers, achieving linear complexity through input-dependent parameterization. However, this selectivity transforms the system from linear time-invariant (LTI) to linear parameter-varying (LPV), where individually stable matrices can produce unbounded trajectories under switching. Existing work focuses on empirical performance, leaving global stability, robustness bounds, and practical certification unresolved. This paper develops a control-theoretic framework providing a comprehensive stability and robustness analysis for selective SSMs. We prove BIBO stability by viewing selective scans as continuous-time LTI sampling and establish two-term robustness bounds with linear growth in sequence length. For general LPV systems, we provide common quadratic Lyapunov function conditions and develop algorithms to extract certificate constants directly from network weights. These results bridge control theory and SSM architectures, providing verifiable certificates for individual selective-SSM layers as a step toward formal guarantees for deployment."},"pdf":{"value":"/pdf/73fda5fc0b72b4b0065e1f6136f7ac2acb9f6a23.pdf"},"lay_summary":{"value":"Mamba is a fast new AI architecture for processing long sequences such as text or sensor data. Its speed comes from continuously adapting its internal computation to each input. This flexibility makes the model effective but mathematically tricky, because such adaptive systems can in principle behave erratically or become unstable.\n\nWe provide the comprehensive mathematical guarantees that Mamba is well-behaved. Its internal state cannot blow up, small input perturbations cause only proportional output changes that grow slowly with sequence length rather than explosively, and these guarantees can be computed directly from a trained model's weights. The analysis bridges Mamba and classical control theory, giving practitioners tools to verify these models before deploying them in settings where reliability matters, such as medical monitoring or industrial control."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Liang_Cao1","~Weide_Liu2","~Zhuo_Chen9","~Yan_Qin4"]},"authors":{"value":["Liang Cao","Weide Liu","Zhuo Chen","Yan Qin"]}},"tmdate":1790067323877,"pdate":1777576296872,"tcdate":1768823404377,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission8139/Authors"],"signatures":["ICML.cc/2026/Conference/Submission8139/Authors"],"forum":"kLFm50q9kO","license":"CC BY 4.0","number":8139,"cdate":1768823404377,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission8139/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission8139/-/Camera_Ready_Revision"],"mdate":1790067323877,"odate":1782341928854,"domain":"ICML.cc/2026/Conference","id":"kLFm50q9kO","version":2},{"content":{"summary":{"value":"This paper studies the ICL ability of Mamba on single-index models by characterizing the optimization dynamics and the sample complexity in pretraining and testing. The theoretical results indicate that Mamba can implement test-time feature learning, which is enabled by the nonlinear gating. The authors also make a comparison between their results and previous works and other archtectures."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Why do you need an MLP layer in your analysis? In line 365, you say the improvement of sample complexity comes from the MLP, then you need to clarify what makes Mamba better than Transformers in some aspect because MLP is not a key difference between Mamba and Transformers from my understanding. You need to distinguish what improvement comes from MLP and what comes from the key component of Mamba, i.e, the nonlinear gating. \n\n2. Why do you formulate $W_B^\\top W_C$ with a simple veector $\\gamma$ and why do you initialize $\\gamma$ in a sepcial way in line 256?\n\n3. It seems you reduce the analysis of a $d+1$-dimensional $w$ to a scalar $\\rho$. What is the reason?\n\n4. Why do you formulate $\\phi(\\theta)$ in a special way in line 154? Can you handle of case of $\\phi(x)=x$?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The theoretical analysis is quite strong and solid.\n\n2. The studied problem on Mamba with single-index models is interesting and novel."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The illustration in Section 4 is confusing to me. \n\n(a) Overall, I think many of this section is not a \"proof overview\" but some important discussion that you can put in previous sections. In other words, in Section 4.1, you introduce too much about some theoretical conclusions or the differences from existing works rather than the logic chain to derive your main theorem, which should be the proof idea from my understanding. \n\n(b). Regarding the role of gating and input embedding, $c_\\beta$ vanishes when $ie(g_*)>2$ if it is linear attention, but what about Mamba? Do you mean $c_\\beta$ will not vanish in this case? When I first read it, I got confused because I thought you would directly answer this question, but you started to talk about the generative exponent. I think it is better to first answer this question or briefly summarize the role of gating at the beginning of this paragraph. \n\n(c). Proposition 4.1 is still not \"informal\" enough. Why not directly write $Mamba(\\cdot)$ as a $g_*(\\cdot)$ function in Eqn. 2? You mention this possibility in line 433 that $g_*(z)$ is a polynomial of $z^{ge(g_*)}$. By the way, I took some time to recall what $P_1$ and $P_2$ are in this proposition. I found they are defined in line 300. It is better to intuitively explain them around this proposition.\n\n2. Not enough experiments are presented. Can you empirically show the advantages of Mamba in sample complexity compared with Transformers, and can you empirically relate it to the nonlinear gating part?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933992375,"tcdate":1761777730762,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20583/Reviewer_cryr"],"signatures":["ICLR.cc/2026/Conference/Submission20583/Reviewer_cryr"],"forum":"3KPsog4mpy","number":2,"license":"CC BY 4.0","cdate":1761777730762,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20583/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933992375,"domain":"ICLR.cc/2026/Conference","replyto":"3KPsog4mpy","id":"zCZnYQxHNl","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We study in-context learning of Mamba on single-index models"},"keywords":{"value":["Mamba","In-Context Learning","Feature Learning","Theory"]},"primary_area":{"value":"learning theory"},"abstract":{"value":"Mamba, a recently proposed linear-time sequence model, has attracted significant attention for its computational efficiency and strong empirical performance. However, a rigorous theoretical understanding of its underlying mechanisms remains limited. In this work, we provide a theoretical analysis of Mamba's in-context learning (ICL) capability by focusing on tasks defined by low-dimensional nonlinear target functions. Specifically, we study in-context learning of a single-index model $y \\approx g_*(\\langle \\boldsymbol{\\beta}, \\boldsymbol{x} \\rangle)$, which depends on only a single relevant direction $\\boldsymbol{\\beta}$, referred to as *feature*. We prove that Mamba, pretrained by gradient-based methods, can achieve efficient ICL via *test-time feature learning*, extracting the relevant direction directly from context examples. Consequently, we establish a test-time sample complexity that improves upon linear Transformers—analyzed to behave like kernel methods—and is comparable to nonlinear Transformers, which have been shown to surpass the Correlational Statistical Query (CSQ) lower bound and achieve near information-theoretically optimal rate in previous works. Our analysis reveals the crucial role of the *nonlinear gating* mechanism in Mamba for feature extraction, highlighting it as the fundamental driver behind Mamba’s ability to achieve both computational efficiency and high performance."},"_bibtex":{"value":"@misc{\noh2026mamba,\ntitle={Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning},\nauthor={Junsoo Oh and Wei Huang and Taiji Suzuki},\nyear={2026},\nurl={https://openreview.net/forum?id=3KPsog4mpy}\n}"},"title":{"value":"Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning"},"pdf":{"value":"/pdf/b872836f3f5ba22e086f394ffe799c2dd9c8c920.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"oh|mamba_can_learn_lowdimensional_targets_incontext_via_testtime_feature_learning"},"authorids":{"value":["~Junsoo_Oh1","~Wei_Huang6","~Taiji_Suzuki1"]},"authors":{"value":["Junsoo Oh","Wei Huang","Taiji Suzuki"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Mixture-of-Memories (MoM), an architecture designed to enhance linear sequence models by addressing their fundamental limitation: compressing entire sequences into a single fixed-size memory state. MoM employs multiple independent memory states with a router network that directs input tokens to specific memories using a top-k selection mechanism. The approach is inspired by theta-gamma oscillations in the hippocampus and borrows conceptually from Mixture-of-Experts (MoE) architectures. The authors demonstrate that MoM maintains O(n) training complexity and O(1) inference complexity while significantly improving performance on recall-intensive tasks. Experiments on 380M and 1.3B parameter models show MoM outperforms existing linear sequence models and approaches Transformer performance on recall tasks, though with modest increases in parameter count. The work provides a general framework compatible with diverse memory update mechanisms and includes a hardware-efficient implementation using varlen operations."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Could shared memory become a new bottleneck? If all tokens access shared memory, will its capacity quickly become saturated? Have you analyzed the actual utilization of shared memory across different layers?\n2. More details about reproduction：\n\ta. What is the computational overhead of the router network?\n\tb. How sensitive is performance to router initialization?\n\tc. What are actual wall-clock training times compared to baselines?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Well-motivated problem formulation: The paper clearly identifies memory interference and limited capacity as core issues in linear sequence models, providing both intuitive explanations and neuroscience-inspired motivation from hippocampal theta-gamma oscillations.\n2. General and flexible framework: MoM's compatibility with various memory update mechanisms (Table 1 lists 11 different methods) demonstrates its generality. This is a significant practical advantage, allowing easy integration with future linear modeling innovations.\n3. Comprehensive experimental evaluation: The paper includes extensive experiments across multiple benchmarks (recall-intensive tasks, long-context tasks, commonsense reasoning), multiple model scales (380M, 1.3B parameters), and thorough ablations (memory count, activation ratios, auxiliary loss, shared memory).\n4. Strong empirical results on target tasks: MoM achieves substantial improvements on recall-intensive tasks (28.16 vs 24.78 average for Gated DeltaNet at 380M; 36.04 vs 32.30 at 1.3B), effectively narrowing the gap with Transformers (31.70 and 37.31 respectively).\n5. Hardware-efficient implementation: Section 3.3 provides a mathematically rigorous description of the varlen-based implementation, demonstrating practical viability. Figure 3 shows clear efficiency advantages over Transformers at long sequences."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. While related work in linear sequence modeling and MoE has been mentioned, it lacks in-depth comparisons and differential analyses with recent methods (such as RWKV-7 and Titans). In particular, Table 1 lists these methods but does not systematically experimentally compare the performance differences of MoM under different memory update mechanisms.\n2. Table 5 claims to have discovered \"specialization\" in different memories, but it is based solely on qualitative observations of an intermediate layer and lacks systematic and statistical significance verification. This \"discovery\" is more like cherry-picking data than rigorous analysis.\n3. The authors claim that the core advantage of MoM is \"eliminates memory interference\" by separating memory states, but also introducing shared memory to capture global information. Does this contradict the core design? In particular, in the ablation of table 6, there seems to be a lack of settings for \"shared memory only\" and \"neither shared memory nor single memory exists\". This lack of ablation experiment is fatal because it directly relates to whether \"mixture\" is truly necessary."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360414338,"tcdate":1761808880900,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8491/Reviewer_SvM8"],"signatures":["ICLR.cc/2026/Conference/Submission8491/Reviewer_SvM8"],"forum":"3PdOq8Rgue","number":2,"license":"CC BY 4.0","cdate":1761808880900,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8491/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360414338,"domain":"ICLR.cc/2026/Conference","replyto":"3PdOq8Rgue","id":"xCRA0zbMMP","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Efficient/Low-Resource Methods for NLP","Linear Sequence Modeling","Machine Learning for NLP"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Linear sequence modeling methods, such as linear attention, state space modeling, and linear RNNs, offer significant efficiency improvements by reducing the complexity of training and inference. However, these methods typically compress the entire input sequence into a single fixed-size memory state, which leads to suboptimal performance on recall-intensive tasks. To address this limitation, we introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes multiple independent memory states, with a router network directing input tokens to specific memory states. This approach greatly enhances the overall memory capacity while minimizing memory interference. MoM serves as a general framework that can be seamlessly combined with diverse memory update mechanisms across linear models. As a result, MoM performs exceptionally well on recall-intensive tasks, surpassing existing linear sequence modeling techniques. Despite incorporating multiple memory states, the computation of each memory state remains linear in complexity, allowing MoM to retain the linear-complexity advantage during training, while constant-complexity during inference. Our experimental results show that MoM outperforms current linear sequence models on downstream language tasks, particularly recall-intensive tasks, and even achieves performance comparable to Transformer models."},"_bibtex":{"value":"@inproceedings{\ndu2026mom,\ntitle={MoM: Linear Sequence Modeling with Mixture-of-Memories},\nauthor={Jusen Du and Weigao Sun and Disen Lan and Jiaxi Hu and Tao Zhang and Yu Cheng},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=3PdOq8Rgue}\n}"},"title":{"value":"MoM: Linear Sequence Modeling with Mixture-of-Memories"},"pdf":{"value":"/pdf/6d8ebeff897bedb6faacfae3dae56e0c956cf7ca.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"du|mom_linear_sequence_modeling_with_mixtureofmemories"},"authorids":{"value":["~Jusen_Du1","~Weigao_Sun1","~Disen_Lan1","~Jiaxi_Hu1","~Tao_Zhang9","~Yu_Cheng1"]},"authors":{"value":["Jusen Du","Weigao Sun","Disen Lan","Jiaxi Hu","Tao Zhang","Yu Cheng"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Universal Intermittent Trajectory Predictor, a universal trajectory prediction framework for autonomous driving that simultaneously handles variable-length historical trajectories and missing observations—common real-world challenges caused by sensor occlusions, communication delays, and temporal sparsity. The authors introduce three key components: (1) a time-aware input representation using timestamps and inter-observation intervals; (2) a Bidirectional Time Decay Mamba module that extends the Mamba architecture to capture forward and backward temporal dependencies while mitigating information gaps from missing data; and (3) a unified prediction module that encodes motion states and models multimodal future distributions. Extensive experiments on Argoverse 1 and Argoverse 2 demonstrate consistent performance gains across diverse observation conditions."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. Can the authors provide inference speed (FPS) or model size comparisons? Is UIT-Pred feasible for real-time deployment on embedded automotive platforms?\n2. Does the bidirectional processing in BTD-Mamba violate causality during inference? If used online, how is the backward pass implemented without future observations?\n3. Have the authors considered modeling missingness as a function of scene context (e.g., occlusion by nearby agents or map geometry), rather than relying solely on time intervals?\n4. Is the model trained on mixed data (full + missing + variable-length), or can it generalize to missing scenarios when trained only on complete trajectories?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The joint modeling of variable-length inputs and missing data aligns closely with practical autonomous driving scenarios, addressing a notable gap in existing methods that assume fixed-length, complete trajectories.\n2. Comprehensive experiments: The evaluation covers four realistic input conditions and compares against strong baselines (e.g., LaKD, RSD, DTO).\n3. The paper includes detailed component-wise analyses (TAIR, BTD-Mamba, prediction module) and auxiliary loss ablations, validating the contribution of each design choice."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The model integrates BTD-Mamba, Transformers, GRUs, cross-attention, and multiple auxiliary tasks, raising concerns about computational cost and real-time deployability, critical for autonomous systems, but these aspects are not discussed.\n2. While the method outperforms variants of HiVT and QCNet, it lacks comparison with cutting-edge approaches such as diffusion-based predictors or hybrid SSM-Transformer models under the same intermittent observation settings.\n3. The use of a negative exponential decay (Eq. 5) is intuitive but not rigorously motivated or compared against alternatives (e.g., learnable gates, linear decay), leaving room for doubt about its optimality.\n4. Missing observations are generated via random drop (RSD), which may not reflect real-world structured missingness (e.g., prolonged occlusions behind large vehicles), limiting the practical validation of robustness."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926860236,"tcdate":1761553638585,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16838/Reviewer_sW8a"],"signatures":["ICLR.cc/2026/Conference/Submission16838/Reviewer_sW8a"],"forum":"ThZeZdRxna","number":2,"license":"CC BY 4.0","cdate":1761553638585,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16838/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926860236,"domain":"ICLR.cc/2026/Conference","replyto":"ThZeZdRxna","id":"SQo2k4aDUY","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Trajectory prediction","Autonomous Driving","Attention","Mamba","Intermittent Data"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Trajectory prediction is a fundamental component of autonomous driving, requiring models that can handle intermittent observation patterns such as variable-length histories and missing data. Existing state-of-the-art methods, however, often assume fixed-length trajectories and complete input, which challenges their applicability in real-world scenarios where sensor occlusions, communication delays, and temporal sparsity are common. Moreover, conventional approaches typically address tasks such as trajectory prediction, variable-length modeling, or missing data handling in isolation, making them less effective in multi-task settings that naturally arise in practice. To address these challenges, we propose Universal Intermittent Trajectory Predictor (UIT-Pred) that processes inputs with the time index features, which capture temporal variations to effectively adapt to diverse input patterns within the domain. Particularly, We extend recent State Space Models (SSMs) by introducing the Bidirectional Time Decay Mamba (BTD-Mamba), designed to capture dependencies both forward and backward along the sequence. By integrating a decay process, BTD-Mamba effectively analyzes trajectories while maintaining relationships under intermittent observation. Furthermore, the proposed prediction module employs state encoding to capture the underlying motion patterns in the input data and models a multimodal trajectory distribution to account for uncertainty in future predictions. These components are fused through a unified fusion module, enabling the model to jointly reason over observed dynamics and potential future behaviors. Extensive experiments on Argoverse 1 and Argoverse 2 datasets validate the effectiveness of the proposed model. By simultaneously handling prediction, variable-length observations, and missing inputs within a universal architecture, the framework proposes to meet the challenges of real-world autonomous driving systems."},"_bibtex":{"value":"@misc{\nbharilya2026uitpred,\ntitle={{UIT}-Pred: Universal Intermittent Trajectory Predictor for Autonomous Driving},\nauthor={Vibha Bharilya and Neetesh Kumar},\nyear={2026},\nurl={https://openreview.net/forum?id=ThZeZdRxna}\n}"},"title":{"value":"UIT-Pred: Universal Intermittent Trajectory Predictor for Autonomous Driving"},"pdf":{"value":"/pdf/09ddf9106e39f3ead7441bc862c70e71d213fcad.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"bharilya|uitpred_universal_intermittent_trajectory_predictor_for_autonomous_driving"},"authorids":{"value":["~Vibha_Bharilya1","~Neetesh_Kumar1"]},"authors":{"value":["Vibha Bharilya","Neetesh Kumar"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.06467v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"liu|point_mamba_a_novel_point_cloud_backbone_based_on_state_space_model_with_octreebased_ordering_strategy"},"authorids":{"value":["~Jiuming_Liu1","https://dblp.org/search/pid/api?q=author:Ruiji_Yu:","https://dblp.org/search/pid/api?q=author:Yian_Wang:","https://dblp.org/search/pid/api?q=author:Yu_Zheng:","~Tianchen_Deng1","~Weicai_Ye1","https://dblp.org/search/pid/api?q=author:Hesheng_Wang_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.06467"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-06467,\n  publtype={informal},\n  author={Jiuming Liu and Ruiji Yu and Yian Wang and Yu Zheng and Tianchen Deng and Weicai Ye and Hesheng Wang},\n  title={Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategy},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.06467},\n  url={https://doi.org/10.48550/arXiv.2403.06467}\n}\n"},"abstract":{"value":"Recently, state space model (SSM) has gained great attention due to its promising performance, linear complexity, and long sequence modeling ability in both language and image domains. However, it is non-trivial to extend SSM to the point cloud field, because of the causality requirement of SSM and the disorder and irregularity nature of point clouds. In this paper, we propose a novel SSM-based point cloud processing backbone, named Point Mamba, with a causality-aware ordering mechanism. To construct the causal dependency relationship, we design an octree-based ordering strategy on raw irregular points, globally sorting points in a z-order sequence and also retaining their spatial proximity. Our method achieves state-of-the-art performance compared with transformer-based counterparts, with 93.4% accuracy and 75.7 mIOU respectively on the ModelNet40 classification dataset and ScanNet semantic segmentation dataset. Furthermore, our Point Mamba has linear complexity, which is more efficient than transformer-based methods. Our method demonstrates the great potential that SSM can serve as a generic backbone in point cloud understanding. Codes are released at https://github.com/IRMVLab/Point-Mamba."},"title":{"value":"Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategy"},"authors":{"value":["Jiuming Liu","Ruiji Yu","Yian Wang","Yu Zheng","Tianchen Deng","Weicai Ye","Hesheng Wang"]}},"tmdate":1741146980856,"pdate":1704067200000,"tcdate":1730685369894,"writers":["~"],"signatures":["~Jiuming_Liu1"],"forum":"Z27J9L3IDQ","license":"CC BY-SA 4.0","number":168935,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1741146980856,"domain":"DBLP.org","id":"Z27J9L3IDQ","version":2},{"content":{"comment":{"value":"> **W2: Baselines seem to be cherry-picked since they are neither the most powerful nor fundamentally relevant sequence models.**\n\nThank you for raising this concern. To further strengthen the credibility of our results, we have expanded our baseline coverage to include several widely adopted and recent architectures. (1) At the 600M-parameter, 15B-token scale, we trained Mamba, Mamba2, and KDA, and (2) at the 2B-parameter, 100B-token scale, we further evaluated Mamba2. The results show that **SSE continues to deliver the strongest performance**, particularly on recall-intensive tasks such as FDA.\n\n\n1. At the 600M-parameter & 15B-token scale, we trained Mamba [1], Mamba2 [2] , and KDA [3] (a recent linear attention method from Moonshot).\n\n| Model            | Wiki↓ | PIQA  | Hella. | Wino. | ARC-e | ARC-c | SIQA  | **CSR-Avg.** | -  | FDA   | SWDE  | SQuAD | **Recall-Avg.**  |\n|------------------|-------|-------|--------|-------|-------|-------|-------|--------------|----|--------|--------|--------|------------------|\n| Transformer      | 33.47 | 61.92 | 31.60  | 51.38 | 48.02 | 23.12 | 37.31 | 42.22        | -  | 74.50 | 60.67 | 32.67 | 55.95 |\n| GLA              | 43.97 | 61.21 | 29.61  | 51.46 | 47.01 | 23.29 | 36.59 | 41.53        | -  | 9.44  | 23.40 | 23.06 | 18.63 |\n| Mamba            | 36.62 | 63.82 | 32.20  | 50.43 | 49.62 | 23.38 | 38.59 | 43.01        | -  | 6.35  | 21.60 | 24.93 | 17.63 |\n| Mamba2           | 34.89 | 62.89 | 32.84  | 48.93 | 49.20 | 23.81 | 37.92 | 42.60        | -  | 20.51 | 30.42 | 27.21 | 26.05 |\n| KDA              | 34.76 | 62.19 | 31.81  | 51.22 | 49.96 | 24.83 | 37.41 | 42.90        | -  | 24.50 | 35.55 | 26.71 | 28.92 |\n| GDN              | 35.26 | 62.79 | 32.16  | 51.85 | 50.46 | 24.23 | 36.80 | **43.05**    | -  | 14.79 | 33.48 | 26.24 | 24.84 |\n| **SSE-n4k1 (Ours)**     | 35.52 | 61.59 | 31.71  | 51.62 | 49.45 | 24.74 | 38.33 | 42.91        | -  | 33.67 | 33.30 | 26.51 | **31.16** |\n| **SSE-n4k2 (Ours)**     | 35.33 | 62.95 | 31.64  | 50.43 | 49.66 | 25.34 | 38.28 | **43.05**    | -  | 30.85 | 36.90 | 27.82 | **31.86** |\n| **SSE-GDN-n4k1 (Ours)** | **34.15** | 63.11 | 32.22 | 50.36 | 50.59 | 23.98 | 37.46 | 42.95 | - | 41.56 | 41.13 | 30.83 | **37.84** |\n\n2. At the 2B-parameter & 100B-token scale, we further conducted experiments for Mamba2. \n\n| Model        | Wiki↓ | PIQA | Hella. | Wino. | ARC-e | ARC-c | SIQA | CSR-Avg. | - | FDA | SWDE | SQuAD | Recall-Avg. |\n|--------------|-------|------|--------|-------|--------|--------|-------|-----------|---|------|-------|--------|--------------|\n| Transformer  | 16.46 | 71.82 | 52.70 | 58.80 | 63.76 | 31.31 | 42.89 | 53.55 | - | 86.48 | 82.99 | 49.53 | 73.00 |\n| GLA          | 21.30 | 68.44 | 42.82 | 54.38 | 59.43 | 28.24 | 41.45 | 49.13 | - | 51.27 | 59.86 | 36.73 | 49.29 |\n| Mamba2       | 17.27 | 71.87 | 53.34 | 57.06 | 64.52 | 32.59 | 42.17 | 53.59 | - | 56.62 | 63.28 | 43.06 | 54.32 |\n| GDN          | 17.08 | 72.25 | 53.20 | 57.70 | 65.61 | 32.68 | 42.89 | 54.05 | - | 54.63 | 66.88 | 39.88 | 53.79 |\n| **SSE-n4k1 (Ours)** | **16.92** | 71.98 | 53.62 | 60.14 | 66.67 | 32.94 | 42.07 | **54.57** | - | 71.23 | 69.94 | 43.20 | **61.46** |\n\n---\n\n**Note:** For fairness, all models differ only in the attention mixer and do *not* include convolutional layers, except for the Mamba family, which inherently uses 1D convolutions and unified blocks.\n\n[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. https://arxiv.org/abs/2312.00752   \n[2] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. https://arxiv.org/abs/2405.21060      \n[3] Kimi Linear: An Expressive, Efficient Attention Architecture. https://arxiv.org/abs/2510.26692\n\n---"},"title":{"value":"[2/3] Response to Reviewer Jg4D (W2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763523659335,"tcdate":1763523659335,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8934/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8934/Authors"],"forum":"R6DrJ4tnGV","number":5,"license":"CC BY 4.0","cdate":1763523659335,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8934/-/Official_Comment"],"mdate":1763523659335,"domain":"ICLR.cc/2026/Conference","replyto":"yOU5Zecd75","id":"7IYuJROgn0","forumContent":{"TLDR":{"value":"SSE is a partitioned state expansion method under a row-sparse update framework for linear attention, improving retrieval and reasoning."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Linear Attention","Language Model"]},"supplementary_material":{"value":"/attachment/6e3475d7248015327464c0fe2192bc1fb8d0142d.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The Transformer architecture, despite its widespread success, struggles with long-context scenarios due to quadratic computation and linear memory growth. While various linear attention variants mitigate these efficiency constraints by compressing context into fixed-size states, they often degrade performance in tasks such as in-context retrieval and reasoning. To address this limitation and achieve more effective context compression, we propose two key innovations. First, we introduce a row-sparse update formulation for linear attention by conceptualizing state updating as information categorization. This enables sparse state updates via softmax-based top-$k$ row selection, thereby extending receptive fields and reducing information interference. Second, we present Sparse State Expansion (SSE) within the sparse framework, which expands the contextual state into multiple partitions, effectively decoupling parameter size from state capacity while maintaining the sparse row-selection paradigm. Supported by efficient parallelized implementations, our design achieves highly discriminative state representations. We extensively validate SSE in both pure linear and hybrid (SSE-H) architectures across language modeling, in-context retrieval, and mathematical reasoning benchmarks. SSE demonstrates strong retrieval performance and scales favorably with state size. Moreover, after reinforcement learning (RL) training, our 2B SSE-H model achieves state-of-the-art mathematical reasoning performance among small reasoning models, scoring 64.5 on AIME24 and 50.2 on AIME25, significantly outperforming similarly sized open-source Transformers. These results highlight SSE as a promising and efficient architecture for long-context modeling."},"_bibtex":{"value":"@inproceedings{\npan2026scaling,\ntitle={Scaling Linear Attention Capacity with Sparse State Expansion},\nauthor={Yuqi Pan and Yongqi An and Zheng Li and Yuhong Chou and Rui-Jie Zhu and Xiaohui Wang and Mingxuan Wang and Jinqiao Wang and Guoqi Li},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=R6DrJ4tnGV}\n}"},"title":{"value":"Scaling Linear Attention Capacity with Sparse State Expansion"},"pdf":{"value":"/pdf/73d9d62767b69cd0649b8caaa9d65c613f858782.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"pan|scaling_linear_attention_capacity_with_sparse_state_expansion"},"authorids":{"value":["~Yuqi_Pan2","~Yongqi_An1","~Zheng_Li48","~Yuhong_Chou1","~Rui-Jie_Zhu2","~Xiaohui_Wang4","~Mingxuan_Wang1","~Jinqiao_Wang1","~Guoqi_Li1"]},"authors":{"value":["Yuqi Pan","Yongqi An","Zheng Li","Yuhong Chou","Rui-Jie Zhu","Xiaohui Wang","Mingxuan Wang","Jinqiao Wang","Guoqi Li"]}},"version":2},{"content":{"summary":{"value":"This work introduces Mamba to FSS. Particularly, it indicates two issues suffered by the original Mamba when being applied to the cross attention case, namely, the support forgetting issue and the intra-class gap issue.\n To address them, a hybrid Mamba block (HMB) is designed, which can effectively incorporate query FG features with the support  \nFG features, thereby leading to better FSS performance"},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"My questions are written in the above Weakness."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. HMB introduces Mamba into FSS, which reduces the computational cost from quadratic complexity to linear complexity.\n2. The illustration of HMB is clear and easy to understand.\n3. Experiments show the method can achieve state-of-the-art."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Introducing mamba into FFS is to reduce the computational cost, but no experimental results support it. Comparing the efficiency of the proposal method with it of cross-attention based methods is needed.\n2. Writing is not standard somewhere. For instance, few-shot segmentation includes few-shot semantic segmentation and few-shot instance segmentation, but the work didn't refer to any instance segmentation task.\n3. The content in coco 'train_data_list' file of the supplementary  code  leaks your identity. I don't know if it violates the double-blind rule."},"limitations":{"value":"This work addresses the limitations(computational cost) in theory， but no experimental results can prove it.\nBesides, this work don't have negative societal impact."}},"nonreaders":[],"tmdate":1730878780409,"tcdate":1720427904219,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2498/Reviewer_noa5"],"signatures":["NeurIPS.cc/2024/Conference/Submission2498/Reviewer_noa5"],"forum":"Qe2BKeCEBC","number":1,"license":"CC BY 4.0","cdate":1720427904219,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2498/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878780409,"domain":"NeurIPS.cc/2024/Conference","replyto":"Qe2BKeCEBC","id":"BLbHDrRgWF","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["few-shot segmentation","mamba","intra-sequence dependencies"]},"supplementary_material":{"value":"/attachment/c2e47c3a1b58104b90d7e4c87b1052435097d66e.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Many few-shot segmentation (FSS) methods use cross attention to fuse support foreground (FG) into query features, regardless of the quadratic complexity. A recent advance Mamba can also well capture intra-sequence dependencies, yet the complexity is only linear. Hence, we aim to devise a cross (attention-like) Mamba to capture inter-sequence dependencies for FSS. A simple idea is to scan on support features to selectively compress them into the hidden state, which is then used as the initial hidden state to sequentially scan query features. Nevertheless, it suffers from (1) support forgetting issue: query features will also gradually be compressed when scanning on them, so the support features in hidden state keep reducing, and many query pixels cannot fuse sufficient support features; (2) intra-class gap issue: query FG is essentially more similar to itself rather than to support FG, i.e., query may prefer not to fuse support features but their own ones from the hidden state, yet the success of FSS relies on the effective use of support information. To tackle them, we design a hybrid Mamba network (HMNet), including (1) a support recapped Mamba to periodically recap the support features when scanning query, so the hidden state can always contain rich support information; (2) a query intercepted Mamba to forbid the mutual interactions among query pixels, and encourage them to fuse more support features from the hidden state. Consequently, the support information is better utilized, leading to better performance. Extensive experiments have been conducted on two public benchmarks, showing the superiority of HMNet. The code is available at https://github.com/Sam1224/HMNet."},"_bibtex":{"value":"@inproceedings{\nxu2024hybrid,\ntitle={Hybrid Mamba for Few-Shot Segmentation},\nauthor={Qianxiong Xu and Xuanyi Liu and Lanyun Zhu and Guosheng Lin and Cheng Long and Ziyue Li and Rui Zhao},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=Qe2BKeCEBC}\n}"},"title":{"value":"Hybrid Mamba for Few-Shot Segmentation"},"pdf":{"value":"/pdf/7fca1561230feeb691e2dbccd7f1ef8f79dfa5ec.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"xu|hybrid_mamba_for_fewshot_segmentation"},"authorids":{"value":["~Qianxiong_Xu1","~Xuanyi_Liu3","~Lanyun_Zhu1","~Guosheng_Lin1","~Cheng_Long1","~Ziyue_Li2","~Rui_Zhao6"]},"authors":{"value":["Qianxiong Xu","Xuanyi Liu","Lanyun Zhu","Guosheng Lin","Cheng Long","Ziyue Li","Rui Zhao"]}},"version":2},{"content":{"comment":{"value":"> The question about state size referred to Mamba state size. 16 was used in language. Video is a different domain, hence an investigation on the influence of Mamba state size could be interesting. Or a discussion on the bottleneck that this fairly small state size brings given that the hidden model dimension is fairly large in comparison (1024).\n\n\nThank you for comments. There are probably some misunderstandings regarding the Mamba state size, which we would like to clarify.\n\nThe state size in Mamba is not a parameter like the hidden dimension which can be arbitrarily large (e.g., 512 or 1024). Instead, it is conceptually closer to **the number of attention heads** in an attention mechanism. As stated in the Mamba2[1] paper (here N refers to the state size):\n\n***\"… a single channel of S6 produces N inner attention matrices ([1] Eq. 16).”***\n\nIn this context, 16 is already considered a large value for the state size. The original Mamba[2] paper only explores state sizes from 1 to 16 (shown in its Tab. 10). Additionally, **setting the state size to 16 has also become a consensus in vision-based Mamba architectures**. For example, both Vim[3] and MambaVision[4] use 16 as their state size. VMamba[5] architecture optimize it to 1 to increase throughput. For video tasks, the default state size for VideoMamba[6] is also 16.\n\nTherefore, we would like to clarify that **16 is not a “fairly small” state size and does not present a bottleneck to our model’s performance**.\n\nIncreasing the state size further can actually significantly **decrease the model’s inference speed**. Since the model's complexity is $O(LN^2)$, where $L$ is the sequence length and $N$ is the state size. In typical settings, $N \\ll L$, but if we double the state size, the computational complexity would increase by a factor of 4.\n\nIn response to your suggestion, we performed an additional comparison with a doubled state size. The results are conducted on K400 with ViT-B/16 backbone and 8 frames input.\n\n| State Size | Top-1 | Top-5 |\n| ---------- | ----- | ----- |\n| 16         | 83.7  | 96.5  |\n| 32         | 83.0  | 95.2  |\n\n\nAs shown in the table, increasing the state size from 16 to 32 hurts the performance. This aligns with the findings from the original Mamba paper ([2] Tab. 10).\n\n\n[1]: The Hidden Attention of Mamba Models\n\n[2]: Mamba: Linear-Time Sequence Modeling with Selective State Spaces\n\n[3]: Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (ICML 24)\n\n[4]: MambaVision: A Hybrid Mamba-Transformer Vision Backbone (CVPR 25)\n\n[5]: VMamba: Visual State Space Model (NIPS 24)\n\n[6]: VideoMamba: State Space Model for Efficient Video Understanding (ECCV 24)\n\n\n> I encourage the authors to use carefully the terms fine-tuning vs PEFT. To me, finetuning means updating end-to-end all the parameters of the model, whereas in PEFT only a subset of parameters are updated, the rest are frozen.\n\nThank you for your suggestion. We will be more careful in differentiating fine-tuning and PEFT and avoid misuse. We have updated our script accordingly to avoid any confusion and ensure greater rigor in our terminology.\n\n\n\nFinally, we would like to sincerely thank you for your detailed review and valuable feedback. We truly appreciate the time and effort you invested in carefully reading our paper and providing insightful comments on both our experiments and writing. We hope our response addresses your concerns, and we would be grateful for your consideration in increasing the score."}},"tmdate":1750269770355,"tcdate":1743941367860,"writers":["ICML.cc/2025/Conference","ICML.cc/2025/Conference/Submission202/Authors"],"signatures":["ICML.cc/2025/Conference/Submission202/Authors"],"forum":"AgAn6xtwWO","number":1,"license":"CC BY 4.0","cdate":1743941367860,"readers":["everyone"],"invitations":["ICML.cc/2025/Conference/Submission202/Official_Review3/Rebuttal1/Rebuttal_Comment1/-/Reply_Rebuttal_Comment","ICML.cc/2025/Conference/-/Edit"],"mdate":1750269770355,"domain":"ICML.cc/2025/Conference","replyto":"ThxF3MwFIl","id":"96e0F3v7b5","forumContent":{"venue":{"value":"ICML 2025 poster"},"TLDR":{"value":"We propose MoMa, an efficient adapter that integrates spatial-temporal modeling into pre-trained image foundation models for improved video understanding, achieving better performance with lower computational cost."},"keywords":{"value":["Video Recognition","PEFT","Mamba","Adapter"]},"primary_area":{"value":"applications->computer_vision"},"abstract":{"value":"Video understanding is a complex challenge that requires effective modeling of spatial-temporal dynamics. \nWith the success of image foundation models (IFMs) in image understanding, recent approaches have explored parameter-efficient fine-tuning (PEFT)  to adapt IFMs for video. \nHowever, most of these methods tend to process\nspatial and temporal information separately,\nwhich may fail to capture the full intricacy of video dynamics. \nIn this paper, we propose MoMa, an efficient adapter framework that achieves full spatial-temporal modeling by integrating Mamba's selective state space modeling into IFMs. \nWe propose a novel SeqMod operation to inject spatial-temporal information into pre-trained IFMs, without disrupting their original features. \nBy incorporating SeqMod into a Divide-and-Modulate architecture, MoMa enhances video understanding while maintaining computational efficiency. \nExtensive experiments on multiple video benchmarks demonstrate the effectiveness of MoMa, achieving superior performance with reduced computational cost. \nCodes will be released upon publication."},"_bibtex":{"value":"@inproceedings{\nyang2025moma,\ntitle={MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition},\nauthor={Yuhuan Yang and Chaofan Ma and Zhenjie Mao and Jiangchao Yao and Ya Zhang and Yanfeng Wang},\nbooktitle={Forty-second International Conference on Machine Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=AgAn6xtwWO}\n}"},"title":{"value":"MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition"},"pdf":{"value":"/pdf/93bc4de6d22278a872000efc916c7dc698e40f29.pdf"},"lay_summary":{"value":"Today’s AI models struggle to fully understand videos. While they excel at analyzing static images, they often treat what’s happening (like a ball flying) separately from where it’s happening (like a basketball court). This limits their ability to grasp complex actions, such as a gymnast’s fluid routine.\nWe developed MoMa, an add-on tool that upgrades image-based AI to understand motion naturally. Inspired by how humans track movement, MoMa uses SeqMod—a technique that weaves together changes over time (temporal) and scene details (spatial) without overwriting the AI’s original knowledge. Its efficient \"Divide-and-Modulate\" design avoids heavy computations.\nMoMa outperformed existing methods on sports, surveillance, and action-recognition tasks while using 40% less computing power."},"venueid":{"value":"ICML.cc/2025/Conference"},"paperhash":{"value":"yang|moma_modulating_mamba_for_adapting_image_foundation_models_to_video_recognition"},"authorids":{"value":["~Yuhuan_Yang1","~Chaofan_Ma1","~Zhenjie_Mao1","~Jiangchao_Yao1","~Ya_Zhang1","~Yanfeng_Wang1"]},"authors":{"value":["Yuhuan Yang","Chaofan Ma","Zhenjie Mao","Jiangchao Yao","Ya Zhang","Yanfeng Wang"]}},"version":2},{"content":{"venue":{"value":"IEEE Transactions on Geoscience and Remote Sensing"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/36/10807682/10976442.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"he|hsimformer_integrating_mamba_and_transformer_experts_for_hyperspectral_image_classification"},"html":{"value":"https://doi.org/10.1109/TGRS.2025.3564167"},"abstract":{"value":"Hyperspectral image (HSI) classification is fundamental to numerous remote sensing applications, enabling detailed analysis of material properties and environmental conditions. Recent Mamba built upon selective state space models (SSMs) (S6) have demonstrated exceptional advantages in long-range sequence modeling with linear computational efficiency, while Transformer based on self-attention mechanisms is particularly adept at capturing short-range dependencies. To leverage the complementary strengths of these models, this article introduces a novel hybrid Mamba-Transformer framework (HSI-MFormer), effectively exploring the multiscale properties of hyperspectral data for HSI classification. Initially, a multiscale token generation (MTG) module is developed, which converts the HSI cube into multiple spatial-spectral token groups across different scales. To adequately capture fine-grained multiscale spatial-spectral patterns, an inner-scale transformer expert (ITE) is designed, which incorporates grouped self-attention operations to perform short-range sequence modeling within token groups at each scale. Meanwhile, a cross-scale Mamba expert (CME) is introduced, which integrates a cross-scale serialization mechanism and bidirectional Mamba block for long-range sequence modeling, further exploring the interactions and complementarity between token groups across different scales. Several hybrid strategies for integrating the ITE and CME are investigated to maximize their complementarity, including parallel, interval, and serial structures. Extensive experiments demonstrate that the proposed HSI-MFormer significantly outperforms the state-of-the-art Transformer-based and Mamba-based HSI classification methods. The code is available at http://github.com/tubingnuist/HSI-MFormer."},"title":{"value":"HSI-MFormer: Integrating Mamba and Transformer Experts for Hyperspectral Image Classification"},"authors":{"value":[{"fullname":"Yan He"},{"fullname":"Bing Tu","username":"~Bing_Tu1"},{"fullname":"Bo Liu"},{"fullname":"Jun Li"},{"fullname":"Antonio Plaza"}]}},"tmdate":1789092712009,"pdate":1735689600000,"externalIds":["doi:10.1109/tgrs.2025.3564167"],"tcdate":1762941874527,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Bing_Tu1"],"forum":"A8perq8q2N","license":"CC BY-SA 4.0","number":14815,"cdate":1760463000888,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/-/Edit"],"mdate":1789092712009,"domain":"OpenReview.net/Public_Article","id":"A8perq8q2N","version":2},{"content":{"summary":{"value":"This paper provides a theoretical analysis of the Mamba architecture’s in-context learning (ICL) capability, focusing on low-dimensional nonlinear target functions. The authors establish that Mamba, when pretrained via a gradient-based algorithm, can achieve efficient in-context feature learning at test time. They prove that Mamba extracts task-relevant features directly from contextual examples, yielding sample complexity comparable to nonlinear Transformers while improving upon linear kernel-based models."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Could the authors consider analyzing Mamba’s training convergence under a more practical optimization process? (Weakness 1)\n\n2. Could the authors provide an analysis of generalization—for example, how the model behaves when the data distribution at test time differs from that during training?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Most existing theoretical works on understanding in-context learning (ICL) focus on Transformers, and this paper’s analysis of Mamba—along with its comparison to Transformers—offers valuable and insightful contributions.\n\n2. Provide conceptual insight: The identification of Mamba’s nonlinear gating as the driver of test-time feature learning is conceptually meaningful for understanding gated recurrent architectures."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Algorithm 1 appears unnatural. Algorithm 1 optimize the Mamba layer and the MLP layer separately, which is not how neural networks are actually optimized in practice. This mismatch with realistic training procedures weakens the contribution of the paper.\n\n2. The input embedding also appears unnatural and overly hand-crafted, raising doubts about whether real models could actually learn such embeddings in practice."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933991811,"tcdate":1761964264372,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20583/Reviewer_XMDG"],"signatures":["ICLR.cc/2026/Conference/Submission20583/Reviewer_XMDG"],"forum":"3KPsog4mpy","number":3,"license":"CC BY 4.0","cdate":1761964264372,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20583/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933991811,"domain":"ICLR.cc/2026/Conference","replyto":"3KPsog4mpy","id":"w5HWd9NQNt","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We study in-context learning of Mamba on single-index models"},"keywords":{"value":["Mamba","In-Context Learning","Feature Learning","Theory"]},"primary_area":{"value":"learning theory"},"abstract":{"value":"Mamba, a recently proposed linear-time sequence model, has attracted significant attention for its computational efficiency and strong empirical performance. However, a rigorous theoretical understanding of its underlying mechanisms remains limited. In this work, we provide a theoretical analysis of Mamba's in-context learning (ICL) capability by focusing on tasks defined by low-dimensional nonlinear target functions. Specifically, we study in-context learning of a single-index model $y \\approx g_*(\\langle \\boldsymbol{\\beta}, \\boldsymbol{x} \\rangle)$, which depends on only a single relevant direction $\\boldsymbol{\\beta}$, referred to as *feature*. We prove that Mamba, pretrained by gradient-based methods, can achieve efficient ICL via *test-time feature learning*, extracting the relevant direction directly from context examples. Consequently, we establish a test-time sample complexity that improves upon linear Transformers—analyzed to behave like kernel methods—and is comparable to nonlinear Transformers, which have been shown to surpass the Correlational Statistical Query (CSQ) lower bound and achieve near information-theoretically optimal rate in previous works. Our analysis reveals the crucial role of the *nonlinear gating* mechanism in Mamba for feature extraction, highlighting it as the fundamental driver behind Mamba’s ability to achieve both computational efficiency and high performance."},"_bibtex":{"value":"@misc{\noh2026mamba,\ntitle={Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning},\nauthor={Junsoo Oh and Wei Huang and Taiji Suzuki},\nyear={2026},\nurl={https://openreview.net/forum?id=3KPsog4mpy}\n}"},"title":{"value":"Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning"},"pdf":{"value":"/pdf/b872836f3f5ba22e086f394ffe799c2dd9c8c920.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"oh|mamba_can_learn_lowdimensional_targets_incontext_via_testtime_feature_learning"},"authorids":{"value":["~Junsoo_Oh1","~Wei_Huang6","~Taiji_Suzuki1"]},"authors":{"value":["Junsoo Oh","Wei Huang","Taiji Suzuki"]}},"version":2},{"content":{"summary":{"value":"The paper focuses on distilling transformers into SSMs for fast inference after training. Specifically, the authors initialize states of SSMs by pretrained transformer components and utilize speculative decoding for multi-step predictions. Experiments show that the distilled hybrid model can achieve competitive performance compared to the teacher network, i.e. Zephyr-7B on standard Chat benchmarks after training with 3B tokens only. The distilled hybrid model can outperform Mamba 7B which is trained from scratch with 1.3T tokens."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"see the weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"How to accelerate the training process of Mamba-based hybrid models is a good research problem. In the paper, the authors propose to utilize distillation from pretrained transformer-based models. The results show the Mamba-based hybrid model can learn efficiently with fewer iterations and samples."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The writing is not well-organized and involves a lot of unclear parts. In Section 2.1, the derivations from softmax-based attention to linear RNNs (Mamba) are not clearly stated, e.g. unexplained symbols in equations. Also the derivations should obey some assumptions. Please refer to Mamba-2.\n2. Some settings of the proposed method are not clear. For example, in Section 2.2, Why is A set as fixed over time? In Section 2.3, which components in the model are from attention and which parts are trainable. Figure 1 is also not informative without explanations of the colors and arrows. In section 4, what are the differences between speculative decoding used in transformer and that used in Mamba here? What are the details of hare-aware designs?\n3. The experiments are not enough. For comparison methods, Hybrid Mamba which is trained from scratch should be included. Besides Mamba, other SSMs and linear RNNs should be included. The benchmarks are only on chat and academic tasks."},"limitations":{"value":"The paper should be generalized to all SSM models (including Mamba-1/2) or linear RNN models, instead of only focusing on Mamba-1."}},"nonreaders":[],"tmdate":1730879010641,"tcdate":1720762006890,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission5543/Reviewer_J8xQ"],"signatures":["NeurIPS.cc/2024/Conference/Submission5543/Reviewer_J8xQ"],"forum":"uAzhODjALU","number":1,"license":"CC BY 4.0","cdate":1720762006890,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission5543/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879010641,"domain":"NeurIPS.cc/2024/Conference","replyto":"uAzhODjALU","id":"DZBWFwbYVn","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","Transformer","Knowledge Distillation","Speculative Decoding"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Linear RNN architectures, like Mamba, can be competitive with Transformer models in language modeling while having advantageous deployment characteristics. Given the focus on training large-scale Transformer models, we consider the challenge of converting these pretrained models for deployment. \nWe demonstrate that it is feasible to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with academic GPU resources. The resulting hybrid model, which incorporates a quarter of the attention layers, achieves performance comparable to the original Transformer in chat benchmarks and outperforms open-source hybrid Mamba models trained from scratch with trillions of tokens in both chat benchmarks and general benchmarks. Moreover, we introduce a hardware-aware speculative decoding algorithm that accelerates the inference speed of Mamba and hybrid models. Overall we show how, with limited computation resources, we can remove many of the original attention layers and generate from the resulting model more efficiently. \nOur top-performing model, distilled from Llama3-8B-Instruct, achieves a 29.61 length-controlled win rate on AlpacaEval 2 against GPT-4 and 7.35 on MT-Bench, surpassing the best 8B scale instruction-tuned linear RNN model.  We also find that the distilled model has natural length extrapolation, showing almost perfect accuracy in the needle-in-a-haystack test at 20x the distillation length. Code and pre-trained checkpoints are open-sourced at [MambaInLlama](https://github.com/jxiw/MambaInLlama) for distillation and [SpeculativeMamba](https://github.com/itsdaniele/speculative\\_mamba) for speculative decoding."},"_bibtex":{"value":"@inproceedings{\nwang2024the,\ntitle={The Mamba in the Llama: Distilling and Accelerating Hybrid Models},\nauthor={Junxiong Wang and Daniele Paliotta and Avner May and Alexander M Rush and Tri Dao},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=uAzhODjALU}\n}"},"title":{"value":"The Mamba in the Llama: Distilling and Accelerating Hybrid Models"},"pdf":{"value":"/pdf/669243b56f6a425c55c92536184abde35c26027c.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"wang|the_mamba_in_the_llama_distilling_and_accelerating_hybrid_models"},"authorids":{"value":["~Junxiong_Wang1","~Daniele_Paliotta1","~Avner_May1","~Alexander_M_Rush1","~Tri_Dao1"]},"authors":{"value":["Junxiong Wang","Daniele Paliotta","Avner May","Alexander M Rush","Tri Dao"]}},"version":2},{"content":{"summary":{"value":"The article presents two main contributions: M-Mamba and GDD-MLP+Channel Mixup. The former is a further exploration of the application of Mamba in time-series tasks, but in my opinion, there is a certain degree of controversy. The latter is a good solution to the problem of time-series channel dependence, with sufficient argumentation and experiments. The overall performance of the model is good, but it mainly comes from the latter contribution."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"Line 64-68, The view put forward is not reasonable in my opinion. Why should MLP be an ideal structure? There is no basis. Can you provide relevant paper citation basis? Has there been any work in the past using MLP to practice between variables or similar work?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The overall writing logic of the article is clear, and the drawings are relatively clear. The results are presented in detail.\nI think the combination of GDD-MLP+Channel mixup is a good contribution point. Moreover, applying it to other frameworks further proves its effectiveness, and an analysis of its efficiency is also conducted. However, on the contrary, as can be seen from Table 3, M-Mamba has no advantage compared to the state-of-the-art."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Line 53-55, There have been many studies on the application of past state space models in temporal dependency. Including: S4, SpaceTime, and the direct application of Mamba itself in temporal tasks has long existed, including the temporal dependency based on patches same with your M-mamba. I think it is worthy of research to consider cross-channel on the basis of temporal for mamba-based models. However, the problem raised here is not very accurate and the theme is not very clear.\n2. Line 61-64, MLP is a multi-layer perceptron. In the Transformer, FFN is a fully connected mapping layer. It is somewhat inappropriate to say that it is part of the Transformer. I don't know what is the meaning of expressing FFN as a part of Attention? Because in iTransformer, the overall method of Attention is applied to channel dependence. The function of FFN in the attention mechanism has no direct connection with the dependence between channels. There is no relevant content in iTransformer. \n3. Line 84-91 This paragraph in the introduction does not meet the basic requirements of paper writing very well. The experimental results proving the effectiveness of the method should be placed in the experimental part.\n4. Line 132-140, you also mentioned S4, and later in Table 12 also compared the effect with the mamba-based model. Still the above problem, there exist studies on SSM in temporal. It should still highlight that the model considers cross-channel on the basis of temporal, rather than generally saying that there is insufficient research on temporal.\n5. For M-mamba (removing convolution and using FI), i am not sure that it is a general and effective improvement. I think there is controversy as a contribution. Mainly because the ablation experiment in section 5.2 is only carried out on one dataset, Weather, and the performance difference is not significant. In addition to the experiment, there is a lack of necessary explanations. For example, In Time-SSM Table 2 gives a detailed comparison with the patch Mamba (Mamba4TS)."}},"nonreaders":[],"tmdate":1732707264014,"tcdate":1730121561496,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5801/Reviewer_ah3a"],"signatures":["ICLR.cc/2025/Conference/Submission5801/Reviewer_ah3a"],"forum":"OclHGmt2ZM","number":2,"license":"CC BY 4.0","cdate":1730121561496,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5801/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732707264014,"domain":"ICLR.cc/2025/Conference","replyto":"OclHGmt2ZM","id":"v2dqvWs2VF","forumContent":{"TLDR":{"value":"We tailor Mamba for multivariate time series forecasting, equipping it with stronger cross-time and cross-channel modeling capabilities, which achieves the SOTA in real-world datasets."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forecasting","Mamba","Channel-Dependent"]},"supplementary_material":{"value":"/attachment/4810e3ea813ec5d058c32afc9ff6a129c09b060a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent advancements in multivariate time series forecasting have been propelled by Linear-based, Transformer-based, and Convolution-based models, with Transformer-based architectures gaining prominence for their efficacy in temporal and cross-channel mixing.\nMore recently, Mamba, a state space model, has emerged with robust sequence and feature mixing capabilities.\nHowever, the suitability of the vanilla Mamba design for time series forecasting remains an open question, particularly due to its inadequate handling of cross-channel dependencies.\nCapturing cross-channel dependencies is critical in enhancing the performance of multivariate time series prediction.\nRecent findings show that self-attention excels in capturing cross-channel dependencies, whereas other simpler mechanisms, such as MLP, may degrade model performance.\nThis is counterintuitive, as MLP, being a learnable architecture, should theoretically capture both correlations and irrelevances, potentially leading to neutral or improved performance.\nDiving into the self-attention mechanism, we attribute the observed degradation in MLP performance to its lack of data dependence and global receptive field, which result in MLP's lack of generalization ability.\nConsidering the powerful sequence modeling capabilities of Mamba and the high efficiency of MLP, the combination of the two is an effective strategy for solving multivariate time series prediction.\nBased on the above insights, we introduce a refined Mamba variant tailored for time series forecasting.\nOur proposed model, \\textbf{CMamba}, incorporates a modified Mamba (M-Mamba) module for temporal dependencies modeling, a global data-dependent MLP (GDD-MLP) to effectively capture cross-channel dependencies, and a Channel Mixup mechanism to mitigate overfitting.\nComprehensive experiments conducted on seven real-world datasets demonstrate the efficacy of our model in improving forecasting performance."},"_bibtex":{"value":"@misc{\nzeng2025cmamba,\ntitle={{CM}amba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting},\nauthor={Chaolv Zeng and Zhanyu Liu and Guanjie Zheng and Linghe Kong},\nyear={2025},\nurl={https://openreview.net/forum?id=OclHGmt2ZM}\n}"},"title":{"value":"CMamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting"},"pdf":{"value":"/pdf/c6642767e189d0d9628d810af94b4cdc57554e7a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zeng|cmamba_channel_correlation_enhanced_state_space_models_for_multivariate_time_series_forecasting"},"authorids":{"value":["~Chaolv_Zeng2","~Zhanyu_Liu1","~Guanjie_Zheng1","~Linghe_Kong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chaolv Zeng","Zhanyu Liu","Guanjie Zheng","Linghe Kong"]}},"version":2},{"content":{"comment":{"value":"### **(W3, Q4) Numerical Stability and Constraints on SSM Parameterization**\nWe appreciate the reviewer’s question regarding the numerical stability of the SSM parameterization. We clarify that our model builds upon the robust diagonal parameterization of structured SSMs (S4 [1], Mamba [2]), which inherently addresses these stability concerns through structural constraints and initialization.\n\nRegarding the stability of the inverse term $\\mathbf{A}_r^{-1}$ (Eq. 10), the transition matrix $\\mathbf{A}_r$ is structured as a **diagonal matrix** initialized with **strictly negative real eigenvalues** based on the HiPPO theory [1,3]. Since these eigenvalues are strictly negative and bounded away from zero, $\\mathbf{A}_r$ is guaranteed to be non-singular, ensuring that the inverse operation is well-conditioned and does not induce numerical instability such as gradient explosion.\n\nWith respect to the timescale parameter $\\Delta_{r,t}$ and system stability (Eq. 11), we enforce strict positivity using a nonlinearity: $\\Delta_{r,t} = \\mathrm{softplus}(\\epsilon_r + f_{\\Delta}(x_{r,t}))$, with $\\epsilon_r$ initialized within a stable range (i.e., $\\mathrm{Uniform}([0.001, 0.1])$) following previous works [1,2]. Consequently, the interplay between the strictly negative eigenvalues of $\\mathbf{A}\\_r$ and the positive sampling step $\\Delta_{r,t}$ ensures that the discrete transition matrix $\\bar{\\mathbf{A}}\\_{r,t} = \\exp(\\Delta\\_{r,t} \\mathbf{A}\\_r)$ always possesses eigenvalues within the unit interval $(0, 1)$. This guarantees that the system dynamics are contractive, leading to state decay rather than explosion, thereby ensuring asymptotic stability.\n\nFor clarity and reproducibility, we have explicitly added these implementation details to **Appendix E.1** of the revised manuscript.\n\n### References\n> [1] Gu et al., “On the parameterization and initialization of diagonal state space models,” NeurIPS, 2022.  \n[2] Gu et al., “Mamba: Linear-time sequence modeling with selective state spaces,\" COLM, 2024.  \n[3] Gu et al., “HiPPO: Recurrent memory with optimal polynomial projections,” NeurIPS, 2020."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764750821838,"tcdate":1764750821838,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8714/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8714/Authors"],"forum":"522R7hU5J0","number":8,"license":"CC BY 4.0","cdate":1764750821838,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8714/-/Official_Comment"],"mdate":1764750821838,"domain":"ICLR.cc/2026/Conference","replyto":"zXbGKnkcas","id":"1R5HthtHsL","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Resting-state fMRI","State-space model","Brain state dynamics"]},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Brain states, observable through resting-state functional magnetic resonance imaging (rs-fMRI), represent dynamic transitions between recurring connectivity patterns and are closely linked to neurological and psychiatric conditions. Therefore, developing a computational model for dynamic brain state estimation has been a long-lasting research interest. Among existing approaches, state-space models (SSMs) provide a principled framework for modeling these dynamics. However, existing methods face key limitations: they fail to preserve the brain's spatial architecture, and they model temporal dynamics without considering co-evolving spatial patterns.\nTo address these limitations, we propose $\\textbf{DyBraSS}$ ($\\textbf{Dy}$namic $\\textbf{Bra}$in $\\textbf{S}$tate-$\\textbf{S}$pace model), a novel structured SSM that unifies spatial and temporal modeling within a single framework, enhancing ROI-level modeling capacity and interpretability through a clustering-based global aggregation module.\nThis module respects the brain's network topology by integrating information from all regions during each local update, and represents evolving brain states as interpretable clusters.\nComprehensive experiments on multiple fMRI datasets demonstrate that our method consistently outperforms state-of-the-art baselines in diagnostic performance across diverse metrics. \nAdditionally, individual- and group-level brain state analyses reveal that the learned dynamics align with known neurobiological alterations, providing clinically relevant insights for understanding neural dysfunction and developing diagnostic biomarkers."},"_bibtex":{"value":"@misc{\njeong2026dybrass,\ntitle={DyBra{SS}: Dynamic Brain State Modeling with State-Space Model},\nauthor={Ah-Yeong Jeong and Seungwoo Jeong and Da-Woon Heo and Heung-Il Suk},\nyear={2026},\nurl={https://openreview.net/forum?id=522R7hU5J0}\n}"},"title":{"value":"DyBraSS: Dynamic Brain State Modeling with State-Space Model"},"pdf":{"value":"/pdf/7ec6c7264a37bbfe18a74f079c78222663cc7d50.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"jeong|dybrass_dynamic_brain_state_modeling_with_statespace_model"},"authorids":{"value":["~Ah-Yeong_Jeong1","~Seungwoo_Jeong2","~Da-Woon_Heo1","~Heung-Il_Suk1"]},"authors":{"value":["Ah-Yeong Jeong","Seungwoo Jeong","Da-Woon Heo","Heung-Il Suk"]}},"version":2},{"content":{"comment":{"value":"###  **W1-a) Reverse modeling is widely used**\n\nWe acknowledge that reverse modeling (bidirectional Mamba) is widely used, as noted in **L46--51**. However, we highlight its inherent limitations in **Table 1**, where the bidirectional Mamba does not consistently yield the best performance and, in some cases, underperforms compared to a single unidirectional Mamba. Additionally, the bidirectional Mamba uses two Mambas for tuning, doubling the computational burden compared to a single Mamba. In light of these limitations, we propose an ***alternative to reverse modeling (bidirectional Mamba)***: a single unidirectional Mamba with regularization, which demonstrates improved performance and efficiency, as shown in **Table 2,5** and **Table 13**, respectively.\n\n&nbsp;\n\n\n---\n###  **W1-b) Removal of 1D-conv is trivial**\n\nWhile removing the 1D-conv might seem minor, many studies in the TS domain often adopt architectures directly from NLP or vision tasks **without accounting for TS-specific properties**. Since the 1D-conv in Mamba [A] was originally designed to capture sequential patterns (**L66--68**), applying it to non-sequential TS channels is an **unnatural choice**, yet it has been commonly used without adaptation.  ***We believe our approach is non-trivial, as it specifically accounts for the distinctive characteristics of TS datasets that have been overlooked in other Mamba-based methods. Furthermore, the proposed method has the potential for application in other domains involving non-sequential data (e.g., tabular data)***.  \n\nFurther details on the removal of the 1D-conv are provided in **Appendix D**.\n\n\n&nbsp;\n\n[A] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv (2024)\n\n&nbsp;\n\n---\n###  **W2.  Performance improvement is not significant.**\nIt is important to clarify that our primary goal is not to achieve substantial performance gains over the baseline, but rather to demonstrate that a ***single unidirectional Mamba with a regularization strategy can effectively replace the widely used bidirectional Mamba***, yielding comparable or even better performance with greater efficiency, as shown in **Table 5** and **Table 13**. Additionally, applying the regularization strategy to the bidirectional Mamba also results in performance gains, as shown in **Table 6**. \n\n\nWhile the optimal model structure (and hyperparameters) may vary between the unidirectional and bidirectional Mamba setups, we used the original hyperparameters from the bidirectional Mamba for the unidirectional Mamba (see **L265--266**), allowing us to evaluate results on the same basis. Optimizing these hyperparameters specifically for the unidirectional Mamba configuration could likely yield additional performance gains.\n\n\n&nbsp;\n\n---\n###  **W3.  Necessity of removing 1D-conv**\n\n\nIt is important to note that we remove 1D-conv primarily ***for general TS datasets, where channels do not inherently have a sequential order***. We believe it is reasonable to design a model architecture suited to these general cases.\n\n\nAs shown in **Table 7** and **Figure 5**, removing the 1D-conv of Mamba in datasets where channel order has inherent structure (e.g., traffic data) does not always lead to performance benefits. However, this finding further supports our approach, indicating that the 1D-conv captures sequential relationships between tokens (channels). When channels have an inherent order, which is identifiable based on the dataset domain or type, removing the 1D convolution may not be desirable.\n\n\nFurther details on the rationale and process for removing the 1D-conv are provided in **Appendix D**.\n\n\n&nbsp;\n\n---\n### **W4.  Comparison with other Mamba models**.\n\nAs mentioned in **Section 2. Related Works**, several concurrent studies have utilized Mamba to capture CD [A,B,C,D], and these methods generally address the sequential order bias by employing the **bidirectional Mamba [A,B,C,D]**. However, aside from our baseline method [A], most of these works have not released code for reproducibility, making direct comparisons with our approach currently infeasible. We look forward to conducting comparisons when these implementations become available.\n\n&nbsp;\n\n[A] Wang, Zihan, et al. \"Is mamba effective for time series forecasting?.\" arXiv (2024)\n\n[B] Liang, Aobo, et al. \"Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting.\" arXiv (2024) \n\n[C] Weng, Zixuan, et al. \"Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting.\" arXiv (2024)\n\n[D] Behrouz, A., Santacatterina, M., & Zabih, R. “Mambamixer: Efficient selective state space models with dual token and channel selection\" arXiv (2024)"}},"tmdate":1731649126367,"tcdate":1731648128275,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission69/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission69/Authors"],"forum":"fyl82vAale","number":12,"license":"CC BY 4.0","cdate":1731648128275,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission69/-/Official_Comment"],"mdate":1731649126367,"domain":"ICLR.cc/2025/Conference","replyto":"zOscO3IUW3","id":"hRcRKCBWeF","forumContent":{"TLDR":{"value":"We propose SOR-Mamba, a time series forecasting method that uses Mamba with regularization to address the sequential order bias in capturing channel dependencies."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","State-space Model","Time Series"]},"supplementary_material":{"value":"/attachment/a3e6bf94e3ea1b0642f98af3f08c4590c0c5d03e.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data.\nHowever, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.\nExtensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios."},"_bibtex":{"value":"@misc{\nlee2025sequential,\ntitle={Sequential Order-Robust Mamba for Time Series Forecasting},\nauthor={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\nyear={2025},\nurl={https://openreview.net/forum?id=fyl82vAale}\n}"},"title":{"value":"Sequential Order-Robust Mamba for Time Series Forecasting"},"pdf":{"value":"/pdf/2172851f730b05caa6144b837024da19c84263c0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"lee|sequential_orderrobust_mamba_for_time_series_forecasting"},"authorids":{"value":["~Seunghan_Lee1","~Juri_Hong1","~Kibok_Lee1","~Taeyoung_Park1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seunghan Lee","Juri Hong","Kibok Lee","Taeyoung Park"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Mamba in the PDE-solving area. It establishes a formal theoretical connection between structured state-space models (SSMs) and neural operators. The proposed method performs relatively well on the considered datasets."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See Weaknesses."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"1. The summary of related work is detailed.\n2. The paper establishes a theoretical connection between the Mamba architecture and neural operators.\n3. Although not comprehensive, the paper includes experiments on several datasets and compares the proposed method with some baseline models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Presentation Issues:** The paper lacks line numbers, which are essential for precise referencing during reviews. Additionally, the appendix is included in the supplementary materials rather than in the main document, which hampers the ease of accessing critical information.\n2. **Writing Clarity:** The manuscript’s writing requires enhancement, as some sentences lack logical coherence. For instance, the statement: *“While Transformers dominate in areas like foundational models and computer vision, the application of SSMs, particularly the Mamba architecture, to neural operators for PDEs is still underexplored.”* The connection between Transformers’ dominance in certain fields and the underexplored application of SSMs in PDEs is unclear.\n3. **Figure 1 Interpretation:** The purpose and contribution of Figure 1 are not clear. The main paper contains excessive mathematical details that could be better placed in the appendix. And a detailed description of the main architecture of MNO is necessary. Specifically, does MNO include a patchification layer as shown in Figure 1? Furthermore, since your baseline models operate on mesh points directly without patchifying, how does this influence your comparisons?\n4. **Novelty Concern:** The paper’s novelty appears limited. Introducing the Mamba architecture into PDE solving is not new. The approach seems to involve simply replacing the Transformer block with a Mamba block without significant modifications. It would be helpful to explain why PDE-specific adaptations are not needed. Additionally,  while you theoretically demonstrate that the MNO is a type of neural operator, this does not enhance the understanding of the benefits of employing Mamba in PDE solving.\n5. **Efficiency Evaluation:** It is well-known that Mamba offers efficiency advantages over Transformers in long sequence data. However, your experiments do not include efficiency comparisons with baseline models, which is a missed opportunity to highlight Mamba’s benefits.\n6. **Dataset and Baseline Limitations:** The datasets and baselines used are not comprehensive enough. The datasets selected are relatively simple. Incorporating more complex datasets, such as the 2D Navier-Stokes dataset from FNO [1], would strengthen your evaluation. Additionally, there are other Transformer-based solvers beyond those utilizing linear attention, such as Transolver [2], which computes attention based on learned physical states.\n7. **Implementation Details:** The implementation details provided are insufficient. Did you keep all hyperparameters consistent across the three variants? What is the total parameter count for each model? It is hard to believe that merely replacing the Transformer block with a Mamba block results in significant performance improvements, especially since such gains have not been observed in NLP and CV domains. Providing more detailed implementation information or sharing inference code would greatly enhance the paper’s credibility.\n8. **Clarity of Figure 3:** The representation in Figure 3 is not clear. What do the different colored lines mean, and do these lines correspond to any ground truth? The sentence *“The change in log amplitude across the frequency range is more uniform, indicating that Mamba effectively balances between capturing necessary high-frequency information and filtering out noise.”* is difficult to understand. It would be helpful if you could provide more explanations or clarify this statement.\n9. **Irregular data:** The paper focuses solely on grid data. I wonder what distinctions do you see between grid-based PDE data and images. Futhermore, how would you extend the MNO to handle irregular data, considering that linear Transformers can readily address such types?\n\n[1] Fourier Neural Operator for Parametric Partial Differential Equations\n\n[2] Transolver: A Fast Transformer Solver for PDEs on General Geometries"}},"nonreaders":[],"tmdate":1731428214414,"tcdate":1729222357414,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4751/Reviewer_vEWw"],"signatures":["ICLR.cc/2025/Conference/Submission4751/Reviewer_vEWw"],"forum":"VtP7CamOR5","number":1,"license":"CC BY 4.0","cdate":1729222357414,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4751/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428214414,"domain":"ICLR.cc/2025/Conference","replyto":"VtP7CamOR5","id":"YKQ9aqfsVT","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Neural Operator","Deep Learning","PDEs"]},"supplementary_material":{"value":"/attachment/3e9fad783a8584dce19c3891f3e3c35ba2506156.pdf"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Partial differential equations (PDEs) are widely used to model complex physical systems, but solving them efficiently remains a significant challenge. Recently, Transformers have emerged as the preferred architecture for PDEs due to their ability to capture intricate dependencies. However, they struggle with representing continuous dynamics and long-range interactions. To overcome these limitations, we introduce the Mamba Neural Operator (MNO), a novel framework that enhances neural operator-based techniques for solving PDEs. MNO establishes a formal theoretical connection between structured state-space models (SSMs) and neural operators, offering a unified structure that can adapt to diverse architectures, including Transformer-based models. By leveraging the structured design of SSMs, MNO captures long-range dependencies and continuous dynamics more effectively than traditional Transformers. Through extensive analysis, we show that MNO significantly boosts the expressive power and accuracy of neural operators, making it not just a complement but a superior framework for PDE-related tasks, bridging the gap between efficient representation and accurate solution approximation."},"_bibtex":{"value":"@misc{\ncheng2024mamba,\ntitle={Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for {PDE}s},\nauthor={Chun-Wun Cheng and Jiahao Huang and Yi Zhang and Guang Yang and Carola-Bibiane Sch{\\\"o}nlieb and Angelica I Aviles-Rivero},\nyear={2024},\nurl={https://openreview.net/forum?id=VtP7CamOR5}\n}"},"title":{"value":"Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for PDEs"},"pdf":{"value":"/pdf/5fbef146ca212b343043790396657ef701fa94d5.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"cheng|mamba_neural_operator_who_wins_transformers_vs_statespace_models_for_pdes"},"authorids":{"value":["~Chun-Wun_Cheng1","~Jiahao_Huang1","~Yi_Zhang59","~Guang_Yang1","~Carola-Bibiane_Schönlieb1","~Angelica_I_Aviles-Rivero1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chun-Wun Cheng","Jiahao Huang","Yi Zhang","Guang Yang","Carola-Bibiane Schönlieb","Angelica I Aviles-Rivero"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a new class of linear recurrent unit (LRU) which is uprising efficient state-of-the-art model class for long-range sequence modeling. The proposed model, RotRNN, utilizes rotation matrix-based parameterization for state transformation and the explicit normalization method. Based on strong mathematical backgrounds, RotRNN could be simply implemented and strictly guarantees preservation of hidden state magnitude across time-steps. This paper compares RotRNN with former LRU and state space models (SSMs) in theory and experiments to give in-depth understanding for readers."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"What does $A^{t-k}$ mean in Equation 2?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":1},"strengths":{"value":"Rigorous mathematical background: The math backgrounds behind the rotation matrix-based parameterization and explicit normalization method are proved with easy-to-read derivations. In addition, those backgrounds lead the simple implementation of RotRNN. \n\nIn-depth comparison between former architectures: The theoretic comparisons between RotRNN and (LRU/SSM) are helpful to posit RotRNN within this field.\n\nStrong, latest baselines: This paper compares RotRNN with latest and state-of-the-art baselines (such as S5 and Liquid-S4) in long-range sequence modeling benchmarks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Majors:\n- Limitation of rotation matrix parameterization: I think there would be drawback with constraining state transformation matrix to be rotation matrix, which might limit expression power of the model. \n- Potential drawback of explicit normalization method: It is unclear that whether the explicit normalization method is beneficial for performance. I understand that this method constrains the operation to target a specific range of dependency based on the trained value of $gamma$, so it looks constraining the model’s expressivity. Although it successfully guarantees the converged norm of hidden states during training, its performance-wise effect is not demonstrated in results. An ablation study would be helpful to see its benefit.\n- Overall, it is questionable how RotRNN could be advantageous in practice. Comparison of computational efficiency (in terms of FLOPS) would be helpful to show that RotRNN is ‘efficient to compute’.\n\nMinors:\n- Weak motivations for rotation matrix parameterization: The three motivations written at the beginning of section 3 aim to simple implementation thanks to the rotation matrix parameterization. However, there is no motivation related to hidden state processing despite of rotation matrix’s unique characteristic (such as regularity). And, it is not clear why real-valued matrix can make it robust to initialization? \n- Arguable interpretations of experiment results: The sentence “we find that our model performs best on domains with more discrete input data, such as ListOps, Text and Retrieval, achieving the highest score of all the baselines” seems not clear from the result table. Except ‘Text’ task, Liquid-S4 and S5 achieved the highest scores in ListOps and Retrieval tasks, respectively.\n- Confronting claims: This paper argues the unnecessity of SSM model’s theory-based initialization method (HiPPO) in practice (Section 2.3). However, this paper aims to build RotRNN on rigorous theoretic backgrounds. I think those claims are confronting with each other."}},"nonreaders":[],"tmdate":1731427699401,"tcdate":1730524524051,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11374/Reviewer_sHHK"],"signatures":["ICLR.cc/2025/Conference/Submission11374/Reviewer_sHHK"],"forum":"z6qmomJW91","number":1,"license":"CC BY 4.0","cdate":1730524524051,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11374/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427699401,"domain":"ICLR.cc/2025/Conference","replyto":"z6qmomJW91","id":"5eLZRkOZba","forumContent":{"TLDR":{"value":"We propose a novel linear recurrent layer for long sequence modelling using rotation matrices for stable and efficient recurrence."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Sequence Modelling","Recurrent Neural Networks","State Space Models","Long Sequences"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Linear recurrent neural networks, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks. Despite their success, their empirical performance is not well understood and they come with a number of drawbacks, most notably their complex initialisation and normalisation schemes. In this work, we address some of these issues by proposing RotRNN – a linear recurrent model which utilises the convenient properties of rotation matrices. We show that RotRNN provides a simple and efficient model with a robust normalisation procedure, and a practical implementation that remains faithful to its theoretical derivation. RotRNN also achieves competitive performance to state-of-the-art linear recurrent models on several long sequence modelling datasets."},"_bibtex":{"value":"@misc{\nbiegun2024rotrnn,\ntitle={Rot{RNN}: Modelling Long Sequences with Rotations},\nauthor={Kai Biegun and Rares Dolga and Harry Jake Cunningham and David Barber},\nyear={2024},\nurl={https://openreview.net/forum?id=z6qmomJW91}\n}"},"title":{"value":"RotRNN: Modelling Long Sequences with Rotations"},"pdf":{"value":"/pdf/78dab72130b925f785999cf1a1e427a2af4f949e.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"biegun|rotrnn_modelling_long_sequences_with_rotations"},"authorids":{"value":["~Kai_Biegun1","~Rares_Dolga1","~Harry_Jake_Cunningham1","~David_Barber1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Kai Biegun","Rares Dolga","Harry Jake Cunningham","David Barber"]}},"version":2},{"content":{"summary":{"value":"The paper presents HeteGraph-Mamba (HGMN), a novel heterogeneous graph learning framework that uses Selective State Space Models (SSSMs) to address the limitations of existing graph neural networks. HGMN introduces a six-step process, including graph tokenization and context-aware node ordering, to improve computational efficiency and capture long-range dependencies across heterogeneous nodes. Experimental results indicate that HGMN achieves higher accuracy and efficiency than other state-of-the-art models on several benchmark datasets."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"Q1: Why was Mamba chosen for graphs? Mamba shows improved efficiency over Transformers when modeling sequences longer than 2K tokens, based on current knowledge. Providing a theoretical analysis to support this choice in graph contexts would add significant value to the paper.\nQ2: Are all six processing steps necessary? A more detailed ablation study on each step would better clarify the contributions and necessity of each component in the model.\nQ3: Performance on tasks beyond node classification: Can the framework achieve similar high performance on other tasks beyond node classification?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"S1：The integration of SSSMs in heterogeneous graph learning is innovative, addressing scalability and long-range dependency challenges not fully tackled by traditional Transformer-based models. HGMN’s tokenization and ordering mechanisms for heterogeneous nodes offer a fresh approach to sequence modeling in graphs.\nS2：The six-step processing flow in HGMN separately addresses homogeneity and heterogeneity dependencies, enhancing the model's expressive capability. Additionally, the intra-type and inter-type node ordering improves the accuracy of context capture.\nS3：The paper provides a detailed overview of each component of the HGMN framework, making it easy to follow the flow and understand the contributions of each design choice.\nS4：HGMN’s performance improvements over 19 baseline models on accuracy and efficiency metrics demonstrate the potential impact of this approach in the heterogeneous graph learning domain."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. The motivation for using Mamba on graphs is unclear. Mamba is faster than Transformer only when modeling sequences longer than 2k.\nW2. Although the paper demonstrates the superiority of the experimental results, it lacks sufficient theoretical analysis of the proposed method, especially in the theoretical basis for improving the efficiency and accuracy of SSSMs. It is recommended to add more mathematical analysis or complexity proof.\nW3. The previous experiments only verified the effect of HGMN in the node classification task. The applicability and performance of other graph tasks (such as edge prediction and graph classification) have not been tested, which affects its wide applicability.\nW4. The model hyperparameters are not explained enough, and the analysis of the functions of each module is also lacking. It is recommended to increase the analysis of hyperparameters and add ablation experiments."}},"nonreaders":[],"tmdate":1731428142256,"tcdate":1730363372371,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5526/Reviewer_T29q"],"signatures":["ICLR.cc/2025/Conference/Submission5526/Reviewer_T29q"],"forum":"mn61GWpEiK","number":2,"license":"CC BY 4.0","cdate":1730363372371,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5526/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428142256,"domain":"ICLR.cc/2025/Conference","replyto":"mn61GWpEiK","id":"p5A9UVmLam","forumContent":{"TLDR":{"value":"a heterogeneous graph mamba network (HGMN) as the first exploration in leveraging the powerful selective state space models (SSSMs) for heterogeneous graph learning."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["graph neural network","state space model"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We propose a heterogeneous graph mamba network (HGMN) as the first exploration in leveraging the selective state space models (SSSMs) for heterogeneous graph learning. Compared with the literature, our HGMN overcomes two major challenges: (i) capturing long-range dependencies among heterogeneous nodes and (ii) adapting SSSMs to heterogeneous graph data. Our key contribution is a general graph architecture that can solve heterogeneous nodes in real-world scenarios, followed an efficient flow. Methodologically, we introduce a two-level efficient tokenization approach that first captures long-range dependencies within identical node types, and subsequently across all node types. Empirically, we conduct comparisons between our framework and 19 state-of-the-art methods on the heterogeneous benchmarks. The extensive comparisons demonstrate that our framework outperforms other methods in both the accuracy and efficiency dimensions."},"_bibtex":{"value":"@misc{\npan2024hetegraphmamba,\ntitle={HeteGraph-Mamba: Heterogeneous Graph Learning via Selective State Space Model},\nauthor={Zhenyu Pan and Yoonsung Jeong and Xiaoda Liu and Han Liu},\nyear={2024},\nurl={https://openreview.net/forum?id=mn61GWpEiK}\n}"},"title":{"value":"HeteGraph-Mamba: Heterogeneous Graph Learning via Selective State Space Model"},"pdf":{"value":"/pdf/f3e375dedad64b3fb6a91ac3fc520e533a0243a1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"pan|hetegraphmamba_heterogeneous_graph_learning_via_selective_state_space_model"},"authorids":{"value":["~Zhenyu_Pan1","~Yoonsung_Jeong1","~Xiaoda_Liu1","~Han_Liu4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zhenyu Pan","Yoonsung Jeong","Xiaoda Liu","Han Liu"]}},"version":2},{"content":{"venue":{"value":"Int. J. Comput. Vis. 2026"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/s11263-025-02597-y.pdf"},"venueid":{"value":"dblp.org/journals/IJCV/2026"},"paperhash":{"value":"chen|video_mamba_suite_state_space_model_as_a_versatile_alternative_for_video_understanding"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Guo_Chen_0006:","https://dblp.org/search/pid/api?q=author:Yifei_Huang_0002:","~Jilan_Xu1","https://dblp.org/search/pid/api?q=author:Baoqi_Pei:","https://dblp.org/search/pid/api?q=author:Jiahao_Wang:","https://dblp.org/search/pid/api?q=author:Zhe_Chen_0017:","https://dblp.org/search/pid/api?q=author:Zhiqi_Li:","https://dblp.org/search/pid/api?q=author:Tong_Lu_0002:","https://dblp.org/search/pid/api?q=author:Limin_Wang_0002:"]},"html":{"value":"https://doi.org/10.1007/s11263-025-02597-y"},"_bibtex":{"value":"@article{DBLP:journals/ijcv/ChenHXPWCLLW26,\n  author={Guo Chen and Yifei Huang and Jilan Xu and Baoqi Pei and Jiahao Wang and Zhe Chen and Zhiqi Li and Tong Lu and Limin Wang},\n  title={Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding},\n  year={2026},\n  month={January},\n  cdate={1767225600000},\n  journal={Int. J. Comput. Vis.},\n  volume={134},\n  number={1},\n  pages={20},\n  url={https://doi.org/10.1007/s11263-025-02597-y}\n}\n"},"abstract":{"value":"State space models (SSMs) have demonstrated remarkable capabilities in natural language processing. Yet, their applicability to video understanding remains underexplored. We present the Video Mamba Suite, a suite of Mamba-based model families together with a unified evaluation protocol covering four roles in video understanding: temporal modeling, cross-modal interaction, temporal adapters, and spatiotemporal modeling. To address limitations of existing bidirectional scanning mechanisms, we propose the Decomposed Bidirectional Mamba (DBM) block, which enhances temporal modeling through parameter-sharing bidirectional pathways with separated input projections. Through extensive experiments spanning 14 SSM-based models across 12 video understanding tasks—including temporal localization, action segmentation, video captioning, and multimodal retrieval—we demonstrate that Mamba consistently achieves competitive or superior performance compared to Transformer baselines while maintaining linear computational complexity. Our findings establish SSMs as viable, versatile alternatives to Transformers for video understanding, offering significant efficiency advantages for long-sequence processing."},"title":{"value":"Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding"},"authors":{"value":["Guo Chen","Yifei Huang","Jilan Xu","Baoqi Pei","Jiahao Wang","Zhe Chen","Zhiqi Li","Tong Lu","Limin Wang"]}},"tmdate":1768991322994,"pdate":1798675200000,"externalIds":["dblp:journals/ijcv/ChenHXPWCLLW26"],"tcdate":1768991293225,"writers":["~"],"signatures":["~Jilan_Xu1"],"forum":"zeiYwkW25O","license":"CC BY-SA 4.0","number":783376,"cdate":1767225600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1768991322994,"domain":"DBLP.org","id":"zeiYwkW25O","version":2},{"content":{"summary":{"value":"The paper studies in-context learning in Mamba-2 using Markov Chains, and shows an interesting finding of a single layer Mamba-2 model being able to very closely track the performance of the optimal in-context Laplacian smoothing estimator. The paper contains several experiments and ablations that focus on comparing a one layer Mamba-2 model to a 1 and 2 layer Transformer, specifically focusing on the Convolution part of the overall architecture as an important piece. The paper also presents several theoretical insights which map well to the observed results.\n\nMy review comes from viewing the paper from the lens of someone familar with the empirical results with larger scale Mamba-2 models and how they relate to Transformers at the whole model (not necessarily one or two layer wise) level.\n\nOverall, I found the paper technically sound and addressing a relevant question to the Mamba line of models. I am slightly learning towards accepting the paper, but do have certain comments and questions that withhold me from a stronger rating (see below)."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"- Re: the importance of convolution, I found these two related comments by the Mamba-2 author on their official implementation:\nhttps://github.com/state-spaces/mamba/issues/525#issuecomment-2290739534\nand \nhttps://github.com/state-spaces/mamba/issues/433#issuecomment-2199557446\nSpecifically, “conv1d helps a bit on perplexity: e.g. at 360M on 7B tokens on the Pile, using GPT2 tokenizer, w conv1d we get around 8.6 perplexity and without conv1d is around 8.9 ppl.”. This represents a slight drop in overall model performance but not as significant as the one seen in the authors experiments in the paper. Do you have any theory about why this may be?\n\n- In the Mamba-2 paper, the original authors illustrate the duality between Transformers and SSM’s. The Mamba-2 layer itself can be seen as a special case of a Transfomer (and vice versa) under certain assumptions and has generally very similar training characteristics. Given this duality, how do the authors feel about the presented results showing that a 1-layer Transformer is significantly worse at ICL than a one layer Mamba?\n\n- Do you have any speculation about how your findings may change / evolve over multiple stacked layers as is the typical case in most models?\n\n- For larger scale practical deployments of Mamba models, do you think we should expect better in-context learning results generally compared to Transformers?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- I found the idea interesting! Thank you for sharing this from a perspective that is intriguing and different to the typical “we ran a bunch of experiments on the same benchmarks and boosted performance by X%” kind of LM study. It is positive to see Mamba models being connected with optimal statistical estimators.\n\n- The paper is well written. Even though it may not be easy to read on a cursory look (I needed to go back and forth to see symbols and notation frequently), it does contain all the information necessary to understand it. The paper also introduces the core idea behind Mamba-2 well.\n\n- In addition to the well motivated experiment operating on random Markov chains sampled from the vocabulary, the authors also repeat the same exercise on a true token distribution from a dataset like WikiText103 that would be more representative of the real world token transition probabilities. Modeling these “biased” next-token probabilities seems like the essence of language modeling and in-context learning to me, and I find it positive that we see perplexity results on real text too.\n\n- Overall, I found the experiments to be well designed and aligned well with the theoretical insights."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- For the non-Markovian setting in Section 5, (ppl on WikiText103 being the only result), I feel like the evaluation could have been a bit more diverse and rigorous. This is the setting that I assume most practitioners (for better or worse) would be interested in, and it may have been useful to do this over a slightly wider range of model sizes (number of layers, dimensionality etc) to help build better intuition.\n\n- There is a lot of related work on architectures that do sub-quadratic attention, e.g. the original Linear Attention work, or related recurrent/State Space style models like DeltaNet, RWKV etc. While I understand the specific focus on Mamba-2 in this work (which in itself needed a lot of analysis), the inclusion of 1-2 other related methods in the comparison (e.g. transformer with Linear Attn in addition to the 1 and 2 layer Transformer) would have strengthened my evaluation of the paper."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927221011,"tcdate":1762153564790,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17277/Reviewer_ntkq"],"signatures":["ICLR.cc/2026/Conference/Submission17277/Reviewer_ntkq"],"forum":"kmK3WSCOCT","number":4,"license":"CC BY 4.0","cdate":1762153564790,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17277/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927221011,"domain":"ICLR.cc/2026/Conference","replyto":"kmK3WSCOCT","id":"1mFrImUweT","forumContent":{"TLDR":{"value":"We uncover an interesting phenomenon where a single-layer Mamba represents the Bayes optimal Laplacian smoothing estimator when trained on Markov chains and we demonstrate it theoretically and empirically."},"venue":{"value":"ICLR 2026 Oral"},"keywords":{"value":["State-space models","Markov chains","In-context learning","Laplacian smoothing"]},"supplementary_material":{"value":"/attachment/7481132bbc16e969531b0db2b1a4e3d8dbafeefc.zip"},"primary_area":{"value":"interpretability and explainable AI"},"abstract":{"value":"While transformer-based language models have driven the AI revolution thus far, their computational complexity has spurred growing interest in viable alternatives, such as structured state space sequence models (SSMs) and Selective SSMs. Among these, Mamba (S6) and its variant Mamba-2 have shown remarkable inference speed-ups over transformers while achieving comparable or superior performance on complex language modeling tasks. However, despite these architectural innovations and empirical successes, the fundamental learning capabilities of Mamba remain poorly understood. In this paper, we address this gap by studying in-context learning (ICL) on Markov chains and uncovering an interesting phenomenon: even a single-layer Mamba efficiently learns the in-context Laplacian smoothing estimator, which is both Bayes and minimax optimal. To explain this, we theoretically characterize the representation capacity of Mamba and reveal the fundamental role of convolution in enabling it to represent the optimal Laplacian smoothing. These theoretical insights align strongly with empirical results and, to the best of our knowledge, represent the first formal connection between Mamba and optimal statistical estimators. Finally, we outline promising research directions inspired by these findings."},"_bibtex":{"value":"@inproceedings{\nbondaschi2026from,\ntitle={From Markov to Laplace: How Mamba In-Context Learns Markov Chains},\nauthor={Marco Bondaschi and Nived Rajaraman and Xiuying Wei and Razvan Pascanu and Caglar Gulcehre and Michael Gastpar and Ashok Vardhan Makkuva},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=kmK3WSCOCT}\n}"},"title":{"value":"From Markov to Laplace: How Mamba In-Context Learns Markov Chains"},"pdf":{"value":"/pdf/7c5974f3688d2ba6f4817747b376897b745b3197.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"bondaschi|from_markov_to_laplace_how_mamba_incontext_learns_markov_chains"},"authorids":{"value":["~Marco_Bondaschi1","~Nived_Rajaraman1","~Xiuying_Wei1","~Razvan_Pascanu1","~Caglar_Gulcehre1","~Michael_Gastpar1","~Ashok_Vardhan_Makkuva1"]},"authors":{"value":["Marco Bondaschi","Nived Rajaraman","Xiuying Wei","Razvan Pascanu","Caglar Gulcehre","Michael Gastpar","Ashok Vardhan Makkuva"]}},"version":2},{"content":{"summary":{"value":"The paper presents RFMamba, a novel frequency-aware state space model for RF-based human-centric perception (HCP), designed to process radio frequency (RF) signals for tasks like human pose estimation, activity recognition, and person re-identification (ReID). Leveraging the capabilities of state space models (SSMs), RFMamba uses a dual-branch architecture that combines frequency and spatiotemporal modeling to effectively capture the characteristics of RF signals. The model introduces an omni-dimensional scanning mechanism, selectively focusing on informative frequency cues. Experiments on the newly introduced THP dataset (covering both free-space and wall-occlusion scenarios) demonstrate RFMamba’s superior performance over existing methods in accuracy."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. What are the specific activities in the THP dataset, and what is their distribution?\n2. Were participants stationary or moving freely during activities?\n3. Would the model generalize to other environments without additional data collection or fine-tuning?\n4. The authors used SFCW radars for evaluation. Would the proposed method generalize to FMCW radar systems as well, and if so, what adjustments might be necessary?\n5. Will the THP dataset be made publicly available to support reproducibility and further research in RF-based human perception?\n6. What are the compute and memory requirements for achieving real-time performance with RFMamba?\n7. The dataset was split into a 4:1 training-testing ratio. How much data was allocated for validation, and what criteria were used for this split?\n8. For pose estimation, the authors used the metric Mean Per Joint Position Error (MPJPE). Is this measured in 3D or 2D space?\n9. Was IRB approval obtained for the human subject data collection, and were participants informed and provided consent for the use of their radar and camera data in this study?"},"rating":{"value":8},"details_of_ethics_concerns":{"value":"The authors mentioned the approval of IRB and receiving consent of the subjects in the rebuttal. No more concerns regarding this."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Motivation and practical use cases: The authors address real-world challenges in RF-based human perception, aiming to support applications like security, health monitoring, and emergency response by enabling through-wall sensing and multi-person tracking where visual methods may be ineffective.\n\n2. Technical novelty with Mamba modules: RFMamba leverages Mamba-based state space modeling specifically adapted for RF signals, incorporating frequency and spatiotemporal Mamba modules to capture long-term dependencies within complex RF data.\n\n3. Generally clear presentation: The paper is well-structured and understandable, though some details are missing, particularly regarding some design choices, data collection and experimental setup (see weaknesses below)."},"flag_for_ethics_review":{"value":["Yes, Responsible research practice (e.g., human subjects, data release)"]},"weaknesses":{"value":"1. Lack of sequence length analysis: While the authors emphasize the model's capability to handle variable and long input sequences, they do not provide an analysis of how sequence length affects performance, efficiency, or stability. It would be useful to see experiments with varying sequence lengths, insights into computational complexity and memory impact, and clarification on the sequence length used in their experiments.\n2. Potential for improved transformer-based model: The lower performance of RadarFormer likely reflects specific design choices rather than a fundamental transformer limitation for RF perception (e.g., using only amplitudes, using two successive frames instead of 12). The authors did not explore enhancements to RadarFormer, such as incorporating phase information, extending the temporal context, or adding RF-specific adaptations, which may have yielded results comparable to or better than RFMamba, questioning the necessity of Mamba-based SSMs.\n3. Unclear design choices: Some design decisions are not fully explained. For instance, in Figure 3(a), the scanning path appears to follow a bidirectional approach converging at the center. It is unclear why the authors did not consider capturing both sequences (from T−6 to T+6 and T+6 to T−6) and concatenating them, which could potentially enrich the temporal representation and improve performance.\n4. Lack of details on activity types, distribution, and environment diversity: The paper lacks specifics on the 19 activities in the THP dataset, their frequency distribution, and whether participants performed these activities statically or while moving freely. This is important for understanding the model’s generalization across activity types and movement patterns in RF-based perception. Additionally, the dataset was collected primarily in a single environment (with and without a wall), limiting environmental diversity. It remains unclear if the model would perform similarly in different settings, whether it would need extensive data collection, or if fine-tuning would be required in new environments.\n5. Limited dataset accessibility: Although the THP dataset provides valuable scenarios for evaluating RF-based human perception, it is not confirmed for public release, restricting reproducibility and broader research applications in this area.\n6. Unclear real-time performance: The paper lacks analysis on the compute and memory requirements needed for real-time performance, leaving practical deployment feasibility uncertain.\n7. Lack of generalization comparison with baseline models: While the authors claim that RFMamba generalizes well across various scenarios, there is no direct comparison with existing methods on generalization performance. Without showing how baseline models perform in similar conditions, it is difficult to assess whether RFMamba’s improvements are specific to the training environment or if they extend to varied, unseen scenarios."}},"nonreaders":[],"tmdate":1732470280807,"tcdate":1730226597427,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4681/Reviewer_aiy3"],"signatures":["ICLR.cc/2025/Conference/Submission4681/Reviewer_aiy3"],"forum":"lG9fjBLb6d","number":1,"license":"CC BY 4.0","cdate":1730226597427,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4681/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732470280807,"domain":"ICLR.cc/2025/Conference","replyto":"lG9fjBLb6d","id":"bRRtQUDtph","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Human-centric perception","State space model","Radio frequency"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Human-centric perception with radio frequency (RF) signals has recently entered a new era of end-to-end processing with Transformers. Considering the long-sequence nature of RF signals, the State Space Model (SSM) has emerged as a superior alternative due to its effective long-sequence modeling and linear complexity. However, integrating SSM into RF-based sensing presents unique challenges including the fundamentally different signal representation, distinct frequency responses in different scenarios, and incomplete capture caused by specular reflection. To address this, we carefully devise a dual-branch SSM block that is characterized by adaptively grasping the most informative frequency cues and the assistant spatial information to fully explore the human representations from radar echoes. Based on these two branchs, we further introduce an SSM-based network for handling various downstream human perception tasks, named RFMamba. Extensive experimental results demonstrate the superior performance of our proposed RFMamba across all three downstream tasks. To the best of our knowledge, RFMamba is the first attempt to introduce SSM into RF-based human-centric perception."},"_bibtex":{"value":"@inproceedings{\nzhang2025rfmamba,\ntitle={{RFM}amba: Frequency-Aware State Space Model for {RF}-Based Human-Centric Perception},\nauthor={Rui Zhang and Ruixu Geng and Yadong Li and Ruiyuan Song and Hanqin Gong and Dongheng Zhang and Yang Hu and Yan Chen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=lG9fjBLb6d}\n}"},"title":{"value":"RFMamba: Frequency-Aware State Space Model for RF-Based Human-Centric Perception"},"pdf":{"value":"/pdf/916ad0a87bbdac678a4df4e6a85d45ce45de18ec.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"zhang|rfmamba_frequencyaware_state_space_model_for_rfbased_humancentric_perception"},"authorids":{"value":["~Rui_Zhang18","~Ruixu_Geng1","~Yadong_Li3","~Ruiyuan_Song1","~Hanqin_Gong1","~Dongheng_Zhang1","~Yang_Hu3","~Yan_Chen15"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Rui Zhang","Ruixu Geng","Yadong Li","Ruiyuan Song","Hanqin Gong","Dongheng Zhang","Yang Hu","Yan Chen"]}},"version":2},{"content":{"research_area_keywords":{"value":"generative models, transfer learning/domain adaptation"},"languages_studied":{"value":"N/A"},"venue":{"value":"ACL ARR 2024 October Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025mambashedder,\ntitle={Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - October 2024},\nyear={2025},\nurl={https://openreview.net/forum?id=wszhnXn96F},\nnote={under review}\n}"},"title":{"value":"Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models"},"contribution_types":{"value":["NLP engineering experiment","Approaches low compute settings-efficiency"]},"abstract":{"value":"Large pre-trained models have achieved outstanding results in sequence modeling in the past few years. The Transformer block and its attention mechanism have been the main drivers of the success of these models. Recently, alternative architectures, such as Structured State Space Models (SSMs), have been proposed to address the inefficiencies of Transformers. This paper explores the compression of the recently proposed Mamba architecture (and their hybrids). We study the sensitivity of these models to the removal of selected components at different granularities to reduce the model size and computational overhead, thus improving their efficiency while maintaining accuracy. The proposed solutions, collectively referred to as Mamba-Shedder, achieve a speed-up of up to 1.39x during inference, demonstrating that model efficiency can be improved by eliminating several redundancies with minimal impact on overall model performance."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/fce98d86985298c337099e798f61585d4e8b2a6e.pdf"},"research_area":{"value":"Machine Learning for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2024/October/Submission"}},"tmdate":1782962332607,"tcdate":1728928934291,"writers":["aclweb.org/ACL/ARR/2024/October","aclweb.org/ACL/ARR/2024/October/Submission645/Authors"],"signatures":["aclweb.org/ACL/ARR/2024/October/Submission645/Authors"],"forum":"wszhnXn96F","license":"CC BY 4.0","number":645,"cdate":1728928934291,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2024/October/-/Submission","aclweb.org/ACL/ARR/2024/October/-/Edit","aclweb.org/ACL/ARR/2024/October/-/Post_Submission","aclweb.org/ACL/ARR/2024/October/-/Desk_Rejected_Submission","aclweb.org/ACL/ARR/2024/October/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2024/October/-/Preprint_Post_Submission"],"mdate":1782962332607,"odate":1736460501709,"domain":"aclweb.org/ACL/ARR/2024/October","id":"wszhnXn96F","version":2},{"content":{"comment":{"value":"> *An ablation study is essential to demonstrate the contribution of individual components in LIME to both acceleration and accuracy. The paper currently lacks such analysis. The authors are encouraged to include an ablation section, comparing LIME against both the standard Transformer and state-of-the-art accelerated Transformer variants, and to report results on model accuracy and computational complexity.*\n\nThat’s a great suggestion. We have added ablation studies (see table 6(a) in section 4.7) for various components of LIME, such as replacing LIME-XOR decoupled attention with \n* sum pooling of the personalized link embeddings\n* dot product between the candidate embedding and link embeddings\n* MLP applied to padded user history embedding sequences on user side and item embeddings on item side (LIME-MLP)\n* XOR-style attention on the user side and MLP on the item side (LIME-XOR-MLP Hybrid)\n\nThese variants performed significantly worse than LIME-XOR and clearly LIME-XOR’s design on the user side is especially crucial to strong performance. This is because LIME-XOR adeptly handles variable length sequences and gives a low-rank approximation to self-attention.\n\n**(a) Ablation Study**\n\n| Model Variant               |   AUC      |\n|-----------------------------|------------|\n| LIME-XOR                    | **0.7448** |\n| LIME-XOR w/ Link Pooling    |  0.7400    |\n| LIME-XOR w/ Dot Product     |  0.7404    |\n| LIME-MLP                    |  0.7407    |\n| LIME-XOR–MLP Hybrid         |  0.7442    |\n\n\n\nRegarding highly optimized transformer variants, our benchmark comparisons are now all based on Flash Attention v2 implemented in Triton (explained more in the question above). We also added several popular linear attention variants to the comparison (see table 6(b) in Section 4.8) which we reproduce below. In terms of both complexity and AUC, LIME-XOR performs better than other popular linear attention alternatives since our low-rank $\\ell \\ll N(U), M(U)$.\n\n**(b) Linear Attention Comparison**\n\n| Model Variant        |   AUC     | Complexity                           |\n|----------------------|-----------|--------------------------------------|\n| HSTU Skyline         |  0.7444   | O(N(U)² + N(U) · M(U))               |\n| Linear Transformer   |  0.7423   | O(N(U) · M(U))                       |\n| Longformer           |  0.7443   | O(N(U) · M(U))                       |\n| LREA (Linformer)     |  0.7408   | O(N(U) · M(U))                       |\n| LIME-XOR             |  0.7448   | O(N(U)ℓ + M(U)ℓ)                     |\n| LIME-XOR+Window      | **0.7453**| O(N(U)ℓ + M(U)ℓ)                     |\n\n\nWe also introduce LIME-XOR+Window, which augments LIME-XOR with a local sliding-window mask of size $\\ell$ to capture additional local interactions. This yields improved results on both KuaiRand and our industrial dataset, while keeping complexity comparable to LIME-XOR. Since any form of linear attention can be integrated into our decoupled transformer LIME framework, in the future we can explore other sophisticated techniques such as Mamba [4], MOBA [5], and Kimi Linear [6].\n\n[4] Gu, Albert, and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. \n\n[5] Pióro, Maciej, Kamil Ciebiera, Krystian Król, Jan Ludziejewski, Michał Krutul, Jakub Krajewski, Szymon Antoniak, Piotr Miłoś, Marek Cygan, and Sebastian Jaszczur. MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts.\n\n[6] K Team. Kimi Linear: An Expressive, Efficient Attention Architecture."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763771036616,"tcdate":1763768778922,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13136/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13136/Authors"],"forum":"Y2tKygPoUp","number":2,"license":"CC BY 4.0","cdate":1763768778922,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13136/-/Official_Comment"],"mdate":1763771036616,"domain":"ICLR.cc/2026/Conference","replyto":"NYC2MZ9Ya4","id":"YKZgpXhUWw","forumContent":{"TLDR":{"value":"We compress user sequence representation via xor-attention, and cache item representation via decoupled target attention, to scale up candidate size and user history length in recommendation systems."},"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"keywords":{"value":["decoupled attention","recommendation system","test time scaling","xor attention","linear attention","ranking"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Scaling large recommendation systems requires advancing three major frontiers: processing longer user histories, expanding candidate sets, and increasing model capacity. While promising, transformers' computational cost scales quadratically with the user sequence length and linearly with the number of candidates. This trade-off makes it prohibitively expensive to expand candidate sets or increase sequence length at inference, despite the significant performance improvements.\n\nWe introduce \\textbf{LIME}, a novel architecture that resolves this trade-off. Through two key innovations, LIME fundamentally reduces computational complexity. First,  low-rank ``link embeddings\" enable pre-computation of attention weights by decoupling user and candidate interactions, making the inference cost nearly independent of candidate set size. Second, a linear attention mechanism, \\textbf{LIME-XOR}, reduces the complexity with respect to user sequence length from quadratic ($O(N^2)$) to linear ($O(N)$).\n\nExperiments on public and industrial datasets show LIME achieves near-parity with state-of-the-art transformers but with a 10$\\times$ inference speedup on large candidate sets or long sequence lengths. When tested on a major recommendation platform, LIME improved user engagement while maintaining minimal inference costs with respect to candidate set size and user history length, establishing a new paradigm for efficient and expressive recommendation systems."},"_bibtex":{"value":"@misc{\nanonymous2026lime,\ntitle={{LIME}: Link-based user-item Interaction Modeling with decoupled xor attention for Efficient test time scaling},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=Y2tKygPoUp}\n}"},"title":{"value":"LIME: Link-based user-item Interaction Modeling with decoupled xor attention for Efficient test time scaling"},"pdf":{"value":"/pdf/e875274ee61fe557999ab1d0847c4efcb8ef2844.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"agarwal|lime_linkbased_useritem_interaction_modeling_with_decoupled_xor_attention_for_efficient_test_time_scaling"},"authorids":{"value":["~Ayush_Agarwal1","~Yunjiang_Jiang1","~Yang_Liu199","~Bi_Xue1"]},"authors":{"value":["Ayush Agarwal","Yunjiang Jiang","Yang Liu","Bi Xue"]}},"version":2},{"content":{"TLDR":{"value":"SSMs fail broad (position-addressed) retrieval not for capacity but accessibility: a curriculum unlocks it and the fix is objective design, not more state."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["state-space models","associative recall","retrieval capacity","optimization accessibility","curriculum learning","sequence models"]},"supplementary_material":{"value":"/attachment/65c1a9ef7676b3ada14da72d92d99ee9d13a25f4.zip"},"primary_area":{"value":"learning theory"},"abstract":{"value":"State-space models (SSMs) such as Mamba solve selective retrieval (associative recall, MQAR) yet fail broad retrieval (uniform-over-positions discrimination) that requires every input position to stay recoverable (Position Recovery, PR). We argue these are two different memory problems: selective retrieval is content-addressed (recall by key), broad retrieval is position-addressed (keep every position discriminable), with distinct capacity requirements and distinct accessibility. Is the broad deficit a capacity limit or an optimization artifact? We model retrieval as distributing a fixed per-position signal-to-noise budget across the sequence. A single linear SSM channel provably cannot solve filler recall, which forces channel specialization: broad retrieval needs retrieval rank scaling with the sequence length, selective only with the number of keys, and the required timescales must be placed to match the queried ages. The theory predicts the structure of selective-retrieval collapse and the specialization signature of successful backbones; both appear in trained weights, including multi-scale RetNet tracking the theoretical 1/m line. A symmetric causal intervention then separates the two explanations: a curriculum unlocks broad retrieval on a standard SSM and transfers to selective, and unlocks selective at its own lower floor where standard SGD fails. At every scale tested, both deficits are accessibility-limited; the barrier persists through 122M parameters. A controlled Pareto frontier shows the objective is the lever: pure language modeling leaves broad at the random floor, and the PPL cost of adding broad is architecture-dependent. The theory holds for the linearized state map; every causal and theoretical claim is validated on this project's own checkpoints, and the frozen-probe absence of broad information replicates on 2.7B open pretrained checkpoints of both families."},"_bibtex":{"value":"@inproceedings{\nanonymous2026selective,\ntitle={Selective but Not Broad: Retrieval Capacity and Accessibility in State-Space Models},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=8EzQDRLyfM},\nnote={under review}\n}"},"title":{"value":"Selective but Not Broad: Retrieval Capacity and Accessibility in State-Space Models"},"pdf":{"value":"/pdf/6580949a8c1b26e89b27d535ea9df9763954419b.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791219728139,"tcdate":1787323546472,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission2754/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission2754/Authors"],"forum":"8EzQDRLyfM","license":"CC BY 4.0","number":2754,"cdate":1787323546472,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791219728139,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"8EzQDRLyfM","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.07522v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"ren|samba_simple_hybrid_state_space_models_for_efficient_unlimited_context_language_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Liliang_Ren:","https://dblp.org/search/pid/api?q=author:Yang_Liu:","~Yadong_Lu1","https://dblp.org/search/pid/api?q=author:Yelong_Shen:","https://dblp.org/search/pid/api?q=author:Chen_Liang_0006:","~Weizhu_Chen1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.07522"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-07522,\n  publtype={informal},\n  author={Liliang Ren and Yang Liu and Yadong Lu and Yelong Shen and Chen Liang and Weizhu Chen},\n  title={Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.07522},\n  url={https://doi.org/10.48550/arXiv.2406.07522}\n}\n"},"abstract":{"value":"Efficiently modeling sequences with infinite context length has been a long-standing problem. Past works suffer from either the quadratic computation complexity or the limited extrapolation ability on length generalization. In this work, we present Samba, a simple hybrid architecture that layer-wise combines Mamba, a selective State Space Model (SSM), with Sliding Window Attention (SWA). Samba selectively compresses a given sequence into recurrent hidden states while still maintaining the ability to precisely recall memories with the attention mechanism. We scale Samba up to 3.8B parameters with 3.2T training tokens and show that Samba substantially outperforms the state-of-the-art models based on pure attention or SSMs on a wide range of benchmarks. When trained on 4K length sequences, Samba can be efficiently extrapolated to 256K context length with perfect memory recall and show improved token predictions up to 1M context length. As a linear-time sequence model, Samba enjoys a 3.73x higher throughput compared to Transformers with grouped-query attention when processing user prompts of 128K length, and 3.64x speedup when generating 64K tokens with unlimited streaming. A sample implementation of Samba is publicly available in https://github.com/microsoft/Samba."},"title":{"value":"Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling"},"authors":{"value":["Liliang Ren","Yang Liu","Yadong Lu","Yelong Shen","Chen Liang","Weizhu Chen"]}},"tmdate":1739674478654,"pdate":1704067200000,"tcdate":1733105640798,"writers":["~"],"signatures":["~Weizhu_Chen1"],"forum":"fMiM1Gg22i","license":"CC BY-SA 4.0","number":251941,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1739674478654,"domain":"DBLP.org","id":"fMiM1Gg22i","version":2},{"content":{"summary":{"value":"This study introduces PhyxMamba, a hybrid forecasting framework that combines a Mamba-based state-space model with physics-informed principles to generate long-term predictions of chaotic dynamical systems based on short-term observations. The proposed method takes low dimensional time series, and then reconstructs the system’s attractor manifold through time-delay embedding, creating a high-dimensional representation to recover global dynamical features. It then applies a generative training strategy with multi-patch prediction and a residual Mamba architecture to model long-range dependencies while maintaining physical coherence. To mitigate error accumulation and preserve the geometry of the strange attractor, PhyxMamba incorporates a student-forcing stage and Maximum Mean Discrepancy (MMD)–based regularization, in order to enforce statistical consistency between predicted and true trajectories. Experiments on simulated chaotic systems (Lorenz63, Rössler, Lorenz96) and real-world datasets (ECG, EEG) show that PhyxMamba achieves strong forecasting accuracy and recovers the invariant measures and geometric integrity of chaotic attractors. Ablation and robustness analyses show that the physics-guided embedding and generative design contribute to stable long-horizon prediction."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"*[Question 1]* I am confused by the finding that “Strong short-term predictive accuracy does not necessarily guarantee robust long-term forecasting performance.” Since the systems that the authors study are generated by ordinary differential equations, wouldn’t the best possible model approximate numerical integration to high precision? How can a model get long-term predictions right without mastering short term first?\n\n*[Question 2]* What does a forecast for “1 Lyapunov time” mean in this context? If I understand Eq. 2 directly, this can be calculated directly from the ODE, but I don’t see how I would get a single number out, even for the “Lorenz” example equations that the authors provide. Do you find this empirically by analyzing your forecasts post-hoc? How does this work for the EEG and other time series?\n\n*[Question 3]* How do you calculate the KL divergence between the forecasted time series and the ground truth? Since the output of PhyxMamba is a set of discrete timepoints, are you binning the output space in order to generate an empirical distribution? Does that bias the estimator in higher dimensions?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The Maximum Mean Discrepancy (MMD) regularizer is a nice approach. While this has previously been applied to time series datasets, it is not a widely-known method in this setting, and it appears to improve the author’s model’s performance on this dataset.\n\nThe residual stacking approach in the Mamba blocks is a great idea. This approach is inspired by the NBEATS papers, and it seems to help in this case.\n\nI found the paper clear, well-structured, and easy-to-follow. The related work section covers the appropriate areas well. While I was not familiar with all terminology in the authors’ subfield, I found that the description was clear enough that I could follow the narrative.\n\nThe experiments are careful and the baselines are appropriate: multiple metrics are used to assess the quality of the forecasts over both long- and short- term intervals, and the authors ablate many of their custom architecture’s components to determine how much they contribute to the model."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The architecture seems ad hoc, like a collection of methods combined together, and I do not feel that it inherits the simplicity or theoretical advantages of Mamba2 and other models like it. The methods section describes a very complicated architecture and multistage training loop. The authors first perform multi-patch next step prediction with teacher forcing. They then perform student forcing, checking that the model’s own predictions capture long-term structure. During the latter loop, the authors enforce regularization terms that ensure history-prediction consistency, as well as prediction-ground-truth consistency. All of these choices introduce substantial complexity into the training loop, including hyperparameters like the number of time delays, the patch size, embedding dimension, mamba depth, multi-patch depth, and the size of the various regularization terms.
\n\n2. Several architectural and methods choices do not fully make sense to me. The authors provide ablations showing that both time delay embedding and patching improve the model. But that doesn’t seem possible, since the two featurizations are redundant:  Since w are going to flatten into an embedding of shape N x (D*V*m), the patch size duplicates the time delays and vice versa. Additionally, my understanding is that Mamba is an SSM architecture that modifies an earlier time series SSM (S4) to work with language data. But the authors are now using Mamba for time series. This seems like an unusual choice, compared to using the S-series models, an RNN with spectral initialization, or a reservoir computer.\n\n3. While I followed the time series modeling claims, I do not follow the chaotic dynamical systems results. The authors are learning to model a dynamical system based on a time series of length T, and they then aim to forecast that system for H >> T. But this does not seem possible. If I were fitting a linear dynamical system on T datapoints in D dimensions, the error of the fit would scale as 1/sqrt{T}. Wouldn’t these small errors exponentially blow up, precluding attaining the correct long-term statistics when H >> T?\n\n4. Several strong and conceptually well-motivated domain-specific forecast architectures have been proposed for chaotic systems over the past few years, including next-generation reservoir computers and teacher-forced methods. While it is good that this particular model performs well on benchmarks, the problem area is narrow enough that I would prefer to see more conceptual novelty. Why wouldn’t a motivated practitioner with an unknown time series use TimeNet, a general model that performs well on general time series benchmarks like GIFT-Eval? Particularly because most time series we encounter in the wild have unknown provenance and thus unknown dynamical structure like stationarity. While not all models have to be intended for practical applications, I would like to see stronger conceptual insights, rather than the current complicated collection of small tweaks to existing architectures."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927558266,"tcdate":1761547653256,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17726/Reviewer_fZEw"],"signatures":["ICLR.cc/2026/Conference/Submission17726/Reviewer_fZEw"],"forum":"KWsrSSNVSB","number":2,"license":"CC BY 4.0","cdate":1761547653256,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17726/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927558266,"domain":"ICLR.cc/2026/Conference","replyto":"KWsrSSNVSB","id":"RHU0N7Z8uC","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We develop PhyxMamba, a framework that integrates a Mamba state-space model with physics-informed principles to forecast long-term chaotic systems using only short-term historical data."},"keywords":{"value":["Chaotic Systems","State-space Models"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Long-term forecasting of chaotic systems remains a fundamental challenge due to the intrinsic sensitivity to initial conditions and the complex geometry of strange attractors. Conventional approaches, such as reservoir computing, typically require training data that incorporates long-term continuous dynamical behavior to comprehensively capture system dynamics. While advanced deep sequence models can capture transient dynamics within the training data, they often struggle to maintain predictive stability and dynamical coherence over extended horizons. Here, we propose PhyxMamba, a framework that integrates a Mamba-based state-space model with physics-informed principles to forecast long-term behavior of chaotic systems given short-term historical observations on their state evolution. We first reconstruct the attractor manifold with time-delay embeddings to extract global dynamical features. After that, we introduce a generative training scheme that enables Mamba to replicate the physical process. It is further augmented by multi-patch prediction and attractor geometry regularization for physical constraints, enhancing predictive accuracy and preserving key statistical properties of systems. Extensive experiments on simulated and real-world chaotic systems demonstrate that PhyxMamba delivers superior forecasting accuracy and faithfully captures essential statistics from short-term historical observations."},"_bibtex":{"value":"@misc{\nliu2026mamba,\ntitle={Mamba Integrated with Physics Principles Masters Long-term Chaotic System Forecasting},\nauthor={Chang Liu and Bohao Zhao and Jingtao Ding and Huandong Wang and Yong Li},\nyear={2026},\nurl={https://openreview.net/forum?id=KWsrSSNVSB}\n}"},"title":{"value":"Mamba Integrated with Physics Principles Masters Long-term Chaotic System Forecasting"},"pdf":{"value":"/pdf/b8497e44ef182aaaf814d83c67b5a9b1c64d0e30.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"liu|mamba_integrated_with_physics_principles_masters_longterm_chaotic_system_forecasting"},"authorids":{"value":["~Chang_Liu29","~Bohao_Zhao1","~Jingtao_Ding1","~Huandong_Wang2","~Yong_Li7"]},"authors":{"value":["Chang Liu","Bohao Zhao","Jingtao Ding","Huandong Wang","Yong Li"]}},"version":2},{"content":{"summary":{"value":"This paper presents the first theoretical analysis of in-context learning (ICL) in Mamba. Specifically, the authors study a one-layer Mamba with a linear attention component followed by a nonlinear gating mechanism and analyze its ability to perform ICL on binary classification tasks—even when prompts contain additive outliers.\nThe paper’s main contributions are:\n1. It provides quantitative convergence and generalization guarantees for Mamba trained on prompts with outliers, showing robustness even when nearly all examples are corrupted.\n2. It theoretically compares Mamba to linear Transformers, proving that Mamba can generalize in the presence of a higher fraction of outliers.\n3. It characterizes the mechanistic role of Mamba’s components: linear attention selects examples sharing relevant patterns with the query, while the gating mechanism suppresses outliers and emphasizes nearby examples.\n\nSynthetic experiments support the theory by showing that Mamba achieves lower classification error than linear Transformers under various outlier corruption schemes."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"How would the gating mechanism behave if the nearest examples to the query are misleading or adversarially labeled? Does the exponential decay still guarantee robustness, or could it amplify incorrect patterns?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"Originality:\nThis work is among the first to give a theoretical treatment of Mamba’s ICL mechanism, especially under the presence of outliers. Previous ICL analyses primarily focus on Transformers. The extension to Mamba—given its nonlinear gating and state-space formulation—is novel and nontrivial.\n\nTechnical quality:\nThe paper presents well-structured theorems (Theorems 1–4) detailing convergence, sample complexity, and robustness. The proofs seem grounded in prior analytical frameworks for Transformers (Li et al., 2024a,b) but adapted to handle the additional nonlinearity of gating. The comparison to linear Transformers provides useful insights into the tradeoff between convergence speed and robustness.\n\nSignificance:\nUnderstanding Mamba’s ICL properties is highly relevant as Mamba-based models gain traction as efficient alternatives to Transformers. The result that Mamba tolerates a higher fraction of outliers could motivate more robust sequence modeling approaches in practice.\n\nClarity and organization:\nThe paper is generally well-written and logically organized."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The task and outlier definition seem not natural and general. This limits the generality of the conclusions. It is unclear whether similar robustness properties would hold for more natural data distributions or real-world text prompts.\n\n2. Several theorems introduce numerous symbols without sufficient intuitive explanation, which makes the results hard to interpret. The meaning of constants such as Θ(·), poly(·), and ranges should be contextualized more clearly.\n\n3. The experiments are purely synthetic and cover only a very limited case. While they match the theory, they provide no evidence that these findings generalize to realistic NLP or vision tasks.\n\n4. Recent empirical work shows that large Transformers can also exhibit robustness to mislabeling or outlier examples in ICL regimes. A discussion or comparison to this line of work is missing.\n\n5. The analysis assumes that “examples close to the query” are beneficial, but if these nearby examples are adversarial or have opposite labels, the gating mechanism could amplify errors instead of mitigating them. This edge case should be acknowledged."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919930126,"tcdate":1762155829347,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7901/Reviewer_v2NB"],"signatures":["ICLR.cc/2026/Conference/Submission7901/Reviewer_v2NB"],"forum":"tswBfpkwHn","number":2,"license":"CC BY 4.0","cdate":1762155829347,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7901/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919930126,"domain":"ICLR.cc/2026/Conference","replyto":"tswBfpkwHn","id":"pUV9wrsc7E","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"This paper provide the first theoretical analysis of the training dynamics of a one-layer Mamba model and characterizes its in-context learning generaization ability with additive outliers in the prompt"},"keywords":{"value":["Mamba","in-context learning","Transformer","generalization","training dynamics","deep learning theory"]},"primary_area":{"value":"learning theory"},"abstract":{"value":"The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks. Like Transformers, Mamba exhibits in-context learning (ICL) capabilities, i.e.,  making predictions for new tasks based on a prompt containing input-label pairs and a query, without requiring fine-tuning.\nDespite its empirical success, the theoretical understanding of Mamba remains limited, largely due to the nonlinearity introduced by its gating mechanism. To the best of our knowledge, this paper presents the first theoretical analysis of the training dynamics of a one-layer Mamba model, which consists of a linear attention component followed by a nonlinear gating layer, and its ICL generalization on unseen binary classification tasks, even when the prompt includes additive outliers. Our analysis shows that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layer to suppress the influence of outliers. By establishing and comparing to the analysis of linear Transformers under the same setting, we show that although Mamba may require more training iterations to converge, it maintains accurate predictions even when the proportion of outliers exceeds the threshold that a linear Transformer can tolerate. These theoretical findings are supported by empirical experiments."},"_bibtex":{"value":"@misc{\nli2026can,\ntitle={Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis},\nauthor={Hongkang Li and Songtao Lu and Xiaodong Cui and Pin-Yu Chen and Meng Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=tswBfpkwHn}\n}"},"title":{"value":"Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis"},"pdf":{"value":"/pdf/771e925a0afdf6afc4b8a950bc4b771d6afabad8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"li|can_mamba_learn_in_context_with_outliers_a_theoretical_generalization_analysis"},"authorids":{"value":["~Hongkang_Li1","~Songtao_Lu1","~Xiaodong_Cui1","~Pin-Yu_Chen1","~Meng_Wang4"]},"authors":{"value":["Hongkang Li","Songtao Lu","Xiaodong Cui","Pin-Yu Chen","Meng Wang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.05316v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"zeng|cmamba_channel_correlation_enhanced_state_space_models_for_multivariate_time_series_forecasting"},"authorids":{"value":["~Chaolv_Zeng2","https://dblp.org/search/pid/api?q=author:Zhanyu_Liu:","https://dblp.org/search/pid/api?q=author:Guanjie_Zheng:","https://dblp.org/search/pid/api?q=author:Linghe_Kong:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.05316"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-05316,\n  publtype={informal},\n  author={Chaolv Zeng and Zhanyu Liu and Guanjie Zheng and Linghe Kong},\n  title={C-Mamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.05316},\n  url={https://doi.org/10.48550/arXiv.2406.05316}\n}\n"},"abstract":{"value":"In recent years, significant progress has been made in multivariate time series forecasting using Linear-based, Transformer-based, and Convolution-based models. However, these approaches face notable limitations: linear forecasters struggle with representation capacities, attention mechanisms suffer from quadratic complexity, and convolutional models have a restricted receptive field. These constraints impede their effectiveness in modeling complex time series, particularly those with numerous variables. Additionally, many models adopt the Channel-Independent (CI) strategy, treating multivariate time series as uncorrelated univariate series while ignoring their correlations. For models considering inter-channel relationships, whether through the self-attention mechanism, linear combination, or convolution, they all incur high computational costs and focus solely on weighted summation relationships, neglecting potential proportional relationships between channels. In this work, we address these issues by leveraging the newly introduced state space model and propose \\textbf{C-Mamba}, a novel approach that captures cross-channel dependencies while maintaining linear complexity without losing the global receptive field. Our model consists of two key components: (i) channel mixup, where two channels are mixed to enhance the training sets; (ii) channel attention enhanced patch-wise Mamba encoder that leverages the ability of the state space models to capture cross-time dependencies and models correlations between channels by mining their weight relationships. Our model achieves state-of-the-art performance on seven real-world time series datasets. Moreover, the proposed mixup and attention strategy exhibits strong generalizability across other frameworks."},"title":{"value":"C-Mamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting"},"authors":{"value":["Chaolv Zeng","Zhanyu Liu","Guanjie Zheng","Linghe Kong"]}},"tmdate":1722954575533,"pdate":1704067200000,"tcdate":1722953889664,"writers":["~"],"signatures":["~Chaolv_zeng1"],"forum":"NQTzWdNidR","license":"CC BY-SA 4.0","number":55560,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1722954575533,"domain":"DBLP.org","id":"NQTzWdNidR","version":2},{"content":{"venue":{"value":"Machine Learning Science and Technology"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"pessoa|mamba_time_series_forecasting_with_uncertainty_quantification"},"authorids":{"value":["~Pedro_Pessoa1","paul.campitelli@gmail.com","douglas.shepherd@asu.edu","~S._Banu_Ozkan1","~Steve_Presse1"]},"html":{"value":"https://doi.org/10.1088/2632-2153/adec3b"},"abstract":{"value":"State space models, such as Mamba, have recently garnered attention in time series forecasting (TSF) due to their ability to capture sequence patterns. However, in electricity consumption benchmarks, Mamba forecasts exhibit a mean error of approximately 8%. Similarly, in traffic occupancy benchmarks, the mean error reaches 18%. This discrepancy leaves us to wonder whether the prediction is simply inaccurate or falls within error given spread in historical data. To address this limitation, we propose a method to quantify the predictive uncertainty of Mamba forecasts. To achieve this, we propose a dual-network framework based on the Mamba architecture for probabilistic forecasting, where one network generates point forecasts while the other estimates predictive uncertainty by modeling variance. We abbreviate our tool, Mamba with probabilistic TSF, as Mamba-ProbTSF and the code for its implementation is available on GitHub https://github.com/PessoaP/Mamba-ProbTSF. Evaluating this approach on synthetic and real-world benchmark datasets, we find Kullback–Leibler divergence between the learned distributions and the data–which, in the limit of infinite data, should converge to zero if the model correctly captures the underlying probability distribution–reduced to the order of 10−3 for synthetic data and 10−1 for real-world benchmark. We find that in both the electricity consumption and traffic occupancy benchmark, the true trajectory stays within the predicted uncertainty interval at the two-sigma level about 95% of the time. We further compare Mamba-ProbTSF against leading probabilistic forecast methods, DeepAR and ARIMA, and show that our method consistently achieves lower forecast errors while offering more reliable uncertainty quantification. We end with a consideration of potential limitations, adjustments to improve performance, and considerations for applying this framework to processes for purely or largely stochastic dynamics where the stochastic changes accumulate as observed, for example, in pure Brownian motion or molecular dynamics trajectories."},"title":{"value":"Mamba time series forecasting with uncertainty quantification"},"authors":{"value":["Pedro Pessoa","Paul Campitelli","Douglas P. Shepherd","S. Banu Ozkan","Steve Presse"]}},"tmdate":1778201215182,"pdate":1753167600000,"tcdate":1778201215182,"writers":["~Pedro_Pessoa1","paul.campitelli@gmail.com","douglas.shepherd@asu.edu","~S._Banu_Ozkan1","~Steve_Presse1"],"signatures":["~Pedro_Pessoa1"],"forum":"iL5xbY9XuI","license":"CC BY 4.0","number":49932,"cdate":1778201215182,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1778201215182,"domain":"OpenReview.net/Archive","id":"iL5xbY9XuI","version":2},{"content":{"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Koopman operator; model predictive control; nonlinear systems"]},"supplementary_material":{"value":"/attachment/0bf4c76c44568ae8539ede6166bb6a47f0d2be3a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The Koopman theory, which enables the transformation of nonlinear systems into linear representations, is a powerful and efficient tool to model and control nonlinear systems. However, the ability of the Koopman operator to model complex systems, particularly time-varying systems, is limited by the fixed linear state-space representation. To address the limitation, the large language model, Mamba, is considered a promising strategy for enhancing modeling capabilities while preserving the linear state-space structure.\nIn this paper, we propose a new framework, the Mamba-based Koopman operator (MamKO), which provides enhanced model prediction capability and adaptability, as compared to Koopman models with constant Koopman operators. Inspired by the Mamba structure, MamKO generates Koopman operators from online data; this enables the model to effectively capture the dynamic behaviors of the nonlinear system over time. A model predictive control system is then developed based on the proposed MamKO model. The modeling and control performance of the proposed method is evaluated through experiments on benchmark time-invariant and time-varying systems. The experimental results demonstrate the superiority of the proposed approach. Additionally, we perform ablation experiments to test the effectiveness of individual components of MamKO. This approach unlocks new possibilities for integrating large language models with control frameworks, and it achieves a good balance between advanced modeling capabilities and real-time control implementation efficiency."},"_bibtex":{"value":"@inproceedings{\nli2025mamko,\ntitle={Mam{KO}: Mamba-based Koopman operator for modeling and predictive control},\nauthor={ZHAOYANG LI and Minghao Han and Xunyuan Yin},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=hNjCVVm0EQ}\n}"},"title":{"value":"MamKO: Mamba-based Koopman operator for modeling and predictive control"},"pdf":{"value":"/pdf/49097a18837a65cc1415fac755f61c717559a104.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"li|mamko_mambabased_koopman_operator_for_modeling_and_predictive_control"},"authorids":{"value":["~ZHAOYANG_LI12","~Minghao_Han2","~Xunyuan_Yin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["ZHAOYANG LI","Minghao Han","Xunyuan Yin"]}},"tmdate":1740141902638,"pdate":1737562741380,"tcdate":1727451044070,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10895/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10895/Authors"],"forum":"hNjCVVm0EQ","license":"CC BY 4.0","number":10895,"cdate":1727451044070,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission10895/-/Full_Submission","ICLR.cc/2025/Conference/Submission10895/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit","ICLR.cc/2025/Conference/Submission10895/-/Camera_Ready_Revision"],"mdate":1740141902638,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"hNjCVVm0EQ","version":2},{"content":{"summary":{"value":"This paper investigates the stability and regularity properties of continuous-time selective state-space models, a class of deep models whose parameters depend on the input and can change abruptly through discontinuous gating. The study draws on concepts from passivity theory and input-to-state stability to build a rigorous control-theoretic foundation for understanding such systems.\n\nThe authors first show that intrinsic energy dissipation ensures exponential forgetting of past states, even when the gating signals are discontinuous. They then prove that any passive selective state-space model possesses a minimal quadratic energy function whose defining matrix satisfies a form of absolute upper semicontinuity, guaranteeing stable energy behavior across switching dynamics. Building on this, the paper derives parametric linear matrix inequality conditions and kernel constraints that limit how gating mechanisms can influence the system while preserving passivity, formalizing a notion of irreversible forgetting. Finally, it provides sufficient conditions for global input-to-state stability, showing that uniform dissipativity leads to robustness under general inputs."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"In general, models with long-term memory tend to be more prone to instability. It would be beneficial if the authors could adopt an approach that explicitly analyzes or discusses the stability of such long-memory dynamics."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"This paper presents a highly original and rigorous treatment of the stability and regularity properties of selective state-space models. Its main theoretical contribution—the formalization of irreversible forgetting—introduces a novel and elegant concept: once a state direction becomes energy-less, it remains so structurally, constraining how future gating can affect the system without violating passivity. This insight not only deepens the theoretical understanding of modern state-space models but also bridges control theory and machine learning in a principled way."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper’s main limitation lies in its narrow empirical validation. While the theoretical framework is rigorous and comprehensive, the experiments are minimal and primarily illustrative rather than demonstrative of broader applicability. As a result, it remains unclear how the proposed theoretical results can be used or extended by the wider machine learning community. The work would benefit from a clearer discussion of its practical relevance, for instance by connecting the theoretical insights to optimization or algorithmic improvements that directly impact model training or design. Without such developments, the contribution risks being perceived as primarily conceptual, with limited immediate value for practitioners working on real-world selective state-space models or related architectures."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924348884,"tcdate":1761014494934,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13823/Reviewer_nz1P"],"signatures":["ICLR.cc/2026/Conference/Submission13823/Reviewer_nz1P"],"forum":"F5jHkdgtYG","number":1,"license":"CC BY 4.0","cdate":1761014494934,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13823/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924348884,"domain":"ICLR.cc/2026/Conference","replyto":"F5jHkdgtYG","id":"oFWxrrFuUZ","forumContent":{"TLDR":{"value":"We provide a rigorous theoretical framework using control theory (passivity, ISS) to analyze and ensure stability and regularity in continuous-time selective SSMs, even when their parameters change discontinuously via gating."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["state space models","passivity theory","input-to-state-stability","regularity","stability","dynamical systems","optimization","control"]},"primary_area":{"value":"learning theory"},"abstract":{"value":"Deep Selective State-Space Models (SSMs), characterized by input-dependent, time-varying parameters, offer significant expressive power but pose challenges for stability analysis, especially with discontinuous gating signals. In this paper, we investigate the stability and regularity properties of continuous-time selective SSMs through the lens of passivity and Input-to-State Stability (ISS). We establish that intrinsic energy dissipation guarantees exponential forgetting of past states. Crucially, we prove that the unforced system dynamics possess an underlying minimal quadratic energy function whose defining matrix exhibits robust $\\text{AUC}_{\\text{loc}}$ regularity, accommodating discontinuous gating. Furthermore, assuming a universal quadratic storage function ensures passivity across all inputs, we derive parametric LMI conditions and kernel constraints that limit gating mechanisms, formalizing \"irreversible forgetting\" of recurrent models. Finally, we provide sufficient conditions for global ISS, linking uniform local dissipativity to overall system robustness. Our findings offer a rigorous framework for understanding and designing stable and reliable deep selective SSMs."},"_bibtex":{"value":"@misc{\nzubic2025regularity,\ntitle={Regularity and Stability Properties of Selective {SSM}s with Discontinuous Gating},\nauthor={Nikola Zubic and Davide Scaramuzza},\nyear={2025},\nurl={https://openreview.net/forum?id=F5jHkdgtYG}\n}"},"title":{"value":"Regularity and Stability Properties of Selective SSMs with Discontinuous Gating"},"pdf":{"value":"/pdf/278dbf15ca49a56cadd83f38d5812eb9c868b3b2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"zubic|regularity_and_stability_properties_of_selective_ssms_with_discontinuous_gating"},"authorids":{"value":["~Nikola_Zubic1","~Davide_Scaramuzza1"]},"authors":{"value":["Nikola Zubic","Davide Scaramuzza"]}},"version":2},{"content":{"venue":{"value":"TIME 2007"},"pdf":{"value":"https://ieeexplore.ieee.org/iel5/4438655/4438656/04438673.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2007"},"paperhash":{"value":"combi|conceptual_modeling_of_temporal_clinical_workflows"},"authorids":{"value":["~Carlo_Combi1","https://dblp.org/search/pid/api?q=author:Matteo_Gozzi:","https://dblp.org/search/pid/api?q=author:José_M._Juárez:","https://dblp.org/search/pid/api?q=author:Barbara_Oliboni:","https://dblp.org/search/pid/api?q=author:Giuseppe_Pozzi:"]},"html":{"value":"https://doi.org/10.1109/TIME.2007.45"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/CombiGJOP07,\n  author={Carlo Combi and Matteo Gozzi and José M. Juárez and Barbara Oliboni and Giuseppe Pozzi},\n  title={Conceptual Modeling of Temporal Clinical Workflows},\n  year={2007},\n  cdate={1167609600000},\n  pages={70-81},\n  url={https://doi.org/10.1109/TIME.2007.45},\n  booktitle={TIME},\n  crossref={conf/time/2007}\n}\n"},"abstract":{"value":"The diffusion of clinical guidelines to describe the proper way to deal with patients' situations is spreading out and opens new issues in the context of modeling and managing (temporal) information about medical activities. Guidelines can be seen as processes describing the sequence of activities to be executed, and thus approaches proposed in the business context can be used to model them. In this paper, we propose a general conceptual workflow model, considering both activities and their temporal properties, and focus on the representation of clinical guidelines by the proposed model."},"title":{"value":"Conceptual Modeling of Temporal Clinical Workflows"},"authors":{"value":["Carlo Combi","Matteo Gozzi","José M. Juárez","Barbara Oliboni","Giuseppe Pozzi"]}},"tmdate":1747652537587,"pdate":1167609600000,"tcdate":1747652518296,"writers":["~"],"signatures":["~Carlo_Combi1"],"forum":"86KheHLugo","license":"CC BY-SA 4.0","number":526590,"cdate":1167609600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747652537587,"domain":"DBLP.org","id":"86KheHLugo","version":2},{"content":{"summary":{"value":"This paper presents a framework called Cross-Modal Contextual Modeling (CM2) to improve Audio-Visual Speech Enhancement (AVSE). CM2 integrates two types of contextual information—semantic and signal context—across audio and visual data to enhance speech quality in noisy environments. The semantic context helps the model infer missing or corrupted speech by maintaining consistency across segments, while the signal context leverages coherence within signal frames. Additionally, the approach emphasizes the importance of visual features, such as the speaker's facial cues, which can correlate with audio frequency characteristics, thus aiding the enhancement process.\n\nThe model uses three main components to build and integrate these contexts: a Semantic Context Module (SeCM) for initial contextual extraction, a Signal Context Module (SiCM) for signal-level context from noisy inputs, and a Cross-Context Fusion Module (CCFM) to combine these contexts. This architecture allows for detailed context fusion across different modalities, effectively improving speech clarity, especially in challenging low signal-to-noise ratios. Experimental results show that CM2 outperforms other state-of-the-art models, demonstrating substantial gains in metrics related to speech quality and intelligibility.\n\nMethodology:\n - SeCM: Processes Video and Audio in Time Domain audio stream, proposes three solutions (V, PV, PAV) from other papers, and test to find the best approach.\n - Audio Encoder: Processes audio in TFM Domain, uses 4 stacked convolutions.\n - SiCM: uses a bidirectional Mamba approach to process along different dimensions and time directions.\n - Time-Frequency Upsampler: two transpose convolutions with BN and prelu. used to take the frequency dim from 1 to match the frequency dim of the other audio route.\n- CSBlock: split across the channels and concatenate cross-modality information\n- AF Block - Generate attention map by applying linear to P, use two SiMC on P^c and E^c, then multiply the mask by both features and they are added together. A 2d conv binds the information together."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"AF Block:\n - Equation 11 is an interesting operation. I would expect something like $MI + (1-M)I$ if $M$ were a Mask. Could you provide some justification/insight into this operation, such as motivation/related work? \n - \"Attention\" usually refers to a specific set of operations. To make this block resemble attention, you could apply two SiCMs to $P^c$ (to make a $K$ and $V$), then one SiCM to $E^c$ (to get $Q$), and then create an attention map by applying cross attention with $Q$, $K$ and $V$. Of course, this would be quadratic in $T_x$, so this may be computationally prehibative. Would it be possible to explore alternative operations? \n\nExperiments:\n - Would it be possible to add Si-SNR(i) and SDR(i) metrics to the results?"},"rating":{"value":3},"details_of_ethics_concerns":{"value":"- Figure 2 looks very similar to RTFS-Net (https://arxiv.org/abs/2309.17189). There is some overlap between the methods introduced in this paper, and the contributions detailed in RTFS-Net and TF-GridNet (https://arxiv.org/abs/2209.03952). While the finer details differ, the overall approach is very similar. Please add citations. \n - CSBlock: This method of switching the dimensions was first introduced by DPRNN in 2019. Other's work should be properly sited and referenced. This method has been used every single year since 2019, such as by DPTNet in 2020, TF-Gridnet in 2022, and RTFS-Net in 2023."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The introduction and literature view show a strong and detailed narrative of their task, their goals and their contributions. It is well researched and has a comprehensive view of current methods, contextualising their methodology and results. Their methods section is long and detailed, with many interesting approaches to the problems that they face. They combine a series of current techniques, such as Mamba and individually processing the time and frequency dimensions, and then applying a global operation down both dimensions. They use a discriminator to add another loss signal to the training process, and in their experiments they cover a range of datasets and evaluation metrics, strengthening their claims. This show good scientific rigour."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The weaknesses mainly come from technical writing problems and lack of clarity. \n\n - SiCM is defined, but it is not clear how it is used. This section appears to define the equations twice, which is unnecessary. Its inputs are not defined - what is $Q_{in}$ and how does it connect with the rest of the model? Use of mamba is interesting, bidirectional seems unusual - and they do not have experiments to back up this choice.\n - CM_2: \n\t- Line 278 - should the notation be $B F_x \\times T_x \\times C$ instead?\n\t- Line 283 \"diverging\" is repeated.\n\t- Line 301 the $F_e$ and $F_p$ are not clearly defined (1d convolutions?). \n\t- On equastions 6 and 7 add some spaces after the \"...\" and after the \",\" otherwise its too hard to read\n\t- Line 302: it should be \"contains\" not \"contain\"\n\t- 3.5 the use of the SiCM notation is a little confusing. Earlier, it refers to a specific set of Mamba-based operations, but here it is for the entire frequency processing and time processing modules. Please keep the nomenclature consistent. \n - The Discriminator is mentioned but undefined. Papers need to be reproducible. \n - The methods they compared to in the results tables are quite old. Would be interesting to compare with modern AVSS models by setting speaker 2 = noise.\n\nFinally, while this is not a strict requirement, open sourcing the code after the paper's release would help understand the methods better, as the methods are quite convoluted."}},"nonreaders":[],"tmdate":1732615206667,"tcdate":1730559188545,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4079/Reviewer_6pDV"],"signatures":["ICLR.cc/2025/Conference/Submission4079/Reviewer_6pDV"],"forum":"EO2hZTtK3M","number":3,"license":"CC BY 4.0","cdate":1730559188545,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4079/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732615206667,"domain":"ICLR.cc/2025/Conference","replyto":"EO2hZTtK3M","id":"4W7phOD2Fl","forumContent":{"TLDR":{"value":"This paper introduces a novel framework, Cross-Modal Contextual Modeling (CM$^2$), which integrates semantic and signal-level contextual information across audio and visual modalities to enhance speech quality in noisy environments."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Speech Enhancement","Audio-Visual","Contextual Modeling"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Audio-Visual Speech Enhancement (AVSE) aims to improve speech quality in noisy environments by utilizing synchronized audio and visual cues.\nIn real-world scenarios, noise is often non-stationary, interfering with speech signals at varying intensities over time.\nDespite these fluctuations, humans can discern and understand masked spoken words as if they were clear.\nThis capability stems from the auditory system's ability to perceptually reconstruct interrupted speech using visual cues and semantic context in noisy environments, a process known as phonemic restoration.\nInspired by this phenomenon, we propose Cross-Modal Contextual Modeling (CM$^2$), integrating contextual information across different modalities and levels to enhance speech quality. \nSpecifically, we target two types of contextual information: semantic-level context and signal-level context.\nSemantic-level context enables the model to infer missing or corrupted content by leveraging semantic consistency across segments.\nSignal-level context further explores coherence within the signals developed from the semantic consistency.\nAdditionally, we particularly highlight the role of visual appearance in modeling the frequency-domain characteristics of speech, aiming to further refine and enrich the expression of these contexts.\nGuided by this understanding, we introduce a Semantic Context Module (SeCM) at the very beginning of our framework to capture the initial semantic contextual information from both audio and visual modalities.\nNext, we propose a Signal Context Module (SiCM) to obtain signal-level contextual information from both  raw noisy audio signal and the previously acquired audio-visual semantic-level context.\nBuilding on this rich contextual information, we finally introduce a Cross-Context Fusion Module (CCFM) to facilitate fine-grained context fusion across different modalities and types of contexts for further speech enhancement process.\nComprehensive evaluations across various datasets demonstrate that our method significantly outperforms current state-of-the-art approaches, particularly in low signal-to-noise ratio (SNR) environments."},"_bibtex":{"value":"@misc{\nwang2025cm,\ntitle={{CM}{\\textasciicircum}2: Cross-Modal Contextual Modeling for Audio-Visual Speech Enhancement},\nauthor={Feixiang Wang and Shuang Yang and Shiguang Shan and Xilin Chen},\nyear={2025},\nurl={https://openreview.net/forum?id=EO2hZTtK3M}\n}"},"title":{"value":"CM^2: Cross-Modal Contextual Modeling for Audio-Visual Speech Enhancement"},"pdf":{"value":"/pdf/160de648b89dbb3edca0448fa250464938595072.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"wang|cm^2_crossmodal_contextual_modeling_for_audiovisual_speech_enhancement"},"authorids":{"value":["~Feixiang_Wang1","~Shuang_Yang4","~Shiguang_Shan2","~Xilin_Chen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Feixiang Wang","Shuang Yang","Shiguang Shan","Xilin Chen"]}},"version":2},{"content":{"venue":{"value":"CVPR 2025"},"pdf":{"value":"https://openaccess.thecvf.com/content/CVPR2025/papers/Sun_Exploring_Historical_Information_for_RGBE_Visual_Tracking_with_Mamba_CVPR_2025_paper.pdf"},"venueid":{"value":"dblp.org/conf/CVPR/2025"},"paperhash":{"value":"sun|exploring_historical_information_for_rgbe_visual_tracking_with_mamba"},"authorids":{"value":["~Chuanyu_Sun1","https://dblp.org/search/pid/api?q=author:Jiqing_Zhang:","https://dblp.org/search/pid/api?q=author:Yang_Wang:","https://dblp.org/search/pid/api?q=author:Huilin_Ge:","https://dblp.org/search/pid/api?q=author:Qianchen_Xia:","https://dblp.org/search/pid/api?q=author:Baocai_Yin:","https://dblp.org/search/pid/api?q=author:Xin_Yang:"]},"html":{"value":"https://openaccess.thecvf.com/content/CVPR2025/html/Sun_Exploring_Historical_Information_for_RGBE_Visual_Tracking_with_Mamba_CVPR_2025_paper.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/cvpr/SunZWGXYY25,\n  author={Chuanyu Sun and Jiqing Zhang and Yang Wang and Huilin Ge and Qianchen Xia and Baocai Yin and Xin Yang},\n  title={Exploring Historical Information for RGBE Visual Tracking with Mamba},\n  year={2025},\n  cdate={1735689600000},\n  pages={6500-6509},\n  url={https://openaccess.thecvf.com/content/CVPR2025/html/Sun_Exploring_Historical_Information_for_RGBE_Visual_Tracking_with_Mamba_CVPR_2025_paper.html},\n  booktitle={CVPR},\n  crossref={conf/cvpr/2025}\n}\n"},"abstract":{"value":"Combining the advantages of conventional and event cameras for robust visual tracing has drawn extensive interest. However, existing tracking approaches heavily engage in complex cross-modal fusion modules, leading to higher computational complexity and training challenges. Besides, these methods generally ignore the effective integration of historical information, which is crucial to grasping the change in the target's appearance and motion trends. Given the recent advancements in Mamba's long-range modeling and linear complexity, we explore its potential in addressing the above issues in RGBE tracking tasks. Specifically, we first propose an efficient fusion module based on Mamba, which utilizes a simple gate-based interaction scheme to achieve effective modality-selective fusion. This module can be seamlessly integrated into the encoding layer of prevalent Transformer-based backbones. Moreover, we further present a novel historical decoder that leverages Mamba's advanced long sequence modeling to effectively capture the target appearance changes with autoregressive queries. Extensive experiments show that our proposed approach achieves state-of-the-art performance on multiple challenging short-term and long-term RGBE benchmarks. Besides, the effectiveness of each key Mamba-based component of our approach is evidenced by our thorough ablation study."},"title":{"value":"Exploring Historical Information for RGBE Visual Tracking with Mamba"},"authors":{"value":["Chuanyu Sun","Jiqing Zhang","Yang Wang","Huilin Ge","Qianchen Xia","Baocai Yin","Xin Yang"]}},"tmdate":1769485526396,"pdate":1767139200000,"externalIds":["dblp:conf/cvpr/SunZWGXYY25"],"tcdate":1769485525001,"writers":["~"],"signatures":["~Chuanyu_Sun1"],"forum":"E58AIYjzYV","license":"CC BY-SA 4.0","number":806126,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769485526396,"domain":"DBLP.org","id":"E58AIYjzYV","version":2},{"content":{"TLDR":{"value":"Hierarchical State Space models for real-world sensory data, and datasets for benchmarking models for continuous sequence prediction."},"venue":{"value":"MFM-EAI@ICML2024 Poster"},"pdf":{"value":"/pdf/7c01675bb0921b7765344ac9193136ab3692f1ef.pdf"},"keywords":{"value":["sequence-to-sequence prediction","deep state space models","sensor datasets"]},"venueid":{"value":"ICML.cc/2024/Workshop/MFM-EAI"},"paperhash":{"value":"bhirangi|hierarchical_state_space_models_for_continuous_sequencetosequence_modeling"},"authorids":{"value":["~Raunaq_Bhirangi1","~Chenyu_Wang9","~Venkatesh_Pattabiraman1","~Carmel_Majidi1","~Abhinav_Gupta1","~Tess_Hellebrekers2","~Lerrel_Pinto1"]},"abstract":{"value":"Reasoning from sequences of raw sensory data is a ubiquitous problem across fields ranging from medical devices to robotics. These problems often involve using long sequences of raw sensor data (e.g. magnetometers, piezoresistors) to predict sequences of desirable physical quantities (e.g. force, inertial measurements). While classical approaches are powerful for locally-linear prediction problems, they often fall short when using real-world sensors. These sensors are typically non-linear, are affected by extraneous variables (e.g. vibration), and exhibit data-dependent drift. For many problems, the prediction task is exacerbated by small labeled datasets since obtaining ground-truth labels requires expensive equipment. In this work, we present Hierarchical State-Space models (HiSS), a conceptually simple, new technique for continuous sequential prediction. HiSS stacks structured state-space models on top of each other to create a temporal hierarchy. Across six real-world sensor datasets, from tactile-based state prediction to accelerometer-based inertial measurement, HiSS outperforms state-of-the-art sequence models such as causal Transformers, LSTMs, S4, and Mamba by at least 23% on MSE. Our experiments further indicate that HiSS demonstrates efficient scaling to smaller datasets and is compatible with existing data-filtering techniques. Code, datasets and videos can be found on https://hiss-csp.github.io."},"_bibtex":{"value":"@inproceedings{\nbhirangi2024hierarchical,\ntitle={Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling},\nauthor={Raunaq Bhirangi and Chenyu Wang and Venkatesh Pattabiraman and Carmel Majidi and Abhinav Gupta and Tess Hellebrekers and Lerrel Pinto},\nbooktitle={Multi-modal Foundation Model meets Embodied AI Workshop @ ICML2024},\nyear={2024},\nurl={https://openreview.net/forum?id=J91zuYxiVO}\n}"},"title":{"value":"Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling"},"authors":{"value":["Raunaq Bhirangi","Chenyu Wang","Venkatesh Pattabiraman","Carmel Majidi","Abhinav Gupta","Tess Hellebrekers","Lerrel Pinto"]}},"tmdate":1725543006833,"pdate":1718728933588,"tcdate":1717108059738,"writers":["ICML.cc/2024/Workshop/MFM-EAI","ICML.cc/2024/Workshop/MFM-EAI/Submission21/Authors"],"signatures":["ICML.cc/2024/Workshop/MFM-EAI/Submission21/Authors"],"forum":"J91zuYxiVO","license":"CC BY 4.0","number":21,"cdate":1717108059738,"readers":["everyone"],"invitations":["ICML.cc/2024/Workshop/MFM-EAI/-/Submission","ICML.cc/2024/Workshop/MFM-EAI/-/Post_Submission","ICML.cc/2024/Workshop/MFM-EAI/-/Edit","ICML.cc/2024/Workshop/MFM-EAI/Submission21/-/Revision"],"mdate":1725543006833,"odate":1725543006821,"domain":"ICML.cc/2024/Workshop/MFM-EAI","id":"J91zuYxiVO","version":2},{"content":{"venue":{"value":"ICML 2024 Poster"},"abstract":{"value":"Reasoning from sequences of raw sensory data is a ubiquitous problem across fields ranging from medical devices to robotics. These problems often involve using long sequences of raw sensor data (e.g. magnetometers, piezoresistors) to predict sequences of desirable physical quantities (e.g. force, inertial measurements). While classical approaches are powerful for locally-linear prediction problems, they often fall short when using real-world sensors. These sensors are typically non-linear, are affected by extraneous variables (e.g. vibration), and exhibit data-dependent drift. For many problems, the prediction task is exacerbated by small labeled datasets since obtaining ground-truth labels requires expensive equipment. In this work, we present Hierarchical State-Space models (HiSS), a conceptually simple, new technique for continuous sequential prediction. HiSS stacks structured state-space models on top of each other to create a temporal hierarchy. Across six real-world sensor datasets, from tactile-based state prediction to accelerometer-based inertial measurement, HiSS outperforms state-of-the-art sequence models such as causal Transformers, LSTMs, S4, and Mamba by at least 23% on MSE. Our experiments further indicate that HiSS demonstrates efficient scaling to smaller datasets and is compatible with existing data-filtering techniques. Code, datasets and videos can be found on https://hiss-csp.github.io."},"_bibtex":{"value":"@inproceedings{\nbhirangi2024hierarchical,\ntitle={Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling},\nauthor={Raunaq Bhirangi and Chenyu Wang and Venkatesh Pattabiraman and Carmel Majidi and Abhinav Gupta and Tess Hellebrekers and Lerrel Pinto},\nbooktitle={Forty-first International Conference on Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=TK7xkOsXDu}\n}"},"title":{"value":"Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling"},"pdf":{"value":"/pdf/dc3783db035c7ac10d98199bc33f47c536bd3bfa.pdf"},"venueid":{"value":"ICML.cc/2024/Conference"},"paperhash":{"value":"bhirangi|hierarchical_state_space_models_for_continuous_sequencetosequence_modeling"},"authorids":{"value":["~Raunaq_Bhirangi1","cw4030@nyu.edu","~Venkatesh_Pattabiraman1","~Carmel_Majidi1","~Abhinav_Gupta1","~Tess_Hellebrekers2","~Lerrel_Pinto1"]},"authors":{"value":["Raunaq Bhirangi","Chenyu Wang","Venkatesh Pattabiraman","Carmel Majidi","Abhinav Gupta","Tess Hellebrekers","Lerrel Pinto"]}},"tmdate":1719287223032,"pdate":1714610277475,"tcdate":1705680021238,"writers":["ICML.cc/2024/Conference","ICML.cc/2024/Conference/Submission936/Authors"],"signatures":["ICML.cc/2024/Conference/Submission936/Authors"],"forum":"TK7xkOsXDu","license":"CC BY 4.0","number":936,"cdate":1705680021238,"readers":["everyone"],"invitations":["ICML.cc/2024/Conference/-/Submission","ICML.cc/2024/Conference/-/Post_Submission","ICML.cc/2024/Conference/-/Edit","ICML.cc/2024/Conference/Submission936/-/Camera_Ready_Revision"],"mdate":1719287223032,"odate":1717693030774,"domain":"ICML.cc/2024/Conference","id":"TK7xkOsXDu","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.15360v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"patro|simba_simplified_mambabased_architecture_for_vision_and_multivariate_time_series"},"authorids":{"value":["~Badri_N._Patro1","https://dblp.org/search/pid/api?q=author:Vijay_Srinivas_Agneeswaran:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.15360"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-15360,\n  publtype={informal},\n  author={Badri N. Patro and Vijay Srinivas Agneeswaran},\n  title={SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.15360},\n  url={https://doi.org/10.48550/arXiv.2403.15360}\n}\n"},"abstract":{"value":"Transformers have widely adopted attention networks for sequence mixing and MLPs for channel mixing, playing a pivotal role in achieving breakthroughs across domains. However, recent literature highlights issues with attention networks, including low inductive bias and quadratic complexity concerning input sequence length. State Space Models (SSMs) like S4 and others (Hippo, Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address the above issues to help handle longer sequence lengths. Mamba, while being the state-of-the-art SSM, has a stability issue when scaled to large networks for computer vision datasets. We propose SiMBA, a new architecture that introduces Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations and uses the Mamba block for sequence modeling. Extensive performance studies across image and time-series benchmarks demonstrate that SiMBA outperforms existing SSMs, bridging the performance gap with state-of-the-art transformers. Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet and transfer learning benchmarks such as Stanford Car and Flower as well as task learning benchmarks as well as seven time series benchmark datasets. The project page is available on this website ~\\url{https://github.com/badripatro/Simba}."},"title":{"value":"SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series"},"authors":{"value":["Badri N. Patro","Vijay Srinivas Agneeswaran"]}},"tmdate":1716535381805,"pdate":1704067200000,"tcdate":1716535373292,"writers":["~"],"signatures":["~Badri_Narayana_Patro1"],"forum":"zPlbRoasWb","license":"CC BY-SA 4.0","number":21239,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1716535381805,"domain":"DBLP.org","id":"zPlbRoasWb","version":2},{"content":{"summary":{"value":"This paper studies the performance of language model architectures with constant state size (*e.g. Mamba).* It is well-known that performance of these architectures degrades on long sequence lengths. This paper focuses on this phenomenon, which the authors call “state collapse”. The paper makes three main claims: \n\n1. That state collapse is due to the state size being *too big* during training, causing the model to \n2. The problem can be mitigated post-training by (a) intervening on the strength of memory decay, (b) using state normalization, and (c) and converting the model to a sliding window model. \n3. The problem can be mitigated during training by simply training on longer sequences."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- *Why are analyses performed at the 38th layer of the Mamba model?*\n- *Can Figure 9 use a log scale on the y-axis?* It is hard to discern differences in perplexity.\n- *What is the color scale in Figure 11 and which mitigation method was used for this experiment?*"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"**Significance.** The paper addresses a very important problem. The perplexity of a language model typically decreases as more context is given to the model (*i.e.* sequence length grows). However, this perplexity vs. token position plot usually flattens out or explodes for token positions beyond the sequence length on which the model was trained. A model with fixed recurrent state size for which perplexity *continues* to decrease indefinitely would represent a very significant advance. \n\n- Note: This paper makes some steps towards realizing this objective, but as I mention in the weaknesses, lacks evidence that the problem is really solved.\n\n**Interesting hypothesis for state collapse (Claim 1).** The authors argue that *“SC arises from state overparameterization relative to the training length. In other words, the state capacity is excessively large for the training length, allowing the model to achieve strong language modeling performance without learning how to forget when the state is about to overflow.”* This hypothesis for the cause of state collapse is interesting and, to my knowledge, original.\n\n**Simple methods.** The mitigation methods proposed are simple and intuitive."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Evidence for state over-parameterization (Claim 1) is unclear and limited.** \n\n- *“Figure 7 shows the memory strength of the first token at different time steps, and we find that the exploded heads (heads 2, 4, and 7 in the 38th layer) have a strong inclination toward retaining all information within the training length, with a memory strength of over 0.8 at t=8K.”* It’s unclear how the layer/heads were chosen. Furthermore, the causal link between high memory strength in this one layer and state collapse is not evident.\n- *“We also notice that Bt explodes earlier than ∆t. Therefore, we conclude that the collapse is largely attributable to Bt.”*  Looking at Figure 6 it’s not clear that $B_t$ “explodes” before $\\Delta_t$. It also isn’t obvious why this would mean that the collapse is “largely attributable to Bt”.\n- *“Finally, as we will show in Section 4.3.2, for any given training length, there exists a state size where SC will be exhibited if and only if the model’s state size is greater.”* Section 4.3.2 does not seem to address this point.\n- *“It shows that SC is only exhibited by checkpoints beyond a certain amount of training, which coincides with behaviors of overfitting—a result of overparameterization.”* I found this argument confusing. Overfitting seems unrelated to the issue at hand. Training data in the language modeling setting is abundant, so it doesn’t seem like overfitting is part of the problem?\n\n**Limitations with training-free mitigation methods (Claim 2).** There are significant limitations with the three training-free mitigation methods, which I am concerned may limit their utility in practice. \n\n- **Methods 1 and 3 induce forgetting.** These methods work by explicitly inducing forgetting. This is concerning given that we don’t want the model to completely forget important earlier context (*e.g.* in passkey retrieval). **\n- **Perplexity still rises on longer sequences with methods 1 and 3.** In Figure 9, it looks like theperplexity at high token positions (30k - 60k) is significantly higher than perplexity at earlier positions (10k). Ideally, the model’s perplexity would continue to decrease with longer sequences (or at least stay constant).\n- **Method 2** **makes training/prefill non-parallelizable.** The authors did a good job of highlighting this limitation, but am raising it again here to emphasize how this would seriously complicate model training and serving.\n- **Method 2** **degrades performance substantially.** Figure 9 shows that Method 2 leads to a significant increase in perplexity at token positions within the training length. This suggests that the model may need to adapt to state normalization *during training,* which is a concern given the non-parallelizability of the architecture.\n\n**Evaluation on a limited set of tasks (Claims 2 + 3).** The experiments supporting *Claim 2* only measures perplexity on RedPajama. Given the comment above (that training-free mitigation methods cause forgetting) the methods should be evaluated on passkey retrieval, at a minimum. Ideally, the methods would also be evaluated on long context evaluations like ∞-BENCH or Loogle [[Zhang 2024](https://aclanthology.org/2024.acl-long.814.pdf), [Li 2024](https://arxiv.org/pdf/2311.04939)]. The experiments supporting *Claim 3* only measure perplexity on the synthetic *newlines* prompt*.*The lack of experiments on real data and hard long context tasks is concerning."}},"nonreaders":[],"tmdate":1731428501205,"tcdate":1730681341324,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12121/Reviewer_gce7"],"signatures":["ICLR.cc/2025/Conference/Submission12121/Reviewer_gce7"],"forum":"cu2CT2VAvs","number":4,"license":"CC BY 4.0","cdate":1730681341324,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12121/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428501205,"domain":"ICLR.cc/2025/Conference","replyto":"cu2CT2VAvs","id":"Qyv9R1G1VS","forumContent":{"TLDR":{"value":"Systematic study of linear diagonal RNNs' length generalization failure and memory capacity in language modeling and contextual information retrieval."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["RNN","foundation models","long-context"]},"supplementary_material":{"value":"/attachment/d71dbd3e7ce14131c3720da88c049fa27def3e94.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"One essential advantage of recurrent neural networks (RNNs) over transformer-based language models is their linear computational complexity concerning the sequence length, which makes them much faster in handling long sequences during inference. However, most publicly available RNNs (e.g., Mamba and RWKV) are trained on sequences with less than 10K tokens, and their effectiveness in longer contexts remains largely unsatisfying so far. In this paper, we study the cause of the inability to process long context for RNNs and suggest critical mitigations. First, we investigate *state explosion* (SE) in Mamba-2 when processing long sequences, a phenomenon where some channels of the state exhibit exploding values that cause severe performance degradation. With controlled experiments, we discover that the model fails to forget the earlier tokens when there is more information than it can remember. We attribute this to overfitting due to the recurrent state being overparameterized for the training length, thereby establishing a relationship between SE and the capacity of the state. To support this hypothesis, we make an important empirical observation: for any given state size, there exists a training length threshold such that SE is exhibited if and only if the training length is greater than this threshold. Empirically searching for this threshold for different state sizes reveals that it is a linear function of the state size. We also search for the maximum context length at which the model can recall contextual information and find that this context length scales exponentially to the state size. Based on this, we empirically train a Mamba-2 370M with near-perfect passkey retrieval accuracy on 256K context length. This suggests a promising future for RNN-based long-context modeling. Code and model checkpoints will be publicly released."},"_bibtex":{"value":"@misc{\nchen2025stuffed,\ntitle={Stuffed Mamba: State Collapse and State Capacity of {RNN}-Based Long-Context Modeling},\nauthor={Yingfa Chen and Xinrong Zhang and Shengding Hu and Xu Han and Zhiyuan Liu and Maosong Sun},\nyear={2025},\nurl={https://openreview.net/forum?id=cu2CT2VAvs}\n}"},"title":{"value":"Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling"},"pdf":{"value":"/pdf/81b99123fa4ffe7fe2c3aa7b6d9993ea1a3889f6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chen|stuffed_mamba_state_collapse_and_state_capacity_of_rnnbased_longcontext_modeling"},"authorids":{"value":["~Yingfa_Chen1","~Xinrong_Zhang1","~Shengding_Hu2","~Xu_Han2","~Zhiyuan_Liu1","~Maosong_Sun1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yingfa Chen","Xinrong Zhang","Shengding Hu","Xu Han","Zhiyuan Liu","Maosong Sun"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Sparse Mamba (S-Mamba), a modification of the Mamba architecture that incorporates controllability and observability structures to the pre input-varying discretization of the SSM. The authors propose two variants: Sparse Controllable Mamba (SC-Mamba) and Sparse Observable Mamba (SO-Mamba), which modify the structure of the state space matrices $A$, $B$, $C$ of the underlying continuous-time system to enforce specific canonical forms. The authors claim improvements in perplexity (5%), training time (3%), and parameter count compared to vanilla Mamba, demonstrating these results across four datasets: CodeParrot, OpenWebText, ArXiv, and Cosmopedia."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. Could you elaborate on why controllability and observability are beneficial properties for a linear SSM layer with input-varying dynamics in a neural network? Specifically, how do could these properties improve the model's ability to learn and generalize?\n\n2. Given that the scan operation becomes more computationally expensive with non-diagonal matrices, how does your implementation handle this trade-off? Could you provide complexity analysis comparing your approach to vanilla Mamba/Mamba2?\n\n3. The parameter reduction achieved is relatively small compared to the total parameter count. Could you explain why this reduction is meaningful and how it affects the model's practical performance?\n\n4. Could you provide statistical significance tests for the reported improvements in perplexity and training time? How consistent are these improvements across different random seeds and training runs?\n\n5. How does your approach ensure stability of the learned SSM, and what impact does the canonical form constraint have on the model's expressivity?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":1},"strengths":{"value":"The paper makes an interesting connection between classical control theory concepts and modern state space models for machine learning, attempting to bring established theoretical frameworks to bear on neural architecture design.\n\n* The experimental evaluation is conducted across multiple datasets of varying sizes and domains, providing some evidence for the generalizability of the approach.\n* The reduction in parameter count while maintaining or improving performance is a potentially valuable contribution, as it could lead to more efficient models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Technical Rigor and Motivation:\n- The paper fails to justify why controllability and observability are desirable properties for an SSM layer in a neural network context. The authors state these properties make analysis easier, but don't explain why such analysis is necessary or beneficial for the learning task.\n- The claim of \"less complexity\" is repeated without proper theoretical or empirical justification.\n- There is no discussion of how BIBO (*bounded-input, bounded-output*) stability is maintained in the proposed architecture. This is notoriously necessary to be enforced.\n\n2. Novelty and Literature Review:\n- The paper does not acknowledge or cite previous work on using companion canonical forms in SSMs [1, 2], making it difficult to assess the novelty of the contribution.\n- The motivation for making the system \"sparse\" is not well-explained, particularly given that the reduction in parameters (∼100K out of 64M) is relatively minor.\n\n3. Methodology and Results:\n- The paper doesn't discuss the computational implications of using non-diagonal state matrices with scan operations, which could significantly impact practical performance.\n- There is no analysis of the statistical significance of the reported improvements in perplexity and training time.\n- The experimental section lacks details about the inference procedure and comparison of different inference algorithms.\n\n4. Presentation:\n- The paper contains numerous issues with notation consistency (e.g., use of bold letters).\n- The writing quality needs significant improvement, with unclear explanations and imprecise language throughout.\n- Referenced materials are not properly used (e.g., where does the Krylov function appears in Krylov 1931?).\n\n[1] Zhang, Michael, et al. \"Effectively modeling time series with simple discrete state spaces.\" (2023).\n\n[2] Parnichkun, Rom N., et al. \"State-Free Inference of State-Space Models: The Transfer Function Approach.\" (2024)."}},"nonreaders":[],"tmdate":1731427831180,"tcdate":1730677232951,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12587/Reviewer_4DXp"],"signatures":["ICLR.cc/2025/Conference/Submission12587/Reviewer_4DXp"],"forum":"vOfDGYGVyj","number":4,"license":"CC BY 4.0","cdate":1730677232951,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12587/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427831180,"domain":"ICLR.cc/2025/Conference","replyto":"vOfDGYGVyj","id":"r8VeXfBnRc","forumContent":{"TLDR":{"value":"Sparse Mamba"},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","state space models","natural language processing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"In this work, we introduce the concept of controllability and observability to the Mamba SSM's architecture in our Sparse-Mamba (S-Mamba) for natural language processing (NLP) applications. The structured state space model (SSM) development in recent studies, such as Mamba and Mamba2, outperformed and solved the computational inefficiency of transformers and large language models at small to medium scale. The Mamba SSMs architecture drops the need for attention layers or multilayer perception blocks in transformers. However, current Mamba models lack reinforcement of controllability in state-space equations for computing the $A$, $B$, $C$, and $D$ matrices at each time step, leading to increased complexity and computational costs. In this paper, we demonstrate a reduction of parameters in comparison to the first published Mamba and Mamba2. We showcase an improvement in perplexity by 5\\% and a decrease in training time by 3\\% after reinforcing controllability and observability on the original Mamba architecture in our proposed S-Mamba. The controllable $n \\times n$ state  matrix $A$ is sparse and it has only $n$ free parameters. Our novel approach will ensure a controllable system which will be the gate key for Mamba3."},"_bibtex":{"value":"@misc{\nhamdan2024sparse,\ntitle={Sparse Mamba: Reinforcing Controllability In Structural State Space Models},\nauthor={Emadeldeen Hamdan and Hongyi Pan and Ahmet Cetin},\nyear={2024},\nurl={https://openreview.net/forum?id=vOfDGYGVyj}\n}"},"title":{"value":"Sparse Mamba: Reinforcing Controllability In Structural State Space Models"},"pdf":{"value":"/pdf/56eae743dc6d789a0b97e323821a349c3bf415ff.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"hamdan|sparse_mamba_reinforcing_controllability_in_structural_state_space_models"},"authorids":{"value":["~Emadeldeen_Hamdan1","~Hongyi_Pan1","~Ahmet_Cetin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Emadeldeen Hamdan","Hongyi Pan","Ahmet Cetin"]}},"version":2},{"content":{"summary":{"value":"The authors introduce a new RNN layer they call Test-Time Training (TTT) layer. The hidden state in this layer is a machine learning model itself, in practice a linear layer or an MLP. The update for this layer is a single step of \"self-supervised\" learning using SGD. The authors show how to efficiently implement their layer on the GPU, or similar hardware. Finally a number of different models using TTT layers are empirically evaluate it terms of perplexity, training FLOPs and inference time on natural language tasks."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Did you experiment with other inner loop losses? What is the intuition behind the mutli-view loss, was the loss selected to produce Theorem 1?\n\nIt does not seem obvious why you'd want to run and iterative optimisation algorithm on the inner loop objective that has a closed form solution, can you give more intuition here? \n\nDid you have any strong justification to think it convolution through time was responsible for the gains when using the Mamba backbone?\n\nI'm not super familiar with the hardware side of LLMs. In section 3.3 you mention wrote two kernels for the TTTlayers. Were these necessary to i) get a fair comparison with the other models? or ii) necessary to make the TTT layers cooperative with other baselines? Without how did the wall clock time compare without these kernels? Can you explain why were these kernel necessary?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The paper is well written and easy to follow.\n\nIt presents a novel idea which it thoroughly investigates at a number of different scales.\n\nSome of the new models using TTT layers seem to present gains over Mamba and Transformers in terms of training efficiency, and inference latency thought the significance of these gains is questionable."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The motivation for the self-supervised loss used is a little weak. The self supervised loss described seem to be learning a mapping from one linear projection of a vector to another. I believe this mapping would have a closed from solution as a function of the projections, at least for the linear case. I would like to see the author expand their justification here and if relevant give examples of other inner loop losses experimented with. Self supervised losses for sequence data conventionally try to predict the next item in a sequence from the preceding elements. Did you try this? If so why not?\n\nThe efficiency gains are not really put into a human understandable form. Additionally the significance of the efficiency gains seem to be very marginal. While I appreciate multiple runs is likely out of scope for experiments of this size its hard to know if the increases are statistically significant.\n\nIt would be great to see if it was indeed the convolution through time that was responsible for the gains when using the Mamba backbone, or if it was some other aspect of the architecture.\n\nThe conclusion section is very weak."}},"nonreaders":[],"tmdate":1731428370723,"tcdate":1729949835080,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5310/Reviewer_ZsP1"],"signatures":["ICLR.cc/2025/Conference/Submission5310/Reviewer_ZsP1"],"forum":"eifW0W0xgt","number":1,"license":"CC BY 4.0","cdate":1729949835080,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5310/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428370723,"domain":"ICLR.cc/2025/Conference","replyto":"eifW0W0xgt","id":"sXxGPGPZuo","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["test-time training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state. Inspired by prior work, we present a practical framework to instantiate sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Both TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer, they can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. With preliminary systems optimization, TTT-Linear is already faster than Transformer at 8k context and matches Mamba in wall-clock time. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research."},"_bibtex":{"value":"@misc{\nsun2025learning,\ntitle={Learning to (Learn at Test Time): {RNN}s with Expressive Hidden States},\nauthor={Yu Sun and Xinhao Li and Karan Dalal and Jiarui Xu and Arjun Vikram and Genghan Zhang and Yann Dubois and Xinlei Chen and Xiaolong Wang and Sanmi Koyejo and Tatsunori Hashimoto and Carlos Guestrin},\nyear={2025},\nurl={https://openreview.net/forum?id=eifW0W0xgt}\n}"},"title":{"value":"Learning to (Learn at Test Time): RNNs with Expressive Hidden States"},"pdf":{"value":"/pdf/be1b3f1cf37ec5812fb041e8bbdd1569d92a9992.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sun|learning_to_learn_at_test_time_rnns_with_expressive_hidden_states"},"authorids":{"value":["~Yu_Sun1","~Xinhao_Li3","~Karan_Dalal1","~Jiarui_Xu1","~Arjun_Vikram1","~Genghan_Zhang1","~Yann_Dubois1","~Xinlei_Chen1","~Xiaolong_Wang3","~Sanmi_Koyejo1","~Tatsunori_Hashimoto1","~Carlos_Guestrin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yu Sun","Xinhao Li","Karan Dalal","Jiarui Xu","Arjun Vikram","Genghan Zhang","Yann Dubois","Xinlei Chen","Xiaolong Wang","Sanmi Koyejo","Tatsunori Hashimoto","Carlos Guestrin"]}},"version":2},{"content":{"summary":{"value":"This paper presents the Multi-Stage State Space Model (MS$^3$M), a new architecture that leverages recent advancements in state space models to improve motion forecasting in autonomous driving systems by effectively modeling spatial-temporal information. This approach aims to address the inefficiencies of traditional transformer models that suffer from quadratic complexity as the number of agents increases. The paper introduces the Temporal Mamba Model and Spatial Mamba Model to handle temporal information and spatial interactions, respectively. Additionally, a hierarchical query-based decoder further enhances both performance and efficiency by structuring scene processing. This paper includes experiments and ablation studies on the Argoverse 2 dataset to demonstrate the effectiveness of MS3M and the trade-off it achieves between latency and performance."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Paper [1] also evaluates the latency of QCNet on the Argoverse dataset (Table 2&8), and the latency is much higher. Could this be due to different settings? \n\n[1] Zhou, Yang, et al. \"SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The paper provides a well-articulated introduction to the methods and thorough explanations of the experiments. It introduces the Mamba architecture to handle long sequences and complex spatial-temporal interactions in motion prediction, which represents an interesting and promising direction especially for lighter model with lower latency solution in autonomous driving."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. It would be beneficial to show the model's performance on additional benchmarks, such as WOMD/NuScenes/Argoverse 1 to strengthen the conclusion that this approach achieves a good trade-off between efficiency and accuracy, given that the best performance of MS$^3$M is still behind the baseline model.\n2. A recent paper [1] also demonstrates significant benefits from the Mamba architecture, and the paper [2] that utilizes LM also shows low latency with reasonable accuracy. It would be interesting to have a deeper analysis of the contributions to model size and latency among different architectures such as CNN/transformer/Mamba.\n3. The caption of Table 3 L493 could be improved to explain the ablation details, making it more readable.\n\n[1] Zhang, Bozhou, Nan Song, and Li Zhang. \"Decoupling Motion Forecasting into Directional Intentions and Dynamic States.\" arXiv preprint arXiv:2410.05982 (2024).\n\n[2] Seff, Ari, et al. \"Motionlm: Multi-agent motion forecasting as language modeling.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023"}},"nonreaders":[],"tmdate":1732637088311,"tcdate":1730499288620,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3447/Reviewer_jjYs"],"signatures":["ICLR.cc/2025/Conference/Submission3447/Reviewer_jjYs"],"forum":"MmOQY71YHw","number":2,"license":"CC BY 4.0","cdate":1730499288620,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3447/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732637088311,"domain":"ICLR.cc/2025/Conference","replyto":"MmOQY71YHw","id":"BHczuckAZh","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"This paper proposes a Mamba-based motion forecasting architecture that achieves superior performance with significantly lower latency."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Motion Forecasting","Autonomous Driving","State Space Model"]},"supplementary_material":{"value":"/attachment/101a980240bdc46f184713f07f4571c1b29e062b.pdf"},"primary_area":{"value":"applications to robotics, autonomy, planning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Motion forecasting is a fundamental component of autonomous driving systems, as it predicts an agent's future trajectories based on its surrounding environment. Transformer architectures have dominated this domain due to their strong ability to model both temporal and spatial information. However, transformers often suffer from quadratic complexity with respect to input sequence length, limiting their ability to efficiently process scenarios involving numerous agents. Additionally, transformers typically rely on positional encodings to represent temporal or spatial relationships, a strategy that may not be as effective or intuitive as the inductive biases naturally embedded in convolutional architectures. To address these challenges, we leverage recent advancements in state space models (SSMs) and propose the Multi-Stage State Space Model (MS$^3$M). In MS$^3$M, the Temporal Mamba Model (TMM) is employed to capture fine-grained temporal information, while the Spatial Mamba Model efficiently handles spatial interactions. By injecting temporal and spatial inductive biases through Mamba’s state-space model structure, the model's capacity is significantly improved. MS$^3$M also strikes an exceptional trade-off between accuracy and efficiency, which is achieved through convolutional computations and near-linear computational strategies in the Mamba architecture. Furthermore, a hierarchical query-based decoder is introduced, further enhancing model performance and efficiency. Extensive experimental results demonstrate that the proposed method achieves superior performance while maintaining low latency, which is crucial for practical real-time autonomous driving systems."},"_bibtex":{"value":"@misc{\nliu2025msm,\ntitle={{MS}\\${\\textasciicircum}3\\$M: Multi-Stage State Space Model for Motion Forecasting},\nauthor={Chunyu Liu and Shijie Li and Xulei Yang and Jianjun Yu},\nyear={2025},\nurl={https://openreview.net/forum?id=MmOQY71YHw}\n}"},"title":{"value":"MS$^3$M: Multi-Stage State Space Model for Motion Forecasting"},"pdf":{"value":"/pdf/b5472cb1e15055084d263ef47ccb9412ed68e871.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"liu|ms^3m_multistage_state_space_model_for_motion_forecasting"},"authorids":{"value":["~Chunyu_Liu2","~Shijie_Li1","~Xulei_Yang2","~Jianjun_Yu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chunyu Liu","Shijie Li","Xulei Yang","Jianjun Yu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes the Free Energy Mixer (FEM), a novel inference-time mechanism that addresses a perceived limitation in standard attention: the inability to perform per-channel selection from the key-value cache. The authors argue that the standard convex combination of values within each attention head synchronizes channels, preventing them from independently selecting different past indices. FEM tackles this by reframing the read operation as a variational free-energy optimization. It uses a value-driven, per-channel \"tilt\" on a prior distribution (e.g., from Q/K scores) to produce a posterior, enabling a smooth interpolation from averaging to near-hard selection. The paper presents a two-level gated instantiation of FEM that is plug-and-play with various backbones (softmax/linear attention, RNNs, SSMs) and demonstrates consistent performance gains on NLP, vision, and time-series tasks under matched parameter budgets."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"For details, please see weakness."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"Motivation and Problem Identification: The paper compellingly argues that the standard per-head convex combination of values creates a \"lossy read\" bottleneck, preventing channel-wise selection. This is a well-articulated and non-trivial insight into a potential limitation of standard attention mechanisms.\n\nTheoretical and Empirical Rigor: The proposed FEM method is grounded in a solid theoretical framework derived from the Donsker-Varadhan variational principle. The paper is supported by thorough theoretical analysis and extensive experiments across diverse domains (synthetic tasks via MAD, language modeling, image classification, time-series forecasting) and model architectures (FEM-SM, FEM-GLA, etc.). \n\nPracticality: A key strength is the method's plug-and-play nature and its ability to preserve the asymptotic time complexity of the underlying prior mechanism (e.g., O(T²) for softmax, O(T) for linear attention), as demonstrated by the latency/throughput analysis for FEM-SM."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.  **Baseline Comparisons:** While the empirical results are comprehensive, the baselines could be strengthened by including more recent architectures that also introduce channel-wise inductive biases. A comparison against methods like Mamba (with its data-dependent SSM) or Multi-head Latent Attention (MLA), which inherently manipulate channel interactions, would provide a more rigorous assessment of FEM's unique contribution beyond simply adding channel-specific gating or modulation.\n2.  **Core Motivation and Justification:** The central premise—that enabling full per-channel selection is a critical goal—may be somewhat overstated.\n    *   It potentially creates a false dichotomy between the roles of the attention mechanism (for token mixing) and the subsequent Feed-Forward Network (FFN, for channel mixing). The universal approximation theorem suggests that a standard Transformer block, with MHA and FFN, is already capable of learning complex functions, and the necessity of extreme channel-wise selection within the attention block itself is not fully justified.\n    *   Many studies indicate that attention heads exhibit significant sparsity and redundancy. Therefore, the performance gains observed with FEM might not stem from achieving perfect channel-wise selection, but rather from introducing a more effective and general form of channel-wise *modulation* or *gating*. Reframing the contribution around enhancing channel *utilization efficiency* rather than enabling a theoretically maximal selection capacity might be a more compelling and accurate narrative.\n3.  **Efficiency Analysis:** The computational cost analysis, while provided for FEM-SM, is incomplete. The overhead of FEM when applied to its linear-time variants (e.g., FEM-GLA, FEM-Mamba) is not thoroughly quantified. In large-scale training and inference, even a modest constant-factor overhead for linear-time models can become a significant practical bottleneck. The performance benefits of FEM might diminish when scaled to very large models and datasets, while the computational cost remains.\n\n**Questions and Suggestions for Authors**\n\n*   Could you include comparisons with other modern sequence models that incorporate strong channel-wise interactions (e.g., Mamba, MLA) to better isolate the benefit of FEM's specific variational formulation?\n*   The paper would be strengthened by a more nuanced discussion on the interplay between attention and the FFN. Could the observed gains be interpreted as FEM providing a more powerful form of channel-wise conditioning *before* the FFN, rather than strictly enabling a previously impossible \"channel-wise selector\"?\n*   Please provide a detailed latency/throughput analysis for the linear-time FEM variants (FEM-GLA, FEM-Mamba) to give a complete picture of the method's computational trade-offs."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764364674659,"tcdate":1761807012452,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18823/Reviewer_bNwC"],"signatures":["ICLR.cc/2026/Conference/Submission18823/Reviewer_bNwC"],"forum":"vjQnKToCnV","number":2,"license":"CC BY 4.0","cdate":1761807012452,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission18823/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764364674659,"domain":"ICLR.cc/2026/Conference","replyto":"vjQnKToCnV","id":"yes1R18dOt","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Sequence Modeling","Attention","Transformer"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Standard attention stores keys/values losslessly but reads them via a per-head convex average, blocking channel-wise selection. We propose the Free Energy Mixer (FEM): a free-energy (log-sum-exp) read that applies a value-driven, per-channel log-linear tilt to a fast prior (e.g., from queries/keys in standard attention) over indices. Unlike methods that attempt to improve and enrich the $(q,k)$ scoring distribution, FEM treats it as a prior and yields a value-aware posterior read at unchanged complexity, smoothly moving from averaging to per-channel selection as the learnable inverse temperature increases, while still preserving parallelism and the original asymptotic complexity ($O(T^2)$ for softmax; $O(T)$ for linearizable variants). We instantiate a two-level gated FEM that is plug-and-play with standard and linear attention, linear RNNs and SSMs. It consistently outperforms strong baselines on NLP, vision, and time-series at matched parameter budgets."},"_bibtex":{"value":"@inproceedings{\nlu2026free,\ntitle={Free Energy Mixer},\nauthor={Jiecheng Lu and Shihao Yang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=vjQnKToCnV}\n}"},"title":{"value":"Free Energy Mixer"},"pdf":{"value":"/pdf/8b7d70089015a6d3314a177be2fcf7ce0c405975.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lu|free_energy_mixer"},"authorids":{"value":["~Jiecheng_Lu1","~Shihao_Yang1"]},"authors":{"value":["Jiecheng Lu","Shihao Yang"]}},"version":2},{"content":{"summary":{"value":"This paper presents Forge, a domain-specific compiler designed specifically for linear attention mechanisms. Forge abstracts the computation into three canonical stages based on chunk-wise parallelization: intra-chunk computation, inter-chunk state propagation, and output merging. This unified abstraction both simplifies the development of new linear attention variants (such as Mamba, RetNet, RWKV, GLA, HGRN, and GDN) and enables systematic optimizations for both single-device and distributed scenarios. Forge leverages Triton-Distributed as its backend to generate high-performance kernels that fuse computation and communication, and incorporates an adaptive scheduling system for parallelism and a static dispatcher to reduce runtime overhead. Experimental results demonstrate that Forge matches or outperforms state-of-the-art hand-written kernels (e.g., Flash Linear Attention) in both performance and scalability, while also requiring much less manual effort to implement new variants."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. According to the experimental results, Flash Linear Attention already supports multiple linear attention variants and delivers strong performance across them. While Forge’s three-phase abstraction enables users to easily implement different variants, Flash Linear Attention is also Triton-based, which inherently provides customization capabilities. How does Forge compare to Flash Linear Attention in terms of flexibility when supporting diverse linear attention variants?\n2. How does the Parallelism Scheduler mentioned in Appendix B adapt to distributed environments? Are there differences in scheduling strategies between single-GPU, intra-node, and inter-node scenarios?\n3. Are there any limitations or edge cases where the three-phase abstraction might restrict the expression of certain linear attention updates, particularly for future architectures that may involve more complex dependencies?\n4. What is the overhead (if any) of using Forge's compilation pipeline compared to maintaining optimized hand-written kernels, especially as new hardware or Triton versions are released?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Proposes a unified, three-phase abstraction for linear attention that captures a wide range of state-of-the-art variants, enabling easy mapping from high-level algorithms to efficient implementations.\n2. Automates the generation of high-performance, hardware-aware kernels from simple PyTorch code, reducing the barrier for researchers to experiment with or deploy new linear attention models.\n3. Integrates native support for fine-grained, tile-level compute-communication fusion using Triton-Distributed, improving distributed execution and network bandwidth utilization compared to existing methods.\n4. Demonstrates strong empirical results, with Forge-generated kernels achieving up to 4.9x speedup over expert-tuned baselines and near-linear weak scaling up to 128 GPUs and 16 million tokens.\n5. Includes practical system-level optimizations such as ahead-of-time compilation and static dispatching to eliminate runtime overheads, particularly beneficial for short to medium sequence lengths.\n6. Provides evidence that the abstraction is expressive enough to cover a broad family of linear attention models with minimal code changes."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. As a compiler-based framework, the proposed abstraction is tailored specifically for linear attention and lacks extensibility to other attention mechanisms, which limits its general applicability.\n2. Although Forge demonstrates competitive performance compared to Flash Linear Attention, the paper does not sufficiently emphasize its advantages, particularly in terms of implementation flexibility and extensibility relative to Triton-based Flash Linear Attention.\n3. While Forge automates optimization, the actual benefits in terms of code brevity or maintainability versus existing Triton-based manual approaches are not quantified in user studies or qualitative analysis.\n4. The distributed scaling results may be somewhat overstated: for example, in Figure 1, the single-GPU latency is 9.2ms, while 4 GPUs achieve 2.7ms. The real scaling efficiency is about 85%, not precisely \"near-linear\" as asserted.\n5. The distributed experiments (Section 4.2) use H20 GPUs (with slower communication) and BatchSize=4, whereas single-GPU experiments use H100 GPUs and BatchSize=1. This difference in hardware and batch size may exaggerate the benefits of compute-communication overlap and makes the experimental comparison less fair."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762920926271,"tcdate":1761068023536,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission9282/Reviewer_r3XC"],"signatures":["ICLR.cc/2026/Conference/Submission9282/Reviewer_r3XC"],"forum":"N4jJQvQSiN","number":1,"license":"CC BY 4.0","cdate":1761068023536,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission9282/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762920926271,"domain":"ICLR.cc/2026/Conference","replyto":"N4jJQvQSiN","id":"2sDHd4U1RU","forumContent":{"TLDR":{"value":"FlexLA is a domain-specific compiler that generates scalable kernels for diverse linear attention variants, from a simple unified abstraction, providing comparable or even better performance than expert-tuned libraries."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["linear attention","efficiency","compiler","kernels"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"The quadratic complexity of softmax attention poses a major bottleneck for long-context modeling, motivating a surge of linear attention variants with linear complexity. Unlike softmax attention, which benefits from optimized kernels, linear attention lacks general-purpose, hardware-efficient support and scalable distributed implementations. We introduce **Flex**ible **L**inear **A**ttention (FlexLA), a domain-specific compiler that automates the generation of high-performance, scalable kernels for a wide range of linear attention models directly from high-level PyTorch code. At its core, FlexLA employs an intuitive programming abstraction that decomposes any linear attention algorithm into three canonical phases: intra-chunk computation, inter-chunk state propagation, and output merging. This unified abstraction enables FlexLA to perform domain-specific optimizations, automatically generating kernels that fuse computation and communication at a fine-grained tile level and eliminating host synchronization. Our evaluation demonstrates that FlexLA combines programmability with performance: a wide range of linear attention variants can be implemented in just a few dozen lines of code, while the generated kernels deliver 1.01x-4.9x the performance of sate-of-the-art expert-optimized library and scale with near-linear efficiency on scalar gated linear attention to 16 million tokens on 128 GPUs, surpassing the state-of-the-art distributed baseline by up to 7.2x."},"_bibtex":{"value":"@inproceedings{\nduanmu2026flexlinearattention,\ntitle={FlexLinearAttention: Compiling a Unified Abstraction into Scalable Kernels for Linear Attention},\nauthor={Haojie Duanmu and Size Zheng and Ningxin Zheng and Jianqiao Lu and Xuegui Zheng and Xingcheng Zhang and Li-Wen Chang and Xin Liu and Dahua Lin},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=N4jJQvQSiN}\n}"},"title":{"value":"FlexLinearAttention: Compiling a Unified Abstraction into Scalable Kernels for Linear Attention"},"pdf":{"value":"/pdf/93b1c1ddfe377d6424c444b7b248e4fccdb4a793.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"duanmu|flexlinearattention_compiling_a_unified_abstraction_into_scalable_kernels_for_linear_attention"},"authorids":{"value":["~Haojie_Duanmu1","~Size_Zheng1","~Ningxin_Zheng1","~Jianqiao_Lu1","~Xuegui_Zheng1","~Xingcheng_Zhang3","~Li-Wen_Chang1","~Xin_Liu51","~Dahua_Lin1"]},"authors":{"value":["Haojie Duanmu","Size Zheng","Ningxin Zheng","Jianqiao Lu","Xuegui Zheng","Xingcheng Zhang","Li-Wen Chang","Xin Liu","Dahua Lin"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2405.17534v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"qi|smr_state_memory_replay_for_long_sequence_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Biqing_Qi:","https://dblp.org/search/pid/api?q=author:Junqi_Gao:","~Kaiyan_Zhang1","https://dblp.org/search/pid/api?q=author:Dong_Li:","https://dblp.org/search/pid/api?q=author:Jianxing_Liu:","~Ligang_Wu1","~Bowen_Zhou8"]},"html":{"value":"https://doi.org/10.48550/arXiv.2405.17534"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2405-17534,\n  publtype={informal},\n  author={Biqing Qi and Junqi Gao and Kaiyan Zhang and Dong Li and Jianxing Liu and Ligang Wu and Bowen Zhou},\n  title={SMR: State Memory Replay for Long Sequence Modeling},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2405.17534},\n  url={https://doi.org/10.48550/arXiv.2405.17534}\n}\n"},"abstract":{"value":"Despite the promising performance of state space models (SSMs) in long sequence modeling, limitations still exist. Advanced SSMs like S5 and S6 (Mamba) in addressing non-uniform sampling, their recursive structures impede efficient SSM computation via convolution. To overcome compatibility limitations in parallel convolutional computation, this paper proposes a novel non-recursive non-uniform sample processing strategy. Theoretical analysis of SSMs through the lens of Event-Triggered Control (ETC) theory reveals the Non-Stable State (NSS) problem, where deviations from sampling point requirements lead to error transmission and accumulation, causing the divergence of the SSM's hidden state. Our analysis further reveals that adjustments of input sequences with early memories can mitigate the NSS problem, achieving Sampling Step Adaptation (SSA). Building on this insight, we introduce a simple yet effective plug-and-play mechanism, State Memory Replay (SMR), which utilizes learnable memories to adjust the current state with multi-step information for generalization at sampling points different from those in the training data. This enables SSMs to stably model varying sampling points. Experiments on long-range modeling tasks in autoregressive language modeling and Long Range Arena demonstrate the general effectiveness of the SMR mechanism for a series of SSM models."},"title":{"value":"SMR: State Memory Replay for Long Sequence Modeling"},"authors":{"value":["Biqing Qi","Junqi Gao","Kaiyan Zhang","Dong Li","Jianxing Liu","Ligang Wu","Bowen Zhou"]}},"tmdate":1744846865894,"pdate":1704067200000,"tcdate":1731470073007,"writers":["~"],"signatures":["~Kaiyan_Zhang1"],"forum":"SCD4iZ8RKC","license":"CC BY-SA 4.0","number":195019,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1744846865894,"domain":"DBLP.org","id":"SCD4iZ8RKC","version":2},{"content":{"venue":{"value":"ACL (Findings) 2024"},"pdf":{"value":"https://aclanthology.org/2024.findings-acl.483.pdf"},"venueid":{"value":"dblp.org/conf/ACL/2024"},"paperhash":{"value":"qi|smr_state_memory_replay_for_long_sequence_modeling"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Biqing_Qi:","https://dblp.org/search/pid/api?q=author:Junqi_Gao:","~Kaiyan_Zhang1","https://dblp.org/search/pid/api?q=author:Dong_Li:","https://dblp.org/search/pid/api?q=author:Jianxing_Liu:","~Ligang_Wu1","~Bowen_Zhou8"]},"html":{"value":"https://doi.org/10.18653/v1/2024.findings-acl.483"},"_bibtex":{"value":"@inproceedings{DBLP:conf/acl/QiGZLLWZ24,\n  author={Biqing Qi and Junqi Gao and Kaiyan Zhang and Dong Li and Jianxing Liu and Ligang Wu and Bowen Zhou},\n  title={SMR: State Memory Replay for Long Sequence Modeling},\n  year={2024},\n  cdate={1704067200000},\n  pages={8102-8116},\n  url={https://doi.org/10.18653/v1/2024.findings-acl.483},\n  booktitle={ACL (Findings)},\n  crossref={conf/acl/2024f}\n}\n"},"abstract":{"value":"Despite the promising performance of state space models (SSMs) in long sequence modeling, limitations still exist. Advanced SSMs like S5 and S6 (Mamba) in addressing non-uniform sampling, their recursive structures impede efficient SSM computation via convolution. To overcome compatibility limitations in parallel convolutional computation, this paper proposes a novel non-recursive non-uniform sample processing strategy. Theoretical analysis of SSMs through the lens of Event-Triggered Control (ETC) theory reveals the Non-Stable State (NSS) problem, where deviations from sampling point requirements lead to error transmission and accumulation, causing the divergence of the SSM’s hidden state. Our analysis further reveals that adjustments of input sequences with early memories can mitigate the NSS problem, achieving Sampling Step Adaptation (SSA).Building on this insight, we introduce a simple yet effective plug-and-play mechanism, State Memory Replay (SMR), which utilizes learnable memories to adjust the current state with multi-step information for generalization at sampling points different from those in the training data. This enables SSMs to stably model varying sampling points. Experiments on long-range modeling tasks in autoregressive language modeling and Long Range Arena demonstrate the general effectiveness of the SMR mechanism for a series of SSM models."},"title":{"value":"SMR: State Memory Replay for Long Sequence Modeling"},"authors":{"value":["Biqing Qi","Junqi Gao","Kaiyan Zhang","Dong Li","Jianxing Liu","Ligang Wu","Bowen Zhou"]}},"tmdate":1744846864764,"pdate":1704067200000,"tcdate":1731470072785,"writers":["~"],"signatures":["~Kaiyan_Zhang1"],"forum":"HQ9OjXGUNU","license":"CC BY-SA 4.0","number":195014,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1744846864764,"domain":"DBLP.org","id":"HQ9OjXGUNU","version":2},{"content":{"summary":{"value":"The manuscript presents a novel target tracking method called TrackMamba, which employs a hybrid Mamba-Transformer model as its backbone. TrackMamba aims to address the issue of high memory consumption in existing single-stream Transformer-based trackers when dealing with high-resolution and long sequences. The main contributions are as follows:\n\n●\tThis manuscript introduces the TrackMamba tracking framework, alleviating computational burdens while ensuring high accuracy.\n●\tThis manuscript designs various scan patterns to better arrange and flip the input sequences within the TrackingMamba Block.\n●\tThis manuscript proposes a Target Enhancement module that utilizes Temporal Tokens and Temporal Mamba to enable effective information propagation across frames, enhancing the aggregation and transmission of target information."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1.\tAlthough TrackMamba performs well at high resolutions, the performance improvement in low-resolution sequences (LaSOT, in Table 1) is not significant. This may limit its generalizability across videos of different resolutions.\n2.\tWith more available templates, the search region is expected to receive target representation at various moments for robust tracking. However, the current framework shows no performance improvement with the addition of more templates (Table 4).\n3.\tAlthough TrackMamba aims to reduce memory consumption, computational complexity may still become a problem as input resolution increases. For resource constrained environments, this can be a challenge."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1.\tTrackMamba introduces several novel Mamba Scan Patterns for better scanning of input sequences. Through the scanning mechanism of Mamba, it successfully achieves the transfer of target features and employs target enhancement techniques to enable high representation transfer across frames.\n2.\tTrackMamba combines tracking efficiency with performance. In comparisons with existing state-of-the-art trackers, including FLOPs and trainingmemory consumption, TrackMamba significantly reduces training memory consumption without decreasing FLOPs, while maintaining high tracking accuracy.\n3.\tFor the first time, TrackMamba effectively integrates Temporal Tokens and Temporal Mamba alongside Template Tokens and Search Tokens, enabling efficient aggregation of target features and propagation of information across frames.\n4.\tThe illustrations in the manuscript are concise and clear, effectively conveying the construction of TrackMamba and the design concepts behind the three Scan Patterns."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\t1.\tAlthough this manuscript proposes a novel Mamba-based tracker with great performance both on tracking accuracy and memory consumption, it lacks the comparisons with the existing representative Mamba-based trackers, such as Mamba-FETrack[1] and so on.\n[1] Huang J, Wang S, Wang S, et al. Mamba-FETrack: Frame-Event Tracking via State Space Model[J]. arXiv preprint arXiv:2404.18174, 2024.\n2.\tThe paper does not provide a detailed explanation of the Enhancement design within the Target Enhancement module(in Figure 1). Specifically, how is the Search Feature refined by the Temporal Token through the Enhancement? I advise providing an extra illustration or pseudocode for how the Temporal Token refines the Search Feature.  \n3.\tIn the comparison experiments of Scan Patterns (in Table 2), the results for sequential-rearrange-whole-flip (#1 in Table 2) far exceed those for interleaved rearrange-whole-flip (#2 in Table 2). However, based on the narrative logic of the manuscript, the interleaved rearrange-whole-flip method should be superior to sequential-rearrange-whole-flip (as discussed in lines 281-289 of the manuscript and section 5.3 as well as in Table 2). I suggest you provide a more in-depth analysis of why this phenomenon exists and thus uncover its potential implications for the proposed method."}},"nonreaders":[],"tmdate":1731428180094,"tcdate":1729672540412,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4593/Reviewer_Ymzw"],"signatures":["ICLR.cc/2025/Conference/Submission4593/Reviewer_Ymzw"],"forum":"V7QRVEZ0le","number":2,"license":"CC BY 4.0","cdate":1729672540412,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4593/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428180094,"domain":"ICLR.cc/2025/Conference","replyto":"V7QRVEZ0le","id":"tw5PVcbXF2","forumContent":{"TLDR":{"value":"A mamba-transformer tracking framework is proposed for resolution-scalable visual tracking."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Single Object Tracking"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Current one-stream Transformer-based trackers are quality but unfriendly to memory consumption of large resolution and long sequence, both of which are crucial keys to tracking tasks. Recently structured state space model (SSM) demonstrates promising performance and efficiency in sequence modeling but struggles to retrieve due to the limited hidden state number. To solve the computation challenge and explore the potential of Mamba, we propose TrackMamba, a Mamba-Transformer tracker containing TrackMamba Blocks and Attention Blocks. In order to better harness the scanning in TrackMamba Blocks for inter- and intra-frame modeling, we introduce various scan patterns for rearrangement and flipping. Furthermore, we propose Target Enhancement, including Temporal Token for target aggregation and search enhancement, and Temporal Mamba for target information cross-frame propagation. Extensive experiments show TrackMamba performs better than the first-generation one-stream Transformer-based tracker at same resolution and mitigates consumption growth when enlarging resolution, exhibiting the potential of Mamba-based model for large-resolution tracking."},"_bibtex":{"value":"@misc{\nzhang2024trackmamba,\ntitle={TrackMamba: Mamba-Transformer Tracking},\nauthor={Jiaming Zhang and Cheng Liang and Yutao Cui and Xiangbo Shu and Gangshan Wu and Limin Wang},\nyear={2024},\nurl={https://openreview.net/forum?id=V7QRVEZ0le}\n}"},"title":{"value":"TrackMamba: Mamba-Transformer Tracking"},"pdf":{"value":"/pdf/b6fd4058e862b558436284a5d3bc1c3711911485.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhang|trackmamba_mambatransformer_tracking"},"authorids":{"value":["~Jiaming_Zhang6","~Cheng_Liang1","~Yutao_Cui1","~Xiangbo_Shu1","~Gangshan_Wu1","~Limin_Wang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jiaming Zhang","Cheng Liang","Yutao Cui","Xiangbo Shu","Gangshan Wu","Limin Wang"]}},"version":2},{"content":{"summary":{"value":"Current State-Space Models have some draw-backs compared to Transformer based models, such as lack certain capabilities, low generalization quality, not hardware-efficient, etc. This paper proposes the Mamba-3 architecture, from inference-first perspective. Specifically, the trapezoidal discretization is used to improve the expressive dynamics, complex-valued state spaces is introduced for state-tracking, and the multi-input multi-output (MIMO) mechanism is proposed for hardware efficiency. Experiments of show that Mamba-3 outperforms Mamba-2 or even transformer based models across 180M to 1.5B parameters."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Can the proposed Mamba-3 be generalized to vision tasks? \n\nDoes Mamba-3 still has advantages when compared to sparse attention based transformers?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The proposed Generalized Trapezoidal Discretization is more accurate than the Euler's rule used in Mamba2, by using the second-order  approximation of the integral. \n2. The complex SSM is novel, which is equivalent to a real SSM with data-dependent rotary embeddings (RoPE). Detailed theoretical analyses are provided.\n3. The MIMO method efficiently solves the I/O problem, which is a key bottleneck for Mamba-2."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"More complex tasks such as reasoning can be explored to fully demonstrate the capability of Mamba-3."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924146944,"tcdate":1761983709714,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13549/Reviewer_5Dn9"],"signatures":["ICLR.cc/2026/Conference/Submission13549/Reviewer_5Dn9"],"forum":"HwCvaJOiCj","number":3,"license":"CC BY 4.0","cdate":1761983709714,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13549/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924146944,"domain":"ICLR.cc/2026/Conference","replyto":"HwCvaJOiCj","id":"FDn90uYjhx","forumContent":{"venue":{"value":"ICLR 2026 Oral"},"TLDR":{"value":"Mamba-3, an inference-first SSM that pushes on core SSM principles: improved discretization for better quality, complex dynamics for new capabilities, and MIMO updates for efficient inference."},"keywords":{"value":["State Space Models","Mamba","LLMs","Subquadratic Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While current Transformer models deliver strong quality, their quadratic compute and linear memory requirements make inference expensive. This has spurred the development of sub-quadratic models with reduced compute and constant memory requirements.\nHowever, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice.\nGuided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule enabling richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation that improves model performance without increasing decode latency.\nTogether with architectural refinements, Mamba-3 achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with the MIMO variant further improving accuracy by an additional 1.2 points, for a total gain of 1.8 points.\nAcross state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half the state size. These results demonstrate that Mamba-3 advances the performance–efficiency frontier."},"_bibtex":{"value":"@inproceedings{\nlahoti2026mamba,\ntitle={Mamba-3: Improved Sequence Modeling using State Space Principles},\nauthor={Aakash Lahoti and Kevin Li and Berlin Chen and Caitlin Wang and Aviv Bick and J Zico Kolter and Tri Dao and Albert Gu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=HwCvaJOiCj}\n}"},"title":{"value":"Mamba-3: Improved Sequence Modeling using State Space Principles"},"pdf":{"value":"/pdf/09c67b76b7d1ad254ea30eeba839b8592de9bd78.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lahoti|mamba3_improved_sequence_modeling_using_state_space_principles"},"authorids":{"value":["~Aakash_Lahoti1","~Kevin_Li3","~Berlin_Chen1","~Caitlin_Wang1","~Aviv_Bick1","~J_Zico_Kolter1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Aakash Lahoti","Kevin Li","Berlin Chen","Caitlin Wang","Aviv Bick","J Zico Kolter","Tri Dao","Albert Gu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a trajectory-aware shifted state space model for online video super-resolution, named as TS-Mamba. TS-Mamba leverages trajectory to select relevant tokens from previous frames, shifted Hilbert scanning to compensate the spatial discontinuities, and a trajectory-aware loss to supervise trajectory generation. Experimental results show the competitive or superior of TS-Mamba compared with SOTA, with reduced complexity."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"(1) It is suggested to add a new set of experiments ablate the temporal window size. This is a low-effort, high-impact experiment that directly validates a core contribution of the paper. \n(2) The authors should revise the introduction and related work sections to include a concise yet specific \"Limitations of Prior Work\" paragraph that directly motivates the design choices of their own model."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The integration of temporal trajectory into Mamba for online VSR is original. \nThe trajectory mechanism for video is a well-motivated and reasonable way to capture the longer temporal dependencies while maintaining low complexity. That is suitable for online video-related tasks. \nNovel use of state space models (SSMs) with elaborately designed shifted operations to compensate for Hilbert scanning discontinuities，and sufficient illustrations enhance understanding and readability. \nThis paper is a well-organized and well-presented, with ablations and detailed training settings. The illustration of the SSMs scan is impressive and clear."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Author proposes a trajectory-aware shifted Mamba aggregation module to achieve the long-term spatio-temporal information aggregation. However, the existing trajectory/temporal alignment methods (e.g., deformable alignment, flow-guided deformable alignment) also can achieve this goal, a more detailed comparison with these methods would strengthen this part. \n\nThe experiments only use standard window sizes, without specific analysis on long-term dependency (e.g., varying temporal window T). A comparison between using the fixed temporal window (i.e., T=15) and other temporal window sizes would help determine the optimal size of temporal window of trajectory modeling for the proposed method.\n\nThe author did not thoroughly analyze the shortcomings of existing methods, and the description is also quite vague. It is suggested that the author conduct a more in-depth analysis of the shortcomings of existing methods to further highlight the advantages of the method proposed in this paper."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919223530,"tcdate":1761532309540,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7012/Reviewer_7jKz"],"signatures":["ICLR.cc/2026/Conference/Submission7012/Reviewer_7jKz"],"forum":"RygnSGcV49","number":1,"license":"CC BY 4.0","cdate":1761532309540,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7012/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919223530,"domain":"ICLR.cc/2026/Conference","replyto":"RygnSGcV49","id":"PfEY5vlUXr","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Video Super-resolution","Online","Mamba","Trajectory"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Online video super-resolution (VSR) is an important technique for many real-world video processing applications, which aims to restore the current high-resolution video frame based on temporally previous frames. Most of the existing online VSR methods solely employ one neighboring previous frame to achieve temporal alignment, which limits long-range temporal modeling of videos. Recently, state space models (SSMs) have been proposed with linear computational complexity and a global receptive field, which significantly improve computational efficiency and performance. In this context, this paper presents a novel online VSR method based on Trajectory-aware Shifted SSMs (TS-Mamba), leveraging both long-term trajectory modeling and low-complexity Mamba to achieve efficient spatio-temporal information aggregation. Specifically, TS-Mamba first constructs the trajectories within a video to select the most similar tokens from the previous frames. Then, a Trajectory-aware Shifted Mamba Aggregation (TSMA) module consisting of proposed shifted SSMs blocks is employed to aggregate the selected tokens. The shifted SSMs blocks are designed based on Hilbert scannings and corresponding shift operations to compensate for scanning losses and strengthen the spatial continuity of Mamba. Additionally, we propose a trajectory-aware loss function to supervise the trajectory generation, ensuring the accuracy of token selection when training our model. Extensive experiments on three widely used VSR test datasets demonstrate that compared with six online VSR benchmark models, our TS-Mamba achieves state-of-the-art performance in most cases and over 22.7% complexity reduction (in MACs). The source code for TS-Mamba is available at https://github.com/QZ1-boy/TS-Mamba."},"_bibtex":{"value":"@inproceedings{\nzhu2026trajectoryaware,\ntitle={Trajectory-aware Shifted State Space Models for Online Video Super-Resolution},\nauthor={Qiang Zhu and Xiandong MENG and Yuxuan Jiang and Fan Zhang and David Bull and Shuyuan Zhu and Bing Zeng and Ronggang Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=RygnSGcV49}\n}"},"title":{"value":"Trajectory-aware Shifted State Space Models for Online Video Super-Resolution"},"pdf":{"value":"/pdf/63207ba1dfa93f1f2594d2cd3d80db2064dc0722.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhu|trajectoryaware_shifted_state_space_models_for_online_video_superresolution"},"authorids":{"value":["~Qiang_Zhu6","~Xiandong_MENG1","~Yuxuan_Jiang8","~Fan_Zhang6","~David_Bull1","~Shuyuan_Zhu1","~Bing_Zeng1","~Ronggang_Wang1"]},"authors":{"value":["Qiang Zhu","Xiandong MENG","Yuxuan Jiang","Fan Zhang","David Bull","Shuyuan Zhu","Bing Zeng","Ronggang Wang"]}},"version":2},{"content":{"summary":{"value":"The paper presents a hybrid architecture that combines Mamba-2 layers with softmax selective attention layers for long sequence modeling. The empirical study validates the effectiveness of the new model up to 1B model parameters."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"It is useful to run ablation experiments to show (1) the gain of using sliding window attention; (2) the gain over using only sliding window attention without selective attention; and (3) the number of tokens selected based on Eq. (1) and (2).\n\nIt is also useful to investigate what tokens are selected, and whether there are any patterns, such as the ones described in https://arxiv.org/pdf/2310.01801."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper is well organized, and the research is well motivated."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The proposed hybrid model is conceptually similar to other hybrid models that combine softmax attention models (Transformers) and modern CNN layers (such as S4, Mamba). Although the gain on small models is encouraging, the paper could be much stronger if a more comprehensive comparison with SOTA Transoformer / Hybrid models can be performed."}},"nonreaders":[],"tmdate":1731429283088,"tcdate":1730320831760,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9385/Reviewer_CMJ8"],"signatures":["ICLR.cc/2025/Conference/Submission9385/Reviewer_CMJ8"],"forum":"9DnKZbOr4r","number":1,"license":"CC BY 4.0","cdate":1730320831760,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9385/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429283088,"domain":"ICLR.cc/2025/Conference","replyto":"9DnKZbOr4r","id":"H47om9hqQ3","forumContent":{"TLDR":{"value":"We introduce Taipan, a novel architecture that combines State Space Models with Selective Attention Layers, achieving superior performance in memory-intensive tasks while enabling efficient language modeling for sequences up to 1 million tokens."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Efficient Language Model","Model Architecture","Long-context Language Model","In-context Retrieval","Hybrid Architecture","Linear Complexity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long sequences due to quadratic computational complexity in training and linearly scaling memory costs during inference. Recent State Space Models (SSMs) such as Mamba offer alternatives with constant memory usage, but they underperform in tasks requiring extensive in-context retrieval. We introduce Taipan, a novel hybrid architecture that combines Mamba-2 with Selective Attention Layers (SALs). These SALs identify tokens requiring long-range interactions, remove less important features, and then augment their representations using the attention module. This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. By constraining the attention budget, Taipan extends accurate predictions to context lengths of up to 1 million tokens while preserving computational efficiency. Our experiments demonstrate Taipan's superior performance across various scales and tasks, offering a promising solution for efficient long-context language modeling."},"_bibtex":{"value":"@misc{\nnguyen2025taipan,\ntitle={Taipan: Efficient and Expressive State Space Language Models with Selective Attention},\nauthor={Chien Van Nguyen and Huy Huu Nguyen and Thang M. Pham and Ruiyi Zhang and Hanieh Deilamsalehy and Puneet Mathur and Ryan A. Rossi and Trung Bui and Viet Dac Lai and Franck Dernoncourt and Thien Huu Nguyen},\nyear={2025},\nurl={https://openreview.net/forum?id=9DnKZbOr4r}\n}"},"title":{"value":"Taipan: Efficient and Expressive State Space Language Models with Selective Attention"},"pdf":{"value":"/pdf/4c569111617737e5532042f859ac730be0ac18ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"nguyen|taipan_efficient_and_expressive_state_space_language_models_with_selective_attention"},"authorids":{"value":["~Chien_Van_Nguyen1","~Huy_Huu_Nguyen1","~Thang_M._Pham1","~Ruiyi_Zhang3","~Hanieh_Deilamsalehy1","~Puneet_Mathur2","~Ryan_A._Rossi2","~Trung_Bui1","~Viet_Dac_Lai1","~Franck_Dernoncourt1","~Thien_Huu_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chien Van Nguyen","Huy Huu Nguyen","Thang M. Pham","Ruiyi Zhang","Hanieh Deilamsalehy","Puneet Mathur","Ryan A. Rossi","Trung Bui","Viet Dac Lai","Franck Dernoncourt","Thien Huu Nguyen"]}},"version":2},{"content":{"summary":{"value":"The paper introduces minimal variants of LSTM and GRU (minLSTM and minGRU) that enable parallel training via the parallel scan algorithm. Key modifications include removing hidden state dependencies and rescaling mechanisms in the LSTM. The authors demonstrate competitive performance with modern architectures on several tasks."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Can the authors provide a detailed comparison with Martin & Cundy (2018), specifically addressing how minGRU differs from GILR, if at all, and what novel contributions this work makes beyond the prior work? Additionally, it would be valuable to understand why these ideas are being revisited now and what new insights are gained.\n\nHow do the proposed new architectures fare at larger data sizes? How do these models perform on standard language modeling benchmarks? Results on datasets with training sequences longer than 1000 steps would be particularly good to see. Understanding the scaling behavior on modern language modeling tasks is crucial for assessing the practical value of this approach.\n\nFinally, more detailed inference speed comparisons would strengthen the paper. How do minGRU and minLSTM compare with Mamba and traditional LSTM/GRU implementations across different sequence lengths and batch sizes?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":1},"strengths":{"value":"The paper's technical exposition is clear, offering good motivation for removing parts of the RNN architectures. The authors provide thorough motivation for each architectural decision, especially regarding the time-independence properties of their models. The connection to parallel scan algorithms makes sense.\n\nThe architectural innovations demonstrate consideration of practical implementation challenges. The rescaling mechanism in minLSTM ensures time-independence while maintaining model expressivity. The authors have given careful thought to numerical stability issues, providing both standard and log-space formulations of their approach. It is nice to see the PyTorch implementation details, which help to make the algorithm very concrete.\n\nThe experimental validation covers a range of tasks and provides evidence of training speed advantages wrt traditional RNNs. The performance on the selective copying task is particularly impressive, demonstrating the model's ability to handle a particular type of long-range dependencies."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The most significant concern is the paper's relationship to prior work, specifically Martin & Cundy (ParalleIizing Linear Recurrent Neural Nets over Sequence Length, ICLR 2018), which appears to have developed very similar ideas. The proposed minGRU architecture appears mathematically equivalent to their GILR architecture when \\(g_t = 1 - z_t\\) and \\(i_t = \\tilde{h}_t\\) in the authors' notation. The parallel scan approach for training is also very similar. While the LSTM rescaling mechanism appears novel, this substantial overlap significantly reduces the paper's claimed novelty.\n\nThe experimental evaluation, while broad, has several limitations. The benefits over baselines are modest in most tasks, with the exception of the selective copying task, which seems carefully chosen to showcase the method's strengths. The language modeling evaluation relies solely on the Shakespeare dataset, which is both too small by modern standards. This choice of dataset is not representative of real-world language modeling challenges, which typically do not have overfitting as a major worry.\n\nThe methodology would benefit from more thorough analysis in several areas. The authors could strengthen their contribution by providing more extensive ablation studies on their architectural choices. The paper would also benefit from a clearer discussion of scaling behavior, to indicate if this architecture could be a potential replacement for transformers or Mamba at the larger scales of modern models."}},"nonreaders":[],"tmdate":1731429058150,"tcdate":1730244259210,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8241/Reviewer_jHLY"],"signatures":["ICLR.cc/2025/Conference/Submission8241/Reviewer_jHLY"],"forum":"GrmFFxGnOR","number":2,"license":"CC BY 4.0","cdate":1730244259210,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8241/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429058150,"domain":"ICLR.cc/2025/Conference","replyto":"GrmFFxGnOR","id":"ZLKbCUEU7B","forumContent":{"TLDR":{"value":"Revisiting decade-old RNNs (LSTMs and GRUs), we introduce minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters, (2) are parallelizable during training, and (3) match the performance of recent sequence models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Recurrent Neural Networks","RNNs","Sequence Modelling","Efficiency","LSTMs","GRUs","Parallel Scan"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers—particularly with respect to sequence length—have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively.\nIn this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers."},"_bibtex":{"value":"@misc{\nfeng2025were,\ntitle={Were {RNN}s All We Needed?},\nauthor={Leo Feng and Frederick Tung and Mohamed Osama Ahmed and Yoshua Bengio and Hossein Hajimirsadeghi},\nyear={2025},\nurl={https://openreview.net/forum?id=GrmFFxGnOR}\n}"},"title":{"value":"Were RNNs All We Needed?"},"pdf":{"value":"/pdf/a56137c0e88e5c5b4003b438b48e61ee149bd102.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"feng|were_rnns_all_we_needed"},"authorids":{"value":["~Leo_Feng1","~Frederick_Tung1","~Mohamed_Osama_Ahmed2","~Yoshua_Bengio1","~Hossein_Hajimirsadeghi1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Leo Feng","Frederick Tung","Mohamed Osama Ahmed","Yoshua Bengio","Hossein Hajimirsadeghi"]}},"version":2},{"content":{"venue":{"value":"ICML 2026 spotlight"},"keywords":{"value":["State Space Models","Geometric Conditioning","Momentum-Augmented Dynamics","Associative Memory","Long-Sequence Modeling"]},"supplementary_material":{"value":"/attachment/2253f6d452d48627ee26f3f6f359a6a2a8490416.zip"},"_bibtex":{"value":"@inproceedings{\nnguyen2026muonssm,\ntitle={Muon{SSM}: Orthogonalizing State Space Models for Sequence Modeling},\nauthor={Thai Khanh Nguyen and Vo Ngoc Bich Uyen and Thieu Vo and Tan Minh Nguyen and Cuong Pham},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=GmP3VcfHi0}\n}"},"title":{"value":"MuonSSM: Orthogonalizing State Space Models for Sequence Modeling"},"paperhash":{"value":"nguyen|muonssm_orthogonalizing_state_space_models_for_sequence_modeling"},"originally_submitted_PDF":{"value":"/pdf/4e4faa2380bff6b1518eed188b1e07d150ba9d3b.pdf"},"TLDR":{"value":"We introduce MuonSSM, which stabilizes state-space models by geometrically conditioning memory updates via momentum and Newton–Schulz iterations, achieving spectrally stable long-horizon performance without sacrificing parallelizability."},"primary_area":{"value":"deep_learning->sequential_models_time_series"},"abstract":{"value":"State space models (SSMs) have emerged as efficient linear-time alternatives to attention for long-sequence modeling. However, existing SSMs often suffer from instability and memory degradation over extended horizons due to poorly conditioned first-order updates and unbalanced update geometry. We introduce MuonSSM, a general framework that stabilizes SSM training by explicitly conditioning the geometry of memory updates rather than the recurrent transition matrix. MuonSSM augments SSMs with a momentum-based pathway and a lightweight Newton-Schulz transformation on low-rank input injections, yielding bounded and spectrally conditioned updates while preserving parallel scan complexity. Theory shows that MuonSSM improves gradient propagation, mitigates spectral amplification, and enriches memory representations over long horizons. Extensive experiments across language, vision, and time-series benchmarks show consistent gains in accuracy, robustness, and long-context performance when integrated into diverse SSM backbones. These results establish geometric conditioning of updates as a principled pathway to stable, scalable sequence modeling."},"link_to_code":{"value":"https://github.com/t-khanusa/MuonSSM/tree/main"},"pdf":{"value":"/pdf/8dbebf6b2c7009cb090bfa4d33c3c46c85b18c61.pdf"},"lay_summary":{"value":"Modern machine learning models often need to process long sequences, such as long documents, videos, sensor signals, or time-series data. Transformers are powerful for these tasks, but they become expensive as the sequence gets longer. State space models offer a more efficient alternative, yet their internal memory can become unstable or gradually lose useful information over long contexts. In this work, we introduce MuonSSM, a method that improves how state space models update and preserve their memory. Instead of only changing the main recurrent transition, MuonSSM adds a momentum-like memory pathway and a lightweight normalization step that keeps each update more balanced. This helps information and gradients travel more reliably across long sequences while preserving the efficient parallel computation that makes state space models attractive. We test MuonSSM on language modeling, long-context retrieval, vision tasks, and human activity recognition. Across these settings, MuonSSM consistently improves accuracy, robustness, and long-context performance over several state space model baselines. These results suggest that better-shaped memory updates can be a simple and effective way to build stable and scalable sequence models."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Thai_Khanh_Nguyen2","~Vo_Ngoc_Bich_Uyen1","~Thieu_Vo1","~Tan_Minh_Nguyen1","~Cuong_Pham1"]},"authors":{"value":["Thai Khanh Nguyen","Vo Ngoc Bich Uyen","Thieu Vo","Tan Minh Nguyen","Cuong Pham"]}},"tmdate":1790067389749,"pdate":1777576609078,"tcdate":1769143052476,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission20651/Authors"],"signatures":["ICML.cc/2026/Conference/Submission20651/Authors"],"forum":"GmP3VcfHi0","license":"CC BY 4.0","number":20651,"cdate":1769143052476,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission20651/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission20651/-/Camera_Ready_Revision"],"mdate":1790067389749,"odate":1782341979430,"domain":"ICML.cc/2026/Conference","id":"GmP3VcfHi0","version":2},{"content":{"summary":{"value":"The paper \"Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models\" presents a new efficient large language and vision model (LLVM), Meteor, which leverages multifaceted rationale to enhance understanding and answering capabilities. The paper  introduces a new concept of traversal of rationale, demonstrating the effectiveness of Meteor in embedding lengthy rationales containing abundant information and improving vision language performances across various evaluation benchmarks requiring diverse capabilities."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please refer to the weakness"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper introduces a novel concept of traversal of rationale and effectively demonstrates the efficiency of embedding lengthy rationales using the Mamba architecture.\n2. The paper provides comprehensive performance comparison with existing LLVMs in Figure 1, 2 and Table 1. Based on this, the experimental results and ablation studies provide strong evidence of the effectiveness of Meteor in improving vision language performances across diverse evaluation benchmarks.\n3. The paper provides a detailed description of the model architecture, training strategy, and evaluation benchmarks, ensuring reproducibility and understanding of the proposed method."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Writing:\n1. All Figures (fonts, line widths, etc.) in the paper are not uniform, there is too much white space. And even Figure 1 shows strange tables and font proportions (fonts are also not clear).\n\nMotivation:\n2. The paper employ the Mamba architecture to embed lengthy rationales containing abundant information, but without setting and demonstrating task scenarios that existing LVLMs does not achieve that require extreme long sequence modeling capabilities.\n3. It appears to share the same motivation as some existing methods, for instance, Vary-toy[1]. Please provide a more detailed explanation of how the proposed \"traversal of rationale\" differs from the design implemented in Vary-toy.\n\nMethod:\n4. Show more compassion with the MMLM models based on Mamba, like Cobra[2] or VL-Mamba[3].\n\n[1] Small Language Model Meets with Reinforced Vision Vocabulary\n[2] Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference\n[3] VL-Mamba: Exploring State Space Models for Multimodal Learning"},"limitations":{"value":"None"}},"nonreaders":[],"tmdate":1730879506562,"tcdate":1720708610891,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission11917/Reviewer_iRmr"],"signatures":["NeurIPS.cc/2024/Conference/Submission11917/Reviewer_iRmr"],"forum":"JVKABhr6mP","number":3,"license":"CC BY 4.0","cdate":1720708610891,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission11917/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879506562,"domain":"NeurIPS.cc/2024/Conference","replyto":"JVKABhr6mP","id":"lC7mEcyQUc","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Large Language and Vision Model"]},"primary_area":{"value":"other"},"abstract":{"value":"The rapid development of large language and vision models (LLVMs) has been driven by advances in visual instruction tuning. Recently, open-source LLVMs have curated high-quality visual instruction tuning datasets and utilized additional vision encoders or multiple computer vision models in order to narrow the performance gap with powerful closed-source LLVMs. These advancements are attributed to multifaceted information required for diverse capabilities, including fundamental image understanding, real-world knowledge about common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions. Drawing from the multifaceted information, we present a new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities. To embed lengthy rationales containing abundant information, we employ the Mamba architecture, capable of processing sequential data with linear time complexity. We introduce a new concept of traversal of rationale that facilitates efficient embedding of rationale. Subsequently, the backbone multimodal language model (MLM) is trained to generate answers with the aid of rationale. Through these steps, Meteor achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities, without scaling up the model size or employing additional vision encoders and computer vision models."},"_bibtex":{"value":"@inproceedings{\nlee2024meteor,\ntitle={Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models},\nauthor={Byung-Kwan Lee and Chae Won Kim and Beomchan Park and Yong Man Ro},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=JVKABhr6mP}\n}"},"title":{"value":"Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models"},"pdf":{"value":"/pdf/d68d63de15506ec221657ddca9ceb58cdd2987ea.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"lee|meteor_mambabased_traversal_of_rationale_for_large_language_and_vision_models"},"authorids":{"value":["~Byung-Kwan_Lee1","~Chae_Won_Kim1","~Beomchan_Park1","~Yong_Man_Ro3"]},"authors":{"value":["Byung-Kwan Lee","Chae Won Kim","Beomchan Park","Yong Man Ro"]}},"version":2},{"content":{"summary":{"value":"The paper introduces MTS-UNMixers, a new framework for multivariate time-series forecasting based on dual-path unmixing mechanisms.  \nThe model decouples shared and variable-specific temporal dependencies through channel-wise and temporal unmixing blocks, further enhanced with a Mamba-based state-space backbone for efficient long-range modeling.  \nThis design aims to mitigate signal aliasing and redundancy problems commonly seen in Transformer-style architectures.  \n\nExperiments on seven public datasets (including ETT, Electricity, Weather, and Traffic) demonstrate strong performance against nine recent baselines, such as PatchTST, FEDformer, and TimeMixer.  \nThe results show consistent improvements in accuracy and effective reconstruction."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Could you include **PathFormer**, **iTransformer**,  and **CARD** as additional baselines in your comparison table to provide a broader empirical view?  \n2. How sensitive is the model to the number of Mamba layers and unmixing depth?  \n3. Can MTS-UNMixers generalize to longer forecasting horizons beyond 720 steps without retraining?  \n4. Is there any benefit in combining MTS-UNMixers with contrastive or self-supervised pretraining?  \n5. How does the dual-path mechanism behave when one modality (e.g., channel group) is noisy or partially missing?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. **Clear and practical architectural motivation.**  \n   The unmixing mechanism offers an intuitive way to separate shared temporal dynamics from variable-specific behaviors, which is valuable for multivariate forecasting tasks.  \n\n2. **Solid empirical results.**  \n   The model achieves competitive or better results than strong baselines including TimeMixer and FEDformer across multiple datasets.  \n\n3. **Integration with Mamba blocks.**  \n   Leveraging a state-space backbone improves both computational efficiency and modeling of long-term dependencies.  \n\n4. **Well-written and structured paper.**  \n   The methodology is clearly described with detailed illustrations and quantitative validation.  \n\n5. **Interpretability through reconstruction visualization.**  \n   The figures effectively demonstrate how dual unmixing captures complementary channel and temporal information."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Missing comparisons with several recent benchmarks.**  \n   While PathTST, TimeMixer, and FEDformer are included, it would strengthen the paper to add comparisons with **PathFormer**, **iTransformer**,  and **CARD** — two strong baselines known for path-level and decomposition-based temporal reasoning.  \n   These would provide more context on how MTS-UNMixers performs against newer architectures targeting similar goals.\n\n2. **Limited analysis on parameter sensitivity.**  \n   The ablation study primarily focuses on architectural settings; however, analyzing the sensitivity of Mamba block size or unmixing depth would help understand robustness.  \n\n3. **Theoretical insight is relatively shallow.**  \n   The paper’s arguments remain empirical; a more formal treatment of the unmixing process (e.g., via subspace decomposition theory) would be valuable.  \n\n4. **Dataset diversity.**  \n   While mid-scale datasets are well covered, evaluations on very large or irregularly sampled datasets (e.g., weather radar or energy trading) would highlight scalability and adaptability."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917506542,"tcdate":1761645196966,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4673/Reviewer_SWHT"],"signatures":["ICLR.cc/2026/Conference/Submission4673/Reviewer_SWHT"],"forum":"PiwfjPYEUI","number":2,"license":"CC BY 4.0","cdate":1761645196966,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4673/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917506542,"domain":"ICLR.cc/2026/Conference","replyto":"PiwfjPYEUI","id":"pu9ShHGdaF","forumContent":{"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"TLDR":{"value":"MTS-UNMixers disentangles temporal and channel patterns via dual-path unmixing for accurate multivariate time series forecasting."},"keywords":{"value":["Time series forecasting","Unmixing","Mamba network","Shared mechanism."]},"supplementary_material":{"value":"/attachment/91adfca16fea8337a66f57973fd6ba60089f8d3e.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Multivariate time series data provide a robust framework for future predictions by leveraging information across multiple dimensions, ensuring broad applicability in practical scenarios. However, their high dimensionality and mixing patterns pose significant challenges in establishing an interpretable and explicit mapping between historical and future series, as well as extracting long-range feature dependencies. To address these challenges, we propose a channel-time dual unmixing network for multivariate time series forecasting (named MTS-UNMixers), which decomposes the entire series into critical bases and coefficients across both the time and channel dimensions. This approach establishes a robust sharing mechanism between historical and future series, enabling accurate representation and enhancing physical interpretability. Specifically, MTS-UNMixers represent sequences over time as a mixture of multiple trends and cycles, with the time-correlated representation coefficients shared across both historical and future time periods. In contrast, sequence over channels can be decomposed into multiple tick-wise bases, which characterize the channel correlations and are shared across the whole series. To estimate the shared time-dependent coefficients, a vanilla Mamba network is employed, leveraging its alignment with directional causality. Correspondlingly, a bidirectional Mamba network is utilized to model the shared channel-correlated bases, accommodating noncausal relationships. Experimental results show that MTS-UNMixers significantly outperform existing methods on multiple benchmark datasets."},"_bibtex":{"value":"@misc{\nanonymous2026mtsunmixers,\ntitle={{MTS}-{UNM}ixers: Multivariate Time Series Forecasting via Channel-Time Dual Unmixing},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=PiwfjPYEUI}\n}"},"title":{"value":"MTS-UNMixers: Multivariate Time Series Forecasting via Channel-Time Dual Unmixing"},"pdf":{"value":"/pdf/ebdc738547ddda3e7bd664663e54f64296439602.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"zhu|mtsunmixers_multivariate_time_series_forecasting_via_channeltime_dual_unmixing"},"authorids":{"value":["~Xuanbing_Zhu1","~Dunbin_Shen1","~Zhongwen_Rao1","~Huiyi_Ma1","~Yingguang_Hao1","~Hongyu_Wang1"]},"authors":{"value":["Xuanbing Zhu","Dunbin Shen","Zhongwen Rao","Huiyi Ma","Yingguang Hao","Hongyu Wang"]}},"version":2},{"content":{"venue":{"value":"NeurIPS 2025 poster"},"TLDR":{"value":"We analyze Mamba's shortcomings in long-range sequence processing through its expressiveness, inductive bias, and training stability, and design a remedy called Block-Biased S6 to achieve state-of-the-art performance on long-range tasks."},"keywords":{"value":["Sequence Model","State Space Model","Mamba","Long Range Arena"]},"supplementary_material":{"value":"/attachment/167f2f60e1c977f69375031aaa972478557e06ae.zip"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Mamba extends earlier state space models (SSMs) by introducing input-dependent dynamics, and has demonstrated strong empirical performance across a range of domains, including language modeling, computer vision, and foundation models. However, a surprising weakness remains: despite being built on architectures designed for long-range dependencies, Mamba performs poorly on long-range sequential tasks. Understanding and addressing this gap is important for improving Mamba's universality and versatility. In this work, we analyze Mamba’s limitations through three perspectives: expressiveness, inductive bias, and training stability. Our theoretical results show how Mamba falls short in each of these aspects compared to earlier SSMs such as S4D. To address these issues, we propose $\\text{B}\\_{2}\\text{S}\\_{6}$, a simple extension of Mamba's S6 unit that combines block-wise selective dynamics with a channel-specific bias. We prove that these changes equip the model with a better-suited inductive bias and improve its expressiveness and stability. Empirically, $\\text{B}\\_{2}\\text{S}\\_{6}$ outperforms S4 and S4D on Long-Range Arena (LRA) tasks while maintaining Mamba's performance on language modeling benchmarks."},"_bibtex":{"value":"@inproceedings{\nyu2025blockbiased,\ntitle={Block-Biased Mamba for Long-Range Sequence Processing},\nauthor={Annan Yu and N. Benjamin Erichson},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=5WKEH9LhAQ}\n}"},"title":{"value":"Block-Biased Mamba for Long-Range Sequence Processing"},"pdf":{"value":"/pdf/30d44e8d9e2ab45e637418c361cdddead113e461.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"yu|blockbiased_mamba_for_longrange_sequence_processing"},"authorids":{"value":["~Annan_Yu1","~N._Benjamin_Erichson1"]},"authors":{"value":["Annan Yu","N. Benjamin Erichson"]}},"tmdate":1783627448779,"pdate":1758216579871,"tcdate":1745314920679,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission3606/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission3606/Authors"],"forum":"5WKEH9LhAQ","license":"CC BY 4.0","number":3606,"cdate":1745314920679,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission3606/-/Full_Submission","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission3606/-/Camera_Ready_Revision"],"mdate":1783627448779,"odate":1761704754698,"domain":"NeurIPS.cc/2025/Conference","id":"5WKEH9LhAQ","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2409.01728v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"cao|shuffle_mamba_state_space_models_with_random_shuffle_for_multimodal_image_fusion"},"authorids":{"value":["","https://dblp.org/search/pid/api?q=author:Xuanhua_He:","https://dblp.org/search/pid/api?q=author:Tao_Hu:","https://dblp.org/search/pid/api?q=author:Chengjun_Xie:","https://dblp.org/search/pid/api?q=author:Jie_Zhang_0033:","https://dblp.org/search/pid/api?q=author:Man_Zhou:","https://dblp.org/search/pid/api?q=author:Danfeng_Hong:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2409.01728"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2409-01728,\n  publtype={informal},\n  author={Ke Cao and Xuanhua He and Tao Hu and Chengjun Xie and Jie Zhang and Man Zhou and Danfeng Hong},\n  title={Shuffle Mamba: State Space Models with Random Shuffle for Multi-Modal Image Fusion},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2409.01728},\n  url={https://doi.org/10.48550/arXiv.2409.01728}\n}\n"},"abstract":{"value":"Multi-modal image fusion integrates complementary information from different modalities to produce enhanced and informative images. Although State-Space Models, such as Mamba, are proficient in long-range modeling with linear complexity, most Mamba-based approaches use fixed scanning strategies, which can introduce biased prior information. To mitigate this issue, we propose a novel Bayesian-inspired scanning strategy called Random Shuffle, supplemented by an theoretically-feasible inverse shuffle to maintain information coordination invariance, aiming to eliminate biases associated with fixed sequence scanning. Based on this transformation pair, we customized the Shuffle Mamba Framework, penetrating modality-aware information representation and cross-modality information interaction across spatial and channel axes to ensure robust interaction and an unbiased global receptive field for multi-modal image fusion. Furthermore, we develop a testing methodology based on Monte-Carlo averaging to ensure the model's output aligns more closely with expected results. Extensive experiments across multiple multi-modal image fusion tasks demonstrate the effectiveness of our proposed method, yielding excellent fusion quality over state-of-the-art alternatives. Code will be available upon acceptance."},"title":{"value":"Shuffle Mamba: State Space Models with Random Shuffle for Multi-Modal Image Fusion"},"authors":{"value":["Ke Cao","Xuanhua He","Tao Hu","Chengjun Xie","Jie Zhang","Man Zhou","Danfeng Hong"]}},"tmdate":1786344076539,"pdate":1704067200000,"tcdate":1747276352599,"writers":["~"],"signatures":["~Ke_Cao2"],"forum":"3MnzvJ52c2","license":"CC BY-SA 4.0","number":470895,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1786344076539,"domain":"DBLP.org","id":"3MnzvJ52c2","version":2},{"content":{"summary":{"value":"This paper proposes the Point Relational Parsing (PointRePar) model, supporting multi-category joint training. Robust tracking is achieved through two core designs: First, the U-shaped Spatial Relational Parsing Module (USRPM) based on Mamba, combined with Dynamic Feature Aggregation (DFA), captures multi-scale spatial point relationships to enhance foreground - background feature separation. Second, it employs dual-level temporal relationship modeling at both point-level (Temporal Scan Mamba) and bounding-box-level (Long-term Motion Trajectory Rectification, LMTR), while introducing Conditional Gaussian Perturbation (CGP) to simulate prediction errors related to scene sparsity. Experiments validated on KITTI, NuScenes, and WOD benchmark datasets demonstrate that PointRePar not only significantly outperforms existing category-unified methods but also rivals current state-of-the-art category-specific approaches. This provides a balanced solution for 3D SOT that optimizes “generalizability, performance, and training efficiency”."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. Regarding Mamba's advantages over USRPM: Compared to Transformers or RNNs, is there quantitative evidence of Mamba's “long sequence modeling advantage” on unstructured point cloud data? For example, under identical FLOPs, how does Mamba's feature separation (e.g., foreground-background cosine distance) compare to Transformers?\n\n2. Regarding CGP's Generalizability: CGP is designed based on the “scene sparsity - error magnitude” correlation. If scene sparsity changes drastically over time (e.g., when an object is momentarily occluded by a building), can CGP's dynamic noise adjustment still function stably? Can qualitative tracking results for such dynamic scenes be provided?\n\n3. To enhance the literature context and highlight research relevance, it is suggested to add references to related work:\n[1] Instance-guided point cloud single object tracking with inception transformer, IEEE TIM 2023;\n[2] Revisiting Siamese-Based 3D Single Object Tracking With a Versatile Transformer, IEEE TPAMI 2025."},"rating":{"value":8},"details_of_ethics_concerns":{"value":"No Ethics Concerns"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":4},"strengths":{"value":"1. Existing methods predominantly employ “category-specific training,” requiring separate models for each category, resulting in poor generalization and low training efficiency. Conversely, few unified category methods (such as CUTrack) suffer from insufficient feature discriminative power and weak motion modeling, leading to performance significantly lagging behind category-specific approaches. PointRePar achieves “unified category training + category-specific SOTA performance” for the first time, resolving the core challenge of balancing “cross-category generalization” with “high-performance tracking.”\n\n2. Spatial modeling breaks traditional frameworks: Combining Mamba (high-efficiency long sequence modeling) with a U-shaped structure to design USRPM, paired with DFA for dynamically adjusting point feature receptive fields — — simultaneously resolving the low feature separability issue in PointNet++ and AdaFormer (validated by Figure 2 t-SNE plot) while outperforming Transformer-based methods in efficiency (36.6 FPS inference speed). This achieves a balance between “multi-scale spatial relationship analysis + low computational overhead.”\n\n3. Temporal Modeling Addresses Fine-Grained & Long-Range Requirements: Point-level modeling captures pixel-level motion features between frames, while bounding-box-level LMTR refines trajectories using historical bounding-box sequences. This dual-level design balances “fine-grained motion details” with “long-range trajectory consistency”; CGP dynamically adjusts noise based on scene sparsity, better matching real-world error patterns than traditional uniform noise (Figure 4 validates sparse scene error patterns), significantly enhancing robustness.\n\n4. Strong Module Synergy: The workflow design—“coarse prediction (lightweight tracker) → spatial feature encoding (USRPM) → temporal relationship analysis (TSM+LMTR) → fine decoding (cross-attention)”—forms a complete “spatio-temporal fusion → error robustness → precise localization” chain, with complementary yet non-redundant module functions.\n\n5. Comprehensive Benchmarking: Performance surpasses category-unified methods (MoCUT, TrackAny3D) by over 15%, while matching category-specific SOTA (SiamMo, MBPTrack), challenging the assumption that category-unified approaches inherently underperform category-specific ones.\n\n6. Thorough ablation studies: Validated the effectiveness of four core modules—DFA, USRPM, LMTR, and CGP (Table 3)—analyzed the impact of hyperparameters (DFA offset points, LMTR sequence length) on performance (Tables 4 and 5), and even supplemented efficiency comparisons between Mamba and self-attention (Table 13), yielding highly credible results."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Insufficient analysis of extreme scenarios: While the paper mentions strong performance in sparse scenarios, it lacks dedicated analysis for extreme cases such as “heavy occlusion (target point cloud < 10 points)” or “rapid viewpoint changes (e.g., sharp vehicle turns).” It also fails to compare behavioral differences with SiamMo in such scenarios, making it difficult to fully support the conclusion that “robustness surpasses category-specific methods.”\n\n2. Lack of core module mechanism visualization: Designs like USRPM's Mamba bidirectional scanning and DFA's dynamic receptive field adjustment are validated solely through performance improvements. Without supplementary feature visualizations (e.g., attention weight maps for different-scale features) or error heatmaps, it remains difficult to intuitively demonstrate “how spatio-temporal relationship analysis enhances tracking accuracy.”"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919500109,"tcdate":1760669674620,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7369/Reviewer_pyGc"],"signatures":["ICLR.cc/2026/Conference/Submission7369/Reviewer_pyGc"],"forum":"DLcnrY5Uqo","number":1,"license":"CC BY 4.0","cdate":1760669674620,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7369/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919500109,"domain":"ICLR.cc/2026/Conference","replyto":"DLcnrY5Uqo","id":"nwe9kpXKc9","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["3D single object tracking","category-unified","point relation parsing"]},"supplementary_material":{"value":"/attachment/24e6be1748f79fdfbc798bc5b15306c0cac6648f.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"3D single object tracking (SOT) remains a highly challenging task due to the inherent crux in learning representations from point clouds to effectively capture both spatial shape features and temporal motion features. Most existing methods employ a category-specific optimization paradigm, training the tracking model individually for each object category to enhance tracking performance, albeit at the expense of generalizability across different categories. In this work, we propose a robust category-unified 3D SOT model, referred to as SpatioTemporal Point Relation Parsing model (*PointRePar*), which is capable of joint training across multiple categories while excelling in unified feature learning for both spatial shapes and temporal motions. Specifically, the proposed *PointRePar* captures and parses the latent point relations across both spatial and temporal domains to learn superior shape and motion characteristics for robust tracking. On the one hand, it models the multi-scale spatial point relations using a Mamba-based U-Net architecture with adaptive point-wise feature refinement. On the other hand, it captures both the point-level and box-level temporal relations to exploit the latent motion features. Extensive experiments across three benchmarks demonstrate that our *PointRePar* not only outperforms the existing category-unified 3D SOT methods significantly, but also compares favorably against the state-of-the-art category-specific methods. Codes will be released."},"_bibtex":{"value":"@inproceedings{\nliu2026pointrepar,\ntitle={PointRePar : SpatioTemporal Point Relation Parsing for Robust Category-Unified 3D Tracking},\nauthor={Juntao Liu and Zikun Zhou and Zhuotao Tian and Guangming Lu and Jun Yu and Wenjie Pei},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=DLcnrY5Uqo}\n}"},"title":{"value":"PointRePar : SpatioTemporal Point Relation Parsing for Robust Category-Unified 3D Tracking"},"pdf":{"value":"/pdf/f1fc8abe2c657c312cae14e03bc1e514184374f0.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"liu|pointrepar_spatiotemporal_point_relation_parsing_for_robust_categoryunified_3d_tracking"},"authorids":{"value":["~Juntao_Liu1","~Zikun_Zhou2","~Zhuotao_Tian1","~Guangming_Lu2","~Jun_Yu1","~Wenjie_Pei1"]},"authors":{"value":["Juntao Liu","Zikun Zhou","Zhuotao Tian","Guangming Lu","Jun Yu","Wenjie Pei"]}},"version":2},{"content":{"summary":{"value":"This paper presents a method for categorizing the oligomeric states of proteins based on their amino acid sequences. The novelty of the research lies in its use of deep learning frameworks, particularly the recently developed Mamba architecture. The authors demonstrates that this approach outperforms other deep learning methods, such as LSTM, RNN, and Transformer models, in terms of accuracy, sensitivity, and computational efficiency."},"strengths":{"value":"The primary objective of this paper appears to be improving computational efficiency in predicting oligomeric states, with a secondary focus on increasing sensitivity in categorization. While the justification for using the Mamba architecture is not clearly articulated, it seems that Mamba has been tested and validated in DNA modeling, a field closely related to protein sequencing. Given that Mamba is a newly emerged framework, testing it's perfomance towards other deep learning architectures represents a valuable contribution to current research on Mamba's potential use cases. The paper thoroughly explores the use of various data augmentation techniques, including ablation studies. It is well-written and adheres to academic standards, introducing the reader to the topic of oligomeric state determination in a clear and concise manner. Additionally, the inclusion of a reference to the codebase used for testing enhances the paper’s transparency and provides deeper insights into its technical aspects."},"weaknesses":{"value":"The literature review mentions the Gradient Boost decision tree-based method FPredX, which reportedly achieves F1 scores of 93.3%, significantly higher than the results presented in this paper. Why is this not addressed in the conclusion? A decision tree approach would also offer superior explainability, a topic that is notably absent from the discussion. While lower computational costs (GPU vs. CPU-based, as mentioned) are advantageous, should this come at the expense of accuracy and sensitivity? Additionally, no direct metric comparison between FPredX and this method is provided, so can we be certain this approach performs better in terms of computational time? Given the brief format of this conference paper, I recommend shortening the somewhat lengthy literature review and focusing more on discussing these critical aspects.\n\nThe paper compares performance to other deep learning frameworks, such as RNN, LSTM, and Transformer, but does not specify which implementations are used. Upon reviewing the code, it appears that standard PyTorch API calls are employed for testing these, but this should be explicitly mentioned in the text.\n\nWhile the paper claims that hyperparameter optimization was performed, it does not detail the method used. Was grid search employed? From what I gather in section 5.3, the smallest Mamba model achieves the best performance... Furthermore, the sensitivity tests in section 5.3 combine various augmentations to evaluate their effects, but only 11 combinations are tested, despite there being 6 different techniques, which could result in 64 possible combinations. The rationale for selecting these 11 baselines is not explained.\n\nAdditionally, the graphs in Figures 3-5 show that the Mamba parameter size stops at 10^8, several orders of magnitude smaller than those of the other methods. Why?\n\nFinally, there are a few minor grammatical issues. Ensure that all acronyms, such as MCC and MAC, are defined. In line 206, the sentence '...overfitting of data, following which a dense layer...' could benefit from rewording. Similarly, in line 264, the phrase '...attempted to reproduce a few test results...' needs clarification for greater precision. Also, in line 147, there is a punctuation mark following a reference, while the sentence continues. Please review for clarity and correctness."},"confidence":{"value":4},"justification":{"value":"The paper regrettably demonstrates no comparison towards the stated methods previously developed by research in this field, forming a major drawback. As such, the paper needs to develop these baselines, preferably before being presented at the conference. Also, some minor flaws were highlighted according to the need for detailing more on experimental setup. However, gaining broader knowledge on the application Mamba as a novel, rather unexplored deep learning framework is advantageous for the community. Moreover, Mamba seems justifiable for protein sequencing, where processing time can be a critical factor, even if the evidence for this was lacking."},"title":{"value":"Review of the paper \"Predicting Oligomeric states of Fluorescent Proteins using Mamba\""},"final_rebuttal_justification":{"value":"Based on the revision and comments from the authours in the process of rebuttal, and the discussions among the reviewers as followed, I belive the paper can be presented at NLDL 2025."},"final_rebuttal_confidence":{"value":3}},"nonreaders":[],"tmdate":1731586640753,"tcdate":1728910223572,"writers":["NLDL.org/2025/Conference","NLDL.org/2025/Conference/Submission50/Reviewer_hT39"],"signatures":["NLDL.org/2025/Conference/Submission50/Reviewer_hT39"],"forum":"KaZzDtUeJY","number":3,"license":"CC BY 4.0","cdate":1728910223572,"readers":["everyone"],"invitations":["NLDL.org/2025/Conference/Submission50/-/Official_Review","NLDL.org/2025/Conference/-/Edit","NLDL.org/2025/Conference/Submission50/Official_Review3/-/Review_Revision"],"mdate":1731586640753,"domain":"NLDL.org/2025/Conference","replyto":"KaZzDtUeJY","id":"VVDQWZhIcJ","forumContent":{"TLDR":{"value":"Use of deep learning model Mamba to predict the Oligomeric state of a protein from its amino acid sequence"},"venue":{"value":"NLDL 2025 Oral"},"keywords":{"value":["Mamba","Fluorescent proteins","Oligomerization state"]},"abstract":{"value":"Fluorescent proteins (FPs) are essential tools in biomedical imaging, known for their ability to absorb and emit light, thereby allowing visualization of biological processes. Understanding the oligomeric state is crucial, as monomeric forms are often preferred in applications to minimize potential artifacts and prevent interference with cellular functions. Experimental methods to find the oligomeric state can be time-consuming and expensive. Most of the current computational model is CPU-based, limiting their speed and scalability. This paper studies the effectiveness of GPU-based deep-learning models in predicting the oligomeric states of fluorescent proteins directly from their amino acid sequences, specifically focusing on the Mamba architecture. Various protein-specific augmentations were also employed to enhance the model's generalizability. Our results indicate that the mamba-based model achieves accuracy and F1 score close to 90\\% and an MCC value of 0.8 with in predicting the oligomeric states of fluorescent proteins directly from its amino acid sequence. The code used in this study is available at [GitHub repository](https://github.com/smlab-niser/FluorMamba)."},"_bibtex":{"value":"@inproceedings{\nrajeev2024predicting,\ntitle={Predicting Oligomeric states of Fluorescent Proteins using Mamba},\nauthor={Agney K Rajeev and Joel Joseph K B and Subhankar Mishra},\nbooktitle={Northern Lights Deep Learning Conference 2025},\nyear={2024},\nurl={https://openreview.net/forum?id=KaZzDtUeJY}\n}"},"title":{"value":"Predicting Oligomeric states of Fluorescent Proteins using Mamba"},"pdf":{"value":"/pdf/80c5dc499c363959a7ab7f01e18b2d69fd297da3.pdf"},"git":{"value":"https://github.com/smlab-niser/FluorMamba"},"venueid":{"value":"NLDL.org/2025/Conference"},"paperhash":{"value":"rajeev|predicting_oligomeric_states_of_fluorescent_proteins_using_mamba"},"authorids":{"value":["~Agney_K_Rajeev1","~Joel_Joseph_K_B2","~Subhankar_Mishra1"]},"authors":{"value":["Agney K Rajeev","Joel Joseph K B","Subhankar Mishra"]}},"version":2},{"content":{"comment":{"value":"We appreciate your constructive feedback.\n\nW1/Q1:\n\nSince Language Modeling (LM) tasks require focusing on important parts of input data, input-adaptive mechanisms like gate mechanisms are known to be suitable for LM tasks.\nTherefore, our experimental results on WikTtext-103, where the baselines achieved performance comparable to ours, are consistent with our statement.\n\nIn contrast, long sequence modeling tasks require pushing gate values towards one to keep previous information regardless of input data and gate mechanisms struggle to memorize tokens under such a situation.\nOur experiments show that the performances of baselines are indeed suboptimal in the copying and the selective copying tasks when the length of the dummy sequence increases from 1000 to 2000.\n\nW2/Q2:\n\nWe would like to clarify that the gate distribution shift occurs caused by the removal of hidden states, and it remains for modern gated linear RNNs such as Mamba and RWKV because they cannot also use previous information through hidden states at the first layer.\nThis suggests that modern gated linear RNNs are still likely to struggle to push gated values close to one.\nThe HGRN2 paper [1] reported that Mamba underperforms compared to linear RNNs such as S4 and S5 on long-sequence modeling tasks.\nWe believe that leveraging our work to modern gated linear RNNs could help identify the underlying causes and improve the ability to capture long-term dependencies.\n\nW3:\n\nOur primary goal is to understand the effect of hidden states on the gate mechanisms, rather than providing state-of-the-art performance on benchmark tasks.\nOur findings can be even more critical for modeling real-world data because they are potentially more complex and large-scale, which makes pushing the gate values towards one more difficult.\nOur experiments on WikiText-103 verifies the versatility of our proposed method, and those on synthetic data show improvement of capturing long-term dependencies.  \nScaling to large models remains future work.\n\nReferences: \n[1] Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. HGRN2: Gated linear RNNs with state expansion. In Proceedings of the Conference on Language Modeling, 2024."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763864850390,"tcdate":1763864850390,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16349/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission16349/Authors"],"forum":"yLA9l9hykz","number":3,"license":"CC BY 4.0","cdate":1763864850390,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16349/-/Official_Comment"],"mdate":1763864850390,"domain":"ICLR.cc/2026/Conference","replyto":"GCOFtnovg1","id":"xMGdLcopbH","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Recurrent Neural Network","Gate mechanism"]},"supplementary_material":{"value":"/attachment/eabf0830918567b555bcceb1949f51417a439308.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Linear Recurrent Neural Networks (RNNs) have attracted attention for their memory and computational efficiency.\nIn particular, gated linear RNNs enable nonlinear transformations through gating mechanisms while still maintaining linear time complexity by removing hidden states from them.\nHowever, the impact of the gate mechanisms and such removal of hidden states from them remains unexplored.\nHere we empirically investigate the impact of these gating mechanisms and find that gate values near zero or one highly depend on hidden states, leading to unintended distribution shifts of gate values when hidden states are removed in gated linear RNNs.\nBased on our findings, we propose an algorithm to mitigate the distribution shifts, which empirically improves performance on long-sequence modeling tasks."},"_bibtex":{"value":"@misc{\niwamori2026towards,\ntitle={Towards Understanding Gated Linear Recurrent Neural Networks},\nauthor={Toshiya Iwamori and Mahito Sugiyama},\nyear={2026},\nurl={https://openreview.net/forum?id=yLA9l9hykz}\n}"},"title":{"value":"Towards Understanding Gated Linear Recurrent Neural Networks"},"pdf":{"value":"/pdf/9b9edb2b974c5293bac768c2ba3eb824740d18c8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"iwamori|towards_understanding_gated_linear_recurrent_neural_networks"},"authorids":{"value":["~Toshiya_Iwamori1","~Mahito_Sugiyama1"]},"authors":{"value":["Toshiya Iwamori","Mahito Sugiyama"]}},"version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Neural Network Pruning","Mamba","Hessian-Based Pruning"]},"_bibtex":{"value":"@inproceedings{\ntuo2026sparsessm,\ntitle={Sparse{SSM}: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot},\nauthor={Kaiwen Tuo and Huan Wang},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=NX0uokLuv7}\n}"},"title":{"value":"SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot"},"paperhash":{"value":"tuo|sparsessm_efficient_selective_structured_state_space_models_can_be_pruned_in_oneshot"},"originally_submitted_PDF":{"value":"/pdf/c019d97e5392d11984e9fe00af4fac21396be1ea.pdf"},"TLDR":{"value":"We introduce an OBS–based pruning methodology tailored to SSM-based architectures such as Mamba, effectively reducing redundancy while preserving model performance."},"primary_area":{"value":"applications->everything_else"},"abstract":{"value":"State-space language models such as Mamba match Transformer quality while permitting linear complexity inference, yet still comprise billions of parameters that hinder deployment. While existing one-shot pruning methods are effective for generic linear and attention blocks, they are not designed with the overall Mamba architecture in mind and fail to account for the time-shared and discretized state-transition matrix at the heart of the selective state-space module (SSM). In this paper, we introduce SparseSSM, the first training-free pruning framework that extends the classic optimal brain surgeon (OBS) framework to state space architectures. Our layer-wise algorithm (i) derives an approximate second-order saliency score that aggregates Hessian-trace information across time steps, (ii) incorporates a component sensitivity analysis to guide feed-forward network (FFN) pruning, which also sheds light on where redundancy resides in mamba architecture, (iii) can be easily extended to semi-structured and structured sparsity, and generalized to other SSM-based architectures. Empirically, we prune 50% of SSM weights without fine-tuning and observe only limited performance degradation, achieving the current state-of-the-art one-shot pruning algorithm for Mamba-based LLMs."},"pdf":{"value":"/pdf/dfb42b0560acd6cb017c71b7a473c4fc27e1e8c4.pdf"},"lay_summary":{"value":"Large language models are powerful, but they are often expensive to run because they contain many parameters. Mamba, a recent alternative to Transformer-based language models, is designed to process long sequences efficiently, yet Mamba models can still be too large for practical deployment. Existing compression methods were mostly built for Transformers and do not account for how Mamba stores and updates information over time.\n\nWe introduce SparseSSM, a method that removes less important parts of Mamba models in one-shot, without extra training. The key idea is to assess the importance of parameters in Mamba’s core modules and identify where redundancy lies within the Mamba architecture, then remove the parts that matter least while protecting more sensitive components.\n\nWe test SparseSSM on Mamba models of multiple sizes and find that it can remove half of the state-space weights with little or no loss in performance. In structured settings, the compressed models also run faster and use less memory. This could make efficient long-sequence language models easier and cheaper to deploy on limited hardware."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Kaiwen_Tuo1","~Huan_Wang3"]},"authors":{"value":["Kaiwen Tuo","Huan Wang"]}},"tmdate":1790067408611,"pdate":1777576706183,"tcdate":1769176337466,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission24916/Authors"],"signatures":["ICML.cc/2026/Conference/Submission24916/Authors"],"forum":"NX0uokLuv7","license":"CC BY 4.0","number":24916,"cdate":1769176337466,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission24916/-/Full_Submission","ICML.cc/2026/Conference/Submission24916/-/Reciprocal_Reviewing_Correction","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission24916/-/Camera_Ready_Revision"],"mdate":1790067408611,"odate":1782341994485,"domain":"ICML.cc/2026/Conference","id":"NX0uokLuv7","version":2},{"content":{"comment":{"value":"Thank you for your insightful follow-up suggestions. We provide our response below.\n\n> *I agree that an explicit and straightforward construction could help further research on the properties of transformers.*\n\nThank you so much for recognizing our contribution. \n\nTo clarify our contribution (simple constructive proof), we have added a discussion of this simple existence proof to Section 1.2.\n\n> *It would be better if the author could prove new results derived from this construction or provide an implementation on real machines to demonstrate the construction*\n\nThank you for the great suggestion! Regarding new results, for example, we believe that our 2-PTM-based framework can also be used to give a simple constructive proof of the Turing completeness of other popular sequence models such as Mamba [TD23]. We choose Transformers in this work because it is the most popular architecture of LLMs. \n\nWe demonstrate our construction via a Python implementation. Our code and results are available at **https://anonymous.4open.science/r/prompting-theory/main.ipynb**\n\nIn the code, we demonstrate the constructed Transformer using two functions `Parity` and `Dyck` (30 examples in total). For example, suppose that we want to compute `Parity(1011110010)`. The prompt for the `Parity` function is:\n```\n^A!++++@ARARA?--@A1ARA1A?+++@ALALALALALA?++@#ARARA0ARA!-----------@A0ALALA?+++@A1A?++@A0ALA?-----------------@$\n```\nThe tokenization of `1011110010` is:\n```\nARARARARARARARARARARARARARARARARARARARARALALA1ALA1ALA1ALALA1ALALA1ALA1ALA1ALA1ALA1ALA1ALA1ALA1ALA1ALALA1ALA1ALA1=--------------------------------------------------------@\n```\nThe output (including CoT) generated by the Transformer is:\n```\n/ARAR=--@ARAR=--@ARAR=--@ARAR=--@ARAR=--@ARAR=--@ARAR=--@ARAR=--@ARAR=--@ARAR/A1ARA1=+++@ALALAL=++@ARARA0AR/A0ALAL/A1=++@AL=-----------------@ALAL=++@ARARA0AR/A0ALAL=+++@A0AL=-----------------@ALAL=++@ARARA0AR=-----------@ALALALALAL=++@ARARA0AR=-----------@ALALALALAL=++@ARARA0AR=-----------@ALALALALAL=++@ARARA0AR/A0ALAL=+++@A0AL=-----------------@ALAL=++@ARARA0AR=-----------@ALALALALAL=++@ARARA0AR/A0ALAL=+++@A0AL=-----------------@ALAL=++@ARARA0AR=-----------@ALALALALAL=++@ARARA0AR=-----------@ALALALALAL/:1$\n```\nWe can see that the generated answer is indeed `1` (the part between `:` and `$` at the end of the output), which is the correct answer. \n\nWe will add this experiment to Appendix D.\n\nReference:\n\n- [TD23] Mamba: Linear-time sequence modeling with selective state spaces."},"title":{"value":"Follow-up response"}},"tmdate":1733191032040,"tcdate":1732781444653,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission7155/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission7155/Authors"],"forum":"AS8SPTyBgw","number":23,"license":"CC BY 4.0","cdate":1732781444653,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission7155/-/Official_Comment"],"mdate":1733191032040,"domain":"ICLR.cc/2025/Conference","replyto":"JIufymH0YV","id":"HWVI2j0WGk","forumContent":{"TLDR":{"value":"We show that prompting is Turing-complete: there exists a finite-size Transformer such that for any computable function, there exists a corresponding prompt following which the Transformer computes the function."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Prompting","Large Language Model","Transformer","Expressive Power"]},"primary_area":{"value":"learning theory"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Since the success of GPT, large language models (LLMs) have revolutionized machine learning and have initiated the so-called *LLM prompting* paradigm. In the era of LLMs, people train a single general-purpose LLM and provide the LLM with different *prompts* to perform different tasks. However, such empirical success largely lacks theoretical understanding. Here, we present the first theoretical study on the LLM prompting paradigm to the best of our knowledge. In this work, we show that prompting is in fact Turing-complete: there exists a finite-size Transformer such that for any computable function, there exists a corresponding prompt following which the Transformer computes the function. Furthermore, we show that even though we use only a single finite-size Transformer, it can still achieve nearly the same complexity bounds as that of the class of all unbounded-size Transformers. Overall, our result reveals that prompting can enable a single finite-size Transformer to be efficiently universal, which establishes a theoretical underpinning for prompt engineering in practice."},"_bibtex":{"value":"@inproceedings{\nqiu2025ask,\ntitle={Ask, and it shall be given: On the Turing completeness of prompting},\nauthor={Ruizhong Qiu and Zhe Xu and Wenxuan Bao and Hanghang Tong},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=AS8SPTyBgw}\n}"},"title":{"value":"Ask, and it shall be given: On the Turing completeness of prompting"},"pdf":{"value":"/pdf/a0f25cd20810f151772f33ee9f97f108f2849afb.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"qiu|ask_and_it_shall_be_given_on_the_turing_completeness_of_prompting"},"authorids":{"value":["~Ruizhong_Qiu1","~Zhe_Xu5","~Wenxuan_Bao1","~Hanghang_Tong3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Ruizhong Qiu","Zhe Xu","Wenxuan Bao","Hanghang Tong"]}},"version":2},{"content":{"venue":{"value":"IEEE Access 2026"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/6287639/11323511/11428172.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"jeon|efficient_and_robust_temporal_modeling_for_lgpr_odometry_with_mamba"},"html":{"value":"https://doi.org/10.1109/ACCESS.2026.3672677"},"_bibtex":{"value":"@article{DBLP:journals/access/JeonLL26,\n  author={Minje Jeon and Hyeonbeen Lee and Jangho Lee},\n  title={Efficient and Robust Temporal Modeling for LGPR Odometry With Mamba},\n  year={2026},\n  cdate={1767225600000},\n  journal={IEEE Access},\n  volume={14},\n  pages={38775-38792},\n  url={https://doi.org/10.1109/ACCESS.2026.3672677}\n}\n"},"abstract":{"value":"Efficient and robust odometry in GPS-denied environments is essential for autonomous navigation, yet conventional sensors such as cameras and LiDAR, as well as Global Navigation Satellite System (GNSS) receivers, suffer from performance degradation under adverse weather, illumination changes, or surface-level variability. Localizing Ground-Penetrating Radar (LGPR) provides complementary robustness by leveraging stable subsurface reflections, but learning from long, noisy B-scan sequences remains challenging. We present the first systematic benchmark of a selective state-space model, Mamba, for LGPR odometry. We build a unified spatio-temporal pipeline with a ResNet-18 radargram encoder and interchangeable temporal backbones, including Linear, LSTM, Transformer, and SSM-based models, and evaluate them under a controlled protocol that fixes preprocessing, spatial encoding, and data splits. On the CMU-GPR dataset, Mamba achieves top-tier displacement accuracy (RMSE 0.06 m,  $R^{2} \\approx ~0.94$ ) while remaining efficient for long-context modeling and embedded deployment. We also observe that domain-specific encoder training is critical for this modality. These results establish a reproducible baseline for subsurface odometry and motivate broader cross-dataset validation. The code will be made available upon publication."},"title":{"value":"Efficient and Robust Temporal Modeling for LGPR Odometry With Mamba"},"authors":{"value":[{"fullname":"Minje Jeon","username":""},{"fullname":"Hyeonbeen Lee","username":""},{"fullname":"Jangho Lee","username":"~Jangho_Lee1"}]}},"tmdate":1791882456465,"pdate":1798675200000,"externalIds":["dblp:journals/access/JeonLL26"],"tcdate":1780154111267,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jangho_Lee1"],"forum":"trYnN83SYo","license":"CC BY-SA 4.0","number":40533,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Author_Removal","OpenReview.net/Public_Article/-/Authorship_Claim"],"mdate":1791882456465,"domain":"OpenReview.net/Public_Article","id":"trYnN83SYo","version":2},{"content":{"summary":{"value":"This paper proposes a new method for temporal knowledge graph reasoning, aiming to address the shortcomings of existing methods in capturing global semantics and long-range dependencies, especially in the prediction tasks of sparse or unseen facts. The paper presents the DSEE-MDiff framework, which includes three core modules: Dynamic-Static Entity Selection Encoder, Mamba-Driven Diffusion Module, and ConvTransE Decoder. Their main contributions are: for the first time, combining a dynamic-static selection encoder with a Mamba-enhanced diffusion model for TKG reasoning, and designing a dynamic-static selection mechanism to adaptively fuse global semantic and local structural information. Their experimental results achieved the competitive performance on four public datasets, with significant improvements especially in the MRR and Hits@1 metrics."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please refer to Weaknesses above."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Integrates dynamic-static semantics with the Mamba diffusion model, leveraging the selective state space mechanism of the Mamba diffusion model to more effectively retain key information in long sequences.\n2. The Mamba model, which is inherently good at capturing dependencies in long sequences during the denoising process, is used to solve the long-distance dependency problem in TKG reasoning, making up for the defect of information attenuation in traditional RNN/Transformer when processing long historical windows.\n3. Diffusion models essentially learn the generative process of data distribution by injecting and removing noise. This mechanism can \"create\" reasonable representations of facts that have not been seen during training, greatly enhancing the ability to generalize and reason about sparse and unseen facts."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"​\t1. How does the dynamic entity selection encoder implement dynamic encoding? When selecting multi-hop neighbors and multi-hop relationships, are the number of hops and the number of relationships fixed?\n\n​\t2. The function of the dynamic entity selection encoder to highlight informative entity signals requires further explanation.\n\n​\t3. What does the frequency signal in Formula 8 refer to?\n\n​\t4. In line 203, \"d\" does not appear in the previous text.\n\n​\t5. Why were other newer baseline models not chosen for comparison? For example, \"DPCL-Diff: The Temporal Knowledge Graph Reasoning based on Graph Node Diffusion Model with Dual-Domain Periodic Contrastive Learning\" and \"Temporal Knowledge Graph Extrapolation via Causal Subhistory Identification\".\n\n​\t6. When evaluating DSEE-MDiff's ability to handle uncertainties arising from rare or unprecedented facts, what are the statistics of the training and test sets of ICEWS14 and ICEWS18, and how were they constructed? Without the proportion of unseen factual data, it cannot be stated that \"under the higher uncertainty of sparse settings, DSEE-MDiff shows greater robustness.\"\n\n​\t7. In Section 5.5, only comparisons with DiffuTKG and RE-GCN are made. Some newer models should be selected for comparison, such as DPCL-Diff.\n\n​\t8. Lack of analysis on the time cost of model training.\n\n​\t9. The effect of DSEE-MDiff on GDELT is not obvious, and there is a lack of in-depth analysis of the experimental results.\n\n​\t10. The motivation for introducing Diffusion and Mamba to temporal knowledge graph reasoning (TKGR) is not sufficiently deep, and the distinction from existing Diffusion TKGR methods is unclear."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916348818,"tcdate":1761896267289,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2729/Reviewer_cYVx"],"signatures":["ICLR.cc/2026/Conference/Submission2729/Reviewer_cYVx"],"forum":"gWyYeO0LWZ","number":4,"license":"CC BY 4.0","cdate":1761896267289,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2729/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916348818,"domain":"ICLR.cc/2026/Conference","replyto":"gWyYeO0LWZ","id":"ZnWPE35pkV","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Temporal knowledge graph","Temporal knowledge graph reasoning","Selective State Space Model","Diffusion"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Temporal Knowledge Graph (TKG) reasoning aims to predict future missing facts based on historical evidence. Prior studies on graph learning and logical rules often overlook global latent semantics and struggle with long-range dependencies, particularly under sparse or unseen facts. To address these limitations, we propose DSEE-MDiff, which frames TKG reasoning as selecting informative history and denoising future signals. Specifically, a Dynamic–Static Entity Selection encoder captures global semantic evolution alongside local structural cues, while a Mamba-based diffusion module injects and removes noise with a selective state-space model to better recover long-range dependencies and mitigate sparsity. The two outputs are fused for prediction through a ConvTransE decoder. Experiments on four public datasets demonstrate that DSEE-MDiff achieves state-of-the-art performance across multiple metrics, validating the effectiveness of the proposed approach."},"_bibtex":{"value":"@misc{\ngeng2026dynamicstatic,\ntitle={Dynamic{\\textendash}Static Representation Learning with Mamba-Enhanced Diffusion for Temporal Knowledge Graph Reasoning},\nauthor={Rushan Geng and Cuicui Luo},\nyear={2026},\nurl={https://openreview.net/forum?id=gWyYeO0LWZ}\n}"},"title":{"value":"Dynamic–Static Representation Learning with Mamba-Enhanced Diffusion for Temporal Knowledge Graph Reasoning"},"pdf":{"value":"/pdf/00bcae00b45c2be35021b53c192d4691538865ca.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"geng|dynamicstatic_representation_learning_with_mambaenhanced_diffusion_for_temporal_knowledge_graph_reasoning"},"authorids":{"value":["~Rushan_Geng1","~Cuicui_Luo1"]},"authors":{"value":["Rushan Geng","Cuicui Luo"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2410.15091v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"xiao|spatialmamba_effective_visual_state_space_models_via_structureaware_state_fusion"},"html":{"value":"https://doi.org/10.48550/arXiv.2410.15091"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-15091,\n  publtype={informal},\n  author={Chaodong Xiao and Minghan Li and Zhengqiang Zhang and Deyu Meng and Lei Zhang},\n  title={Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.15091},\n  url={https://doi.org/10.48550/arXiv.2410.15091}\n}\n"},"abstract":{"value":"Selective state space models (SSMs), such as Mamba, highly excel at capturing long-range dependencies in 1D sequential data, while their applications to 2D vision tasks still face challenges. Current visual SSMs often convert images into 1D sequences and employ various scanning patterns to incorporate local spatial dependencies. However, these methods are limited in effectively capturing the complex image spatial structures and the increased computational cost caused by the lengthened scanning paths. To address these limitations, we propose Spatial-Mamba, a novel approach that establishes neighborhood connectivity directly in the state space. Instead of relying solely on sequential state transitions, we introduce a structure-aware state fusion equation, which leverages dilated convolutions to capture image spatial structural dependencies, significantly enhancing the flow of visual contextual information. Spatial-Mamba proceeds in three stages: initial state computation in a unidirectional scan, spatial context acquisition through structure-aware state fusion, and final state computation using the observation equation. Our theoretical analysis shows that Spatial-Mamba unifies the original Mamba and linear attention under the same matrix multiplication framework, providing a deeper understanding of our method. Experimental results demonstrate that Spatial-Mamba, even with a single scan, attains or surpasses the state-of-the-art SSM-based models in image classification, detection and segmentation. Source codes and trained models can be found at https://github.com/EdwardChasel/Spatial-Mamba."},"title":{"value":"Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion"},"authors":{"value":[{"fullname":"Chaodong Xiao","username":""},{"fullname":"Minghan Li","username":""},{"fullname":"Zhengqiang Zhang","username":""},{"fullname":"Deyu Meng","username":""},{"fullname":"Lei Zhang","username":""}]}},"tmdate":1779771696642,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2410-15091"],"tcdate":1779771588042,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~ChaoDong_Xiao1"],"forum":"DABLE1x8rq","license":"CC BY-SA 4.0","number":26205,"cdate":1704067200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Author_Removal"],"mdate":1779771696642,"domain":"OpenReview.net/Public_Article","id":"DABLE1x8rq","version":2},{"content":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Representation learning","Visual state space models","Structure-aware state fusion"]},"supplementary_material":{"value":"/attachment/bfe4420ea6a429fc96daffcd3ec6b05d08ace381.pdf"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSMs), such as Mamba, highly excel at capturing long-range dependencies in 1D sequential data, while their applications to 2D vision tasks still face challenges. Current visual SSMs often convert images into 1D sequences and employ various scanning patterns to incorporate local spatial dependencies. However, these methods are limited in effectively capturing the complex image spatial structures and the increased computational cost caused by the lengthened scanning paths. To address these limitations, we propose Spatial-Mamba, a novel approach that establishes neighborhood connectivity directly in the state space. Instead of relying solely on sequential state transitions, we introduce a structure-aware state fusion equation, which leverages dilated convolutions to capture image spatial structural dependencies, significantly enhancing the flow of visual contextual information. Spatial-Mamba proceeds in three stages: initial state computation in a unidirectional scan, spatial context acquisition through structure-aware state fusion, and final state computation using the observation equation. Our theoretical analysis shows that Spatial-Mamba unifies the original Mamba and linear attention under the same matrix multiplication framework, providing a deeper understanding of our method. Experimental results demonstrate that Spatial-Mamba, even with a single scan, attains or surpasses the state-of-the-art SSM-based models in image classification, detection and segmentation. Source codes and trained models can be found at \\url{ https://github.com/EdwardChasel/Spatial-Mamba }."},"_bibtex":{"value":"@inproceedings{\nxiao2025spatialmamba,\ntitle={Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion},\nauthor={Chaodong Xiao and Minghan Li and Zhengqiang ZHANG and Deyu Meng and Lei Zhang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=iDe1mtxqK5}\n}"},"title":{"value":"Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion"},"pdf":{"value":"/pdf/8219b6919d5e23d90a43ddcdec43ba242eac39c4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"xiao|spatialmamba_effective_visual_state_space_models_via_structureaware_state_fusion"},"authorids":{"value":["~Chaodong_Xiao2","~Minghan_Li1","~Zhengqiang_ZHANG1","~Deyu_Meng1","~Lei_Zhang2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chaodong Xiao","Minghan Li","Zhengqiang ZHANG","Deyu Meng","Lei Zhang"]}},"tmdate":1740544573862,"pdate":1737562286842,"tcdate":1727025939716,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2673/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission2673/Authors"],"forum":"iDe1mtxqK5","license":"CC BY 4.0","number":2673,"cdate":1727025939716,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission2673/-/Full_Submission","ICLR.cc/2025/Conference/Submission2673/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit","ICLR.cc/2025/Conference/Submission2673/-/Camera_Ready_Revision"],"mdate":1740544573862,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"iDe1mtxqK5","version":2},{"content":{"summary":{"value":"This paper addresses a key limitation in linear attention models, termed global context collapse, where the model’s representational diversity diminishes as sequence length increases.\nTo overcome this, the authors propose Multi-Head Linear Attention (MHLA), which partitions the token sequence into multiple blocks and computes local key–value summaries within each.\nBy introducing a multi-head mixing mechanism that adaptively combines these local summaries, MHLA restores query-conditioned diversity while maintaining linear computational complexity.\nExtensive experiments demonstrate that MHLA achieves significant improvements over existing linear attention baselines in image classification (ImageNet-1K), image generation (DiT and SANA models), and NLP benchmarks, with minimal computational overhead."},"soundness":{"value":3},"confidence":{"value":2},"questions":{"value":"See weakness"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Insightful analysis of linear attention – The authors conduct a rigorous examination of why linear attention underperforms, identifying global context collapse as the root cause. Their analysis of rank deficiency and entropy in attention maps provides a strong theoretical foundation.\n\n2. Innovative multi-head design – MHLA introduces a token-level multi-head mechanism that mixes local key–value summaries through learnable coefficients. This design cleverly restores query-dependent diversity and sparsity while keeping linear complexity.\n\n3. Strong empirical results – On ImageNet classification, MHLA improves accuracy by 3.6% over standard linear attention, achieving state-of-the-art results when integrated into DeiT and VLT architectures. On image generation, it delivers up to 12.6% gains over baselines while matching or surpassing softmax attention performance in large-scale DiT models. On NLP tasks, MHLA yields 2.1% improvements over existing linear attention methods, even without additional positional embeddings."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"### 1. Limited and Inconclusive Experimental Evaluation\nThe major limitation of this paper lies in its lack of comprehensive experiments to support its central claim—improving long-term modeling capability.\nIn the vision domain, experiments are restricted to image classification on 224×224 inputs, which are too short and fail to reflect long-range dependencies.\nIn the NLP domain, the evaluation covers only three small-scale reasoning benchmarks (ARC-c, WinoGrande, CoPA), which do not assess long-context understanding, open-ended text generation, or large-scale language modeling.\nConsequently, it remains unclear whether MHLA truly benefits long-sequence scaling.\nMoreover, the NLP model used (340 M parameters trained on 5 B tokens) is relatively small; while suitable for controlled experiments, it is insufficient to demonstrate MHLA’s effectiveness in large-scale language models, where long-context efficiency becomes critical.\nTo better validate the method’s claims, I recommend incorporating:\n\nVision tasks such as image captioning, VQA, or video captioning, which require multi-modal and temporal reasoning.\n\nNLP datasets such as LongBench, PG19, or BookSum, which directly evaluate scalability and diversity preservation in long-sequence scenarios.\n\n### 2. Incomplete Discussion of the Computational Trade-off\nThe proposed method essentially trades extra computation for improved representational rank.\nThis trade-off is reasonable when the additional $O(M^2d^2)$ cost remains small, but it does not fundamentally solve the scalability problem for very long input sequences. Linear attention mechanisms are expected to handle arbitrarily long contexts, yet MHLA’s computational complexity scales with block size $M$, limiting its practicality for extremely long sequences.\nThe paper should more clearly analyze this scaling limitation and discuss how MHLA behaves as sequence length increases.\n\n### 3. No Throughput or Speed Analysis\nThe paper does not report throughput, latency, or efficiency comparisons.\nAlthough MHLA maintains linear complexity in theory, the method introduces an additional $O(M^2d^2)$ term due to multi-head block mixing.\nThis overhead may be negligible in small-scale settings like image classification or ARC-c, but it could become significant in truly long-sequence scenarios.\nWithout explicit runtime or memory benchmarks, the paper fails to demonstrate that MHLA preserves an effective balance between accuracy and computational cost under long-context conditions.\n\n### 4. Weak Literature Review and Terminology Issues\nThe paper’s literature review is notably shallow.\nThe introduction and related work sections lack a comprehensive discussion of the extensive prior research on linear attention, a highly active field with numerous variants and analyses.\nThe authors should consult and reference existing survey papers on linear attention to position MHLA more accurately within the broader landscape.\nAdditionally, the writing style introduces several non-standard or self-coined terms—such as “adapt to each query individually” and “KV summary”—which are not widely adopted in the community.\nThe authors are encouraged to revise these descriptions using standard terminology and provide clearer, community-recognized definitions to improve readability and academic rigor."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360429838,"tcdate":1761783435613,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1352/Reviewer_2bkY"],"signatures":["ICLR.cc/2026/Conference/Submission1352/Reviewer_2bkY"],"forum":"340QjF3jJP","number":1,"license":"CC BY 4.0","cdate":1761783435613,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1352/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360429838,"domain":"ICLR.cc/2026/Conference","replyto":"340QjF3jJP","id":"gtTUCQPH6w","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"Multi-Head Linear Attention addresses the performance degradation in linear attention by preserving representational diversity through head-wise token dimension computation."},"keywords":{"value":["Linear Attention","Model Architecture","Efficiency"]},"primary_area":{"value":"generative models"},"abstract":{"value":"While the Transformer architecture dominates many fields, its quadratic self-attention complexity hinders its use in large-scale applications. **Linear attention** offers an efficient alternative, but its direct application often degrades performance, with existing fixes typically re-introducing computational overhead through extra modules (e.g., depthwise separable convolution and few self-attention blocks) that defeat the original purpose. In this work, we identify a key failure mode in these methods: **global context collapse**, where the model loses representational diversity. To address this, we propose **Multi-Head Linear Attention (MHLA)**, which preserves this diversity by computing attention within divided heads along the token dimension. We prove that MHLA maintains linear complexity while recovering much of the expressive power of softmax attention, and verify its effectiveness across multiple domains, achieving a **3.6%** improvement on ImageNet classification, a **6.3%** gain on NLP, a **12.6%** improvement in image generation tasks and a **41%** enhancement in video generation tasks with the same computational complexity."},"_bibtex":{"value":"@inproceedings{\nzhang2026mhla,\ntitle={{MHLA}: Restoring Expressivity of Linear Attention via Token-Level Multi-Head},\nauthor={Kewei Zhang and Ye Huang and Yufan Deng and Jincheng YU and Junsong Chen and Huan Ling and Enze Xie and Daquan Zhou},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=340QjF3jJP}\n}"},"title":{"value":"MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head"},"pdf":{"value":"/pdf/cd6bb3047f1271e4adc285113e13ed1cfccb2109.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|mhla_restoring_expressivity_of_linear_attention_via_tokenlevel_multihead"},"authorids":{"value":["~Kewei_Zhang2","~Ye_Huang4","~Yufan_Deng4","~Jincheng_YU1","~Junsong_Chen1","~Huan_Ling1","~Enze_Xie1","~Daquan_Zhou1"]},"authors":{"value":["Kewei Zhang","Ye Huang","Yufan Deng","Jincheng YU","Junsong Chen","Huan Ling","Enze Xie","Daquan Zhou"]}},"version":2},{"content":{"comment":{"value":"**Regarding Weakness 3 (Connection between Theoretical properties and Model Design):** Thank you for the question. In our response to Weakness 2 we also touched on that aspect. Namely, the learning mechanism of the ARMA coefficient includes a normalization term that is influenced by the theoretical understanding, to maintain stability and allow long-range propagation. This was discussed in Section 3.2, but we now better highlighted the connection between this subsection, and the theoretical properties shown in Section 4. Thank you. \n\n**Regarding Weakness 4 (GRAMA vs State-of-the-art in terms of performance and computational efficiency):** We appreciate your question. In our paper, we included extensive comparisons between GRAMA and recent state-of-the-art methods, both in Section 5 and the Appendix. For your convenience, we now provide a Table that summarizes the results on all datasets from the main paper, and compared it with the best performing method. As can be seen, in most cases, our GRAMA offers better performance, and similar in others. We also added the table to our revised paper for added clarity. Thank you.\n\n\n| Model / Dataset                         | Diameter ($log_{10}(MSE)$↓) | SSSP ($log_{10}(MSE)$↓) | Eccentricity ($log_{10}(MSE)$↓) | Peptides-func (AP ↑) | Peptides-struct (MAE ↓) | MalNet-Tiny (Acc ↑) | Roman-empire (Acc ↑) | Amazon-ratings (Acc ↑) | Minesweeper (AUC ↑) | Tolokers (AUC ↑) | Questions (AUC ↑) |\n|-----------------------------------------|-----------------------------|--------------------------|---------------------------------|----------------------|-------------------------|---------------------|----------------------|-----------------------|---------------------|------------------|------------------|\n| Best baseline out of all methods in Tables | -0.5981 ± 0.1145            | -3.599 ± 0.1949         | -0.0739 ± 0.2190               | **71.50 ± 0.44**     | 0.2478 ± 0.0016         | 94.22 ± 0.24        | 91.37 ± 0.35         | **54.17 ± 0.37**      | 97.31 ± 0.41       | 84.52 ± 0.21     | 80.02 ± 0.86     |\n| GRAMA (Best performing model out of 3 variants) | **-0.8663 ± 0.0514**       | **-4.1289 ± 0.0988**    | **-1.3012 ± 0.1258**           | 70.93 ± 0.78         | **0.2436 ± 0.0022**     | **94.37 ± 0.36**    | **91.82 ± 0.39**     | 53.71 ± 0.57         | **98.33 ± 0.55**    | **86.23 ± 1.10** | **80.47 ± 1.09** |\n\n\n\nSecondly,  we understand that the Reviewer would like to also see a comparison in terms of time and downstream performance with recent state-of-the-art methods, and to address your question, we now added a Table, below, that compares the required runtime (training and inference) of our GRAMA and other models, as well as the obtained downstream performance. As can be seen, our GRAMA requires similar time to other methods like GPS+Mamba and GMN, which are also selective models, while requiring significantly less time than GPS. For reference, we also provide the training runtimes and accuracy of GCN as a baseline. We believe that this comparison further highlights the applicability of GRAMA, and the results were added to the revised paper in the appendix. Thank you. \n\n| **Model**                    | **Training runtime (ms)** | **Accuracy (%)**   |\n|------------------------------|---------------------------|--------------------|\n| GatedGCN                     | 18.38                    | 73.69 ± 0.74       |\n| GPS                          | 1139.05                  | 82.00 ± 0.61       |\n| GPS + Mamba [R2]             | 320.39                   | 83.10 ± 0.28       |\n| GMN [R2]                     | 387.04                   | 87.69 ± 0.50       |\n| GRAMA (Non-selective, Ours)  | 249.68                   | 85.13 ± 0.36       |\n| GRAMA (Ours)                 | 362.41                   | 88.61 ± 0.43       |\n\n\n\n**References:**\n\n\n[R1] Wang C., et al. (2024). Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint arXiv:2402.00789.\n\n[R2] Behrouz A., and Hashemi A. (2024). Graph mamba: Towards learning on graphs with state space models. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.\n\n[R3] Huang Y., Miao S., Li P.\t(2024).\tWhat Can We Learn from State Space Models for Machine Learning on Graphs?. arXiv preprint arXiv:2406.05815.\n\n[R4] Gu A., Goel K., Ré  C. (2022).\tEfficiently Modeling Long Sequences with Structured State Spaces. International Conference on Learning Representations (ICLR).\n\n[R5] Dao T., Gu A. (2024).\tTransformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. International Conference on Machine Learning (ICML)."},"title":{"value":"Rebuttal Part 2"}},"tmdate":1732189519660,"tcdate":1732189519660,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6789/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission6789/Authors"],"forum":"ZuOXuS7yDw","number":16,"license":"CC BY 4.0","cdate":1732189519660,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6789/-/Official_Comment"],"mdate":1732189519660,"domain":"ICLR.cc/2025/Conference","replyto":"LbOnFdIDU1","id":"RTjoJMt5ew","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We introduce a selective graph neural ARMA-based method, drawing links to Graph SSMs, to address oversquashing in graphs."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Graph Neural Networks","Auto-regressive Moving Average","State Space Models"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Graph State Space Models (SSMs) have recently been introduced to enhance Graph Neural Networks (GNNs) in modeling long-range interactions. Despite their success, existing methods either compromise on permutation equivariance or limit their focus to pairwise interactions rather than sequences. Building on the connection between Autoregressive Moving Average (ARMA)  and SSM, in this paper, we introduce GRAMA, a Graph Adaptive method based on a learnable Autoregressive Moving Average (ARMA) framework that addresses these limitations. By transforming from static to sequential graph data, GRAMA leverages the strengths of the ARMA framework, while preserving permutation equivariance. Moreover, GRAMA incorporates a selective attention mechanism for dynamic learning of ARMA coefficients, enabling efficient and flexible long-range information propagation. We also establish theoretical connections between GRAMA and Selective SSMs, providing insights into its ability to capture long-range dependencies. Extensive experiments on 14 synthetic and real-world datasets demonstrate that GRAMA consistently outperforms backbone models and performs competitively with state-of-the-art methods."},"_bibtex":{"value":"@misc{\neliasof2025grama,\ntitle={{GRAMA}: Adaptive Graph Autoregressive Moving Average Models},\nauthor={Moshe Eliasof and Alessio Gravina and Andrea Ceni and Claudio Gallicchio and Davide Bacciu and Carola-Bibiane Sch{\\\"o}nlieb},\nyear={2025},\nurl={https://openreview.net/forum?id=ZuOXuS7yDw}\n}"},"title":{"value":"GRAMA: Adaptive Graph Autoregressive Moving Average Models"},"pdf":{"value":"/pdf/3e53920f196b3400350b0e071a686ed9f4810b50.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"eliasof|grama_adaptive_graph_autoregressive_moving_average_models"},"authorids":{"value":["~Moshe_Eliasof1","~Alessio_Gravina1","~Andrea_Ceni1","~Claudio_Gallicchio1","~Davide_Bacciu1","~Carola-Bibiane_Schönlieb1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Moshe Eliasof","Alessio Gravina","Andrea Ceni","Claudio Gallicchio","Davide Bacciu","Carola-Bibiane Schönlieb"]}},"version":2},{"content":{"venue":{"value":"IEEE Trans. Consumer Electron. 2025"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/30/11128999/10839254.pdf"},"venueid":{"value":"dblp.org/journals/TCE/2025"},"paperhash":{"value":"chen|selective_state_space_model_for_monaural_speech_enhancement"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Moran_Chen:","https://dblp.org/search/pid/api?q=author:Qiquan_Zhang:","https://dblp.org/search/pid/api?q=author:Mingjiang_Wang:","https://dblp.org/search/pid/api?q=author:Xiangyu_Zhang_0005:","~Hexin_Liu1","https://dblp.org/search/pid/api?q=author:Eliathamby_Ambikairajah:","https://dblp.org/search/pid/api?q=author:Deying_Chen:"]},"html":{"value":"https://doi.org/10.1109/TCE.2024.3523297"},"_bibtex":{"value":"@article{DBLP:journals/tce/ChenZWZLAC25,\n  author={Moran Chen and Qiquan Zhang and Mingjiang Wang and Xiangyu Zhang and Hexin Liu and Eliathamby Ambikairajah and Deying Chen},\n  title={Selective State Space Model for Monaural Speech Enhancement},\n  year={2025},\n  month={May},\n  cdate={1746057600000},\n  journal={IEEE Trans. Consumer Electron.},\n  volume={71},\n  number={2},\n  pages={5414-5424},\n  url={https://doi.org/10.1109/TCE.2024.3523297}\n}\n"},"abstract":{"value":"Voice user interfaces (VUIs) have facilitated the efficient interactions between humans and machines through spoken commands. Since real-word acoustic scenes are complex, speech enhancement plays a critical role for robust VUI. Transformer and its variants, such as Conformer, have demonstrated cutting-edge results in speech enhancement. However, both of them suffers from the quadratic computational complexity with respect to the sequence length, which hampers their ability to handle long sequences. Recently a novel State Space Model called Mamba, which shows strong capability to handle long sequences with linear complexity, offers a solution to address this challenge. In this paper, we propose a novel hybrid convolution-Mamba backbone, denoted as MambaDC, for speech enhancement. Our MambaDC marries the benefits of convolutional networks to model the local interactions and Mamba’s ability for modeling long-range global dependencies. We conduct comprehensive experiments within both basic and state-of-the-art (SoTA) speech enhancement frameworks, on two commonly used training targets. The results demonstrate that MambaDC outperforms Transformer, Conformer, and the standard Mamba across all training targets. Built upon the current advanced framework, the use of MambaDC backbone showcases superior results compared to existing SoTA systems. This sets the stage for efficient long-range global modeling in speech enhancement."},"title":{"value":"Selective State Space Model for Monaural Speech Enhancement"},"authors":{"value":["Moran Chen","Qiquan Zhang","Mingjiang Wang","Xiangyu Zhang","Hexin Liu","Eliathamby Ambikairajah","Deying Chen"]}},"tmdate":1767754713367,"pdate":1735689600000,"externalIds":["dblp:journals/tce/ChenZWZLAC25"],"tcdate":1767754711210,"writers":["~"],"signatures":["~Hexin_Liu1"],"forum":"sjZwyWpnC0","license":"CC BY-SA 4.0","number":723295,"cdate":1746057600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1767754713367,"domain":"DBLP.org","id":"sjZwyWpnC0","version":2},{"content":{"summary":{"value":"This paper studies Mamba, a state-space model (SSM) that has previously been introduced as an efficient alternative to Transformers. The authors analyze Mamba's stability in mixed-precision and parameter-efficient fine-tuning by applying Lyapunov exponents, theoretically and empirically confirming its resilience against input perturbations. Experiments show that Mamba maintains stability better than comparable Transformers under mixed precision and achieves notable memory and computation efficiency. Additionally, instruction tuning enhances Mamba's in-context learning, suggesting that SSMs can approach or surpass Transformer models in certain few-shot tasks."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"See weakness."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"This work provides promising evidence that fine-tuning Mamba on instruction-tuning data can improve its in-context learning abilities."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Line 51:  For some training framework it does not keep master weight in fp32 which problematic for Mamba. But Mamba was still trained with mixed precision using AMP.\n\n- Theory 1 seems shallow: Isn't Lemma 1 trivial to derive? \\( dx_{t+1}/dx_t \\) directly represents the decay rate, which must be \\(\\leq 1\\) to prevent exponential growth in cumulative products. This insight is obvious for any recurrent model, making Theorem 1’s assertion about exponential decay rather redundant.\n\n- Theory 2 seems unnecessary: It’s clear that concatenating multiple weight matrices with LoRA applied to the larger matrix creates dependencies. However, there's no practical advantage demonstrated here, nor is there an empirical basis for this approach.\n\n- Line 274: Why use MMLU as a proxy metric? For models at this scale, MMLU scores are generally around random (i.e., ~25%), leading to limited interpretability. Additionally, only the differential score is provided, without the original MMLU results, which weakens MMLU’s reliability as a metric in this context.\n\n- Instruction tuning experiments: Pythia feels like a weak baseline. And is it plausible to claim Mamba has an “emergent ability” (line 439) for in-context learning after instruction tuning, given SSMs' inherent limitations, as noted in [1,2]? Figure 3 also shows a performance drop for Mamba2 as shot count rises, likely due to state-space models’ well-known difficulties with long-sequence processing and in-context learning.\n\n[1] https://arxiv.org/abs/2402.18510\n\n[2] https://arxiv.org/abs/2402.18668"}},"nonreaders":[],"tmdate":1732744910723,"tcdate":1730650417797,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9040/Reviewer_jrwp"],"signatures":["ICLR.cc/2025/Conference/Submission9040/Reviewer_jrwp"],"forum":"i9RTCC6whL","number":1,"license":"CC BY 4.0","cdate":1730650417797,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9040/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732744910723,"domain":"ICLR.cc/2025/Conference","replyto":"i9RTCC6whL","id":"MaDMWExHrR","forumContent":{"TLDR":{"value":"Fine-tuning Mamba LLMs is theoretically stable, and much more stable than Transformer LLMs in practice."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Mamba SSMs","SSMs","LLMs","PEFT","LoRA","Transformers"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba state-space models (SSMs) were recently shown to outperform state-of-the-art (SOTA) Transformer large language models (LLMs) across various tasks.  Despite subsequent widespread adaptation, little work has focused on Mamba LLMs' amenability for fine-tuning frameworks ubiquitously used for Transformer-based LLMs, e.g., mixed-precision fine-tuning (MPFT) and parameter-efficient fine-tuning (PEFT).  For the former, it currently remains an open question whether Mamba's recurrent dynamics are robust to small input changes, such as those encountered during MPFT.  Using dynamical systems theory (in particular, Lyapunov exponents), we answer this question in the affirmative.  We empirically validate this result through several experiments, showing that Mamba SSMs are significantly more stable to changes introduced by mixed-precision than comparable Transformers, even when both MPFT and PEFT are combined.  For PEFT, we show how targeting specific memory buffers in Mamba's customized CUDA kernels for low-rank adaptation regularizes SSM parameters, thus providing both parameter efficient learning and computational savings. Finally, with both MPFT and PEFT enabled, we explore the impact of instruction tuning Mamba SSMs for in-context learning (ICL) on natural language tasks.  While pretrained Mamba and Mamba-2 models only achieve 38% and 82% (respectively) of the ICL improvements of comparable Transformer-based LLMs, we show that instruction tuning allows Mamba models to narrow this gap to 81% and Mamba-2 models to skyrocket over this gap to 132%."},"_bibtex":{"value":"@misc{\nhalloran2025mamba,\ntitle={{MAMBA} {STATE}-{SPACE} {MODELS} {ARE} {LYAPUNOV}-{STABLE} {LEARNERS}},\nauthor={John Timothy Halloran and Manbir S Gulati and Paul F Roysdon},\nyear={2025},\nurl={https://openreview.net/forum?id=i9RTCC6whL}\n}"},"title":{"value":"MAMBA STATE-SPACE MODELS ARE LYAPUNOV-STABLE LEARNERS"},"pdf":{"value":"/pdf/c8197bfc3ebf6e57772b5572cc4c46a84fd0d994.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"halloran|mamba_statespace_models_are_lyapunovstable_learners"},"authorids":{"value":["~John_Timothy_Halloran1","~Manbir_S_Gulati1","~Paul_F_Roysdon1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["John Timothy Halloran","Manbir S Gulati","Paul F Roysdon"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2510.03279v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"wang|memmamba_rethinking_memory_patterns_in_state_space_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Youjin_Wang:","https://dblp.org/search/pid/api?q=author:Yangjingyi_Chen:","https://dblp.org/search/pid/api?q=author:Jiahao_Yan:","~Jiaxuan_Lu1","https://dblp.org/search/pid/api?q=author:Xiao_Sun:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2510.03279"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2510-03279,\n  publtype={informal},\n  author={Youjin Wang and Yangjingyi Chen and Jiahao Yan and Jiaxuan Lu and Xiao Sun},\n  title={MemMamba: Rethinking Memory Patterns in State Space Model},\n  year={2025},\n  month={October},\n  cdate={1759276800000},\n  journal={CoRR},\n  volume={abs/2510.03279},\n  url={https://doi.org/10.48550/arXiv.2510.03279}\n}\n"},"abstract":{"value":"With the explosive growth of data, long-sequence modeling has become increasingly important in tasks such as natural language processing and bioinformatics. However, existing methods face inherent trade-offs between efficiency and memory. Recurrent neural networks suffer from gradient vanishing and explosion, making them hard to scale. Transformers can model global dependencies but are constrained by quadratic complexity. Recently, selective state-space models such as Mamba have demonstrated high efficiency with O(n) time and O(1) recurrent inference, yet their long-range memory decays exponentially. In this work, we conduct mathematical derivations and information-theoretic analysis to systematically uncover the memory decay mechanism of Mamba, answering a fundamental question: what is the nature of Mamba's long-range memory and how does it retain information? To quantify key information loss, we further introduce horizontal-vertical memory fidelity metrics that capture degradation both within and across layers. Inspired by how humans distill and retain salient information when reading long documents, we propose MemMamba, a novel architectural framework that integrates state summarization mechanism together with cross-layer and cross-token attention, which alleviates long-range forgetting while preserving linear complexity. MemMamba achieves significant improvements over existing Mamba variants and Transformers on long-sequence benchmarks such as PG19 and Passkey Retrieval, while delivering a 48% speedup in inference efficiency. Both theoretical analysis and empirical results demonstrate that MemMamba achieves a breakthrough in the complexity-memory trade-off, offering a new paradigm for ultra-long sequence modeling."},"title":{"value":"MemMamba: Rethinking Memory Patterns in State Space Model"},"authors":{"value":["Youjin Wang","Yangjingyi Chen","Jiahao Yan","Jiaxuan Lu","Xiao Sun"]}},"tmdate":1767786717758,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2510-03279"],"tcdate":1767786712752,"writers":["~"],"signatures":["~Jiaxuan_Lu1"],"forum":"EruTGtd6Bv","license":"CC BY-SA 4.0","number":725692,"cdate":1759276800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1767786717758,"domain":"DBLP.org","id":"EruTGtd6Bv","version":2},{"content":{"summary":{"value":"This paper proposes an motion prediction model based on the Mamba architecture, which consists of the Temporal Mamba Model and the Spatial Mamba Model that handle temporal and spatial dependencies respectively. By leveraging the Mamba architecture's near-linear computational complexity with respect to sequence length, the proposed method demonstrates efficiency advantages compared to QCNet, a transformer-based approach."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. Among the compared methods, some SOTA approaches listed on the leaderboard, such as SEPT and LOF, are not included. Please clarify the criteria used for selecting the baselines.\n2. In the scene encoding, the tokens corresponding to lane should have a natural permutation invariance property. Why do the authors assert that 'the order of tokens is crucial'? Additionally, what is the specific advantage of using a state-space model, a typical temporal model, to handle the spatial structural relationships in traffic scenes?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper is relatively well-organized and easy to read.\n2. The paper introduces the concept of state space models into the scene encoding of motion prediction, which is a noteworthy innovation."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The authors mention in the contributions that 'By injecting inductive biases of temporal and spatial dependency through Mamba’s state-space model structure, the model’s capacity is significantly improved.' The authors should clarify for this improvement relative to which specific methods.\n2. Based on the test results from Argoverse 2, the proposed method shows a significant performance gap in various accuracy metrics compared to Transformer-based methods like QCNet. Although MS$^3$M has the best performance on minFDE$_1$, this is achieved by excluding some more advanced methods such as SEPT and LOF. Therefore, the experiments do not support the claim in the paper that \"Mamba was proposed as a more advanced foundation model\".\n3. The experimental results are not sufficiently convincing, as the authors only conducted experiments on the Argoverse 2. It is recommended to include experimental results on the Argoverse 1 or other datasets.\n4. In the test for inference time, the paper presents some unusual data, such as '16.56±26.38.' I suggest that the authors need to explain why the variance for the MS$^3$M is so large."}},"nonreaders":[],"tmdate":1731427842613,"tcdate":1730369532098,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3447/Reviewer_Dxg8"],"signatures":["ICLR.cc/2025/Conference/Submission3447/Reviewer_Dxg8"],"forum":"MmOQY71YHw","number":1,"license":"CC BY 4.0","cdate":1730369532098,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3447/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427842613,"domain":"ICLR.cc/2025/Conference","replyto":"MmOQY71YHw","id":"XfrdKR6pNS","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"This paper proposes a Mamba-based motion forecasting architecture that achieves superior performance with significantly lower latency."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Motion Forecasting","Autonomous Driving","State Space Model"]},"supplementary_material":{"value":"/attachment/101a980240bdc46f184713f07f4571c1b29e062b.pdf"},"primary_area":{"value":"applications to robotics, autonomy, planning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Motion forecasting is a fundamental component of autonomous driving systems, as it predicts an agent's future trajectories based on its surrounding environment. Transformer architectures have dominated this domain due to their strong ability to model both temporal and spatial information. However, transformers often suffer from quadratic complexity with respect to input sequence length, limiting their ability to efficiently process scenarios involving numerous agents. Additionally, transformers typically rely on positional encodings to represent temporal or spatial relationships, a strategy that may not be as effective or intuitive as the inductive biases naturally embedded in convolutional architectures. To address these challenges, we leverage recent advancements in state space models (SSMs) and propose the Multi-Stage State Space Model (MS$^3$M). In MS$^3$M, the Temporal Mamba Model (TMM) is employed to capture fine-grained temporal information, while the Spatial Mamba Model efficiently handles spatial interactions. By injecting temporal and spatial inductive biases through Mamba’s state-space model structure, the model's capacity is significantly improved. MS$^3$M also strikes an exceptional trade-off between accuracy and efficiency, which is achieved through convolutional computations and near-linear computational strategies in the Mamba architecture. Furthermore, a hierarchical query-based decoder is introduced, further enhancing model performance and efficiency. Extensive experimental results demonstrate that the proposed method achieves superior performance while maintaining low latency, which is crucial for practical real-time autonomous driving systems."},"_bibtex":{"value":"@misc{\nliu2025msm,\ntitle={{MS}\\${\\textasciicircum}3\\$M: Multi-Stage State Space Model for Motion Forecasting},\nauthor={Chunyu Liu and Shijie Li and Xulei Yang and Jianjun Yu},\nyear={2025},\nurl={https://openreview.net/forum?id=MmOQY71YHw}\n}"},"title":{"value":"MS$^3$M: Multi-Stage State Space Model for Motion Forecasting"},"pdf":{"value":"/pdf/b5472cb1e15055084d263ef47ccb9412ed68e871.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"liu|ms^3m_multistage_state_space_model_for_motion_forecasting"},"authorids":{"value":["~Chunyu_Liu2","~Shijie_Li1","~Xulei_Yang2","~Jianjun_Yu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chunyu Liu","Shijie Li","Xulei Yang","Jianjun Yu"]}},"version":2},{"content":{"summary":{"value":"This paper focuses on a specific class of Markov Decision Processes (MDPs) in which a state can be decomposed into a stochastic component and a pseudo-stochastic component. The transition dynamics of stochastic states depend solely on the current stochastic state and the action taken, while the transition dynamics of pseudo-stochastic states are determined deterministically by the current state, the next stochastic state, and the action. Such MDPs are suitable for modeling applications like queueing systems.\n\nThe authors initially consider a scenario with finite state spaces and batch reinforcement learning. Given a set of independent and identically distributed (i.i.d.) transition and reward data, they propose a data augmentation algorithm that generates m new virtual samples from each data point. This is achieved by replacing the pseudo-stochastic components with newly sampled ones. Subsequently, they propose the fitted Q iteration with an augmented dataset. Under certain coverage assumptions for stochastic states and with n initial samples, they establish an error bound on the Q function, which scales as 1/\\sqrt{n} + 1/\\sqrt{m}. The authors extend their results to scenarios with an infinite pseudo-stochastic state space, along with additional assumptions. Simulations are conducted to demonstrate the benefits of the proposed algorithm."},"soundness":{"value":"3 good"},"confidence":{"value":"3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked."},"questions":{"value":"In line 93, the authors claim \"far less memory space.\" Could you provide a quantification, such as the complexity mentioned in line 91?\n\nIn the simulations, how is the reward function determined? How are the state sampling distributions \\beta_k set? Additionally, what is the number of virtual samples used?\n\nRegarding Figures 2 and 3, how is an episode defined in the context of the simulations?\n\nIn Figure 3 (a, b, c), it appears that the performance of DQN-ASG does not show improvement after training with episodes. Furthermore, why is the performance in episode 0, where the algorithm should not have learned anything yet, already better than other algorithms? The same question applies to Figure 4 in the appendix.\n\nIn lines 24 and 25, from \"time-consuming\" to the period before \"In this paper,\" the text does not form a complete sentence.\n\nIn line 56, the last \"and\" is unnecessary.\n\nIn line 222, please change \"satisfies\" to \"satisfying.\""},"rating":{"value":"7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":"3 good"},"contribution":{"value":"4 excellent"},"strengths":{"value":"Using reinforcement learning to determine optimal policies for systems with large state spaces is a well-known challenge. However, this paper presents a valuable and intuitive framework that effectively captures the inherent structure of environments, specifically queueing systems, through the introduction of pseudo-stochastic states. This innovative modeling approach is commendable. The concept of utilizing fixed transition functions to generate new data points is novel and has practical relevance for systems influenced by limited stochastic inputs. The theoretical results provided in the paper are sound and contribute to the overall strength of the work."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"In Section 2.1, the authors provide an example where the pseudo-stochastic state space is not finite, which goes beyond the scope of the model as indicated by the assumption of a finite state space in line 148. Although the authors later extend their results, it remains unclear how this example aligns with Assumption 1.\n\nBuilding upon point 1, Assumption 1 poses challenges in designing the function class F for the Q function. It seems difficult to define a class where the value of an atypical state under any function within the class is consistently close to the optimal Q value. The authors need to provide better justification for why this property can be easily achieved. One approach could be to establish a broad enough function class that satisfies Assumption 1 for a simple queueing system. Justifying the reasonableness of Assumption 1 is crucial, especially considering the paper's claim that the theory and method are applicable to queueing systems with unbounded state spaces.\n\nThe simulation section lacks sufficient explanation and some of the plots appear unreasonable. Further details and clarification are needed to address these issues.\n\nThere are several typos that need to be corrected throughout the paper."},"limitations":{"value":"N/A"}},"nonreaders":[],"tmdate":1702411394597,"tcdate":1687747207557,"writers":["NeurIPS.cc/2023/Conference","NeurIPS.cc/2023/Conference/Submission12687/Reviewer_ciJb"],"signatures":["NeurIPS.cc/2023/Conference/Submission12687/Reviewer_ciJb"],"forum":"LTbIUkN95h","number":1,"license":"CC BY 4.0","cdate":1687747207557,"mdate":1702411394597,"readers":["everyone"],"invitations":["NeurIPS.cc/2023/Conference/Submission12687/-/Official_Review","NeurIPS.cc/2023/Conference/-/Edit"],"domain":"NeurIPS.cc/2023/Conference","replyto":"LTbIUkN95h","id":"mWk9Wog0LZ","forumContent":{"venue":{"value":"NeurIPS 2023 spotlight"},"keywords":{"value":["Reinforcement Learning","Mixed Systems","Queueing Network","Sample Efficient"]},"supplementary_material":{"value":"/attachment/26a5b58edf7d428193249415efa4ea74e8c1fadf.zip"},"_bibtex":{"value":"@inproceedings{\nwei2023sample,\ntitle={Sample Efficient Reinforcement Learning in Mixed Systems through Augmented Samples and Its Applications to Queueing Networks},\nauthor={Honghao Wei and Xin Liu and Weina Wang and Lei Ying},\nbooktitle={Thirty-seventh Conference on Neural Information Processing Systems},\nyear={2023},\nurl={https://openreview.net/forum?id=LTbIUkN95h}\n}"},"title":{"value":"Sample Efficient Reinforcement Learning in Mixed Systems through Augmented Samples and Its Applications to Queueing Networks"},"paperhash":{"value":"wei|sample_efficient_reinforcement_learning_in_mixed_systems_through_augmented_samples_and_its_applications_to_queueing_networks"},"TLDR":{"value":"We propose a sample efficient RL method that accelerates learning in mixed systems."},"abstract":{"value":"This paper considers a class of reinforcement learning problems, which involve systems with two types of states: stochastic and pseudo-stochastic. In such systems, stochastic states follow a stochastic transition kernel while the transitions of pseudo-stochastic states are deterministic {\\em given} the stochastic states/transitions. We refer to such systems as mixed systems, which are widely used in various applications, including Manufacturing systems, communication networks, and queueing networks. We propose a sample-efficient RL method that accelerates learning by generating augmented data samples. The proposed algorithm is data-driven (model-free), but it learns the policy from data samples from both real and augmented samples. This method significantly improves learning by reducing the sample complexity such that the dataset only needs to have sufficient coverage of the stochastic states. We analyze the sample complexity of the proposed method under Fitted Q Iteration (FQI) and demonstrate that the optimality gap decreases as  $O\\left(\\sqrt{\\frac{1}{n}}+\\sqrt{\\frac{1}{m}}\\right),$ where $n$ represents the number of real samples, and $m$ is the number of augmented samples per real sample. It is important to note that without augmented samples, the optimality gap is $O(1)$ due to the insufficient data coverage of the pseudo-stochastic states. Our experimental results on multiple queueing network applications confirm that the proposed method indeed significantly accelerates both deep Q-learning and deep policy gradient."},"pdf":{"value":"/pdf/69edfe02e4a5907f1172ddcb774d1138aee62e60.pdf"},"venueid":{"value":"NeurIPS.cc/2023/Conference"},"authorids":{"value":["~Honghao_Wei2","~Xin_Liu14","~Weina_Wang1","~Lei_Ying1"]},"authors":{"value":["Honghao Wei","Xin Liu","Weina Wang","Lei Ying"]}},"version":2},{"content":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["sequence parallelism","distributed training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Sequence parallelism (SP) serves as a prevalent strategy to handle long sequences that exceed the memory limit of a single device. However, for linear sequence modeling methods like linear attention, existing SP approaches do not take advantage of their right-product-first feature, resulting in sub-optimal communication efficiency and usability. In this paper, we introduce Linear Attention Sequence Parallelism (LASP), an efficient SP approach designed for linear attention-based transformer models. Specifically, we design an efficient point-to-point ring-style communication mechanism to leverage the right-product kernel trick of linear attention, which sharply decreases the communication overhead, comparing with existing SP methods. We enhance the computation efficiency of LASP by performing kernel fusion and intermediate state caching, making the implementation of LASP hardware-friendly on GPUs. Furthermore, we meticulously ensure the compatibility of sequence-level LASP with all types of batch-level data parallel methods, which is vital for distributed training on large clusters with very-long sequences. We also discuss the generalization of LASP on other linear sequence modeling methods. Extensive experiments on linear attention-based models are conducted with varying sequence lengths from 2K to 4096K. LASP scales sequence length up to 4096K on 128 GPUs, which is 8$\\times$ longer than existing SP methods."},"_bibtex":{"value":"@misc{\nsun2025linear,\ntitle={Linear Attention Sequence Parallelism},\nauthor={Weigao Sun and Zhen Qin and Dong Li and Xuyang Shen and Yu Qiao and Yiran Zhong},\nyear={2025},\nurl={https://openreview.net/forum?id=oVnfVnwh6y}\n}"},"title":{"value":"Linear Attention Sequence Parallelism"},"pdf":{"value":"/pdf/91f0bb8ec553fe88b189a2550d301b39cef48faa.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"sun|linear_attention_sequence_parallelism"},"authorids":{"value":["~Weigao_Sun1","~Zhen_Qin6","~Dong_Li11","~Xuyang_Shen1","~Yu_Qiao1","~Yiran_Zhong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Weigao Sun","Zhen Qin","Dong Li","Xuyang Shen","Yu Qiao","Yiran Zhong"]}},"tmdate":1736391500067,"tcdate":1727326528725,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5885/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5885/Authors"],"forum":"oVnfVnwh6y","license":"CC BY 4.0","number":5885,"cdate":1727326528725,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission5885/-/Full_Submission","ICLR.cc/2025/Conference/-/Withdrawn_Submission"],"mdate":1736391500067,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"oVnfVnwh6y","version":2},{"content":{"TLDR":{"value":"DAG-Mamba: An Order-Robust Structural Interface for State Space Modeling on Directed Acyclic Graphs"},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["DAG-MAMBA; Selective State Space Model; Directed Acyclic Graph (DAG); Structure-aware Representation; Long-range Dependency Modeling; Graph Representation Learning"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"The directed acyclic graph (DAG) induces a partial order, but does not induce a unique node sequence, which causes a structural mismatch between it and the order-sensitive state space model (SSM). Arbitrary topological serialization may make the loop model depend on the relative order of incomparable nodes, even if the order is not part of the graph. We study how to align order-sensitive sequence models with DAGs without committing to an arbitrary linear extension. We propose \\method, a sequential robust structural interface, which separates the node-level dependency information from the coarse-grained depth sequence. The local and bounded reachability branches retain the direct relationship and the predecessor-follower relationship, and the nodes with the same longest path depth are pooled into a norm-front token sequence ; we then use Mamba as a learned cross-frontier mixer. The frontier represents that the node index and the selected topology order remain unchanged, but is intentionally designed to be non-injective, so it complements rather than replaces the graph structure. On the code, reference, and neural architecture DAG, complete instantiation is significantly superior to sequential naive Mamba on several tasks and remains competitive with strong graph/DAG baselines. A diagnostic ablation further demonstrated that replacing the learned Mamba transfer with an identical leading-edge mixer on three ablation tasks could match or exceed the complete model. This distinguishes two confusing effects : sequential robustness is provided by the DAG interface, and the marginal value of the learned cross-frontier loop depends on the task. Therefore, we position \\method as a Mamba instantiation of a sequential robust DAG interface, rather than as evidence that selective scanning is generally necessary for DAG learning. The simulation code can be obtained in \\url{https://github.com/zongjin130/DAG-MAMBA}."},"_bibtex":{"value":"@inproceedings{\nanonymous2026dagmamba,\ntitle={{DAG}-{MAMBA}: Structure-aware selectivity for directed acyclic graphs  State space model},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=rMuWCjdsTc},\nnote={under review}\n}"},"title":{"value":"DAG-MAMBA: Structure-aware selectivity for directed acyclic graphs  State space model"},"pdf":{"value":"/pdf/dbdb8c79e22d73fdb51ad39d22ea8bb4766142c5.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791220780901,"tcdate":1787888146911,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission5642/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission5642/Authors"],"forum":"rMuWCjdsTc","license":"CC BY 4.0","number":5642,"cdate":1787888146911,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission5642/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791220780901,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"rMuWCjdsTc","version":2},{"content":{"summary":{"value":"The paper is aimed at predicting gene expression from DNA sequence. In addition to the sequence itself, it also incorporates epigenetic information at each based. The model consists of two MAMBA-based Caduceus networks: the first network is trained to estimate a sparse mask over the bases to select regulatory regions, and the second one uses the selected DNA regions to predict the expression of the gene."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"Details about the network training (and pre-training, if used), including architectural hyperparameters, as well as training procedure details (# steps, etc)."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The paper showcases a successful application of the recently-proposed Selective State Space models (specifically, MAMBA-based Caduceus) on relatively long sequences (200k). \n\nThe model goes beyond most current approaches by combining sequence with epigenetic signaling in a single, jointly optimized architecture. The architecture is sound, and uses information bottleneck principle to promote sparsity of the mask for selecting sequence regions, and then a straight-through estimator to allow differentiability through the hard mask being produced by the first network.\n\nExperimental evaluation shows that the proposed architecture leads to improved prediction, in comparison with Transformer- and State-space-based models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While the manuscript is mostly well written, it could use a more detailed explanation in several places, specifically related to:\n- gene expression Y: it is often just described as \"target variable Y\", or \"$Y \\in R$. From 5.1.4 we read that ultimately the model is applied to a 200kbp region around a specific target gene, and one can understand that Y is the expression of that gene; it would be helpful to provide this information clearly earlier in the paper. \n- model architecture: it also deferred to 5.1.4, and only described in one sentence (Caduceus-Ph). One would expect it to be in 4.1. entitled \"Model architecture\". Also, key details are missing: what are the specific hyperparameters (#layers, hidden dimension, etc.)? Is this a network trained de-novo for the expression task, or pre-trained (Caduceus has pretrained models of up to 131kbp, this one is 200kbp, if it was pre-trained by the authors, more details should be provided)."}},"nonreaders":[],"tmdate":1731429074313,"tcdate":1730718494759,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12644/Reviewer_KQDw"],"signatures":["ICLR.cc/2025/Conference/Submission12644/Reviewer_KQDw"],"forum":"Mfnh1Sqdwf","number":3,"license":"CC BY 4.0","cdate":1730718494759,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12644/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429074313,"domain":"ICLR.cc/2025/Conference","replyto":"Mfnh1Sqdwf","id":"HejRtowXPJ","forumContent":{"venue":{"value":"ICLR 2025 Oral"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Gene Expression","Deep Learning","Sequence Modeling"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We consider the problem of predicting gene expressions from DNA sequences. A key challenge of this task is to find the regulatory elements that control gene expressions. Here, we introduce Seq2Exp, a Sequence to Expression network explicitly designed to discover and extract regulatory elements that drive target gene expression, enhancing the accuracy of the gene expression prediction. Our approach captures the causal relationship between epigenomic signals, DNA sequences and their associated regulatory elements. Specifically, we propose to decompose the epigenomic signals and the DNA sequence conditioned on the causal active regulatory elements, and apply an information bottleneck with the Beta distribution to combine their effects while filtering out non-causal components. Our experiments demonstrate that Seq2Exp outperforms existing baselines in gene expression prediction tasks and discovers influential regions compared to commonly used statistical methods for peak detection such as MACS3. The source code is released as part of the AIRS library (https://github.com/divelab/AIRS/)."},"_bibtex":{"value":"@inproceedings{\nsu2025learning,\ntitle={Learning to Discover Regulatory Elements for Gene Expression Prediction},\nauthor={Xingyu Su and Haiyang Yu and Degui Zhi and Shuiwang Ji},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=Mfnh1Sqdwf}\n}"},"title":{"value":"Learning to Discover Regulatory Elements for Gene Expression Prediction"},"pdf":{"value":"/pdf/d42999487f2a6b7b3ec98eb801262dafd17c9443.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"su|learning_to_discover_regulatory_elements_for_gene_expression_prediction"},"authorids":{"value":["~Xingyu_Su2","~Haiyang_Yu6","~Degui_Zhi1","~Shuiwang_Ji1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xingyu Su","Haiyang Yu","Degui Zhi","Shuiwang Ji"]}},"version":2},{"content":{"summary":{"value":"The authors present Saturn, an RL-approach using a Mamba backbone for generative molecular design. Design choices surrounding implementation are thoroughly explored and motivated towards increasing sample efficiency to enable the use of higher fidelity (and more computationally expensive) objective functions."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- Is there a possibility that the molecules generated by GraphGA impair the quality of the replay buffer molecules as this is an imperfect generative approach itself, would it be better to use a higher fidelity method to breed the members of the buffer."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The authors focus on a weakness of the current trend in generative chemistry to couple an optimization algorithm with an insilico oracle function, suggesting two avenues for improvement: (i) increase the fidelity of the fast oracle, or (ii) increase the sample efficiency of the generative process to enable more costly oracles at reasonable throughput. In this paper they focus on (ii) which is a valuable contribution to the field.\n\nThe authors previously described the RL paradigm and here focus on substituting the language model backbone within the constraints that it is sufficiently simple to fine-tune on the fly. Given the recent success of Mamba as a linear time sequence modelling architecture this choice seems well motivated and suitable.\n\nThe reinforcement learning paradigm including SMILES augmentation, and experience replay buffer update with a genetic algorithm is intricately described along with its effect on the diversity of generated outputs."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Major:\n- The main weakness of the method seems to be the diversity reduction exhibited upon moving to Mamba from an RNN backbone. Therefore the majority of my main questions are surrounding this observation.\n  - The authors describe a logical diversity filter on the augmented memory buffer that uses bemis murcko as a filter. I am surprised that this addition does not seem to mitigate the reduction in diversity. Could the authors comment on this, and possibly show the murcko scaffold diversity along with their results in Table 2.\n  - Following the argument at Line 292 that Mamba generates repeat SMILES because its loss is lower during pre-training, this argument would suggest overfitting. If the authors use the Mamba checkpoint with a loss matched to the RNN, is the performance restored to the RNN levels of diversity? At what sample efficiency cost?\n\nMinor:\n- The introduction should mention the use of QSAR predictive oracles since these are very common in addition to docking.\n- The paper is motivated by an intent to use molecular dynamics as an oracle but does not achieve this goal, could the authors add a comment on what it would take to achieve this in the discussion section.\n- The authors discuss the trade off of sample efficiency and diversity, this is similar to the common fidelity vs diversity tradeoff among image generative models. The authors might consider a 2D summary plot showing the tradeoff of these while varying batch size and n augmentation rounds since its a lot of information to digest in a tabular form.\n- I am not clear on the logic presented in the section “squeezing the likelihood of Augmented SMILES” the authors might consider clarifying.\n- Figure 2d compares the trajectory of generation throughout augmentation rounds. It is unclear that the Mamba path is more linear from these plots in my opinion. (i) The UMAP should be fit to the same set in order to compare and (ii) the RNN should also have 10 rounds."}},"nonreaders":[],"tmdate":1731427497463,"tcdate":1730902607618,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1831/Reviewer_SmbD"],"signatures":["ICLR.cc/2025/Conference/Submission1831/Reviewer_SmbD"],"forum":"Hbpzrh7JbN","number":5,"license":"CC BY 4.0","cdate":1730902607618,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1831/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427497463,"domain":"ICLR.cc/2025/Conference","replyto":"Hbpzrh7JbN","id":"RbGLsSDOzu","forumContent":{"TLDR":{"value":"state-of-the-art sample efficiency using mamba for generative molecular design"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["generative design","drug discovery","sample efficiency","language models","reinforcement learning","scaling"]},"supplementary_material":{"value":"/attachment/4735cc8666aadddbad349c1f2433393b9784f551.pdf"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Generative molecular design for drug discovery has very recently achieved a wave\nof experimental validation, with language-based backbones being the most common\narchitectures employed. The most important factor for downstream success is\nwhether an *in silico* oracle is well correlated with the desired end-point. To this end,\ncurrent methods use cheaper proxy oracles with higher throughput before evaluating\nthe most promising subset with high-fidelity oracles. The ability to directly optimize\nhigh-fidelity oracles would greatly enhance generative design and be expected to\nimprove hit rates. However, current models are not efficient enough to consider such\na prospect, exemplifying the sample efficiency problem. In this work, we introduce\n**Saturn**, which leverages the Augmented Memory algorithm and demonstrates the\nfirst application of the Mamba architecture for generative molecular design. We\nelucidate *how* experience replay with data augmentation improves sample efficiency\nand *how* Mamba synergistically exploits this mechanism. Saturn outperforms 22\nmodels on multi-parameter optimization tasks relevant to drug discovery and may\npossess sufficient sample efficiency to consider the prospect of directly optimizing\nhigh-fidelity oracles."},"_bibtex":{"value":"@misc{\nguo2025saturn,\ntitle={Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation},\nauthor={Jeff Guo and Philippe Schwaller},\nyear={2025},\nurl={https://openreview.net/forum?id=Hbpzrh7JbN}\n}"},"title":{"value":"Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation"},"pdf":{"value":"/pdf/c0929a667bd3f510fb8322a9d402f95c457e38e1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"guo|saturn_sampleefficient_generative_molecular_design_using_memory_manipulation"},"authorids":{"value":["~Jeff_Guo1","~Philippe_Schwaller1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jeff Guo","Philippe Schwaller"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a simple yet effective Graph and Skipped Transformer (G-SFormer) for 3D human pose estimation. It leverages a Part-based Adaptive GNN and a Frameset-based Skipped Transformer to capture both detailed pose representations and multi-perspective dynamic representations. Experimental results show that G-SFormer achieves competitive performance on benchmark datasets, including Human3.6M, MPI-INF-3DHP, and HumanEva."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"See Weaknesses"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The paper is well-structured and clearly presented. \nThe idea is simple, intuitive yet effective."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The proposed method does not compare with the state-of-the-art method [1,2]. For example, MotionAGFormer achives 38.4 mm in MPJPE on Human3.6M, which achive better performance than the proposed method. \n- Although the authors claim that their method can address joint noise, no clear evidence or quantitative results are provided to substantiate this claim. It is recommended that the authors include experimental comparisons and present evidence demonstrating why the proposed method effectively addresses joint noise to strengthen their argument.\n- Given the paper’s emphasis on efficiency, a time-comparison table (inference time) should be included, as a FLOPs comparison alone is insufficient. Furthermore, since G-SFormer only regresses the 3D pose of the center frame, it appears to achieve a much lower speed compared to seq2seq-based methods [1,2,3]. \n- The paper lacks a comparison with other efficient methods, while I think it is necessary. Such as flash-attention [4] and state-space models [5]. \n\n[1] MotionAGFormer: Enhancing 3D Human Pose Estimation with a Transformer-GCNFormer Network\n\n[2] MotionBERT: Unified Pretraining for Human Motion Analysis\n\n[3] MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video\n\n[4] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\n\n[5] Mamba: Linear-Time Sequence Modeling with Selective State Spaces"}},"nonreaders":[],"tmdate":1731427708324,"tcdate":1730167270024,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3002/Reviewer_2qgK"],"signatures":["ICLR.cc/2025/Conference/Submission3002/Reviewer_2qgK"],"forum":"XHvdM04T0l","number":3,"license":"CC BY 4.0","cdate":1730167270024,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3002/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427708324,"domain":"ICLR.cc/2025/Conference","replyto":"XHvdM04T0l","id":"CdWcv5Ly2d","forumContent":{"TLDR":{"value":"We propose a compact Graph and Skipped Transformer architecture to realise efficient and robust 2D-to-3D Human Pose Estimation."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["3D pose estimation","Part-based Adaptive GNN","Frameset-based Skipped Transformer","efficient and robust"]},"supplementary_material":{"value":"/attachment/9fa2505afc4c9e9a1fdbbe5881ea1b5b99f6dad2.pdf"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent works in 2D-to-3D pose uplifting for monocular 3D Human Pose Estimation (HPE) have shown significant progress. However, two key challenges persist in real-world applications: vulnerability to joint noise and high computational costs. These issues arise from the dense joint-frame connections and iterative correlations typically employed by mainstream GNN-based and Transformer-based methods. To address these challenges, we propose a novel approach that leverages human physical structure and long-range dynamics to learn spatial part- and temporal frameset-based representations. This method is inherently robust to missing or erroneous joints while also reducing model parameters. Specifically, in the Spatial Encoding stage, coarse-grained body parts are used to construct structural correlations with a fully adaptive graph topology. This spatial correlation representation is integrated with muti-granularity pose attributes to generate a comprehensive pose representation for each frame. In Temporal Encoding and Decoding stages, Skipped Self-Attention is performed in framesets to establish long-term temporal dependencies from multiple perspectives of movement. On this basis, a compact Graph and Skipped Transformer (G-SFormer) is proposed, which realises efficient and robust 3D HEP in both experimental and practical scenarios. Extensive experiments on Human3.6M, MPI-INF-3DHP and Human-Eva benchmarks demonstrate that G-SFormer series models can compete and outperform the state-of-the-arts but takes only a fraction of parameters and around 1\\% computational cost. It also exhibits outstanding robustness to inaccurately detected 2D poses. The source code will be available at https://sites.google.com/view/g-sformer."},"_bibtex":{"value":"@misc{\ncui2025learning,\ntitle={Learning Structure-Dynamics-aware Representations for Efficient and Robust 3D Pose Estimation},\nauthor={Mengmeng Cui and Wei Wang and Kunbo Zhang and Zhenan Sun},\nyear={2025},\nurl={https://openreview.net/forum?id=XHvdM04T0l}\n}"},"title":{"value":"Learning Structure-Dynamics-aware Representations for Efficient and Robust 3D Pose Estimation"},"pdf":{"value":"/pdf/50f6afdeab21d4f19889b51280b7d115c5b181ed.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"cui|learning_structuredynamicsaware_representations_for_efficient_and_robust_3d_pose_estimation"},"authorids":{"value":["~Mengmeng_Cui1","~Wei_Wang4","~Kunbo_Zhang1","~Zhenan_Sun1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Mengmeng Cui","Wei Wang","Kunbo Zhang","Zhenan Sun"]}},"version":2},{"content":{"summary":{"value":"The paper titled \"CMamba: Channel Correlation Enhanced State Space Model for Multivariate Time Series Prediction\" introduces a new model called CMamba, which builds upon the existing Mamba state space model and addresses the key limitation of capturing cross channel dependencies that are crucial for multivariate time series prediction (MTSF). The CMamba architecture integrates two main components: (1) the M-Mamba module for modeling temporal dependencies, and (2) the Global Data Dependency MLP (GDD-MLP) for capturing cross channel correlations. In addition, a channel mixing strategy was introduced to improve generalization ability and alleviate overfitting. The author validated their method through extensive experiments on seven benchmark datasets, demonstrating that CMamba outperforms state-of-the-art models in long-term prediction tasks."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Has the model been tested on datasets outside the field explored in this article, such as financial or biological time series? Understanding the generalization level of CMamba to different types of multivariate time series would be valuable.\nCan the author provide more insights into the computational efficiency of CMamba compared to Transformer based models such as iTransformer? A more detailed analysis of computational trade-offs may be helpful for practitioners considering large-scale deployment of the model."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"Originality: M-Mamba is used for time-dependent analysis, while GDD-MLP is used for cross channel correlation, which creatively enhances the state space model and solves the limitations of existing MTSF methods.\nQuality: This article presents a set of well executed experiments that validate the effectiveness of CMamba in multiple datasets and prediction tasks. The consistent and statistically significant results demonstrate the robustness of the proposed model.\nClarity: The model architecture and experimental setup are clearly described and accompanied by useful charts (as shown in Figure 3). Tables and charts are well organized for easy interpretation of results.\nMeaning: CMamba solves an important problem in time series prediction - modeling cross channel dependencies - which has implications for many real-world applications such as traffic management and energy consumption prediction."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Long term dependency exploration: Although the GDD-MLP module has shown promising results in mid-term time series prediction, its effectiveness in handling very long-term dependencies (such as prediction ranges exceeding 720) has not been fully explored. Additional experiments on long-term prediction tasks can further validate the robustness of the model.\nGeneralization in different fields: This article mainly focuses on standard datasets such as electricity and transportation, but it is beneficial to test CMamba's generalization ability on other types of time series data (such as financial or biological datasets), which may have different time and cross channel characteristics."}},"nonreaders":[],"tmdate":1731428496720,"tcdate":1729000937362,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5801/Reviewer_Wmo3"],"signatures":["ICLR.cc/2025/Conference/Submission5801/Reviewer_Wmo3"],"forum":"OclHGmt2ZM","number":1,"license":"CC BY 4.0","cdate":1729000937362,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5801/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428496720,"domain":"ICLR.cc/2025/Conference","replyto":"OclHGmt2ZM","id":"Z8gy12AAxX","forumContent":{"TLDR":{"value":"We tailor Mamba for multivariate time series forecasting, equipping it with stronger cross-time and cross-channel modeling capabilities, which achieves the SOTA in real-world datasets."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forecasting","Mamba","Channel-Dependent"]},"supplementary_material":{"value":"/attachment/4810e3ea813ec5d058c32afc9ff6a129c09b060a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent advancements in multivariate time series forecasting have been propelled by Linear-based, Transformer-based, and Convolution-based models, with Transformer-based architectures gaining prominence for their efficacy in temporal and cross-channel mixing.\nMore recently, Mamba, a state space model, has emerged with robust sequence and feature mixing capabilities.\nHowever, the suitability of the vanilla Mamba design for time series forecasting remains an open question, particularly due to its inadequate handling of cross-channel dependencies.\nCapturing cross-channel dependencies is critical in enhancing the performance of multivariate time series prediction.\nRecent findings show that self-attention excels in capturing cross-channel dependencies, whereas other simpler mechanisms, such as MLP, may degrade model performance.\nThis is counterintuitive, as MLP, being a learnable architecture, should theoretically capture both correlations and irrelevances, potentially leading to neutral or improved performance.\nDiving into the self-attention mechanism, we attribute the observed degradation in MLP performance to its lack of data dependence and global receptive field, which result in MLP's lack of generalization ability.\nConsidering the powerful sequence modeling capabilities of Mamba and the high efficiency of MLP, the combination of the two is an effective strategy for solving multivariate time series prediction.\nBased on the above insights, we introduce a refined Mamba variant tailored for time series forecasting.\nOur proposed model, \\textbf{CMamba}, incorporates a modified Mamba (M-Mamba) module for temporal dependencies modeling, a global data-dependent MLP (GDD-MLP) to effectively capture cross-channel dependencies, and a Channel Mixup mechanism to mitigate overfitting.\nComprehensive experiments conducted on seven real-world datasets demonstrate the efficacy of our model in improving forecasting performance."},"_bibtex":{"value":"@misc{\nzeng2025cmamba,\ntitle={{CM}amba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting},\nauthor={Chaolv Zeng and Zhanyu Liu and Guanjie Zheng and Linghe Kong},\nyear={2025},\nurl={https://openreview.net/forum?id=OclHGmt2ZM}\n}"},"title":{"value":"CMamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting"},"pdf":{"value":"/pdf/c6642767e189d0d9628d810af94b4cdc57554e7a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zeng|cmamba_channel_correlation_enhanced_state_space_models_for_multivariate_time_series_forecasting"},"authorids":{"value":["~Chaolv_Zeng2","~Zhanyu_Liu1","~Guanjie_Zheng1","~Linghe_Kong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chaolv Zeng","Zhanyu Liu","Guanjie Zheng","Linghe Kong"]}},"version":2},{"content":{"summary":{"value":"The paper tackles prediction over large cross-sections of units (loans, stocks, customers) where each unit is a time series. The authors propose a Set-Sequence architecture that factors the problem into two natural dimensions: Set modeling across entities at each time step (the “cross-section”). Sequence modeling through time for each entity (the “temporal” dimension). Experiments on (i) a synthetic contagion/default task, (ii) U.S. equity portfolio construction, and (iii) mortgage risk prediction (~5M loan-months; 52 features) show consistent gains, plus interpretable set summaries that correlate with latent factors."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- Did you compare early-vs-late insertion or stacking multiple Set layers? Any diminishing returns?\n- Will synthetic generator, equity feature pipelines, and mortgage preprocessing be released to enable reproduction?\n- This is just an open question: can the Set-Sequence concept be verified beyond financial or factor-style datasets?\n- To be honest, I am not sure about the assumption of exchangeability in real-world applications. Are they only approximately exchangeable or any exceptions?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"-  The idea is clear: set over units → sequence over time.\n- The method is scalable,  with theoretical complexity analysis, and can be extended when M is large.\n- Empirical results are broad and strong, both in Synthetic contagion and real-world datasets.\n- The model gives the Interpretability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The Set-Sequence architecture extends the idea of multiple instance learning mean pooling to the temporal domain. There are a few related works that are encouraged to be discussed or compared.\n\n[1] Zaheer, Manzil, et al. \"Deep sets.\" Advances in neural information processing systems 30 (2017).\n\n[2] Ilse, Maximilian, Jakub Tomczak, and Max Welling. \"Attention-based deep multiple instance learning.\" International conference on machine learning. PMLR, 2018.\n\n[3] Chen, Xiwen, et al. \"TimeMIL: Advancing multivariate time series classification via a time-aware multiple instance learning.\" ICML'24\n\n\n- The empirical section mainly compares against early Transformer-style and S4/LongConv models. Given the rapid development of efficient long-sequence and cross-sectional architectures, incorporating or at least discussing stronger, contemporaneous methods would be encouraged."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922780984,"tcdate":1761868552744,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11754/Reviewer_VTYC"],"signatures":["ICLR.cc/2026/Conference/Submission11754/Reviewer_VTYC"],"forum":"c1A3DCni3K","number":3,"license":"CC BY 4.0","cdate":1761868552744,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11754/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922780984,"domain":"ICLR.cc/2026/Conference","replyto":"c1A3DCni3K","id":"utbmiEPgpl","forumContent":{"TLDR":{"value":"When numerous time series are exchangeable, the Set-Sequence model efficiently captures cross-sectional dependencies"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Set modeling","Sequence modeling","High-dimensional time series","Cross-sectional dependencies","Interpretable AI","Synthetic data","Mortgage risk prediction","Exchangeability","Time series prediction","Financial AI","Deep learning"]},"supplementary_material":{"value":"/attachment/325e76334150d1a8a8e96c235d088931acd8c9ae.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Many prediction problems across science and engineering, especially in finance and economics, involve large cross-sections of individual time series, where each unit (e.g., a loan, stock, or customer) is driven by unit-level features and latent cross-sectional dynamics. While sequence models have advanced per-unit temporal prediction, capturing cross-sectional effects often still relies on hand-crafted summary features. We propose Set-Sequence, a model that learns cross-sectional structure directly, enhancing expressivity and eliminating manual feature engineering. At each time step, a permutation-invariant Set module summarizes the unit set; a Sequence module then models each unit’s dynamics conditioned on both its features and the learned summary. The architecture accommodates unaligned series, supports varying numbers of units at inference, integrates with standard sequence backbones (e.g., Transformers), and scales linearly in cross-sectional size. Across a synthetic contagion task and two large-scale real-world applications—equity portfolio optimization and loan risk prediction—Set-Sequence significantly outperforms strong baselines, delivering higher Sharpe ratios, improved AUCs, and interpretable cross-sectional summaries."},"_bibtex":{"value":"@misc{\nepstein2026a,\ntitle={A Set-Sequence Model for Time Series},\nauthor={Elliot L Epstein and Apaar Sadhwani and Kay Giesecke},\nyear={2026},\nurl={https://openreview.net/forum?id=c1A3DCni3K}\n}"},"title":{"value":"A Set-Sequence Model for Time Series"},"pdf":{"value":"/pdf/af39a1344d262612bd1bf64c15a33dbcc3a7f6ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"epstein|a_setsequence_model_for_time_series"},"authorids":{"value":["~Elliot_L_Epstein1","~Apaar_Sadhwani1","~Kay_Giesecke1"]},"authors":{"value":["Elliot L Epstein","Apaar Sadhwani","Kay Giesecke"]}},"version":2},{"content":{"summary":{"value":"The paper proposes a method for robot reasoning and manipulation by developing a multimodal large language model (MLLM) based on the Mamba state space model (SSM). The joint model is able to both attack robot reasoning tasks assessed via VQA, and solve robot manipulation tasks by fine-tuning a dedicated head that contains only a small fraction of the model parameters."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- Can you please provide results regarding comparison with SOTA methods in terms of time efficiency?\n- Can the training strategy, including the fine-tuning of only the manipulation policy head, be applied to other MLLMs?\n- Is the performance in the manipulation tasks affected by the number of parameters in the corresponding head and how?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The paper introduces an MLLM based on the Mamba model built specifically for robotic tasks. The exposition of the method is clear and well motivated. The proposed method takes advantage of the linear complexity of Mamba and, in combination to the reduced number of parameters used for the manipulation policy head, achieves higher efficiency and better performance with respect to state-of-the-art robot MLLM methods. Another strong point is the combination of natural language reasoning abilities and task specific manipulation policies based on the MLLM output. \n\nThe experimental evaluation is comprehensive, considering both simulated and real-world experiments, where the proposed method achieves competitive results with respect to SOTA methods across multiple benchmarks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Although time efficiency is one of the major claimed contributions of the work, the aspect of time efficiency is not well covered in the experimental evaluation. The fact that the proposed method is 7 times faster than SOTA MLLM methods is mentioned a couple of times, however detailed results regarding inference times are not provided. \n\nAnother aspect not covered in detail, regards the effect the number of parameters in the manipulation head has on the performance and efficiency of the manipulation task.\n\nAnother question that comes to mind, is whether it is possible to apply the idea of fine-tuning only a small-sized manipulation head to other MLLMs. Does Mamba work better with this constrained policy head and, if yes, why?\n\nFinally, another issue regards reproducibility, as details regarding the structure of the projection layers and the MLPs used in the manipulation policy head are not provided. \n\n### Minor comments\n- L.13: \"action pose prediction abilities\" is not very clear\n- L.100 \"ManipVQA, enhancing robotic\" verb missing\n- L.192: what is the angle representation considered?\n- L.235: \"on when\""},"limitations":{"value":"Limitations are discussed in the text."}},"nonreaders":[],"tmdate":1730879040558,"tcdate":1720969135417,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission5916/Reviewer_QK1g"],"signatures":["NeurIPS.cc/2024/Conference/Submission5916/Reviewer_QK1g"],"forum":"JxOQeg1NkH","number":5,"license":"CC BY 4.0","cdate":1720969135417,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission5916/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879040558,"domain":"NeurIPS.cc/2024/Conference","replyto":"JxOQeg1NkH","id":"PEnnHlPPCn","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Robot Manipulation","Multimodal Large Language Model","State Space Model"]},"supplementary_material":{"value":"/attachment/a8ec83e43dde9f6210ec7009438418ab8d320803.zip"},"primary_area":{"value":"robotics"},"abstract":{"value":"A fundamental objective in robot manipulation is to enable models to comprehend visual scenes and execute actions. Although existing Vision-Language-Action (VLA) models for robots can handle a range of basic tasks, they still face challenges in two areas: (1) insufficient reasoning ability to tackle complex tasks, and (2) high computational costs for VLA model fine-tuning and inference. The recently proposed state space model (SSM) known as Mamba demonstrates promising capabilities in non-trivial sequence modeling with linear inference complexity. Inspired by this, we introduce RoboMamba, an end-to-end robotic VLA model that leverages Mamba to deliver both robotic reasoning and action capabilities, while maintaining efficient fine-tuning and inference. Specifically, we first integrate the vision encoder with Mamba, aligning visual tokens with language embedding through co-training, empowering our model with visual common sense and robotic-related reasoning. To further equip RoboMamba with SE(3) pose prediction abilities, we explore an efficient fine-tuning strategy with a simple policy head. We find that once RoboMamba possesses sufficient reasoning capability, it can acquire manipulation skills with minimal fine-tuning parameters (0.1\\% of the model) and time. In experiments, RoboMamba demonstrates outstanding reasoning capabilities on general and robotic evaluation benchmarks. Meanwhile, our model showcases impressive pose prediction results in both simulation and real-world experiments, achieving inference speeds 3 times faster than existing VLA models."},"_bibtex":{"value":"@inproceedings{\nliu2024robomamba,\ntitle={RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation},\nauthor={Jiaming Liu and Mengzhen Liu and Zhenyu Wang and Pengju An and Xiaoqi Li and Kaichen Zhou and Senqiao Yang and Renrui Zhang and Yandong Guo and Shanghang Zhang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=JxOQeg1NkH}\n}"},"title":{"value":"RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation"},"pdf":{"value":"/pdf/d121b003663e64b010981e0a4b9e271adebd7c2d.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"liu|robomamba_efficient_visionlanguageaction_model_for_robotic_reasoning_and_manipulation"},"authorids":{"value":["~Jiaming_Liu2","~Mengzhen_Liu1","~Zhenyu_Wang11","~Pengju_An1","~Xiaoqi_Li3","~Kaichen_Zhou1","~Senqiao_Yang1","~Renrui_Zhang1","~Yandong_Guo2","~Shanghang_Zhang4"]},"authors":{"value":["Jiaming Liu","Mengzhen Liu","Zhenyu Wang","Pengju An","Xiaoqi Li","Kaichen Zhou","Senqiao Yang","Renrui Zhang","Yandong Guo","Shanghang Zhang"]}},"version":2},{"content":{"summary":{"value":"UMZero is a hybrid ZSL framework combining CNN and state space models to capture both local and global features. Its core includes a High-order Global Aggregator for channel dependencies, a Mamba Interaction Module for cross-modal fusion, and prototype learning for discriminative representations, effectively solving cross-modal interaction challenges."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"See above."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"UMZero's architecture comprises three specialized modules: the High-order Global Aggregator (HGA) enhances features through cross-channel dependency modeling; the Mamba Interaction Module (MIM) enables bidirectional local-global cross-modal fusion; and the prototype learning unit constructs a discriminative semantic embedding space. Together, these components effectively address cross-modal interaction challenges in zero-shot learning. The integrated modules significantly strengthen cross-modal representation learning. Besides, decent performance is obtained on three benchmarks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Despite the interesting contributions, I have several significant concerns regarding the manuscript:\n1. The manuscript justifies the use of Mamba over Transformer by citing its linear complexity. However, this claim lacks empirical support. To substantiate this design choice, it is essential to include an efficiency analysis comparing GPU memory, inference time, number of parameters, and GFLOPs. Furthermore, a performance comparison with a self-attention-based baseline is necessary to validate the advantage of the proposed approach.\n2. The related work and experiments primarily focus on embedding-based methods. To better position the contribution, the review should be expanded to include more recent generative approaches for zero-shot learning, particularly state-of-the-art methods from 2024-2025.\n3. To better interpret the model's behavior, the authors should provide attention visualizations on real samples, illustrating what the model focuses on during prediction.\n4. The performance improvement on AWA2 is notably more significant than on CUB and SUN. The authors should analyze and discuss the potential reasons for this dataset-dependent performance variation.\n5. The authors should discuss the potential of integrating large language models (LLMs) or vision-language models (VLMs) like CLIP. Exploring how their method could leverage these powerful pre-trained representations would strengthen the discussion and highlight future research directions."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942597945,"tcdate":1761957135005,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23305/Reviewer_7XYH"],"signatures":["ICLR.cc/2026/Conference/Submission23305/Reviewer_7XYH"],"forum":"UZJPcLtQkx","number":4,"license":"CC BY 4.0","cdate":1761957135005,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23305/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942597945,"domain":"ICLR.cc/2026/Conference","replyto":"UZJPcLtQkx","id":"VXaoMpEn6j","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Zero-shot learning (ZSL)；State space models；Cross-modal interactions; Prototype Contrastive"]},"supplementary_material":{"value":"/attachment/3db11224bd373e868c09ba3e0ed90d1d03585fb8.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"In zero-shot learning (ZSL), accurately recognizing unseen classes relies heavily on deeply understanding both local details and global correlations between visual and semantic modalities. While prior methods have leveraged CNNs for local feature extraction or utilized Transformers and Mamba models to capture global contextual information, these approaches often lack an integrated mechanism to jointly model both aspects. This limitation hampers their ability to fully exploit complex cross-modal interactions. To overcome this challenge, we propose UMZero, a novel hybrid ZSL framework that synergistically combines a pretrained CNN with a state space model (SSM), effectively uniting fine-grained local feature extraction with long-range dependency modeling. UMZero is composed of three core modules: the High-order Global Aggregator (HGA), the Mamba Interaction Module (MIM), and a prototype learning unit. Specifically, the HGA enhances feature expressiveness by capturing high-order statistical dependencies across channels; the MIM enables deep cross-modal fusion and alignment by jointly modeling local-global interactions and supporting bidirectional information flow between modalities; and the prototype learning module constructs a semantically structured embedding space, promoting compact intra-class and discriminative inter-class representations. Extensive evaluations on several well-established ZSL datasets confirm that UMZero surpasses current state-of-the-art methods, delivering superior performance and demonstrating robust generalization capabilities. The source code will be publicly released upon paper acceptance."},"_bibtex":{"value":"@misc{\nlei2026umzero,\ntitle={{UMZ}ero: A Unified {CNN}-Mamba Framework for Zero-Shot learning},\nauthor={Yu Lei and Junping Wang and Quanxue Gao and ShunxinCheng and Qin Li},\nyear={2026},\nurl={https://openreview.net/forum?id=UZJPcLtQkx}\n}"},"title":{"value":"UMZero: A Unified CNN-Mamba Framework for Zero-Shot learning"},"pdf":{"value":"/pdf/5a00bafead3301124fd587813fe4aba52bc85553.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"lei|umzero_a_unified_cnnmamba_framework_for_zeroshot_learning"},"authorids":{"value":["~Yu_Lei12","~Junping_Wang3","~Quanxue_Gao1","~ShunxinCheng1","~Qin_Li6"]},"authors":{"value":["Yu Lei","Junping Wang","Quanxue Gao","ShunxinCheng","Qin Li"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2606.04493v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"wang|sfmambanet_spectralfrequency_enhanced_selective_state_space_model_for_correspondence_pruning"},"html":{"value":"https://doi.org/10.48550/arXiv.2606.04493"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2606-04493,\n  publtype={informal},\n  author={Zhihua Wang and Yanping Li and Yizhang Liu},\n  title={SFMambaNet: Spectral-Frequency Enhanced Selective State Space Model for Correspondence Pruning},\n  year={2026},\n  month={June},\n  cdate={1780272000000},\n  journal={CoRR},\n  volume={abs/2606.04493},\n  url={https://doi.org/10.48550/arXiv.2606.04493}\n}\n"},"abstract":{"value":"Correspondence pruning aims to identify inliers from an initial set of correspondences. Most existing Graph Neural Network (GNN)-based methods rely on geometric features mapped from coarse Euclidean coordinates, which struggle to capture the subtle geometric consistencies presented by inliers. While Mamba-based methods possess global receptive fields and long sequence modeling capabilities, they tend to accumulate substantial inconsistent features within the hidden state space, making it difficult to distinguish inliers from outliers. In this paper, we integrate frequency domain perception into this task for the first time and propose SFMambaNet, a novel Spectral-Frequency enhanced Mamba-based two-view correspondence pruning network. Our method is collaboratively composed of two components: First, we design a Local Spectral-Geometric Attention (LSGA) block. LSGA incorporates spectral positional encoding into local graph interactions and introduces multi-scale Mamba processing to enhance the capture of subtle geometric consistencies and improve local feature discriminability. Building upon this, we design a Spectral-Integrated Global Mamba (SIGM) block. SIGM embeds a frequency gating mechanism within the state space, utilizing the frequency information provided by LSGA to explicitly suppress high-frequency noise accumulation within hidden states and mitigate the propagation of inconsistent features. This enhances inlier-outlier separability and achieves robust global context modeling capabilities with nearly linear complexity. Extensive experiments demonstrate that SFMambaNet outperforms current state-of-the-art methods on several challenging tasks. The code is available at https://github.com/Kirito14IT/SFMambaNet."},"title":{"value":"SFMambaNet: Spectral-Frequency Enhanced Selective State Space Model for Correspondence Pruning"},"authors":{"value":[{"fullname":"Zhihua Wang","username":""},{"fullname":"Yanping Li","username":""},{"fullname":"Yizhang Liu","username":"~Yizhang_Liu1"}]}},"tmdate":1784902079328,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2606-04493"],"tcdate":1784902075409,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Yizhang_Liu1"],"forum":"KE0jJRJFhk","license":"CC BY-SA 4.0","number":100178,"cdate":1780272000000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784902079328,"domain":"OpenReview.net/Public_Article","id":"KE0jJRJFhk","version":2},{"content":{"summary":{"value":"This paper proposes Mamba, a linear-time sequence model with an intra-layer combination of Selective S4D, Short Convolution and Gated Linear Unit. The paper introduces input-dependent gating to SSM and provides a hardware-aware efficient implementation for the selective parallel scan operator. Mamba shows strong performance in language, DNA and audio modeling compared to models based on attention/RNN/convolution."},"ethics_flag":{"value":1},"reasons_to_reject":{"value":"1. Lack of Ablation Study: Besides the proposed Selective S4D module, the Mamba layer also includes the Short Convolution module which can also be added to other linear recurrent models. It is not clear how Short Convolution contributes to the performance of Mamba.\n2. Weak Baselines: The table 2 does not include the state-of-the-art transformer++ architecture that is used by Llama models, and the included baselines either have suboptimal architectural configurations or bad optimization/initialization recipes.\n3. Lack of Training Efficiecy Measurement: The strong performance of Mamba can be due to the fact that it has far more layers than other models. The authors should show the training cost of Mamba when compared with other language models in Table 2.\n4. The relationship to other input selection mechnism: While the authors claims that they \"use selection to refer to the mechanistic **action** of a model to select or ignore inputs\", Mamba does not make any discrete actions to select the input, but rather uses a soft gating techinque similar to Gated Recurrent Units. If the authors still want to use the terminology of selection, they should discuss its relationship to the works [1, 2, 3, 4] that have real discrete selections of inputs.\n5. The length extrapolation experiments are only conducted on the synthetic setting. The synthetic induction head task is too simple and the authors make a strong claim of perfect length extrapolation. However, this doesn't even translate to the language modeling setting where Mamba is known to be not extrapolable as tested in [5] and [6]. I would suggets authors to (i) conduct experiments on a more challenging synthetic setting such as Multi-Query Associative Recall (MQAR), and (ii) avoid claiming the infinite extrapolation ability of Mamba.\n6. The settings of the scaling law experiments are unclear and somewhat misleading. See question 6 below for more details. Also, in Figure 4, while increasing the sequence length from 2K to 8K, Mamba seems to have a worse scaling trend compared with Transformer++ when the training FLOPs is larger than $10^{20}$.\n\n---\n[1] Efficient Content-Based Sparse Attention with Routing Transformers (TACL 2020)\n\n[2] Reformer: The Efficient Transformer (ICLR 2020)\n\n[3] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (JMLR 2022)\n\n[4] Sparse Modular Activation for Efficient Sequence Modeling (NeurIPS 2023)\n\n[5] https://github.com/jzhang38/LongMamba\n\n[6] Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024)"},"confidence":{"value":5},"reasons_to_accept":{"value":"1. The strong performance of the model on both synthetic and the real world tasks.\n2. The proposed Selective State Space Models and its efficient implementation are significant contributions to the community of SSM and linear recurrence."},"rating":{"value":6},"questions_to_authors":{"value":"1. How is the training wall time of Mamba compared to other models of similar sizes?\n2. How is the performance of Mamba compared to the Llama architecture on downstream NLP tasks?\n3. Is the short convolution module critical to Mamba's performance? What if we remove it?\n4. It seems that in the Mamba official implementation, the $\\mathbf{B}$ parameter in S6 is not using discretization, but rather a simple outer product. How does this discrepency affect the performance?  Does this indicate discretization is not neccessary for gated linear recurrent models? What about removing discretization for A? How is the discretization important generally?\n5. Why is Mamba not length extrapolable in real-world language modeling?\n6. How is the FLOPs calcuated in the scaling law experiments? Why using the Chinchilla protocol which is built for Transformers instead of fitting a new curve for Mamba?"}},"nonreaders":[],"tmdate":1724633724869,"tcdate":1715538265625,"writers":["colmweb.org/COLM/2024/Conference","colmweb.org/COLM/2024/Conference/Submission1039/Reviewer_F6hZ"],"signatures":["colmweb.org/COLM/2024/Conference/Submission1039/Reviewer_F6hZ"],"forum":"tEYskw1VY2","number":2,"license":"CC BY 4.0","cdate":1715538265625,"readers":["everyone"],"invitations":["colmweb.org/COLM/2024/Conference/Submission1039/-/Official_Review","colmweb.org/COLM/2024/Conference/-/Edit"],"mdate":1724633724869,"domain":"colmweb.org/COLM/2024/Conference","replyto":"tEYskw1VY2","id":"XSB8vvovkv","forumContent":{"TLDR":{"value":"This paper introduces the Mamba architecture."},"venue":{"value":"COLM"},"code_of_ethics":{"value":["I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"]},"keywords":{"value":"sequence model, deep learning, state space model, S4, Mamba"},"abstract":{"value":"Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5x higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation."},"_bibtex":{"value":"@inproceedings{\ngu2024mamba,\ntitle={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},\nauthor={Albert Gu and Tri Dao},\nbooktitle={First Conference on Language Modeling},\nyear={2024},\nurl={https://openreview.net/forum?id=tEYskw1VY2}\n}"},"title":{"value":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces"},"research_area":{"value":["Compute efficient LMs","Learning algorithms for LMs"]},"pdf":{"value":"/pdf/05ba820a837c2f704f5b5bb1ccdb66c503b0e180.pdf"},"venueid":{"value":"colmweb.org/COLM/2024/Conference"},"paperhash":{"value":"gu|mamba_lineartime_sequence_modeling_with_selective_state_spaces"},"authorids":{"value":["~Albert_Gu1","~Tri_Dao1"]},"author_guide":{"value":["I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"]},"authors":{"value":["Albert Gu","Tri Dao"]}},"version":2},{"content":{"summary":{"value":"Point cloud segmentation is crucial for 3D understanding, traditionally tackled using CNNs or Transformers. Recently, Mamba has emerged as an efficient solution for contextual modeling, though it has struggled to outperform leading CNN and Transformer methods. This work identifies key requirements for effective segmentation: strong spatial locality and robust contextual understanding. Enhancing Mamba, the authors remove causality and introduce a Strided Bidirectional SSM to address directional biases in unordered point clouds. The resulting architecture, MEEPO, merges the strengths of CNNs and Mamba, achieving up to +0.8 mIoU over state-of-the-art methods on benchmarks like ScanNet and nuScenes, with notable gains in speed and memory efficiency."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"see above weakness"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper is well-structured with a clear analysis-driven approach.\n2. The experimental results are impressive."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. In lines 238-241, it's unclear why Mamba is considered effective for local processing, given its design goal of modeling long sequences with linear complexity. The reference to a \"locally-biased forget gate\" needs further explanation and detailed analysis or statistical data beyond visualization.\n\n2. In lines 300-302, the statement that long-range attention is unnecessary may only partially reflect the issue. The core reason might be the sparsity of 3D point clouds. Transformers generally require extensive data and prolonged training to surpass CNNs, so it might be more accurate to say that insufficient data limits the full potential of Transformers rather than implying long-range attention is irrelevant.\n\n3. The innovations appear somewhat limited, with minimal structural changes. The modifications, such as Causal-Free Mamba and Bidirectional Strided SSM, feel more like tricks.\n\n4. Presenting results on downstream perception tasks like segmentation and detection would add further value.\n\nMinor Errors:\n- Line 251: Fig.4 should be Fig.5.\n- clarify the point cloud processing order in Mamba. Is it similar to PTv3's order?"}},"nonreaders":[],"tmdate":1732504018003,"tcdate":1729172074533,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1247/Reviewer_fEVg"],"signatures":["ICLR.cc/2025/Conference/Submission1247/Reviewer_fEVg"],"forum":"E1ML0nEReb","number":1,"license":"CC BY 4.0","cdate":1729172074533,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1247/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732504018003,"domain":"ICLR.cc/2025/Conference","replyto":"E1ML0nEReb","id":"6g87iOxqEj","forumContent":{"TLDR":{"value":"This paper analyzes the key components that make a model architecture efficient and effective for point cloud segmentation and explores various methods to implement these components successfully."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["point cloud segmentation","efficient","contextual modeling"]},"supplementary_material":{"value":"/attachment/62537ba3107759f903cfd3e8f9ea61fd79348a56.zip"},"primary_area":{"value":"applications to robotics, autonomy, planning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Point cloud segmentation is an important topic in 3D understanding that has traditionally been tackled using either the CNN or Transformer. Recently, Mamba has emerged as a promising alternative, offering efficient long-range contextual modeling capabilities without the quadratic complexity associated with Transformer's attention mechanisms. However, despite Mamba's potential, early efforts have all failed to achieve better performance than the best CNN-based and Transformer-based methods. In this work, we address this challenge by identifying the key components of an effective and efficient point cloud segmentation architecture. Specifically, we show that: 1) Spatial locality and robust contextual understanding are critical for strong performance, and 2) Mamba features linear computational complexity, offering superior data and inference efficiency compared to Transformers, while still being capable of delivering strong contextual understanding. Additionally, we further enhance the standard Mamba specifically for point cloud segmentation by identifying its two key shortcomings. First, the enforced causality in the original Mamba is unsuitable for processing point clouds that have no such dependencies. Second, its unidirectional scanning strategy imposes a directional bias, hampering its ability to capture the full context of unordered point clouds in a single pass. To address these issues, we carefully remove the causal convolutions and introduce a novel Bidirectional Strided SSM to enhance the model's capability to capture spatial relationships. Our efforts culminate in a novel architecture named MEEPO that effectively integrates the strengths of CNN and Mamba. MEEPO surpasses the previous state-of-the-art method, PTv3, by up to +0.8 mIoU on multiple key benchmark datasets, while being 42.1\\% faster and 5.53$\\times$ more memory efficient. Our code will be released."},"_bibtex":{"value":"@misc{\nchng2025exploring,\ntitle={Exploring contextual modeling with linear complexity for point cloud segmentation},\nauthor={Yong Xien Chng and Xuchong QIU and Henry Zheng and Yizeng Han and Yifan Pu and Jiewei Cao and Gao Huang},\nyear={2025},\nurl={https://openreview.net/forum?id=E1ML0nEReb}\n}"},"title":{"value":"Exploring contextual modeling with linear complexity for point cloud segmentation"},"pdf":{"value":"/pdf/6670e7aa64624951a356099a0f6c5ec34dd15c15.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chng|exploring_contextual_modeling_with_linear_complexity_for_point_cloud_segmentation"},"authorids":{"value":["~Yong_Xien_Chng1","~Xuchong_QIU1","~Henry_Zheng1","~Yizeng_Han1","~Yifan_Pu1","~Jiewei_Cao2","~Gao_Huang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yong Xien Chng","Xuchong QIU","Henry Zheng","Yizeng Han","Yifan Pu","Jiewei Cao","Gao Huang"]}},"version":2},{"content":{"summary":{"value":"This paper presents a new autoregressive image generation framework, M-VAR, which leverages bidirectional self-attention for intra-scale modeling and the Mamba mechanism for inter-scale modeling.\nThe proposed method seems improved both the computational efficiency and the quality of generated images compared to VAR."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"see weaknesses"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"(1) The statistics of attention score and computation cost of the attention in VAR are interesting and inspiring.\n\n(2) The combination of intra-scale self-attention and inter-scale linear modeling seems a reasonable solution to improve the computational efficiency of VAR.\n\n(3) The largest model of M-VAR achieves SOTA FIDs on ImageNet dataset."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"(1) The decoupling of scale-wise autoregressive modeling seems reasonable, but why we must adopt Mamba? Other efficient self-attention variants should also be considered.\n\n(2) In Table 2, M-VAR-dX usually has more parameters than VAR-dX. Are these additional parameters help M-VAR for better performance? \n\n(3) The computational FLOPS are not discussed in this article, since the number of paramters is not the only factor affecting computational efficiency. \n\n(4) The curve shown in Figure 5 might seem counter-intuitive, why global attention performances worse despite its global modeling capacity?  \n\n(5) Some typos, e.g. L480 'As shown in Table 5', it should be 'Figure 5'?"}},"nonreaders":[],"tmdate":1731427516707,"tcdate":1730710426234,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1959/Reviewer_sAvT"],"signatures":["ICLR.cc/2025/Conference/Submission1959/Reviewer_sAvT"],"forum":"BYoN2c0o6M","number":4,"license":"CC BY 4.0","cdate":1730710426234,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1959/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427516707,"domain":"ICLR.cc/2025/Conference","replyto":"BYoN2c0o6M","id":"iaB6ZrDIYh","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Scale-wise Autoregressive Model"]},"primary_area":{"value":"generative models"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"There exists recent work in computer vision, named VAR, that proposes a new autoregressive paradigm for image generation. Diverging from the vanilla next-token prediction, VAR structurally reformulates the image generation into a coarse to fine next-scale prediction. \nIn this paper, we show that this scale-wise autoregressive framework can be effectively decoupled into \\textit{intra-scale modeling}, which captures local spatial dependencies within each scale, and \\textit{inter-scale modeling}, which models cross-scale relationships progressively from coarse-to-fine scales.\nThis decoupling structure allows to rebuild VAR in a more computationally efficient manner. Specifically, for intra-scale modeling --- crucial for generating high-fidelity images --- we retain the original bidirectional self-attention design to ensure comprehensive modeling; for inter-scale modeling, which semantically connects different scales but is computationally intensive, we apply linear-complexity mechanisms like Mamba to substantially reduce computational overhead. \nWe term this new framework M-VAR. Extensive experiments demonstrate that our method outperforms existing models in both image quality and generation speed. For example, our 1.5B model, with fewer parameters and faster inference speed, outperforms the largest VAR-d32-2B. Moreover, our largest model M-VAR-d32 impressively registers 1.78 FID on ImageNet 256$\\times$256 and outperforms the prior-art autoregressive models LlamaGen/VAR by 0.4/0.19 and popular diffusion models LDM/DiT by 1.82/0.49, respectively."},"_bibtex":{"value":"@misc{\nren2024mvar,\ntitle={M-{VAR}: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation},\nauthor={Sucheng Ren and Yaodong Yu and Nataniel Ruiz and Feng Wang and Alan Yuille and Cihang Xie},\nyear={2024},\nurl={https://openreview.net/forum?id=BYoN2c0o6M}\n}"},"title":{"value":"M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation"},"pdf":{"value":"/pdf/8835246a72db2706a543e2db5e5821a0a7a9965d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"ren|mvar_decoupled_scalewise_autoregressive_modeling_for_highquality_image_generation"},"authorids":{"value":["~Sucheng_Ren1","~Yaodong_Yu4","~Nataniel_Ruiz1","~Feng_Wang14","~Alan_Yuille1","~Cihang_Xie3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Sucheng Ren","Yaodong Yu","Nataniel Ruiz","Feng Wang","Alan Yuille","Cihang Xie"]}},"version":2},{"content":{"summary":{"value":"The paper systematically investigates the effects of various normalization techniques and their combinations within the Mamba architecture, finding that applying normalization after the S6 Module improves performance and stability. It proposes practical guidelines for selecting optimal normalization strategies to enhance the efficiency and robustness of Mamba networks. But there are still many problems that need to be paid attention to."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Can the authors include additional diverse tasks from different domains to validate the effectiveness of normalization techniques more broadly?\n2. Can the authors provide a more in-depth analysis or explanation for the contradictory findings between vision and sequence tasks, and the potential causes for these differences?\n3. Can the authors provide detailed information on the experimental settings, including hyperparameters, evaluation metrics, and data preprocessing steps, to ensure reproducibility?\n4. Can the authors improve the visualization in Figure 3 by standardizing the x-axes, adding clearer labels, and ensuring consistent plotting styles across subfigures?\n5. Can the authors provide more empirical evidence supporting the Weight Domination hypothesis or explain the apparent contradictions in the results related to the BN→None configuration?\n6. Can the authors strive to derive more generalizable conclusions or provide guidance on when specific normalization techniques may be more appropriate, rather than relying heavily on task-specific findings?"},"rating":{"value":3},"details_of_ethics_concerns":{"value":"None."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Relevance and Importance: \nThe study addresses a critical aspect of deep learning model stability, particularly in the context of the Mamba architecture, which is gaining attention for its efficiency in handling long sequences.\n\n2. Practical Insights: \nThe paper provides valuable insights into the positioning of normalization layers, which can be directly applicable to practitioners working with Mamba networks.\n\n3. Comprehensive Analysis: \nThe study systematically evaluates various normalization techniques and their combinations, offering a detailed analysis of their effects on model performance and stability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Limited Task Validation:\nThe study is limited to only two tasks: long sequence modeling with the LRA ListOps dataset, and image classification with ImageNet-100. To draw broader conclusions about the generalizability of normalization techniques across different domains, the authors should consider including additional diverse tasks such as natural language processing, speech recognition, or reinforcement learning tasks.\n2. Contradictory Findings and Lack of Explanation: \nThe findings appear to be task-dependent, with contradictory results between vision and sequence tasks. For instance, the paper suggests that normalization after the S6 Module is beneficial for sequence modeling (Section 4.1), but this does not hold for image classification tasks (Section 4.3). The authors should provide a more in-depth analysis or explanation of why the results differ between tasks, as this could lead to valuable insights about how normalization techniques interact with different types of data and model architectures.\n3. Methodological Issues: \n3.1 Reproducibility Issues: The experimental settings are not well-documented, making it difficult to replicate the results. Crucial details about accuracy metrics in tables are missing. For example, Table 2 and Table 3 lack information on the specific accuracy metrics used. To ensure reproducibility, the authors should provide detailed information about the experimental setup, including hyperparameters, evaluation metrics, and any data preprocessing steps.\n3.2 Inconsistent Visualization: The visualization in Figure 3 is inconsistent, with varying x-axes and plotting styles, which hampers the interpretation of the results. For instance, the x-axes in subfigures (2), (3), and (4) are not clearly labeled, making it hard to compare the weight norms across different configurations. The authors should standardize the x-axes, add clearer labels, and ensure consistent plotting styles across subfigures to improve the interpretability of the results.\n4. Theoretical Foundation:\nWeight Domination Hypothesis: The Weight Domination hypothesis, which is central to the paper's findings, lacks sufficient supporting evidence. The contradictory results regarding the BN→None performance weaken the credibility of this hypothesis. For example, the paper states that the BN→None configuration leads to significant disparity in weight norms (Section 4.1), but this is not consistently observed across all experiments. The authors should provide more empirical evidence supporting the Weight Domination hypothesis or explain the apparent contradictions in the results.\n5. Limited Generalizability: \n5.1 Task-Specific Conclusions: The conclusions appear to be highly task-specific, which limits their generalizability. For instance, the results from Vision Mamba, which show that LN is sufficient, contradict the findings from sequence modeling. The paper states that LN+RMSN underperformed relative to LN+LN (Section 4.3), but this contradicts the findings in sequence modeling tasks. The authors should strive to provide more general insights that could replace task-specific ablation studies and offer a unified framework for understanding normalization techniques across different tasks.\n5.2 Ablation Studies: The work falls short of providing general insights that could replace task-specific ablation studies. For example, the paper concludes that BN is particularly effective when placed after the Mamba block (Section 4.3), but this is not generalized to other tasks. The authors should aim to derive more generalizable conclusions or provide guidance on when specific normalization techniques may be more appropriate."}},"nonreaders":[],"tmdate":1731428661550,"tcdate":1730347402601,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9166/Reviewer_tLtQ"],"signatures":["ICLR.cc/2025/Conference/Submission9166/Reviewer_tLtQ"],"forum":"YK8eO7BEkJ","number":1,"license":"CC BY 4.0","cdate":1730347402601,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9166/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428661550,"domain":"ICLR.cc/2025/Conference","replyto":"YK8eO7BEkJ","id":"XVfwhbwDLL","forumContent":{"TLDR":{"value":"We conduct an emprical study on the role of extra normalization layers in enhancing model performance and stability during the training for Mamba architecture"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","long-sequence modeling","normalization","performance","stability"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Normalization layers are crucial for improving the training efficiency and stability of deep neural network architectures. The recently proposed Mamba network has demonstrated significant potential in competing with Transformers. However, as with many deep architectures, the training stability of Mamba remains a significant challenge, and normalization techniques are key to addressing this issue. In this paper, we systematically investigate the effects of normalization type, position and combinations on the Mamba Block. On the one hand, we conducted extensive experiments to evaluate the impact of applying various normalization layers before or after the SSM module(the core module of Mamba Block). On the other hand, we performed thorough experiments to assess the effects of combining diverse normalization techniques before and after the SSM module. Our analysis encompasses both long sequence modeling and image classification tasks. The results show that applying normalization layers after the SSM module (if used only once) and combining different normalization layers before and after the SSM module can enhance training stability and improve Mamba performance. Furthermore, we provide practical recommendations for selecting appropriate normalization techniques in designing Mamba  architectures and validated them on other datasets. We hope that our insights will help mitigate training instabilities in deep learning and foster the development of more robust architectures. All codes and models used in this study will be open-sourced on GitHub."},"_bibtex":{"value":"@misc{\nfeng2025an,\ntitle={An Empirical Study on Normalization in Mamba},\nauthor={Peilin Feng and Yuanshuai Wang and Yunhao Ni and Zekun Li and Wenjun Wu and Lei Huang},\nyear={2025},\nurl={https://openreview.net/forum?id=YK8eO7BEkJ}\n}"},"title":{"value":"An Empirical Study on Normalization in Mamba"},"pdf":{"value":"/pdf/9e73f11d8e2a5213c03f3049e0d3b4489169eff2.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"feng|an_empirical_study_on_normalization_in_mamba"},"authorids":{"value":["~Peilin_Feng1","~Yuanshuai_Wang1","~Yunhao_Ni1","~Zekun_Li11","~Wenjun_Wu2","~Lei_Huang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Peilin Feng","Yuanshuai Wang","Yunhao Ni","Zekun Li","Wenjun Wu","Lei Huang"]}},"version":2},{"content":{"summary":{"value":"This paper proposes UIT-Pred, a trajectory prediction framework for autonomous driving that is designed to work under intermittent observation, e.g. variable-length histories and missing inputs. It uses time-aware input representation so the model doesn’t need to rely on fixed windows or masks. It uses a Bidirectional Time Decay Mamba (BTD-Mamba) to capture sequential dependencies in both forward and backward directions across input observations. Experiments on Argoverse 1 and 2 Datasets show consistent gains over baselines (HiVT, QCNet, DeMo, Forecast-MAE)."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"There are many clear typos and small grammar issues. Here is a list:\n1. Line 21: “Particularly, We extend” → “Particularly, we extend”\n2. Line 40: “are are designed” → “are designed”\n3. Line 41: “observations are often intermitten” → intermittent\n4. Line 61: “variale-length” → “variable-length”\n5. Line 72-73: “intervals feature” → “interval features”\n6. Line 85: “to effectively captures” → “to effectively capture”\n7. Line 107: “pedestrians, or cyclist” → “pedestrians, or cyclists”\n8. Line 420: “Observations Lengths” → “Observation Lengths”"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The time-aware input representation allows the model reason about intermittency without validity masks.\n2. It extends Mamba to BTD-Mamba to preserve temporal coherence under missing observations.\n3. On Argoverse 1 and 2 Datasets, the model outperforms mask-based baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The notations in this paper are very confusing. Usually we use $x_t$ for position at time $t$, but this paper uses $t_x$ for position and $t_{vel}$ for velocity. The paper also uses $t_i$ for timestamps, $\\Delta t_i$ for gaps, and $t’, t’’$ for temporal embeddings. \n2. The core ideas of time-aware input features (scaled timestamps, inter-observation gaps) and a Bidirectional Time-Decay Mamba are reasonable, but read as modest adaptations of known ingredients (temporal features + bidirectional/decay in SSMs) rather than a clearly new idea.\n3. The baselines this paper is comparing against are old. For example, on the current Argoverse 2 leaderboard, top models can achieve MR$_6$ of 0.11 or 0.12, and minADE$_1$ of 1.42.\n4. No complexity/runtime is given."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926858313,"tcdate":1761983960103,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16838/Reviewer_cpBU"],"signatures":["ICLR.cc/2026/Conference/Submission16838/Reviewer_cpBU"],"forum":"ThZeZdRxna","number":4,"license":"CC BY 4.0","cdate":1761983960103,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16838/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926858313,"domain":"ICLR.cc/2026/Conference","replyto":"ThZeZdRxna","id":"3Q3DL7Oihq","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Trajectory prediction","Autonomous Driving","Attention","Mamba","Intermittent Data"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Trajectory prediction is a fundamental component of autonomous driving, requiring models that can handle intermittent observation patterns such as variable-length histories and missing data. Existing state-of-the-art methods, however, often assume fixed-length trajectories and complete input, which challenges their applicability in real-world scenarios where sensor occlusions, communication delays, and temporal sparsity are common. Moreover, conventional approaches typically address tasks such as trajectory prediction, variable-length modeling, or missing data handling in isolation, making them less effective in multi-task settings that naturally arise in practice. To address these challenges, we propose Universal Intermittent Trajectory Predictor (UIT-Pred) that processes inputs with the time index features, which capture temporal variations to effectively adapt to diverse input patterns within the domain. Particularly, We extend recent State Space Models (SSMs) by introducing the Bidirectional Time Decay Mamba (BTD-Mamba), designed to capture dependencies both forward and backward along the sequence. By integrating a decay process, BTD-Mamba effectively analyzes trajectories while maintaining relationships under intermittent observation. Furthermore, the proposed prediction module employs state encoding to capture the underlying motion patterns in the input data and models a multimodal trajectory distribution to account for uncertainty in future predictions. These components are fused through a unified fusion module, enabling the model to jointly reason over observed dynamics and potential future behaviors. Extensive experiments on Argoverse 1 and Argoverse 2 datasets validate the effectiveness of the proposed model. By simultaneously handling prediction, variable-length observations, and missing inputs within a universal architecture, the framework proposes to meet the challenges of real-world autonomous driving systems."},"_bibtex":{"value":"@misc{\nbharilya2026uitpred,\ntitle={{UIT}-Pred: Universal Intermittent Trajectory Predictor for Autonomous Driving},\nauthor={Vibha Bharilya and Neetesh Kumar},\nyear={2026},\nurl={https://openreview.net/forum?id=ThZeZdRxna}\n}"},"title":{"value":"UIT-Pred: Universal Intermittent Trajectory Predictor for Autonomous Driving"},"pdf":{"value":"/pdf/09ddf9106e39f3ead7441bc862c70e71d213fcad.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"bharilya|uitpred_universal_intermittent_trajectory_predictor_for_autonomous_driving"},"authorids":{"value":["~Vibha_Bharilya1","~Neetesh_Kumar1"]},"authors":{"value":["Vibha Bharilya","Neetesh Kumar"]}},"version":2},{"content":{"summary":{"value":"The paper introduces Linear Attention Sequence Parallelism (LASP), a novel sequence parallelism approach specifically designed for linear attention models to enhance training on long sequences across multiple GPUs. Unlike traditional sequence parallelism (SP) methods, LASP leverages the unique properties of linear attention, such as the right-product kernel trick, which optimizes both communication and computation efficiency. LASP also incorporates GPU-friendly optimizations like kernel fusion and key-value (KV) state caching."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. One limitation of linear attention is that it may cause obvious accuracy drops in real models, like GPT and Llama models. Could authors provide an accuracy comparison of real models or applications when adopting proposed methods?\n2. The comparisons among DeepSpeed-Ulysses, Megatron-SP are not quite fair. The other methods use origin attentions (softmax) which definitely has high computation, which causes low throughput.  Could we provide some explanations?\n3. Authors claim \"sharply decreases the communication overhead\", could authors provide more data about reduction in communication?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The approach leverages the right-product kernel trick specifically for linear attention models and focuses on a method that capitalizes on linear attention's unique properties.\n2. LASP incorporates GPU-friendly optimizations like kernel fusion and key-value (KV) state caching, which improve its execution efficiency on GPUs.\n3. LASP is compatible with various distributed data-parallel methods and supports integration with tensor and pipeline parallelism."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. While LASP is optimized for linear attention models, its applicability may be limited when dealing with standard attention models, such as those relying on Softmax.\n2. The claim system-level optimizations contribution like kernel fusion and KV state caching is not strong since these techniques are pure engineering implementation. \n3. The comparisons among DeepSpeed-Ulysses, Megatron-SP are not quite fair. The other methods use origin attentions (softmax) which definitely has high computation, which causes low throughput."}},"nonreaders":[],"tmdate":1731428086116,"tcdate":1730645032712,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5885/Reviewer_WPHj"],"signatures":["ICLR.cc/2025/Conference/Submission5885/Reviewer_WPHj"],"forum":"oVnfVnwh6y","number":3,"license":"CC BY 4.0","cdate":1730645032712,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5885/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428086116,"domain":"ICLR.cc/2025/Conference","replyto":"oVnfVnwh6y","id":"MQHK7IoZ6H","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["sequence parallelism","distributed training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Sequence parallelism (SP) serves as a prevalent strategy to handle long sequences that exceed the memory limit of a single device. However, for linear sequence modeling methods like linear attention, existing SP approaches do not take advantage of their right-product-first feature, resulting in sub-optimal communication efficiency and usability. In this paper, we introduce Linear Attention Sequence Parallelism (LASP), an efficient SP approach designed for linear attention-based transformer models. Specifically, we design an efficient point-to-point ring-style communication mechanism to leverage the right-product kernel trick of linear attention, which sharply decreases the communication overhead, comparing with existing SP methods. We enhance the computation efficiency of LASP by performing kernel fusion and intermediate state caching, making the implementation of LASP hardware-friendly on GPUs. Furthermore, we meticulously ensure the compatibility of sequence-level LASP with all types of batch-level data parallel methods, which is vital for distributed training on large clusters with very-long sequences. We also discuss the generalization of LASP on other linear sequence modeling methods. Extensive experiments on linear attention-based models are conducted with varying sequence lengths from 2K to 4096K. LASP scales sequence length up to 4096K on 128 GPUs, which is 8$\\times$ longer than existing SP methods."},"_bibtex":{"value":"@misc{\nsun2025linear,\ntitle={Linear Attention Sequence Parallelism},\nauthor={Weigao Sun and Zhen Qin and Dong Li and Xuyang Shen and Yu Qiao and Yiran Zhong},\nyear={2025},\nurl={https://openreview.net/forum?id=oVnfVnwh6y}\n}"},"title":{"value":"Linear Attention Sequence Parallelism"},"pdf":{"value":"/pdf/91f0bb8ec553fe88b189a2550d301b39cef48faa.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"sun|linear_attention_sequence_parallelism"},"authorids":{"value":["~Weigao_Sun1","~Zhen_Qin6","~Dong_Li11","~Xuyang_Shen1","~Yu_Qiao1","~Yiran_Zhong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Weigao Sun","Zhen Qin","Dong Li","Xuyang Shen","Yu Qiao","Yiran Zhong"]}},"version":2},{"content":{"summary":{"value":"The paper proposed a fully time-continuous version of the gated recurrent unit (GRU) RNN model. Different from the standard ODE-RNN of Rubanova et al. 2019, which is comprised of a continuous-time ODE and a discrete RNN, the Cont-GRU is fully continuous. The paper shows that the Cont-GRU outperforms alternative time-continuous architectures experimentally."},"presentation":{"value":"3 good"},"contribution":{"value":"3 good"},"soundness":{"value":"3 good"},"strengths":{"value":"# Pros:\n- Interesting and novel idea to make a discrete time RNN continuous while maintaining the same discrete time semantics by using time-delayed differential equations.\n- The paper is mostly well written"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"# Cons:\n- The experiments are primarily based on outdated RNN architectures and miss breakthrough time-continuous architectures of the past few years (Gu et al. 2021, Rusch et al. 2021, and Rusch et al. 2022).\n\n\n# References\n- Gu et al. 2021, Efficiently Modeling Long Sequences with Structured State Spaces.\n- Rusch et al. 2021, Unicornn: A recurrent model for learning very long time dependencies.\n- Rusch et al. 2022, Long expressive memory for sequence modeling."},"confidence":{"value":"3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked."},"questions":{"value":"Why did you not compare with the methods in Rusch et al. 2021, Rusch et al. 2022, and Gu et al. 2021?"},"rating":{"value":"3: reject, not good enough"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636307985,"tcdate":1698695357817,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission3539/Reviewer_LsJ8"],"signatures":["ICLR.cc/2024/Conference/Submission3539/Reviewer_LsJ8"],"forum":"FRpCyW7pWp","number":2,"license":"CC BY 4.0","cdate":1698695357817,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission3539/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636307985,"domain":"ICLR.cc/2024/Conference","replyto":"FRpCyW7pWp","id":"XIJTYoQgR7","forumContent":{"TLDR":{"value":"Previous GRU-based models are piece-wise continuous. We proposed the first fully continuous GRU."},"venue":{"value":"ICLR 2024 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time-series","Continuous GRU"]},"supplementary_material":{"value":"/attachment/d129f814c1c367c878e53cfe9e14a7afcac61fd7.pdf"},"primary_area":{"value":"general machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"For a long time, RNN-based models, such as RNNs, LSTMs and GRUs, have been used to process time series. However, RNN-based models do not fit well with sporadically (or irregularly) observed real-world data. To this end, some methods \\emph{partially} continuously model RNNs/GRUs using ordinary differential equations (ODEs). In this paper, however, we propose Cont-GRU, which models GRUs as delay differential equations (DDEs). By redefining GRUs as DDEs, we show that i) all the parts of GRUs (the hidden state, the reset gate, the update gate, and the update vector) can be interpreted \\emph{fully} continuously, and ii) our method does not inherit the limitations of ODEs.  In our experiments using 5 real-world datasets and 17 baselines, Cont-GRU outperforms all baselines by non-trivial margins."},"_bibtex":{"value":"@misc{\njhin2024contgru,\ntitle={Cont-{GRU}: Fully Continuous Gated Recurrent Units for Irregular Time Series},\nauthor={Sheo yon Jhin and Noseong Park},\nyear={2024},\nurl={https://openreview.net/forum?id=FRpCyW7pWp}\n}"},"title":{"value":"Cont-GRU: Fully Continuous Gated Recurrent Units for Irregular Time Series"},"pdf":{"value":"/pdf/0de559ce60aab3325f6df6ad2f26d674e0657e41.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Withdrawn_Submission"},"paperhash":{"value":"jhin|contgru_fully_continuous_gated_recurrent_units_for_irregular_time_series"},"authorids":{"value":["~Sheo_yon_Jhin1","~Noseong_Park1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Sheo yon Jhin","Noseong Park"]}},"version":2},{"content":{"summary":{"value":"This paper proposes ViP, a visual intervention probing method for hallucination detection in medical VLMs. The method applies mild visual distortions to the input image, measures the hidden-state shift between the original and intervened inputs, and combines this shift with the original hidden representation in a linear probe. Experiments on MedGemma and LLaVA-Med across five medical VQA datasets show that ViP generally outperforms probability, entropy, consistency, and standard linear-probe baselines."},"comments":{"value":"The authors should soften the causal language and frame ViP as an intervention-based empirical signal. Please add sensitivity analysis for perturbation strength and type, including separate results for rotation, translation, and Gaussian noise. The paper should also report effect sizes, not only significance tests, since several gains are very small. More human validation of hallucination labels is needed, especially for open-ended medical answers. Also, adding more medical VLMs and clearer implementation details would make the empirical claims stronger."},"clarity":{"value":"7/10 The main idea and structure are clear, but several experimental details are under-specified."},"review":{"value":"The paper addresses an important problem: detecting unreliable outputs from medical VLMs. The proposed idea is intuitive and reasonably motivated. If a response is visually grounded, its internal representation should be more stable under mild visual intervention, while hallucinated responses may show larger shifts. The evaluation covers multiple datasets and includes an OOD leave-one-out setting, which is a positive aspect.\n\nHowever, the paper overstates some claims. The method provides an intervention-based feature, but it does not establish a rigorous causal mechanism. A hidden-state shift under image distortion may reflect general robustness, image degradation sensitivity, or perturbation artifacts, not necessarily visual grounding. Also, the gain over Linear Probe is very small in several settings, such as 0.11 or 0.10 AUROC for MedGemma on some datasets, so the practical significance is not fully convincing. The reliance on Qwen3-generated hallucination labels is another concern, even though the authors report agreement checks."},"strengths":{"value":"1. The paper covers uncertainty-based, consistency-based, probing-based, and vision-aware hallucination detection methods, and positions ViP as an extension of hidden-state probing with visual intervention.\n2. The method is simple but interesting. Adding an intervention-induced representation shift to a linear probe is a clean and interpretable design.\n3. The paper evaluates two medical VLMs on five medical VQA datasets, includes OOD testing, and provides ablations over token positions and output-level distribution shifts.\n4. Mostly easy to follow."},"reproducibility":{"value":"moderate"},"weaknesses":{"value":"1. The causal and mechanism-level claims are too strong. The evidence supports correlation under intervention, not causal grounding. The Δ feature may capture perturbation sensitivity rather than hallucination-specific visual reliance.\n2.  ViP requires labeled hallucination data to train the probe, so it is not fully reference-free. Improvements over Linear Probe are often marginal. The paper does not sufficiently analyze the impact of intervention type and strength.\n3. The paper should report dataset sizes, hallucination class balance, train-validation-test splits, exact prompts, perturbation parameters, and answer-labeling details more clearly.\n4. The paper would benefit from comparisons with stronger medical VLMs, calibration methods, selective prediction, OOD uncertainty methods, and more discussion of how ViP differs from VASE."},"confidence":{"value":3},"rating":{"value":2},"title":{"value":"Review"}},"parentInvitations":"MICCAI.org/2026/Workshop/SAFER/-/Official_Review","nonreaders":[],"tmdate":1786628578301,"tcdate":1783255585481,"writers":["MICCAI.org/2026/Workshop/SAFER","MICCAI.org/2026/Workshop/SAFER/Submission23/Reviewer_RzuM"],"signatures":["MICCAI.org/2026/Workshop/SAFER/Submission23/Reviewer_RzuM"],"forum":"AeYpYtPL5Y","number":1,"license":"CC BY 4.0","cdate":1783255585481,"readers":["everyone"],"invitations":["MICCAI.org/2026/Workshop/SAFER/Submission23/-/Official_Review","MICCAI.org/2026/Workshop/SAFER/-/Edit"],"mdate":1786628578301,"domain":"MICCAI.org/2026/Workshop/SAFER","replyto":"AeYpYtPL5Y","id":"k5jc1twbN3","forumContent":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"We propose a Visual Intervention Probing (ViP) framework that detects hallucinations in medical vision-language models by measuring intervention-induced shifts in internal representations as a vision-grounded reliability signal."},"venue":{"value":"Submitted to SAFER 2026"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["Hallucination detection","Medical vision-language models","Visual intervention probing"]},"abstract":{"value":"Medical vision-language models (VLMs) hold significant potential for clinical decision support, but a key limitation undermines their reliability: hallucinations, whereby the model generates clinically plausible outputs unsupported by the underlying visual data. Existing hallucination detection methods primarily rely on output statistics (e.g., probability or entropy) or multi-sample consistency, which provide indirect proxies and can be either prone to confident errors or computationally expensive. Probing hidden states improves detection but remains purely observational, failing to explicitly determine whether the generated response is causally grounded in the visual evidence. Empirically, we find that hallucinated responses are more sensitive to visual intervention, while evidence-supported responses exhibit comparatively higher representation invariance. Motivated by this, we propose ViP (Visual Intervention Probing), a novel detector that goes beyond observational heuristics by introducing controlled visual interventions. By quantifying the intervention-induced shift in multimodal representations, ViP establishes a mechanism-level grounding cue that explicitly assesses a response’s reliance on visual input for hallucination detection. Extensive experiments on two medical VLMs across five diverse benchmarks (spanning radiology, pathology, and ophthalmology) demonstrate that ViP consistently outperforms competitive baselines. Notably, ViP exhibits pronounced robustness under domain shifts, highlighting its potential for real-world clinical deployment."},"_bibtex":{"value":"@misc{\nliu2026a,\ntitle={A Visual Intervention Probing for Hallucination Detection in Medical Vision-Language Models},\nauthor={Lei Liu and Shangkun Li and Yi Guo and Yuanyuan Wang and Zeju Li},\nyear={2026},\nurl={https://openreview.net/forum?id=AeYpYtPL5Y}\n}"},"title":{"value":"A Visual Intervention Probing for Hallucination Detection in Medical Vision-Language Models"},"venueid":{"value":"MICCAI.org/2026/Workshop/SAFER/Rejected_Submission"},"paperhash":{"value":"liu|a_visual_intervention_probing_for_hallucination_detection_in_medical_visionlanguage_models"},"authorids":{"value":["~Lei_Liu51","~Shangkun_Li4","~Yi_Guo11","~Yuanyuan_Wang12","~Zeju_Li2"]},"authors":{"value":["Lei Liu","Shangkun Li","Yi Guo","Yuanyuan Wang","Zeju Li"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer again for their thoroughness, including reading the other reviewers' comments, and for recognizing the value of our contribution by raising the score.\n\nBelow, we provide our responses:\n\n> Statistics for the iterations in LrcSSM\n\n| Dataset      | Min | Q1   | Median | Q3   | Max   |\n|--------------|-----|------|--------|------|-------|\n| Heartbeat    | 3   | 3    | 3      | 4    | 5.25  |\n| SCP1         | 2   | 2    | 2      | 2    | 4     |\n| SCP2         | 3   | 4    | 4      | 5    | 7.25  |\n| EthanolConcentration      | 3   | 3    | 4.75   | 6    | 6     |\n| MotorImagery        | 3   | 3    | 3      | 3    | 3     |\n| EigenWorms   | 3   | 3.75 | 4      | 4    | 4     |\n\nWe will provide this in a boxplot format in the paper.\n\n> Table 7\n\nThank you for the suggestion. We agree that standard error is more appropriate, as it better reflects confidence in the mean than standard deviation.\n\nFollowing your recommendation, we will revise Table 7 to report standard errors instead. For reference, the updated values are:\n65.4 ± 8.02, 65.0 ± 8.04, 63.8 ± 8.37\n\nWe will also apply this adjustment consistently across all relevant tables in the paper.\n\n> Mamba on EigenWorms\n\nWe agree that the reviewer raises a valid point and appreciate the opportunity to further share our thoughts. Our understanding is that the additions introduced in Mamba were designed to improve performance on the specific datasets evaluated in the Mamba paper. However, in the context of the tasks evaluated here, particularly the Eigenworms dataset, these enhancements do not appear to offer the same benefit.\n\nIt is important to note that the results we report for both S6 and Mamba come from a different paper [1], not the original Mamba paper [2]. While we did not reproduce these results ourselves, we assume they are valid and reported in good faith. Nonetheless, the observed performance gap between S6 and Mamba on this dataset remains a task for future investigation.\n\n> quasi-DEER or quasi-ELK\n\nThank you for catching this, the reviewer is absolutely right. This is a mistake on our part. We used quasi-DEER, which is in fact a specialization of quasi-ELK when the lambda values are close to zero. We will correct the terminology in the paper to accurately reflect this, and we appreciate you pointing it out.\n\n> NSSM discussion, dependency and (non)-linearity\n\nWe agree with the reviewer on the key distinction regarding dependency: Mamba is input-dependent, while LrcSSM is both input- and state-dependent in $A$, this is well understood.\n\n\nWe would like to clarify some aspects regarding parallel computations. \n\nMamba uses linear projections [2] and includes exponential terms for discretization for $\\bar A$, where the time-varying component comes from input dependency. The Mamba authors were able to construct the computations in an associative manner, with a customized parallel scan. However, this approach does not generalize to arbitrary non-linear projections in $A$. In the general case, methods like ELK are needed to achieve parallelization. \n\nOverall, when we refer to non-linear SSMs, the non-linear nature is a core part of $A$.\n\nRegarding xLSTM [3], including experiments on it is a valuable suggestion. However, due to time constraints, we are unable to provide these during the rebuttal period. We hypothesize that its performance is reflected in our LstmSSM ablation study, which we have included in the rebuttal.\n\n[1] Benjamin Walker, Andrew Donald McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. \"Log neural controlled differential equations: The lie brackets make a difference.\" Forty-first International Conference on Machine Learning (2024).\n\n[2] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023)."}},"parentInvitations":"NeurIPS.cc/2025/Conference/-/Official_Comment","tmdate":1761724769549,"tcdate":1754052019128,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission17571/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission17571/Authors"],"forum":"ENYvdnyhLl","number":2,"license":"CC BY 4.0","cdate":1754052019128,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/Submission17571/-/Official_Comment","NeurIPS.cc/2025/Conference/-/Edit"],"mdate":1761724769549,"domain":"NeurIPS.cc/2025/Conference","replyto":"wUAIIPHaSw","id":"SXTvoRLrTv","forumContent":{"venue":{"value":"NeurIPS 2025 poster"},"TLDR":{"value":"LrcSSM is a nonlinear RNN with parallel prefix-scan updates, formal gradient stability, and compute-optimal scaling. It outperforms LRU, S5, and Mamba on long-range tasks - matching SSM speed without attention or FFT overhead."},"keywords":{"value":["state space model","non-linear RNN","liquid neural network","parallelization","sequence modeling"]},"supplementary_material":{"value":"/attachment/72a1758b1e8d1aeb5a379ade6be10e8f7ece6b00.zip"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"We present LrcSSM, a $\\textit{non-linear}$ recurrent model that processes long sequences as fast as today's linear state-space layers. By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel, giving $\\mathcal{O}(TD)$ computational work and memory and only $\\mathcal{O}(\\log T)$ sequential depth, for input-sequence length $T$ and a state dimension $D$. Moreover, LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 and Mamba do not provide. Importantly, the diagonal Jacobian structure of our model results in no performance loss compared to the original model with dense Jacobian, and the approach can be generalized to other non-linear recurrent models, demonstrating broader applicability. On a suite of long-range forecasting tasks, we demonstrate that LrcSSM outperforms Transformers, LRU, S5, and Mamba."},"_bibtex":{"value":"@inproceedings{\nfarsang2025parallelization,\ntitle={Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling},\nauthor={M{\\'o}nika Farsang and Radu Grosu},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=ENYvdnyhLl}\n}"},"title":{"value":"Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling"},"pdf":{"value":"/pdf/6102c945cda7f097a0fe360fe046def01669279b.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"farsang|parallelization_of_nonlinear_statespace_models_scaling_up_liquidresistance_liquidcapacitance_networks_for_efficient_sequence_modeling"},"authorids":{"value":["~Mónika_Farsang1","~Radu_Grosu1"]},"authors":{"value":["Mónika Farsang","Radu Grosu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes using a unified dynamical systems framework (DSF) to analyze and compare the behavior of various sequence models, including SSMs (S4, LRU, Mamba-2) and different attention mechanisms (softmax, linear, norm). The core idea is to represent these models as dynamical systems and study their eigenvalue spectra to find \"spectral signatures\" that correlate with specific task requirements, such as long-term memory or selective forgetting. The authors conduct an empirical study across several benchmarks (LRA, MQAR, WikiText) to support their claims."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See Weaknesses."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The primary strength of this work is the application of the DSF to provide a common ground for analyzing seemingly disparate architectures like SSMs and attention. The goal of finding a unified principle to understand their internal dynamics is valuable and timely.\n-  The paper includes an extensive set of experiments across multiple model classes and a diverse set of tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The authors frame their work as a novel application of eigenvalue analysis of sequential models. However, the paper fails to cite or engage with a crucial body of literature on analyzing non-linear dynamical systems using linear operators. The work misses key papers such as [1]. This is particularly relevant, as it explicitly analyzes the eigenspectrum of the Koopman operator to model the non-linear dynamics of sequence models. This paper's core method—using the DSF to derive a linear-like state transition matrix $\\Lambda_i$ and analyzing its spectrum—is conceptually similar. The paper would be significantly stronger if it contextualized its approach within this existing work or compare with it.\n\n- The paper notes that linear attention retains its initialization's spectral shape, “raising the question of the potential importance of a task-dependent initialization.” This question remains unaddressed. The paper poses this as an open question but fails to engage with significant, recent work that directly addresses it. The work [2] (and related works) demonstrates that the initialization of weights (e.g., from a pre-trained model) can significantly improve the performance of softmax attention-based models on the LRA benchmark—the same benchmark used in this paper. This is a critical omission. It strongly suggests that a \"good\" spectral initialization is vital for success. The paper misses a key opportunity to connect its analysis to this important line of work and investigate, for example, how the initial eigenvalues of a pre-trained model compare to the random initializations used in this study.\n\n- The analysis of architectural modifications in Section 5 only partially demonstrates the framework's explanatory power, leaving the overall contribution and utility of the method in question (for this application).\n\n\n[1] \"An operator theoretic approach for analyzing sequence neural networks\", Naiman et al. (AAAI23)\n\n[2] \"Never Train from Scratch: Fair Comparison of Long Sequence Models Requires Data-Driven Priors\", Amos et al. (ICLR 2024)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762940622143,"tcdate":1762161635446,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21047/Reviewer_T1HT"],"signatures":["ICLR.cc/2026/Conference/Submission21047/Reviewer_T1HT"],"forum":"ALf0xHCCaP","number":3,"license":"CC BY 4.0","cdate":1762161635446,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission21047/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762940622143,"domain":"ICLR.cc/2026/Conference","replyto":"ALf0xHCCaP","id":"tMmILJK9RL","forumContent":{"TLDR":{"value":"This paper shows that eigenvalues offer a metric for memory retention and selective forgetting in sequence models, guiding model design choices to meet task requirements."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Sequence models","dynamical systems","memory dynamics"]},"supplementary_material":{"value":"/attachment/a34ab7106844c8b1974675fdf300085c394ec367.pdf"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Although softmax attention drives state-of-the-art performance for sequence models, its quadratic complexity limits scalability, motivating linear alternatives such as state space models (SSMs). While these alternatives improve efficiency, their fundamental differences in information processing remain poorly understood. In this work, we leverage the recently proposed dynamical systems framework to represent softmax, norm and linear attention as dynamical systems, enabling a structured comparison with SSMs by analyzing their respective eigenvalue spectra. Since eigenvalues capture essential aspects of dynamical system behavior, we conduct an extensive empirical analysis across diverse sequence models and benchmarks. We first show that eigenvalues influence essential aspects of memory and long-range dependency modeling, revealing spectral signatures that align with task requirements. Building on these insights, we then investigate how architectural modifications in sequence models impact both eigenvalue spectra and task performance.  This correspondence further strengthens the position of eigenvalue analysis as a principled metric for interpreting, understanding, and ultimately improving the capabilities of sequence models."},"_bibtex":{"value":"@misc{\nrickenbach2026tasklevel,\ntitle={Task-Level Insights from Eigenvalues across Sequence Models},\nauthor={Rahel Rickenbach and Jelena Trisovic and Alexandre Didier and Jerome Sieber and Melanie Zeilinger},\nyear={2026},\nurl={https://openreview.net/forum?id=ALf0xHCCaP}\n}"},"title":{"value":"Task-Level Insights from Eigenvalues across Sequence Models"},"pdf":{"value":"/pdf/ef15be1e4ca44ec34adb76dbbbb5e30448f4997c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"rickenbach|tasklevel_insights_from_eigenvalues_across_sequence_models"},"authorids":{"value":["~Rahel_Rickenbach1","~Jelena_Trisovic1","~Alexandre_Didier1","~Jerome_Sieber1","~Melanie_Zeilinger1"]},"authors":{"value":["Rahel Rickenbach","Jelena Trisovic","Alexandre Didier","Jerome Sieber","Melanie Zeilinger"]}},"version":2},{"content":{"summary":{"value":"This paper proposes DPMFormer, a dual-branch architecture that integrates Transformer and Mamba modules for efficient image super-resolution. The Transformer branch employs a Spatial–Channel Correlation attention and a depthwise SwiGLU feed-forward (DW-SwiFFN) to enhance local feature modeling, while the Mamba branch introduces a Lightweight Bidirectional Mamba (LBi-Mamba) for linear-time global dependency capture. The two branches interact via Cross-Attention Layers (CAL) within blocks and Inter-branch Exchange Bridges (IEB) across stages. Experiments on five benchmark datasets show that DPMFormer achieves competitive PSNR/SSIM with fewer parameters and FLOPs than prior lightweight SR models, such as HiT-SR and MambaIR."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1.How sensitive is the performance to the choice of window sizes (r) in ETL and CAL?\n\n2.Could the proposed LBi-Mamba be applied to other low-level tasks (e.g., denoising, deblurring)?\n\n3.Is the training stable when coupling both branches with IEB — any gradient conflict observed?\n\n4.How does DPMFormer scale to higher resolutions (e.g., 4 K images) given linear Mamba dynamics?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1.Proposes a clearly motivated dual-path fusion leveraging complementary strengths of Mamba and Transformer.\n\n2.Strong quantitative performance–efficiency trade-off, reducing FLOPs by ~20 % vs HiT-SR with comparable PSNR.\n\n3.Extensive ablations (DW-SwiFFN, RMSNorm, IEB variants) demonstrate careful engineering and reproducibility.\n\n4.Reproducibility statement is complete and code release is promised."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.Innovation marginal: The dual-branch idea has been explored in prior hybrid SR models; more theoretical or analytic justification of the coupling design would enhance novelty.\n\n2.Limited qualitative diversity: Most visual comparisons are standard; additional challenging scenes or real-world degradations would strengthen claims.\n\n3.Missing complexity analysis: An explicit breakdown of runtime cost per module (ETL vs LBi-Mamba vs IEB) would help understand where efficiency gains arise.\n\n4.Minor clarity issues: Equations (2)–(6) lack dimensional definitions; figure readability (font size) could be improved."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762932908459,"tcdate":1761644246108,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20005/Reviewer_38Ex"],"signatures":["ICLR.cc/2026/Conference/Submission20005/Reviewer_38Ex"],"forum":"yk95hwXbUZ","number":1,"license":"CC BY 4.0","cdate":1761644246108,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20005/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762932908459,"domain":"ICLR.cc/2026/Conference","replyto":"yk95hwXbUZ","id":"b5If8k2nSK","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Transformer","Mamba","Dual-Path Network","Lightweight Network"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Vision Transformers have achieved outstanding performance in image super-resolution (SR), but existing lightweight models rely on window-based attention, limiting their ability to model global dependencies essential for high-quality reconstruction. To address these challenges, we present DPMFormer, a Dual-Path Mamba–Transformer architecture for lightweight image super-resolution.  Rather than a simple combination of a state-space model and a Transformer, the design couples two streams throughout the network. On the Transformer side, an Enhanced Transformer Layer (ETL) replaces self-attention with Spatial–Channel Correlation (SCC) and a Depthwise-SwiGLU Feed-Forward (DW-SwiFFN). On the Mamba side, Lightweight Bi-directional Mamba Layers (LBi-ML) implement single-pass bidirectionality via channel split and sequence reversal with additive cross coupling. The streams interact at two levels: within each block, a Cross-Attention Layer (CAL) performs fixed, non-overlapping cross fusion,\nand across blocks, Inter-branch Exchange Bridges (IEB) use resolution-preserving 1 × 1 adapters around tokenization to align channel spaces in both directions. Besides, we employ RMSNorm to reduce normalization overhead and, under our setup, observe modest, configuration-dependent gains. Extensive experiments show that DPMFormer reduces FLOPs by 47.3G (21\\%) and parameters by 21K under 2 × upsampling compared to HiT-SR, while almost achieving state-of-the-art performance across five benchmarks. Measured on an RTX 4090, our method reaches 668 ms latency, yielding 1.59 × and 2.33 × speedups over MambaIR and CATANet, respectively. The code will be publicly released."},"_bibtex":{"value":"@misc{\nyin2026dpmformer,\ntitle={{DPMF}ormer: Dual-Path Mamba-Transformer for Efficient Image Super\\nobreakdash-Resolution},\nauthor={Zhengnan Yin},\nyear={2026},\nurl={https://openreview.net/forum?id=yk95hwXbUZ}\n}"},"title":{"value":"DPMFormer: Dual-Path Mamba-Transformer for Efficient Image Super‑Resolution"},"pdf":{"value":"/pdf/933d7f41da3c9fd9c83cfe8eb7c75a778305cf0a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yin|dpmformer_dualpath_mambatransformer_for_efficient_image_superresolution"},"authorids":{"value":["~Zhengnan_Yin2"]},"authors":{"value":["Zhengnan Yin"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.10140v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"qiao|himamba_hierarchical_mamba_for_efficient_image_superresolution"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Junbo_Qiao:","https://dblp.org/search/pid/api?q=author:Jincheng_Liao:","~Wei_Li60","https://dblp.org/search/pid/api?q=author:Yulun_Zhang_0001:","https://dblp.org/search/pid/api?q=author:Yong_Guo:","https://dblp.org/search/pid/api?q=author:Yi_Wen:","https://dblp.org/search/pid/api?q=author:Zhangxizi_Qiu:","~Jiao_Xie1","https://dblp.org/search/pid/api?q=author:Jie_Hu_0021:","https://dblp.org/search/pid/api?q=author:Shaohui_Lin:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.10140"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-10140,\n  publtype={informal},\n  author={Junbo Qiao and Jincheng Liao and Wei Li and Yulun Zhang and Yong Guo and Yi Wen and Zhangxizi Qiu and Jiao Xie and Jie Hu and Shaohui Lin},\n  title={Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.10140},\n  url={https://doi.org/10.48550/arXiv.2410.10140}\n}\n"},"abstract":{"value":"State Space Models (SSM), such as Mamba, have shown strong representation ability in modeling long-range dependency with linear complexity, achieving successful applications from high-level to low-level vision tasks. However, SSM's sequential nature necessitates multiple scans in different directions to compensate for the loss of spatial dependency when unfolding the image into a 1D sequence. This multi-direction scanning strategy significantly increases the computation overhead and is unbearable for high-resolution image processing. To address this problem, we propose a novel Hierarchical Mamba network, namely, Hi-Mamba, for image super-resolution (SR). Hi-Mamba consists of two key designs: (1) The Hierarchical Mamba Block (HMB) assembled by a Local SSM (L-SSM) and a Region SSM (R-SSM) both with the single-direction scanning, aggregates multi-scale representations to enhance the context modeling ability. (2) The Direction Alternation Hierarchical Mamba Group (DA-HMG) allocates the isomeric single-direction scanning into cascading HMBs to enrich the spatial relationship modeling. Extensive experiments demonstrate the superiority of Hi-Mamba across five benchmark datasets for efficient SR. For example, Hi-Mamba achieves a significant PSNR improvement of 0.29 dB on Manga109 for $\\times3$ SR, compared to the strong lightweight MambaIR."},"title":{"value":"Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution"},"authors":{"value":["Junbo Qiao","Jincheng Liao","Wei Li","Yulun Zhang","Yong Guo","Yi Wen","Zhangxizi Qiu","Jiao Xie","Jie Hu","Shaohui Lin"]}},"tmdate":1784650603387,"pdate":1704067200000,"tcdate":1739076585883,"writers":["~"],"signatures":["~Wei_Li60"],"forum":"MgMtUL1hGK","license":"CC BY-SA 4.0","number":313637,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784650603387,"domain":"DBLP.org","id":"MgMtUL1hGK","version":2},{"content":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Image Super-Resolution","Mamba","Structured State Space Models"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSM), such as Mamba, have shown strong representation ability in modeling long-range dependency with linear complexity, achieving successful applications from high-level to low-level vision tasks. However, SSM's sequential nature necessitates multiple scans in different directions to compensate for the loss of spatial dependency when unfolding the image into a 1D sequence. This multi-direction scanning strategy significantly increases the computation overhead and is unbearable for high-resolution image processing. To address this problem, we propose a novel Hierarchical Mamba network, namely, Hi-Mamba, for image super-resolution (SR).  Hi-Mamba consists of two key designs: (1) The Hierarchical Mamba Block (HMB) assembled by a Local SSM (L-SSM) and a Region SSM (R-SSM) both with the single-direction scanning, aggregates multi-scale representations to enhance the context modeling ability. (2) The Direction Alternation  Hierarchical Mamba Group (DA-HMG) allocates the isomeric single-direction scanning into cascading HMBs to enrich the spatial relationship modeling.\nExtensive experiments demonstrate the superiority of Hi-Mamba across five benchmark datasets for efficient SR. For example, Hi-Mamba achieves a significant PSNR improvement of 0.29 dB on Manga109 for $\\times3$ SR, compared to the strong lightweight MambaIR."},"_bibtex":{"value":"@misc{\nqiao2024himamba,\ntitle={Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution},\nauthor={Junbo Qiao and Jincheng Liao and Wei Li and Yulun Zhang and Yong Guo and Yi Wen and Zhangxizi Qiu and Jiao Xie and Jie Hu and Shaohui Lin},\nyear={2024},\nurl={https://openreview.net/forum?id=iPYwddLhXR}\n}"},"title":{"value":"Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution"},"pdf":{"value":"/pdf/e75e168c76e93b40ac8e29dd9015c5e7d72edc8b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"qiao|himamba_hierarchical_mamba_for_efficient_image_superresolution"},"authorids":{"value":["~Junbo_Qiao1","~Jincheng_Liao1","~Wei_Li60","~Yulun_Zhang1","~Yong_Guo1","~Yi_Wen3","~Zhangxizi_Qiu2","~Jiao_Xie1","~Jie_Hu8","~Shaohui_Lin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Junbo Qiao","Jincheng Liao","Wei Li","Yulun Zhang","Yong Guo","Yi Wen","Zhangxizi Qiu","Jiao Xie","Jie Hu","Shaohui Lin"]}},"tmdate":1732524641293,"tcdate":1727171698705,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3583/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission3583/Authors"],"forum":"iPYwddLhXR","license":"CC BY 4.0","number":3583,"cdate":1727171698705,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission3583/-/Full_Submission","ICLR.cc/2025/Conference/-/Withdrawn_Submission"],"mdate":1732524641293,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"iPYwddLhXR","version":2},{"content":{"summary":{"value":"This paper presents a novel hierarchical embedding method, which uses depth-dissimilarity and boundary-dissimilarity to assimilate Euclidean spaces to hyperbolic spaces. The authors proved that, by choosing appropriate depth-dissimilarity functions, the method equals traditional hyperbolic embedding models. In other words, the method is a generalization of hyperbolic embedding to allow tailored hierarchy embedding."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Can the authors provide more insights on Weakness 3 and Weakness 4?\n\nWhat are the \"richer\" structures that your method applies better to than simple hyperbolic embedding?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The authors generalize hyperbolic embedding models by assigning depth-dissimilarity and boundary-dissimilarity functions to Euclidean spaces. It allows sub-exponential distance scaling to adapt to richer structures than hierarchy.\n\n2. The authors connect non-linear distance metrics with ball/box representations. It allows more expressiveness while maintaining the semantic modeling capabilities needed for ontology.\n\n3. Clear mathematical derivations and proofs."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Citation format errors: many of the citations should be in brackets.\n\n2. The authors motivated the choice of a Euclidean alternative to the traditional hyperbolic embedding by saying \"However, as shown in Table 1, hyperbolic methods often rely on specialized constructed objects as embedding candidates, limiting their generalizability to data that encodes richer semantics beyond hierarchy\" (Line 51-53). This is a very vague, descriptive motivation that may confuse readers -- does a specialized constructed object necessarily mean bad generalizability to non-hierarchy structures? Is there any theoretical support in literature for your motivation?\n\nYou may consider referring to this paper (https://arxiv.org/pdf/2407.16641?) -- the \"curvature\" of a subspace you need to sufficiently embed a certain size of hierarchy relies on the *capacity* of the subspace. There is trade-off between capacity and numerical stability -- that is, it is beneficial to use a subspace that has just adequate capacity. This can be a good support for your motivation -- your method effectively allows users to specify sub-exponential distance (i.e. between linear distance in Euclidean spaces and exponential distance in hyperbolic spaces) so that the capacity of subspaces match the sizes of hierarchical structures.\n\n3. As Sala et al. (http://proceedings.mlr.press/v80/sala18a.html) discusses, equipping a coordinate space with exponential distance metrics (e.g. Poincare ball model or Lorentz model) poses an upper limit of numerical stability -- the reflective boundary. Due to the finite digits of float numbers, the depth of the hierarchy that can be losslessly embedded is limited -- i.e., the practical expressiveness of hyperbolic spaces is not infinite as the theory permits. What is the limit of numerical stability of the proposed Euclidean space alternative? From how I understand it, there is trade-off between how fast the depth-dissimilarity increases and how deep a hierarchy the method can stably embed. What is the trade-off for your method?\n\n4. Equation 7 and 8 lack theoretical motivation. Why can and should the two dissimilarities be weighted summed? Is there any convex guarantee?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922939180,"tcdate":1761982268388,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11938/Reviewer_RDte"],"signatures":["ICLR.cc/2026/Conference/Submission11938/Reviewer_RDte"],"forum":"fu7iSvTwXm","number":3,"license":"CC BY 4.0","cdate":1761982268388,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11938/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922939180,"domain":"ICLR.cc/2026/Conference","replyto":"fu7iSvTwXm","id":"QYj5jqdBQR","forumContent":{"TLDR":{"value":"RegD: A new embedding framework that could represents entities as arbitrary Euclidean regions and theoretically achieves hyperbolic-like expressiveness."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["hierarchy","partial order embedding","representation learning"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Hierarchical data is common in many domains like life sciences and e-commerce, and its embeddings often play a critical role. While hyperbolic embeddings offer a theoretically grounded approach to representing hierarchies in low-dimensional spaces, current methods often rely on specific geometric constructs as embedding candidates. This reliance limits their generalizability and makes it difficult to integrate with techniques that model semantic relationships beyond pure hierarchies, such as ontology embeddings. In this paper, we present RegD, a flexible Euclidean framework that supports the use of arbitrary geometric regions---such as boxes and balls---as embedding representations. Although RegD operates entirely in Euclidean space, we formally prove that it achieves hyperbolic-like expressiveness by incorporating a depth-based dissimilarity between regions, enabling it to emulate key properties of hyperbolic geometry, including exponential growth. Our empirical evaluation on diverse real-world datasets shows consistent performance gains over state-of-the-art methods and demonstrates RegD’s potential for broader applications such as the ontology embedding task that goes beyond hierarchy. Code and data are available at https://anonymous.4open.science/r/RegD-F4E3."},"_bibtex":{"value":"@misc{\nyang2026achieving,\ntitle={Achieving Hyperbolic-Like Expressiveness with Arbitrary Euclidean Regions: A New Approach to Hierarchical Embeddings},\nauthor={Hui Yang and Jiaoyan Chen},\nyear={2026},\nurl={https://openreview.net/forum?id=fu7iSvTwXm}\n}"},"title":{"value":"Achieving Hyperbolic-Like Expressiveness with Arbitrary Euclidean Regions: A New Approach to Hierarchical Embeddings"},"pdf":{"value":"/pdf/f9981fa7620aee93287a013c348dc7c76dc6a938.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yang|achieving_hyperboliclike_expressiveness_with_arbitrary_euclidean_regions_a_new_approach_to_hierarchical_embeddings"},"authorids":{"value":["~Hui_Yang5","~Jiaoyan_Chen1"]},"authors":{"value":["Hui Yang","Jiaoyan Chen"]}},"version":2},{"content":{"venue":{"value":"MICCAI (5) 2024"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-031-72086-4_13.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"yang|cardiovascular_disease_detection_from_multiview_chest_xrays_with_bimamba"},"html":{"value":"https://doi.org/10.1007/978-3-031-72086-4_13"},"_bibtex":{"value":"@inproceedings{DBLP:conf/miccai/YangZWKY24,\n  author={Zefan Yang and Jiajin Zhang and Ge Wang and Mannudeep K. Kalra and Pingkun Yan},\n  title={Cardiovascular Disease Detection from Multi-view Chest X-Rays with BI-Mamba},\n  year={2024},\n  cdate={1704067200000},\n  pages={134-144},\n  url={https://doi.org/10.1007/978-3-031-72086-4_13},\n  booktitle={MICCAI (5)},\n  crossref={conf/miccai/2024-5}\n}\n"},"abstract":{"value":"Accurate prediction of Cardiovascular disease (CVD) risk in medical imaging is central to effective patient health management. Previous studies have demonstrated that imaging features in computed tomography (CT) can help predict CVD risk. However, CT entails notable radiation exposure, which may result in adverse health effects for patients. In contrast, chest X-ray emits significantly lower levels of radiation, offering a safer option. This rationale motivates our investigation into the feasibility of using chest X-ray for predicting CVD risk. Convolutional Neural Networks (CNNs) and Transformers are two established network architectures for computer-aided diagnosis. However, they struggle to model very high resolution chest X-ray due to the lack of large context modeling power or quadratic time complexity. Inspired by state space sequence models (SSMs), a new class of network architectures with competitive sequence modeling power as Transfomers and linear time complexity, we propose Bidirectional Image Mamba (BI-Mamba) to complement the unidirectional SSMs with opposite directional information. BI-Mamba utilizes parallel forward and backward blocks to encode long-range dependencies of multi-view chest X-rays. We conducted extensive experiments on images from 10,395 subjects in National Lung Screening Trail (NLST). Results show that BI-Mamba outperforms ResNet-50 and ViT-S with comparable parameter size, and saves significant amount of GPU memory during training. Besides, BI-Mamba achieves promising performance compared with previous state of the art in CT, unraveling the potential of chest X-ray for CVD risk prediction."},"title":{"value":"Cardiovascular Disease Detection from Multi-view Chest X-Rays with BI-Mamba"},"authors":{"value":[{"fullname":"Zefan Yang","username":""},{"fullname":"Jiajin Zhang","username":"~Jiajin_Zhang2"},{"fullname":"Ge Wang","username":""},{"fullname":"Mannudeep K. Kalra","username":""},{"fullname":"Pingkun Yan","username":""}]}},"tmdate":1780217019219,"pdate":1735603200000,"externalIds":["dblp:conf/miccai/YangZWKY24"],"tcdate":1780217014348,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jiajin_Zhang2"],"forum":"h0RbF6sxpu","license":"CC BY-SA 4.0","number":41233,"cdate":1704067200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1780217019219,"domain":"OpenReview.net/Public_Article","id":"h0RbF6sxpu","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2405.18533v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"yang|cardiovascular_disease_detection_from_multiview_chest_xrays_with_bimamba"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zefan_Yang:","~Jiajin_Zhang2","https://dblp.org/search/pid/api?q=author:Ge_Wang_0001:","https://dblp.org/search/pid/api?q=author:Mannudeep_K._Kalra:","~Pingkun_Yan1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2405.18533"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2405-18533,\n  publtype={informal},\n  author={Zefan Yang and Jiajin Zhang and Ge Wang and Mannudeep K. Kalra and Pingkun Yan},\n  title={Cardiovascular Disease Detection from Multi-View Chest X-rays with BI-Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2405.18533},\n  url={https://doi.org/10.48550/arXiv.2405.18533}\n}\n"},"abstract":{"value":"Accurate prediction of Cardiovascular disease (CVD) risk in medical imaging is central to effective patient health management. Previous studies have demonstrated that imaging features in computed tomography (CT) can help predict CVD risk. However, CT entails notable radiation exposure, which may result in adverse health effects for patients. In contrast, chest X-ray emits significantly lower levels of radiation, offering a safer option. This rationale motivates our investigation into the feasibility of using chest X-ray for predicting CVD risk. Convolutional Neural Networks (CNNs) and Transformers are two established network architectures for computer-aided diagnosis. However, they struggle to model very high resolution chest X-ray due to the lack of large context modeling power or quadratic time complexity. Inspired by state space sequence models (SSMs), a new class of network architectures with competitive sequence modeling power as Transfomers and linear time complexity, we propose Bidirectional Image Mamba (BI-Mamba) to complement the unidirectional SSMs with opposite directional information. BI-Mamba utilizes parallel forward and backwark blocks to encode longe-range dependencies of multi-view chest X-rays. We conduct extensive experiments on images from 10,395 subjects in National Lung Screening Trail (NLST). Results show that BI-Mamba outperforms ResNet-50 and ViT-S with comparable parameter size, and saves significant amount of GPU memory during training. Besides, BI-Mamba achieves promising performance compared with previous state of the art in CT, unraveling the potential of chest X-ray for CVD risk prediction."},"title":{"value":"Cardiovascular Disease Detection from Multi-View Chest X-rays with BI-Mamba"},"authors":{"value":["Zefan Yang","Jiajin Zhang","Ge Wang","Mannudeep K. Kalra","Pingkun Yan"]}},"tmdate":1780217014652,"pdate":1704067200000,"tcdate":1724938814922,"writers":["~"],"signatures":["~Pingkun_Yan1"],"forum":"tqcZf8Erqy","license":"CC BY-SA 4.0","number":77066,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1780217014652,"domain":"DBLP.org","id":"tqcZf8Erqy","version":2},{"content":{"summary":{"value":"This paper presents a new algorithm for sequence parallelism that focuses on linear attention models. In contrast to sequence parallelism methods that focus on standard attention, the paper introduces two general modifications for the linear attention setting: exploiting the right product rule in attention and introducing a ring-style communication scheme that leverages it. The paper also discusses some optimizations in the implementation and how hybrid parallelism with data or other model parallelism might be performed. Results presented include scaling of the method for increasingly long sequences and a direct comparison of throughput and final losses with other sequence parallelism approaches using two linear attention models. The proposed sequence parallelism scheme demonstrates increased throughput versus existing approaches, as well is longer sequences supported versus some methods."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Any thoughts on weaknesses suggested would be helpful. I really struggle with the 8x being appropriate to say here."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"The tests in the results are very well presented – they correctly use throughput as the appropriate metric and the clarity of the presentation is high. The level of technical detail throughout is high and, to my read, appears correct.\n\nThe fundamental contribution of the paper is to see that linear attention allows for the right product rule and then to develop a communication scheme around it such that communication size does not scale with N. At the same time, memory usage remains moderate enough to enable linear scaling of the maximum sequence length with the number of resources. This represents a clear improvement on what RingAttention, designed for standard attention, does by default in the linear setting.\n\nThe paper is fairly well written and the notable typo, ‘casual’ instead of ‘causal’, made me laugh :)"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I found myself repeatedly praising the sobriety and reasonableness of the claims of the paper, until I got to the last line of the abstract/paper and saw the ‘8x longer sequence’ claim. The Megatron sequence parallelism implementation cited is a couple years out of date and can’t be reasonably be considered state-of-the-art in this setting; claiming an 8x versus it as the headline result of the paper doesn’t really seem reasonable . [As it is, the goalpost may already have been shifted by FPDT (https://arxiv.org/pdf/2408.16978), although I’m admittedly not aware if it is available ‘off the shelf’ yet and don’t expect the authors to have compared with it given its very recent posting.]\n\nReally, I’d have liked to see a more focused comparison with RingAttention and digging into the details for how the outperformance happens. The appendix does have some of this information but, to me, it’s perhaps the most interesting part of the paper."}},"nonreaders":[],"tmdate":1731428086074,"tcdate":1730597644757,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5885/Reviewer_AVTR"],"signatures":["ICLR.cc/2025/Conference/Submission5885/Reviewer_AVTR"],"forum":"oVnfVnwh6y","number":2,"license":"CC BY 4.0","cdate":1730597644757,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5885/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428086074,"domain":"ICLR.cc/2025/Conference","replyto":"oVnfVnwh6y","id":"E8VxfLLP4T","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["sequence parallelism","distributed training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Sequence parallelism (SP) serves as a prevalent strategy to handle long sequences that exceed the memory limit of a single device. However, for linear sequence modeling methods like linear attention, existing SP approaches do not take advantage of their right-product-first feature, resulting in sub-optimal communication efficiency and usability. In this paper, we introduce Linear Attention Sequence Parallelism (LASP), an efficient SP approach designed for linear attention-based transformer models. Specifically, we design an efficient point-to-point ring-style communication mechanism to leverage the right-product kernel trick of linear attention, which sharply decreases the communication overhead, comparing with existing SP methods. We enhance the computation efficiency of LASP by performing kernel fusion and intermediate state caching, making the implementation of LASP hardware-friendly on GPUs. Furthermore, we meticulously ensure the compatibility of sequence-level LASP with all types of batch-level data parallel methods, which is vital for distributed training on large clusters with very-long sequences. We also discuss the generalization of LASP on other linear sequence modeling methods. Extensive experiments on linear attention-based models are conducted with varying sequence lengths from 2K to 4096K. LASP scales sequence length up to 4096K on 128 GPUs, which is 8$\\times$ longer than existing SP methods."},"_bibtex":{"value":"@misc{\nsun2025linear,\ntitle={Linear Attention Sequence Parallelism},\nauthor={Weigao Sun and Zhen Qin and Dong Li and Xuyang Shen and Yu Qiao and Yiran Zhong},\nyear={2025},\nurl={https://openreview.net/forum?id=oVnfVnwh6y}\n}"},"title":{"value":"Linear Attention Sequence Parallelism"},"pdf":{"value":"/pdf/91f0bb8ec553fe88b189a2550d301b39cef48faa.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"sun|linear_attention_sequence_parallelism"},"authorids":{"value":["~Weigao_Sun1","~Zhen_Qin6","~Dong_Li11","~Xuyang_Shen1","~Yu_Qiao1","~Yiran_Zhong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Weigao Sun","Zhen Qin","Dong Li","Xuyang Shen","Yu Qiao","Yiran Zhong"]}},"version":2},{"content":{"summary":{"value":"TTT-UNet adds test time training layers into the UNet framework, adjusting parameters on inference, and reporting superior performance to state of the art methods. The authors insert themselves as a new approach different from transformer unets / nnunet / u-mamba. They provide reproducibility in code derived from u-mamba. \n\nThe work has already appeared as a preprint and seems to have gained early traction."},"justification_of_final_rating":{"value":"As I said in my initial review, this is impressive original work that would be of interest to most of the MIDL audience, with strong results, and now with some statistical validation. I still recommend the authors to perform more extensive statistical validation in multiple tasks against multiple architectures, to better ground your contribution. They mentioned this is the plan for the final version. Since the authors have addressed my main concerns, I will raise the rating to Strong Accept."},"confidence":{"value":3},"final_rating":{"value":5},"justification_of_the_preliminary_rating":{"value":"This is impressive and original work. I already mentioned this but I believe statistical significance testing is mandatory to support the claims of \" TTT-UNet achieves notable improvements over a range of state-of-the-art methods\" and improvement of the rating."},"confidentiality_llm_acknowledgment":{"value":"Yes"},"strengths":{"value":"The proposed architecture follows a similar structure to U-Mamba, but with different operations performed after the flattening of convolutional features. The intent is to treat the state acumulation as a trainable problem using learnable projections that update on the fly trough self supervision. The paper is well written, results are strong and the idea is sound. The Appendix is detailed and well-written, providing answers to many questions one might have reading the main manuscript."},"weaknesses":{"value":"In my opinion, Figure 1 could have more details in explaining the TTT Layer, the main proposal of the manuscript.\n\nResults are impressive for such a different architecture but appear to be very close to the nnUNet and U-Mamba approaches. Given the small margins and overlapping standard deviations, statistical validation showing if there is a significant difference among the tested methods is missing.\n\nI understand the multiple projections and the self supervision idea in test-time, but we need more details for the self-supervised optimization and how it is done in the main manuscript, in a summarized way."},"detailed_comments":{"value":"There is some repetition in writing, how the TTT-UNet is built at the top level is mentioned in many places, with repeated references to Figure 1. \n\nThe explanation on TTT building blocks on Section 3.2 would benefit of coming before Section 3.1, in my opinion.\n\nFor better understanding, I am not sure if using Q K V in the diagrams is the best choice, considering they are not used in a similar way to self attention / multi head attention. It confuses a reader that expects a transformer block when seeing Q K V, in my opinion, and TTT Layer does not use attention. Maybe more expressive names (authors use training view and label view in the method's description) would help. Just a suggestion."},"questions_to_address_in_the_rebuttal":{"value":"Why were the multiple projections adopted? I understand the idea of being adaptable to different pertubations and inputs, but I wonder how the \"naive\" approach would compare in performance."},"preliminary_rating":{"value":4}},"parentInvitations":"MIDL.io/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1771530050219,"tcdate":1768015696070,"writers":["MIDL.io/2026/Conference","MIDL.io/2026/Conference/Submission342/Reviewer_aXx1"],"signatures":["MIDL.io/2026/Conference/Submission342/Reviewer_aXx1"],"forum":"3dKmXxITJB","number":2,"license":"CC BY 4.0","cdate":1768015696070,"readers":["everyone"],"invitations":["MIDL.io/2026/Conference/Submission342/-/Official_Review","MIDL.io/2026/Conference/-/Edit"],"mdate":1771530050219,"domain":"MIDL.io/2026/Conference","replyto":"3dKmXxITJB","id":"Ri539QMQ8u","forumContent":{"venue":{"value":"MIDL 2026 Poster"},"midl_latex_submission_checklist":{"value":["The paper compiles correctly using the pdflatex compiler.","Created a single midl26_NNN.zip file with midl26_NNN.tex, midl26_NNN.bib, all necessary figures and files.","The LaTeX file includes the correct header commands before \\title.","Hyperref is preloaded; do not modify it.","The times package is not used.","All co-authors are listed with correct firstname, lastname, and any suffix/prefix.","Math in the title and abstract uses valid LaTeX.","References are provided through the .bib file only.","Tables and figures stay within the page margins.","The zip archive contains all necessary figures and no unused files.","Special formatting from rebuttal has been removed.","All special characters use LaTeX commands.","Appendices and supplementary material are included in the same PDF after references.","The main paper does not exceed 12 pages."]},"keywords":{"value":["Biomedical Image Segmentation","U-Net","Test-Time Training"]},"reproducibility":{"value":"https://github.com/rongzhou7/TTT-UNet"},"read_cfp_and_author_instructions":{"value":"Yes"},"originality_policy":{"value":"Yes"},"abstract":{"value":"Biomedical image segmentation is crucial for accurately diagnosing and analyzing various diseases. However, Convolutional Neural Networks (CNNs) and Transformers, the most commonly used architectures for this task, struggle to effectively capture long-range dependencies due to the inherent locality of CNNs and the computational complexity of Transformers. To address this limitation, we introduce TTT-UNet, a novel framework that integrates Test-Time Training (TTT) layers into the traditional U-Net architecture for biomedical image segmentation. TTT-UNet dynamically adjusts model parameters during the test time, enhancing the model's ability to capture both local and long-range features. We evaluate TTT-UNet on multiple medical imaging datasets, including 3D abdominal organ segmentation in CT and MR images, instrument segmentation in endoscopy images, and cell segmentation in microscopy images. The results demonstrate that TTT-UNet consistently outperforms state-of-the-art CNN-based and Transformer-based segmentation models across all tasks. The code is available at https://github.com/rongzhou7/TTT-UNet."},"_bibtex":{"value":"@inproceedings{\nzhou2026tttunet,\ntitle={{TTT}-{UN}et: Enhancing U-Net with Test-Time Training Layers for Biomedical Image Segmentation},\nauthor={Rong Zhou and Zhengqing Yuan and Zhiling Yan and Weixiang Sun and Kai Zhang and Yiwei Li and Yanfang Ye and Xiang Li and Lichao Sun and Lifang He},\nbooktitle={Medical Imaging with Deep Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=3dKmXxITJB}\n}"},"title":{"value":"TTT-UNet: Enhancing U-Net with Test-Time Training Layers for Biomedical Image Segmentation"},"latex_code":{"value":"/attachment/d783cd6904b59ce55127ab682cca563620d1d991.zip"},"secondary_subject_area":{"value":"Unsupervised Learning and Representation Learning"},"pdf":{"value":"/pdf/b72a44efe9ad631c4c68501c25f976e6156aa86e.pdf"},"copyright_form":{"value":"/attachment/2385d53a70aea7663b4b212241cc9a17373939be.pdf"},"visa":{"value":"Yes"},"single_blind_notice":{"value":"Yes"},"venueid":{"value":"MIDL.io/2026/Conference"},"paperhash":{"value":"zhou|tttunet_enhancing_unet_with_testtime_training_layers_for_biomedical_image_segmentation"},"primary_subject_area":{"value":"Segmentation"},"authorids":{"value":["~Rong_Zhou3","~Zhengqing_Yuan2","~Zhiling_Yan2","~Weixiang_Sun1","~Kai_Zhang14","~Yiwei_Li2","~Yanfang_Ye1","~Xiang_Li14","~Lichao_Sun1","~Lifang_He1"]},"registration":{"value":"Yes"},"authors":{"value":["Rong Zhou","Zhengqing Yuan","Zhiling Yan","Weixiang Sun","Kai Zhang","Yiwei Li","Yanfang Ye","Xiang Li","Lichao Sun","Lifang He"]},"llm_policy_acknowledgment":{"value":"Yes"}},"version":2},{"content":{"summary":{"value":"The paper introduces a image matching method that combines State Space Models (SSMs)—specifically the Mamba architecture—with transformer-style attention. \n\nThe SSM branch delivers linear-time visual feature processing, while the attention branch supplies global context. An adaptive fusion module balances the local correlations captured by Mamba with the long-range dependencies modeled by attention. \n\nExperiments demonstrate higher feature-matching precision and recall."},"soundness":{"value":1},"confidence":{"value":3},"questions":{"value":"Please provide qualitative visualizations for both the matching and the SLAM results.\n\nThe current narrative blurs the line between a image matcher and a SLAM‐specific contribution. As the work essentially replaces the SLAM matching module with a “LoFTR + Mamba” variant, would it be clearer to reframe the paper around feature matching with an application to SLAM?\n\nThe manuscript lacks discussion of closely related work, notably\n\n    MambaGlue: Fast and Robust Local Feature Matching with Mamba\n    JamMa: Ultra-lightweight Local Feature Matching with Joint Mamba\n\nPlease include quantitative comparisons against the following strong baselines:\n\n    Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed\n    MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training\n    RoMa: Robust Dense Feature Matching\n    UFM: A Simple Path towards Unified Dense Correspondence with Flow"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":1},"strengths":{"value":"The paper merge a State Space Model (Mamba) with transformer attention in a SLAM front-end, offering a way to exploit their complementary properties (linear efficiency vs. global context).\n\nThe proposed adaptive fusion goes beyond naive concatenation, balancing the two pathways according to the input. The experimental protocol shows gains in both precision and recall.\n\nAchieving superior feature matching in scenes with repetitive patterns, illumination changes, and viewpoint shifts addresses a long-standing challenge in SLAM."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Figure 1 is not informative: it is almost identical to LoFTR, and some text is obscured by the images.\n\nFigure 2 is unclear and again closely resembles LoFTR; the figure should be revised to highlight the new contributions.\n\nThe Method section repeats much of LoFTR; please separate preliminaries from your own contributions more explicitly."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922113494,"tcdate":1761910434704,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10912/Reviewer_NG27"],"signatures":["ICLR.cc/2026/Conference/Submission10912/Reviewer_NG27"],"forum":"7lddcnHLCI","number":3,"license":"CC BY 4.0","cdate":1761910434704,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10912/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922113494,"domain":"ICLR.cc/2026/Conference","replyto":"7lddcnHLCI","id":"MD6mBZG8Nc","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["SLAM","Feature Matching","State Space Models"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"SLAM (Simultaneous Localization and Mapping) systems depend on front-end components for feature detection and matching. Traditional methods use handcrafted or learning-based features, but both have limitations in robustness and efficiency. We propose a new SLAM front-end framework that fuses recent State Space Models (SSMs), specifically the Mamba architecture, with transformer-based attention. Our method exploits the linear efficiency of SSMs for visual feature processing and the global modeling of attention. By integrating these techniques, we achieve better feature matching on challenging datasets while keeping computation efficient. The fusion strategy adaptively balances local relationships from Mamba and global dependencies from attention. Experiments show our approach surpasses state-of-the-art methods in feature matching precision and recall, especially in scenes with repetitive patterns, lighting and viewpoint shifts."},"_bibtex":{"value":"@misc{\nli2025mambamatch,\ntitle={MambaMatch: {SLAM} Front-End Feature Matching with State Space Models},\nauthor={Shijun Li and Jingping qiao and Jiechao Gao and Yifan Liu and Hengyu Liu and Luyao Tang and Haochen You and Chenxin Li and Zheng Zhu},\nyear={2025},\nurl={https://openreview.net/forum?id=7lddcnHLCI}\n}"},"title":{"value":"MambaMatch: SLAM Front-End Feature Matching with State Space Models"},"pdf":{"value":"/pdf/6a3bf43c34b080e67dbbf216a0a26e52650ec583.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"li|mambamatch_slam_frontend_feature_matching_with_state_space_models"},"authorids":{"value":["~Shijun_Li4","~Jingping_qiao2","~Jiechao_Gao5","~Yifan_Liu8","~Hengyu_Liu2","~Luyao_Tang1","~Haochen_You1","~Chenxin_Li1","~Zheng_Zhu1"]},"authors":{"value":["Shijun Li","Jingping qiao","Jiechao Gao","Yifan Liu","Hengyu Liu","Luyao Tang","Haochen You","Chenxin Li","Zheng Zhu"]}},"version":2},{"content":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"Simple hybrid state space models outperform SOTA Transformers and SSMs."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Models;Length Extrapolation;Efficiency;Hybrid State Space Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Efficiently modeling sequences with infinite context length has long been a challenging problem. Previous approaches have either suffered from quadratic computational complexity or limited extrapolation ability in length generalization. In this\nwork, we present Samba, a simple hybrid architecture that layer-wise combines\nMamba, a selective State Space Model (SSM), with Sliding Window Attention\n(SWA). Samba selectively compresses a given sequence into recurrent hidden\nstates while still maintaining the ability to precisely recall recent memories with the\nattention mechanism. We scale Samba up to 3.8B parameters with 3.2T training\ntokens and demonstrate that it significantly outperforms state-of-the-art models\nacross a variety of benchmarks. Pretrained on sequences of 4K length, Samba\nshows improved perplexity in context lengths of up to 1M in zero-shot. When\nfinetuned on 4K-length sequences, Samba efficiently extrapolates to a 256K context length with perfect memory recall on the Passkey Retrieval task, and exhibits\nsuperior retrieval extrapolation on the challenging Phonebook task compared to\nfull-attention models. As a linear-time sequence model, Samba achieves a 3.73×\nhigher throughput compared to Transformers with grouped-query attention for user\nprompts of 128K length, and a 3.64× speedup when generating 64K tokens with\nunlimited streaming."},"_bibtex":{"value":"@inproceedings{\nren2025samba,\ntitle={Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling},\nauthor={Liliang Ren and Yang Liu and Yadong Lu and yelong shen and Chen Liang and Weizhu Chen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=bIlnpVM4bc}\n}"},"title":{"value":"Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling"},"pdf":{"value":"/pdf/afcc61b2a3af1b85ec0707ec1a8d10ef4f128970.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"ren|samba_simple_hybrid_state_space_models_for_efficient_unlimited_context_language_modeling"},"authorids":{"value":["~Liliang_Ren1","~Yang_Liu50","~Yadong_Lu1","~yelong_shen1","~Chen_Liang3","~Weizhu_Chen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Liliang Ren","Yang Liu","Yadong Lu","yelong shen","Chen Liang","Weizhu Chen"]}},"tmdate":1740708685829,"pdate":1737562882284,"tcdate":1727477465161,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12408/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission12408/Authors"],"forum":"bIlnpVM4bc","license":"CC BY 4.0","number":12408,"cdate":1727477465161,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission12408/-/Full_Submission","ICLR.cc/2025/Conference/Submission12408/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit","ICLR.cc/2025/Conference/Submission12408/-/Camera_Ready_Revision"],"mdate":1740708685829,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"bIlnpVM4bc","version":2},{"content":{"summary":{"value":"This paper presents DyGMamba, a modeling framework designed for continuous-time dynamic graphs. Built on a state space model (SSM), DyGMamba captures and leverages hidden temporal patterns from historical graph data. These temporal patterns guide the selection of critical information from a node’s temporal neighbors, enhancing the model's ability to predict dynamic links. DyGMamba achieves state-of-the-art results across seven datasets in dynamic link prediction tasks. Furthermore, extensive experiments have been conducted to demonstrate the framework's efficiency."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. In the experiment illustrated in Fig. 4, which evaluates the scalability with varying numbers of neighbors, the analysis is conducted using the Enron dataset. However, the Enron dataset has an average degree of 676, meaning that only a small number of nodes have more than 676 neighbors. Have you considered the implications of this situation, where only a few nodes exceed this average degree? Could you discuss how this may influence the results presented in Fig. 4?\n\n\n2. In Figure 3a, why does the performance of DyGFormer decrease as the number of patches decreases, while the performance of DyGMamba increases? Could you provide some reasoning or discussion to explain this phenomenon?\n\n3. DyGFormer is a general framework designed to address the Continuous-Time Dynamic Graph (CTDG) problem, employing mean pooling to merge final node representations. In contrast, DyGMamba utilizes a short time-difference sequence to parameterize a weighted pooling mechanism instead of mean pooling. Based on the results from Variant A, I believe that Section 3.2 contributes to the observed performance gains.\nHowever, I have the following questions: \n- Is the Mamba structure necessary within the Time-Level SSM Block? From my perspective, both Mamba and Transformer can capture long-sequence dependencies, and Mamba has been shown to operate as a linear attention mechanism [2]. Therefore, the performance upper bound might be that of the Transformer itself or a Transformer with causal attention. **Could you explain why Mamba is used for such short sequences in Section 3.2? Additionally, please provide a comparison with the Transformer structure**. \n- Is time-difference encoding essential for the pooling strategy? After experimenting with your code and removing Equations (6a–7c) by setting $\\beta=Softmax(Linear(H))$ in Eq. (7d) [1], I observed that the performance remains strong. This raises **the question of whether time-difference encoding is necessary in your design**. It seems likely that the performance gain results from weighted pooling versus mean pooling, rather than from time-difference encoded weighted pooling.\n- To justify the inclusion of Mamba in Section 3.2, you should consider applying \"Dynamic Information Selection\" to DyGFormer. Possible comparisons could include: **1. DyGFormer with dynamic information selection; 2. DyGFormer with dynamic information selection using the Transformer structure instead;** \n\n\n4. As analyzed in W1, the introduction of Mamba appears to harm the performance of the DyGFormer structure, so further analysis of the reasons behind this would be helpful. More importantly, **beyond the efficiency gains from the Mamba structure and patching techniques, could you please clarify the contributions of DyGMamba?**\n\n\n\n5. In Section 3.2, lines 250–254, the part regarding \"enabling batch processing\" is difficult to follow. It’s unclear how using only 10 neighbors could lead to such a large value. **Could you provide details on the batch processing—specifically, whether it follows the same batching process as DyGFormer?** Additionally, how was the figure $10^{10}$ derived?\n\n\n\n6. The explanation of \"Dynamic Information Selection\" is generally unclear. There is no evidence about why the specific design in Equation 7 was chosen or what advantages it offers. Additionally, it’s not clear how this design effectively achieves selection. Could you provide a more thorough explanation of the intuition behind the Dynamic Information Selection mechanism and clarify **its benefits compared to other weighted pooling mechanisms?**\n\n\n7. Furthermore, other references for the State space model on dynamic graph learning should be added, such as [3], and the state-space model's applications on graphs should be discussed, such as [4].\n\n8. Finally, if the Mamba structure is not a key component of your design and the performance gains stem primarily from the pooling mechanism, demonstrating its effectiveness across other CTDG methods would be better for the paper. **Integrating \"Dynamic Information Selection\" with other methods**, such as FreeDyG, GraphMixer, and CTAN, to show performance improvements could clear the value of this component.\n\n\n**References:** \\\n[1] Language Modeling with Gated Convolutional Networks, ICML, 2017. \\\n[2] Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arxiv, 2023. \\\n[3] STG-Mamba: Spatial-Temporal Graph Learning via Selective State Space Model, arxiv, 2024. \\\n[4] Graph Mamba: Towards Learning on Graphs with State Space Models, KDD, 2024."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":1},"strengths":{"value":"1. The efficiency improvement introduced by Mamba is sound compared to the Transformer.\n2. This paper gives a clear and detailed description of the algorithm.\n3. Both link prediction and node classification tasks are discussed.\n4. The extensive experiments demonstrate that the Mamba structure significantly improves computational and memory efficiency compared to the Transformer."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Low Novelty:** \n- Based on my understanding of the algorithm, Section 3.1 is identical to DyGFormer, except that it replaces the Transformer with Mamba. In Section 3.2, the mean pooling mechanism in DyGFormer is modified to a \"Dynamic Information Selection\" strategy\nIt appears, then, that **the primary contribution lies in the design of the pooling strategy**, which is more like a minor technical improvement and totally **unrelated to the Mamba structure**. This is somewhat misleading, given that both the name DyGMamba and the abstract emphasize the Mamba component as a key feature. \n- Furthermore, in the ablation study, Variant A uses mean pooling for the output in Equation 5. This modification makes Variant A essentially identical to DyGFormer, with the sole difference being the replacement of Transformer with Mamba. However, the performance of Variant A drops compared to DyGFormer, suggesting that incorporating the Mamba structure into DyGFormer negatively impacts performance. This observation implies that the performance improvements in DyGFormer stem from the \"Dynamic Information Selection\" mechanism, while the efficiency gains are due to the Mamba structure and the patching techniques utilized within DyGFormer.\nFor this reason, **DyGMamba’s design offers limited novelty, as its efficiency gains are entirely derived from prior studies**—specifically, the Mamba structure and the patching techniques originally developed for DyGFormer.  \n- In addition, from Equation 6, the recent interaction number k is selected as a very small value, such as 10 or 30. Given this, is the Mamba block necessary in this context? Since Mamba primarily enhances efficiency, its advantages may not be realized with such short sequences. I argue that the Mamba structure is unnecessary in Section 3.2. And it contradicts the motivation of the paper, suggesting that **the Mamba structure may not necessary to your design**.\n\n\n\n\n2. **Reference issues:** \n - In line 176, from the description of the time encoding function, it is not the one from TGAT [1], but from GraphMixer [2]. \n - In line 180, the Node Interaction Frequency Encoding is followed FreeDyG [3], which should be Yu et al. [4], as it closely aligns with Yu's neighbor co-occurrence scheme. Specifically, the only difference between your description and DyGFormer’s neighbor co-occurrence scheme is the appending of the self-node to the last position in the sequence, which DyGFormer also implements.\n\n\n\n3. **Some key content illustrations are vague and hard to follow.** \n- In Section 3.2, lines 250–254, the part regarding \"enabling batch processing\" is difficult to follow. Specifically, how was the number $10^{10}$ derived? Is it accurate that using only 10 neighbors could result in such a large figure?\n- The overall explanation of \"Dynamic Information Selection\" is also unclear. It’s not evident why this particular design of Eq.7 was chosen or what advantages it offers. Additionally, it’s unclear how this design effectively achieves selection.\n- In the \"Ablation Study,\" the design of Variant B is unclear, which is described as removing the Mamba SSM layers in Equation 4. Does this mean completely removing the layers, or does it involve replacing Mamba with Transformer? In my opinion, simply removing these layers lacks purpose, and replacing Mamba with Transformer would be more meaningful. \n\n4. **Lacking Experiments:** \n- From W1, it appears that the performance gains are entirely due to the \"Dynamic Information Selection\" mechanism. To thoroughly evaluate the impact of this design and the effectiveness of the time-difference encoding, a more detailed ablation study focused on this component would be beneficial. However, only the entirety of Section 3.2 is ablated as Variant A, which presents a rough experiment. To me, the only observation from Variant A is that the Mamba structure negatively affects DyGFormer’s performance.\n- The paper lacks tuning key hyperparameters such as $\\alpha$ and $\\beta$ in time encoding, the $\\gamma$ in Time-Level SSM Block. These hyperparameters are crucial for model performance, and optimal settings may vary across different datasets. Additionally, the reason behind the introduction of $\\gamma$ is not discussed. \n- The model was only evaluated on seven datasets out of 13 datasets used in DyGFormer. Could be discussed more thoroughly about the statistics of data sparsity and give the reason why only seven datasets are used. This would improve the persuasion of the experiments part.\n- Recently, a very related study, FreeDyG [3], has been proposed and might need to be compared in your experiments.\n\n\n5. **There are no discussions regarding the limitations of this paper and possible solutions.**\n\n**Conclusion:** \\\nThis paper has a good example in the Introduction and is well-organized; however, the description of the key component in Section 3.2 is vague and difficult to follow. Additionally, the paper presents extensive experiments on efficiency comparisons, mainly attributed to the Mamba structure and patching techniques from the prior works. However, the effectiveness of the key component, \"Dynamic Information Selection,\" is inadequately evaluated through experiments. \\\nMost importantly, the novelty of the work is low, and the design of the algorithm raises concerns, particularly regarding the use of Mamba. Such as the results from Variant A indicate that incorporating Mamba can negatively impact performance, and the use of mamba in very short sequences, as shown in Equation 6. Therefore, I think this paper is insufficient for ICLR.\n\n\n\n**Reference**: \\\n[1] Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks, ICLR, 2022. \\\n[2] Do We Really Need Complicated Model Architectures For Temporal Networks?, ICLR, 2023. \\\n[3] FreeDyG: Frequency Enhanced Continuous-Time Dynamic Graph Model for Link Prediction, ICLR, 2024. \\\n[4] Towards Better Dynamic Graph Learning: New Architecture and Unified Library, NIPS, 2023."}},"nonreaders":[],"tmdate":1731428326515,"tcdate":1730135332256,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5146/Reviewer_Gzn2"],"signatures":["ICLR.cc/2025/Conference/Submission5146/Reviewer_Gzn2"],"forum":"f3gCs2a4ZD","number":1,"license":"CC BY 4.0","cdate":1730135332256,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5146/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428326515,"domain":"ICLR.cc/2025/Conference","replyto":"f3gCs2a4ZD","id":"0V8qMVAcdu","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["dynamic graph","state space model"]},"supplementary_material":{"value":"/attachment/2dc1a3502a1d53b635f3c846cad8277b987ac60b.zip"},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Learning useful representations for continuous-time dynamic graphs (CTDGs) is challenging, due to the concurrent need to span long node interaction histories and grasp nuanced temporal details. In particular, two problems emerge: (1) Encoding longer histories requires more computational resources, making it crucial for CTDG models to maintain low computational complexity to ensure efficiency; (2) Meanwhile, more powerful models are needed to identify and select the most critical temporal information within the extended context provided by longer histories. To address these problems, we propose a CTDG representation learning model named DyGMamba, originating from the popular Mamba state space model (SSM). DyGMamba first leverages a node-level SSM to encode the sequence of historical node interactions. Another time-level SSM is then employed to exploit the temporal patterns hidden in the historical graph, where its output is used to dynamically select the critical information from the interaction history. We validate DyGMamba experimentally on the dynamic link prediction task. The results show that our model achieves state-of-the-art in most cases. DyGMamba also maintains high efficiency in terms of computational resources, making it possible to capture long temporal dependencies with a limited computation budget."},"_bibtex":{"value":"@misc{\nding2025dygmamba,\ntitle={Dy{GM}amba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models},\nauthor={Zifeng Ding and Yifeng Li and Yuan He and Antonio Norelli and Jingcheng Wu and Volker Tresp and Yunpu Ma and Michael M. Bronstein},\nyear={2025},\nurl={https://openreview.net/forum?id=f3gCs2a4ZD}\n}"},"title":{"value":"DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models"},"pdf":{"value":"/pdf/0921c754d017d250e0d2f4b59a779766e47b30a6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ding|dygmamba_efficiently_modeling_longterm_temporal_dependency_on_continuoustime_dynamic_graphs_with_state_space_models"},"authorids":{"value":["~Zifeng_Ding1","~Yifeng_Li4","~Yuan_He5","~Antonio_Norelli2","~Jingcheng_Wu3","~Volker_Tresp1","~Yunpu_Ma1","~Michael_M._Bronstein1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zifeng Ding","Yifeng Li","Yuan He","Antonio Norelli","Jingcheng Wu","Volker Tresp","Yunpu Ma","Michael M. Bronstein"]}},"version":2},{"content":{"venue":{"value":"IEEE Transactions on Neural Networks and Learning Systems"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/5962385/6104215/11175044.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"liu|vision_mamba_a_comprehensive_survey_and_taxonomy"},"html":{"value":"https://doi.org/10.1109/TNNLS.2025.3610435"},"abstract":{"value":"State space model (SSM) is a mathematical model used to describe and analyze the behavior of dynamic systems. This model has witnessed numerous applications in several fields, including control theory, signal processing, economics, and machine learning. In the field of deep learning, SSMs are used to process sequence data, such as time series analysis, natural language processing (NLP), and video understanding. By mapping sequence data to state space, long-term dependencies in the data can be better captured. In particular, modern SSMs have shown strong representational capabilities in NLP, especially in long sequence modeling, while maintaining linear time complexity. In particular, based on the latest SSMs, Mamba merges time-varying parameters into SSMs toward efficient training and inference. Given its impressive efficiency and strong long-range dependency modeling capability, Mamba is expected to become a new AI architecture that may be capable of surpassing Transformer. Recently, a number of works attempt to study the potential of Mamba in various fields, such as general vision, multimodal learning, medical image analysis, and remote sensing image analysis, by extending Mamba from natural language domain to visual domain. To fully understand Mamba in the visual domain, we conduct a comprehensive survey and present a taxonomy study. This survey focuses on Mamba’s application to a variety of visual tasks and data types, and discusses its predecessors, recent advances, and far-reaching impact on a wide range of domains."},"title":{"value":"Vision Mamba: A Comprehensive Survey and Taxonomy"},"authors":{"value":[{"fullname":"Xiao Liu"},{"fullname":"Chenxu Zhang","username":"~Chenxu_Zhang2"},{"fullname":"Fuxiang Huang","username":"~Fuxiang_Huang2"},{"fullname":"Shuyin Xia"},{"fullname":"Guoyin Wang"},{"fullname":"Lei Zhang"}]}},"tmdate":1789091553465,"pdate":1735689600000,"externalIds":["doi:10.1109/tnnls.2025.3610435"],"tcdate":1762518942228,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Chenxu_Zhang2"],"forum":"TEz2TAcI5x","license":"CC BY-SA 4.0","number":10588,"cdate":1758589098276,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Authorship_Claim","OpenReview.net/-/Edit"],"mdate":1789091553465,"domain":"OpenReview.net/Public_Article","id":"TEz2TAcI5x","version":2},{"content":{"venue":{"value":"ICONIP (3) 2024"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-981-96-6582-2_14.pdf"},"venueid":{"value":"dblp.org/conf/ICONIP/2024"},"paperhash":{"value":"lei|mamamba_multiagent_reinforcement_learning_with_state_space_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Yuan_Lei:","~Jian_Xue2","https://dblp.org/search/pid/api?q=author:Lin_Zhao:","https://dblp.org/search/pid/api?q=author:Chao_Yue:","https://dblp.org/search/pid/api?q=author:Ke_Lu_0002:"]},"html":{"value":"https://doi.org/10.1007/978-981-96-6582-2_14"},"_bibtex":{"value":"@inproceedings{DBLP:conf/iconip/LeiXZYL24,\n  author={Yuan Lei and Jian Xue and Lin Zhao and Chao Yue and Ke Lu},\n  title={MA-Mamba: Multi-Agent Reinforcement Learning with State Space Model},\n  year={2024},\n  cdate={1704067200000},\n  pages={198-210},\n  url={https://doi.org/10.1007/978-981-96-6582-2_14},\n  booktitle={ICONIP (3)},\n  crossref={conf/iconip/2024-3}\n}\n"},"abstract":{"value":"In recent years, there has been a notable advancement in the field of Multi-Agent Reinforcement Learning (MARL). The Multi-Agent Transformer (MAT) represents the multi-agent decision-making problem as a sequence modeling problem and introduces the Transformer as a solution to the cooperative MARL problem. Mamba is a promising efficient State Space Model (SSM) that exhibits comparable performance to Transformers while having fewer parameters on long sequences. To enhance the efficacy of MARL in heterogeneous-agent and complex scenarios, we propose a more efficient and adaptable solution, the MA-Mamba method based on Mamba. In order to solve the problem of sparse temporal information and difficulty in modeling agent relationships in MARL, we propose to introduce a multi-channel mechanism to enhance the ability of MA-Mamba model to extract high-dimensional information. Moreover, we have implemented a hybrid module that integrates Mamba and attention mechanisms to facilitate the efficient encoding and decoding of information, thereby enabling the generation of more precise action sequences. Our method was tested on the StarCraftII Multi-Agent Challenge (SMAC), a widely utilized multi-Agent testing platform for MARL. The experimental results demonstrate that MA-Mamba achieves the best performance on all 13 tested maps, with four of them outperforming the best algorithm MAT."},"title":{"value":"MA-Mamba: Multi-Agent Reinforcement Learning with State Space Model"},"authors":{"value":["Yuan Lei","Jian Xue","Lin Zhao","Chao Yue","Ke Lu"]}},"tmdate":1754035851853,"pdate":1704067200000,"externalIds":["dblp:conf/iconip/LeiXZYL24"],"tcdate":1754035848621,"writers":["~"],"signatures":["~Jian_Xue2"],"forum":"wApQRKl4KW","license":"CC BY-SA 4.0","number":608541,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1754035851853,"domain":"DBLP.org","id":"wApQRKl4KW","version":2},{"content":{"summary":{"value":"The authors explore a key research question: Can the attention-free Mamba model effectively handle meta-continual learning (MCL) tasks? They reframe State Space Models (SSM) and Mamba as sequence-prediction-based continual learners, training them via meta-learning across continual learning episodes. To enhance this training, they introduce a selectivity regularization technique. Extensive experiments reveal that Mamba consistently performs well in various MCL settings, significantly surpassing other attention-free approaches and often equaling or surpassing Transformer models in performance—all while using fewer parameters and computational resources. Notably, Mamba demonstrates strong reliability, generalization, and robustness in complex scenarios."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"N/A"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"* It is interesting to explore how  Mamba performs in a meta-continual learning setting."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The conclusion of this paper is unsurprising, as Mamba's MCL performance aligns closely with its results on standard benchmarks.\n\n* There is insufficient analysis explaining how and why Mamba outperforms other attention-free architectures and achieves comparable results to Transformer-based models."}},"nonreaders":[],"tmdate":1731427182599,"tcdate":1730007531941,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9/Reviewer_BA8y"],"signatures":["ICLR.cc/2025/Conference/Submission9/Reviewer_BA8y"],"forum":"1TXDtnDIsV","number":1,"license":"CC BY 4.0","cdate":1730007531941,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427182599,"domain":"ICLR.cc/2025/Conference","replyto":"1TXDtnDIsV","id":"yTRFSeYK5q","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Continual Learning","Sequence Modelling"]},"primary_area":{"value":"transfer learning, meta learning, and lifelong learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Continual learning (CL) aims to efficiently learn and accumulate knowledge from a data stream with different distributions. By formulating CL as a sequence prediction task, meta-continual learning (MCL) enables to meta-learn an efficient continual learner based on the recent advanced sequence models, e.g., Transformers. Although attention-free models (e.g., Linear Transformers) can ideally match CL's essential objective and efficiency requirements, they usually perform not well in MCL. Considering that the attention-free Mamba achieves excellent performances matching Transformers' on general sequence modeling tasks, in this paper, we aim to answer a question -- Can attention-free Mamba perform well on MCL? By formulating Mamba with a selective state space model (SSM) for MCL tasks, we propose to meta-learn Mamba as a continual learner, referred to as MambaCL. By incorporating a selectivity regularization, we can effectively train MambaCL. Through comprehensive experiments across various CL tasks, we also explore how Mamba and other models perform in different MCL scenarios. Our experiments and analyses highlight the promising performance and generalization capabilities of Mamba in MCL."},"_bibtex":{"value":"@misc{\nzhao2025learning,\ntitle={Learning Mamba as a Continual Learner},\nauthor={Chongyang Zhao and Dong Gong},\nyear={2025},\nurl={https://openreview.net/forum?id=1TXDtnDIsV}\n}"},"title":{"value":"Learning Mamba as a Continual Learner"},"pdf":{"value":"/pdf/65bb972e176ce4ef1a4d56629ad29b0e78847ac1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhao|learning_mamba_as_a_continual_learner"},"authorids":{"value":["~Chongyang_Zhao1","~Dong_Gong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chongyang Zhao","Dong Gong"]}},"version":2},{"content":{"TLDR":{"value":"First application of structured state space sequence models (S4s) to de novo design."},"venue":{"value":"ML4LMS Poster"},"keywords":{"value":["drug discovery","de novo design","chemical language modeling","state space models","s4"]},"supplementary_material":{"value":"/attachment/2593df673ce284ce6d086b83e37e0d10af1d3c2a.pdf"},"abstract":{"value":"Generative deep learning is reshaping drug design. Chemical language models (CLMs) — which generate molecules in the form of molecular strings — bear particular promise for this endeavor. Here, we introduce a recent deep learning architecture, termed Structured State-Space Sequence (S4) model, into de novo drug design. In addition to its unprecedented performance in various fields, S4 has shown remarkable capabilities to learn the global properties of sequences. This aspect is intriguing in chemical language modeling, where complex molecular properties like bioactivity can 'emerge' from separated portions in the molecular string. This observation gives rise to the following question: Can S4 advance chemical language modeling for de novo design? To provide an answer, we systematically benchmark S4 with state-of-the-art CLMs on an array of drug discovery tasks, such as the identification of bioactive compounds, and the design of drug-like molecules and natural products. S4 showed a superior capacity to learn complex molecular properties, while at the same time exploring diverse scaffolds. Finally, when applied prospectively to kinase inhibition, S4 designed eight of out ten molecules that were predicted as highly active by molecular dynamics simulations. Taken together, these findings advocate for the introduction of S4 into chemical language modeling — uncovering its untapped potential in the molecular sciences."},"_bibtex":{"value":"@inproceedings{\nanonymous2024chemical,\ntitle={Chemical Language Modeling with Structured State Spaces},\nauthor={Anonymous},\nbooktitle={ICML'24 Workshop ML for Life and Material Science: From Theory to Industry Applications},\nyear={2024},\nurl={https://openreview.net/forum?id=xqGMG37x5P}\n}"},"title":{"value":"Chemical Language Modeling with Structured State Spaces"},"pdf":{"value":"/pdf/132106c8f143e55bf5334c1f9ac7fe16f07ab109.pdf"},"venueid":{"value":"ICML.cc/2024/Workshop/ML4LMS"},"paperhash":{"value":"özçelik|chemical_language_modeling_with_structured_state_spaces"},"authorids":{"value":["~Rıza_Özçelik2","~Sarah_de_Ruiter1","e.criscuolo@tue.nl","~Francesca_Grisoni2"]},"poster":{"value":"/attachment/13ca0136f2cfb240472f2ce020d757a31a69a29f.pdf"},"authors":{"value":["Rıza Özçelik","Sarah de Ruiter","Emanuele Criscuolo","Francesca Grisoni"]}},"tmdate":1721143954469,"pdate":1718661419083,"tcdate":1714480384184,"writers":["ICML.cc/2024/Workshop/ML4LMS","ICML.cc/2024/Workshop/ML4LMS/Submission5/Authors"],"signatures":["ICML.cc/2024/Workshop/ML4LMS"],"forum":"xqGMG37x5P","license":"CC BY 4.0","number":5,"cdate":1714480384184,"readers":["everyone"],"invitations":["ICML.cc/2024/Workshop/ML4LMS/-/Submission","ICML.cc/2024/Workshop/ML4LMS/-/Post_Submission","ICML.cc/2024/Workshop/ML4LMS/-/Edit","ICML.cc/2024/Workshop/ML4LMS/Submission5/-/Revision"],"mdate":1721143954469,"odate":1716510540449,"domain":"ICML.cc/2024/Workshop/ML4LMS","id":"xqGMG37x5P","version":2},{"content":{"summary":{"value":"This work introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State Space Model (SSM). The Gaussian reconstructor propagates causal context containing multi-view information for cross-view self-refinement while generating a long sequence of Gaussians for fine-detail modeling with linear complexity. With off-the-shelf multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Extensive experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only 0.1x of the model size."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See the weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper is of good written quality.\n2. The inference speed is relatively fast compared to optimization-based methods.\n3. This work introduces the new architecture, Mamba, into the field of 3D generative models, which is novel.\n4. The computation cost of the training is lower than LGM and LRM.\n5. The method is robust to multiview inconsistency."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. In Fig. 6(b), the performance is continuously increasing as the token length increase. Although the authors' motivation is to lower the computation cost, Fig. 6(b) indicates the the performance grows as the computation cost grows.\n2. No failure cases are shown to demonstrate the limitation of this work.\n3. Fig 2(b) may be wrong. Mamba has linear complexity instead of constant complexity.\n4. The video results in the supplementary materials are only marginally better than LGM's results."},"limitations":{"value":"See the weaknesses."}},"nonreaders":[],"tmdate":1730878754880,"tcdate":1718524078764,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2178/Reviewer_5iTN"],"signatures":["NeurIPS.cc/2024/Conference/Submission2178/Reviewer_5iTN"],"forum":"AprsVxrwXT","number":1,"license":"CC BY 4.0","cdate":1718524078764,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2178/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878754880,"domain":"NeurIPS.cc/2024/Conference","replyto":"AprsVxrwXT","id":"3kL6xK9F9d","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["3D Generation and Reconstruction","State Space Model"]},"supplementary_material":{"value":"/attachment/836f2031251d5559938f7200420a18aa0a39f08c.zip"},"primary_area":{"value":"generative_models"},"abstract":{"value":"Recent 3D large reconstruction models (LRMs) can generate high-quality 3D content in sub-seconds by integrating multi-view diffusion models with scalable multi-view reconstructors. Current works further leverage 3D Gaussian Splatting as 3D representation for improved visual quality and rendering efficiency. However, we observe that existing Gaussian reconstruction models often suffer from multi-view inconsistency and blurred textures. We attribute this to the compromise of multi-view information propagation in favor of adopting powerful yet computationally intensive architectures (\\eg, Transformers). \nTo address this issue, we introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State Space Model (SSM). Our Gaussian reconstructor propagates causal context containing multi-view information for cross-view self-refinement while generating a long sequence of Gaussians for fine-detail modeling with linear complexity.\nWith off-the-shelf multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Extensive experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only $0.1\\times$ of the model size. The codes are available at \\url{https://github.com/SkyworkAI/MVGamba}."},"_bibtex":{"value":"@inproceedings{\nyi2024mvgamba,\ntitle={{MVG}amba: Unify 3D Content Generation as State Space Sequence Modeling},\nauthor={Xuanyu Yi and Zike Wu and Qiuhong Shen and Qingshan Xu and Pan Zhou and Joo Hwee Lim and Shuicheng YAN and Xinchao Wang and Hanwang Zhang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=AprsVxrwXT}\n}"},"title":{"value":"MVGamba: Unify 3D Content Generation as State Space Sequence Modeling"},"pdf":{"value":"/pdf/d1260f2186da4119a6e9c5f99613609eaa771e3e.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"yi|mvgamba_unify_3d_content_generation_as_state_space_sequence_modeling"},"authorids":{"value":["~Xuanyu_Yi1","~Zike_Wu1","~Qiuhong_Shen1","~Qingshan_Xu1","~Pan_Zhou3","~Joo_Hwee_Lim1","~Shuicheng_YAN3","~Xinchao_Wang1","~Hanwang_Zhang3"]},"authors":{"value":["Xuanyu Yi","Zike Wu","Qiuhong Shen","Qingshan Xu","Pan Zhou","Joo Hwee Lim","Shuicheng YAN","Xinchao Wang","Hanwang Zhang"]}},"version":2},{"content":{"summary":{"value":"The paper addresses the problem of low FLOP utilization with TTT layers. They proposes Large-Chunk Test-Time Training (LaCT): updating fast weights over very large chunks compared to prior work to reach high FLOP utilization. As updates of fast weights treats token in the current chunk as sets, TTT layers are paired with sliding window attention to recover locality. The method is validated on novel view synthesis (NVS, up to ~1M tokens), language modeling (per-position loss & RULER/S-NIAH retrieval), and autoregressive video diffusion showing competitive performance with linear time models and Transformers."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Is it possible to ablate the effect of the chunk size on performance?\n2. Can you clarify the end-to-end complexity where the chunk size equals the full sequence length?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The suggested solution of combining large chunks with SWA is effective, simple and easy to implement.\n2. Evaluation is very robust spanning multiple modalities and tasks, comparing to relevant baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Combining linear time sequence models with attention is not new and has been shown to improve the performance, see e.g., [1], although not motivated from low throughput.\n2. The method strongly relies on large chunk sizes yet the effect of chunk sizes on performance (while maintaining high throughput) is not clear from the results.\n\n[1] An Empirical Study of Mamba-based Language Models"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923778089,"tcdate":1761916251144,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13043/Reviewer_21cR"],"signatures":["ICLR.cc/2026/Conference/Submission13043/Reviewer_21cR"],"forum":"Tb9qAxT3xv","number":1,"license":"CC BY 4.0","cdate":1761916251144,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13043/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923778089,"domain":"ICLR.cc/2026/Conference","replyto":"Tb9qAxT3xv","id":"hsgyq8g6F5","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"Large chunk test-time training enables efficient training of large nonlinear RNN."},"keywords":{"value":["Test-Time Training","Sequence Model","Long Context Model"]},"supplementary_material":{"value":"/attachment/03d3be37bbd96a2868c5744e6d632c6afc6affaa.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Test-Time Training (TTT) models context dependencies by adapting part of the model's weights (often referred to as fast weights) at inference time. This adapted fast weight, similar to recurrent states in RNNs, stores temporary memories of past tokens in the current sequence. Existing TTT methods have struggled to demonstrate effectiveness in handling long-sequence data, due to their computational inefficiency on modern GPUs. The TTT layers in many of these approaches operate with extremely low FLOPs utilization (often below 5%) because they deliberately apply small online mini-batch sizes (e.g., updating fast weights every 16 or 64 tokens). Moreover, a small mini-batch implies fine-grained block-wise causal dependencies in the data, making them unsuitable for data beyond 1D ordered sequences, like sets or N-dimensional grids such as images or videos. In contrast, we pursue the opposite direction by proposing an extremely large chunk update, ranging from 2K to 1M tokens across tasks of varying modalities, which we refer to as Large Chunk Test-Time Training (LaCT). This approach improves hardware utilization by orders of magnitude, and more importantly, facilitates scaling of nonlinear state size (up to 40% of model parameter size), hence substantially improving state capacity, all without requiring cumbersome and error-prone custom kernel implementations. It also allows easy integration of sophisticated optimizers like Muon for online memory updates. We validate our approach across diverse data modalities and tasks, including novel view synthesis from image sets, language models, and auto-regressive video diffusion models. Our approach can scale up to 14-billion-parameter auto-regressive video diffusion models handling sequences of up to 56K tokens. In our longest sequence experiment, we perform novel view synthesis with more than one million context length. Our results highlight the computational and performance benefits of large-chunk test-time training, paving the way for more efficient and scalable long-context sequence modeling. We hope that this work will inspire and accelerate new research in the field of long-context modeling and test-time training. See visual results on project website \\url{https://tianyuanzhang.com/projects/ttt-done-right/}."},"_bibtex":{"value":"@inproceedings{\nzhang2026testtime,\ntitle={Test-Time Training Done Right},\nauthor={Tianyuan Zhang and Sai Bi and Yicong Hong and Kai Zhang and Fujun Luan and Songlin Yang and Kalyan Sunkavalli and William T. Freeman and Hao Tan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=Tb9qAxT3xv}\n}"},"title":{"value":"Test-Time Training Done Right"},"pdf":{"value":"/pdf/c7cc05baf499da2d19da2f9c556356bb1ebd1c14.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|testtime_training_done_right"},"authorids":{"value":["~Tianyuan_Zhang2","~Sai_Bi1","~Yicong_Hong1","~Kai_Zhang7","~Fujun_Luan2","~Songlin_Yang1","~Kalyan_Sunkavalli1","~William_T._Freeman1","~Hao_Tan1"]},"authors":{"value":["Tianyuan Zhang","Sai Bi","Yicong Hong","Kai Zhang","Fujun Luan","Songlin Yang","Kalyan Sunkavalli","William T. Freeman","Hao Tan"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a new visual compression network called MambaVC. Based on the State Space Model (SSM), MambaVC uses Visual State Space (VSS) blocks and a 2D Selective Scan (2DSS) module to enhance global context modeling, improving performance and efficiency in image compression.  MambaVC outperforms traditional Convolutional Neural Networks (CNN) and Transformer models in image compression benchmarks, showing clear advantages in computational and memory efficiency. The paper also presents experiments and evaluations demonstrating MambaVC's potential in high-resolution image compression, comparing it with other network designs to validate its advantages in compression efficiency, computational resource consumption, and memory usage."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. **Training For Video**: Can the authors provide more details about how to adapt the image compression method to video compression? How does the quality of video datasets influence the performance?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. **2DSS**: Authors introduce VSS blocks, which incorporate the 2D Selective Scan (2DSS) mechanism, enabling selective scanning along four predefined paths. This design allows the model to process spatial information in parallel, enhancing its ability to capture global context while reducing unnecessary information redundancy.\n\n2. **Performance on Benchmarks**: Experiments on benchmark datasets show superior performance and competitive efficiency of MambaVC on image and video compression.\n\n3. **High-Res Results**: Authors show the effectiveness and scalability to high-resolution compression"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Lack Baseline Comparision for 2DSS**: Although the authors compare MambaVC with ConvVC and SwinVC, the core innovation of the paper, 2DSS, is not analyzed in depth. It remains unclear how significant the impact of 2DSS is. The authors should compare the 2DSS mechanism with vanilla 2D inputs handling method of Mamba model, such as Vision Mamba[R1]’s approach, to demonstrate the effectiveness of 2DSS.\n\n2. **Variations of 2DSS**: A deeper exploration of 2DSS appears to be missing. The current 2DSS uses cardinal scan patterns, but how would alternative patterns (e.g., diagonal) affect performance? Additionally, what is the impact of varying the number of scan patterns on the results?\n\n[R1] Zhu, Lianghui, et al. \"Vision mamba: Efficient visual representation learning with bidirectional state space model.\" arXiv preprint arXiv:2401.09417 (2024)."}},"nonreaders":[],"tmdate":1731427251762,"tcdate":1730710509597,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission408/Reviewer_kCv8"],"signatures":["ICLR.cc/2025/Conference/Submission408/Reviewer_kCv8"],"forum":"KgJwbsfN7G","number":4,"license":"CC BY 4.0","cdate":1730710509597,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission408/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427251762,"domain":"ICLR.cc/2025/Conference","replyto":"KgJwbsfN7G","id":"0icMfvYmop","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Selective State Spaces; Learned Visual Compression"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Learned visual compression is an important and active task in multimedia. Existing approaches have explored various CNN- and Transformer-based designs to model content distribution and eliminate redundancy, where balancing efficacy (i.e., rate-distortion trade-off) and efficiency remains a challenge. Recently, state-space models (SSMs) have shown promise due to their long-range modeling capacity and efficiency. Inspired by this, we take the first step to explore SSMs for visual compression. We introduce MambaVC, a simple, strong and efficient compression network based on SSM. MambaVC develops a visual state space (VSS) block with a 2D selective scanning (2DSS) module as the nonlinear activation function after each downsampling, which helps to capture informative global contexts and enhances compression. On compression benchmark datasets, MambaVC achieves superior rate-distortion performance with lower computational and memory overheads. Specifically, it outperforms CNN and Transformer variants by 7.2% and 15.2% on Kodak, respectively, while reducing computation by 42% and 24%, and saving 12% and 71% of memory. MambaVC shows even greater improvements with high-resolution images, highlighting its potential and scalability in real-world applications. We also provide a comprehensive comparison of different network designs, underscoring MambaVC's advantages. Code is available at https://anonymous.4open.science/r/MambaVC-408 and will be open-sourced."},"_bibtex":{"value":"@misc{\nqin2024mambavc,\ntitle={Mamba{VC}: Exploring Selective State Spaces for Learned Visual Compression},\nauthor={Shiyu Qin and Jinpeng Wang and Yimin Zhou and Tianci Luo and Bin Chen and Baoyi An and Tao Dai and Shu-Tao Xia and Yaowei Wang},\nyear={2024},\nurl={https://openreview.net/forum?id=KgJwbsfN7G}\n}"},"title":{"value":"MambaVC: Exploring Selective State Spaces for Learned Visual Compression"},"pdf":{"value":"/pdf/ddff1f38bd940bf2fd5e03ceba2546c9f1081429.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"qin|mambavc_exploring_selective_state_spaces_for_learned_visual_compression"},"authorids":{"value":["~Shiyu_Qin1","~Jinpeng_Wang3","~Yimin_Zhou1","~Tianci_Luo1","~Bin_Chen4","~Baoyi_An1","~Tao_Dai3","~Shu-Tao_Xia1","~Yaowei_Wang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Shiyu Qin","Jinpeng Wang","Yimin Zhou","Tianci Luo","Bin Chen","Baoyi An","Tao Dai","Shu-Tao Xia","Yaowei Wang"]}},"version":2},{"content":{"summary":{"value":"The paper presents NeuralWalker, which combines random walks and message-passing mechanisms to capture both local and long-range dependencies in graphs. The model processes graphs by treating random walks as sequences, thereby leveraging sequence models to handle long-range interactions."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1.\tIs the motivation for using the random walk sampler driven by memory constraints? Additionally, in Figure 3, why does the ZINC dataset display a different trend compared to the other two datasets?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1.\tExisting graph transformers works typically incorporate random walks as positional encodings; directly integrating them into token design is an interesting approach.\n\n2.\tThe theoretical analysis provided is thorough and comprehensive.\n\n3.\tExtensive experiments are conducted, with detailed results presented both in the main text and the appendix."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tThe primary concern is the unclear motivation. Why do the authors choose to integrate random walks into token design, and how could this approach benefit over graph transformers, which can already use positional encodings to capture structural information? A preliminary study exploring this choice would be helpful.\n\n2.\tBuilding on this, the purpose of incorporating multiple techniques, such as the walk aggregator, also lacks clarity. For instance, why use average pooling in the walk aggregator? Do different random walks carry the same importance?\n\n3.\tAnother concern is the lack of technical innovation. It’s difficult to discern a clear distinction between NeuralWalker and graph transformers, as NeuralWalker appears to follow general design principles of graph transformers, treating the graph as a sequence and representing tokens as the combination of nodes, edges, and positional encodings, albeit based on random walks.\n\n4.\tCould the authors clarify why SSE performs better than transformers? Since SSE and Mamba are designed based on RNNs, have bidirectional RNNs been tried as a sequence model?\n\n5.\tI also question the motivation for including global message passing although the ablation study shows it could help. When the length of random walks is long, they should already capture global information inherently. Besides, the authors directly borrow existing graph transformer methods for this, could you provide an ablation study on the choice of graph transformer in this part?\n\n6.\tFollowing this, an empirical efficiency comparison—specifically running time and memory usage—against current graph transformer baselines like Polynormer would be beneficial.\n\n7.\tAnother major issue is the absence of several recent baselines. In traditional GNNs, several works, like LazyGNN[1], aim to capture long-range dependencies. There are also state-of-the-art graph transformers, such as VCR-Graphormer[2] and GOAT[3], and crucially, since the authors use SSE as a sequence model, Graph Mamba[4] should be included as a baseline.\n\n8.\tThe datasets are not large enough, especially for node classification tasks. Commonly used datasets like the medium-sized ogbn-products or the large-sized ogbn-papers100m are not included. The current datasets are not common and cannot be considered as large-scale.\n\n[1] LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagation\n\n[2] VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connections\n\n[3] GOAT: A Global Transformer on Large-scale Graphs\n\n[4] Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces"}},"nonreaders":[],"tmdate":1732764254751,"tcdate":1730606819342,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission7025/Reviewer_CxEX"],"signatures":["ICLR.cc/2025/Conference/Submission7025/Reviewer_CxEX"],"forum":"kJ5H7oGT2M","number":2,"license":"CC BY 4.0","cdate":1730606819342,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission7025/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732764254751,"domain":"ICLR.cc/2025/Conference","replyto":"kJ5H7oGT2M","id":"FLk763uQqk","forumContent":{"TLDR":{"value":"A novel random-walk based neural architecture for graph representation learning"},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Graph neural networks","Graph transformers","Random walks","Long-range dependencies"]},"supplementary_material":{"value":"/attachment/50c6aae8c04a492040eabc7613d483878f67c58a.zip"},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Message-passing graph neural networks (GNNs) excel at capturing local relationships but struggle with long-range dependencies in graphs. In contrast, graph transformers (GTs) enable global information exchange but often oversimplify the graph structure by representing graphs as sets of fixed-length vectors. This work introduces a novel architecture that overcomes the shortcomings of both approaches by combining the long-range information of random walks with local message passing. By treating random walks as sequences, our architecture leverages recent advances in sequence models to effectively capture long-range dependencies within these walks. Based on this concept, we propose a framework that offers (1) more expressive graph representations through random walk sequences, (2) the ability to utilize any sequence model for capturing long-range dependencies, and (3) the flexibility by integrating various GNN and GT architectures. Our experimental evaluations demonstrate that our approach achieves competitive performance on 19 graph and node benchmark datasets, notably outperforming existing methods by up to 13\\% on the PascalVoc-SP and COCO-SP datasets.\nCode: https://github.com/BorgwardtLab/NeuralWalker"},"_bibtex":{"value":"@inproceedings{\nchen2025learning,\ntitle={Learning Long Range Dependencies on Graphs via Random Walks},\nauthor={Dexiong Chen and Till Hendrik Schulz and Karsten Borgwardt},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=kJ5H7oGT2M}\n}"},"title":{"value":"Learning Long Range Dependencies on Graphs via Random Walks"},"pdf":{"value":"/pdf/dc3d39aafa3105645e5407cb2008e4053e257ad9.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"chen|learning_long_range_dependencies_on_graphs_via_random_walks"},"authorids":{"value":["~Dexiong_Chen1","~Till_Hendrik_Schulz1","~Karsten_Borgwardt2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Dexiong Chen","Till Hendrik Schulz","Karsten Borgwardt"]}},"version":2},{"content":{"summary":{"value":"The authors present a protein model which is based on the Mamba backbone which runs in bidirectional mode with parameter sharing. The authors compare their method to ESM-2 for the pre-training data (UniRef90) and for downstream structure prediction tasks. For protein protein interaction tasks, the authors present a graph-based version of their model."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"- Wrt the PPI section: It seems a bit counterintuitive to force the graph-structure like information into tokenized text only to be able to naively apply a language model to it. Why shouldn't it be possible to simply train a GNN on the given data?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The results indicate that the proposed model outperforms the ESM-2 baseline on both pretraining MLM objective and the downstream tasks.\n- The authors perform experiments for a wide range of model sizes ranging from 130M parameters to 1400M parameters."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Little novelty**: \n  * Protein modeling with Mamba based backbone architecture is done here [1]. Notably, in a sense [1] also allows for bidirectional information sharing by their fill-in-the-middle objective.\n  * Bidirectional Mamba blocks with parameter sharing was already done here [2].\n\n- **Lack of comparison**:\n  * If the authors thought, their work is substantially different from [1], they have to compare to [1] and discuss benefits/differences.\n  * For all experiments, e.g. ProteinGym experiment (Table 4), SOTA methods and their performances should be reported along with ESM-2 and LC-PLM-*.\n\n- Figure 2: Mistake wrt the orientation of the third linear projection block.\n\n\n\n[1] ProtMamba: a homology-aware but alignment-free protein state space model\n\n[2] Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling"}},"nonreaders":[],"tmdate":1731429172386,"tcdate":1730673966051,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8832/Reviewer_sY9s"],"signatures":["ICLR.cc/2025/Conference/Submission8832/Reviewer_sY9s"],"forum":"Et0SIGDpP5","number":3,"license":"CC BY 4.0","cdate":1730673966051,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8832/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429172386,"domain":"ICLR.cc/2025/Conference","replyto":"Et0SIGDpP5","id":"3zu2yV6v6W","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["protein language model"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. \nMost protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. Such protein LMs cannot extrapolate to longer proteins and protein complexes well. They also fail to account for the underlying biological mechanisms carried out by biomolecular interactions and dynamics i.e., proteins often interact with other proteins, molecules, and pathways in complex biological systems. \nIn this work, we propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built off selective structured state-space models, to learn high-quality universal protein representations at the amino acid token level using masked language modeling. We also introduce its graph-contextual variant, LC-PLM-G, which contextualizes protein-protein interaction (PPI) graphs for a second stage of training. LC-PLM demonstrates favorable neural scaling laws, better length extrapolation capability, and a 7\\% to 34\\% improvement on protein downstream tasks than Transformer-based ESM-2. LC-PLM-G further trained within the context of PPI graphs shows promising results on protein structure and function prediction tasks. Our study demonstrates the benefit of increasing the context size with computationally efficient LM architecture (e.g. structured SSMs) in learning universal protein representations and incorporating molecular interaction contexts contained in biological graphs."},"_bibtex":{"value":"@misc{\nwang2025longcontext,\ntitle={Long-context Protein Language Model},\nauthor={Yingheng Wang and Zichen Wang and Gil Sadeh and Luca Zancato and Alessandro Achille and George Karypis and Huzefa Rangwala},\nyear={2025},\nurl={https://openreview.net/forum?id=Et0SIGDpP5}\n}"},"title":{"value":"Long-context Protein Language Model"},"pdf":{"value":"/pdf/3dea956cc09965e552e48b137c346e62db12eb35.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"wang|longcontext_protein_language_model"},"authorids":{"value":["~Yingheng_Wang1","~Zichen_Wang3","~Gil_Sadeh2","~Luca_Zancato1","~Alessandro_Achille1","~George_Karypis1","~Huzefa_Rangwala2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yingheng Wang","Zichen Wang","Gil Sadeh","Luca Zancato","Alessandro Achille","George Karypis","Huzefa Rangwala"]}},"version":2},{"content":{"TLDR":{"value":"We show that expanding the eigenvalue range of Linear RNN from [0, 1] to [-1,1] enhances their state-tracking capabilities, enabling them to solve complex tasks like parity and modular counting, while preserving their efficiency in language modeling."},"venue":{"value":"M3L Oral"},"is_neurips_submission":{"value":"No"},"pdf":{"value":"/pdf/318c0cbbe4e559f8a71e6e7b7b724a4536e544e8.pdf"},"keywords":{"value":["State Tracking","state space","mamba","Linear RNN","Linear Attention","GLA","DeltaNet","Formal Languages"]},"venueid":{"value":"NeurIPS.cc/2024/Workshop/M3L"},"paperhash":{"value":"grazzi|unlocking_statetracking_in_linear_rnns_through_negative_eigenvalues"},"authorids":{"value":["~Riccardo_Grazzi2","~Julien_Siems1","~Jörg_K.H._Franke1","~Arber_Zela1","~Frank_Hutter1","~Massimiliano_Pontil4"]},"abstract":{"value":"Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers in large language modeling, offering linear scaling with sequence length and improved training efficiency. However, LRNNs struggle to perform state-tracking which may impair performance in tasks such as code evaluation or tracking a chess game. Even parity, the simplest state-tracking task, which non-linear RNNs like LSTM handle effectively, cannot be solved by current LRNNs. Recently, Sarrof et al. (2024) demonstrated that the failure of LRNNs like Mamba to solve parity stems from restricting the value range of their diagonal state-transition matrices to $[0, 1]$ and that incorporating negative values can resolve this issue. We extend this result to non-diagonal LRNNs, which have recently shown promise in models such as DeltaNet. We prove that finite precision LRNNs with state-transition matrices having only positive eigenvalues cannot solve parity, while complex eigenvalues are needed to count modulo $3$. Notably, we also prove that LRNNs can learn any regular language when their state-transition matrices are products of identity minus vector outer product matrices, each with eigenvalues in the range $[-1, 1]$. Our empirical results confirm that extending the eigenvalue range of models like Mamba and DeltaNet to include negative values not only enables them to solve parity but consistently improves their performance on state-tracking tasks. Furthermore, pre-training LRNNs with an extended eigenvalue range for language modeling achieves comparable performance and stability while showing promise on code and math data. Our work enhances the expressivity of modern LRNNs, broadening their applicability without changing the cost of training or inference."},"_bibtex":{"value":"@inproceedings{\ngrazzi2024unlocking,\ntitle={Unlocking State-Tracking in Linear {RNN}s Through Negative Eigenvalues},\nauthor={Riccardo Grazzi and Julien Siems and J{\\\"o}rg K.H. Franke and Arber Zela and Frank Hutter and Massimiliano Pontil},\nbooktitle={NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=EXGAodFkQX}\n}"},"title":{"value":"Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues"},"authors":{"value":["Riccardo Grazzi","Julien Siems","Jörg K.H. Franke","Arber Zela","Frank Hutter","Massimiliano Pontil"]}},"tmdate":1740735283294,"pdate":1728604917889,"tcdate":1727786676694,"writers":["NeurIPS.cc/2024/Workshop/M3L","NeurIPS.cc/2024/Workshop/M3L/Submission54/Authors"],"signatures":["NeurIPS.cc/2024/Workshop/M3L/Submission54/Authors"],"forum":"EXGAodFkQX","license":"CC BY 4.0","number":54,"cdate":1727786676694,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Workshop/M3L/-/Submission","NeurIPS.cc/2024/Workshop/M3L/-/Post_Submission","NeurIPS.cc/2024/Workshop/M3L/-/Edit","NeurIPS.cc/2024/Workshop/M3L/Submission54/-/Revision"],"mdate":1740735283294,"odate":1731228599767,"domain":"NeurIPS.cc/2024/Workshop/M3L","id":"EXGAodFkQX","version":2},{"content":{"summary":{"value":"The paper presents a Mamba-based U-shape model for 3D volumetric medical image segmentation. By comparison with TransformerSRA-based U-shape models, Mamba-based model can achieve a better performance. The authors replace the original 1D depthwise convolution by multiple multi-scale depthwise convolutions. Meanwhile, the authors present the Tri-scan outperforms the exiting scan methods for SSM. This paper is easy to follow."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"see the weakness"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The authors compare Mamba-based U-shape model with a simpler Transformer-based model. \n2. The authors design a multi-scale depthwise convolution before SSM. \n3. The authors present a Tri-scan design for 3D medical images. \n4. This paper is easy to follow."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. To demonstrate the capabilities of modeling long-term dependencies, the authors develop a baseline Transformer-based model utilizing a simpler spatial-reduction attention (SRA) layer. While the vanilla Transformer causes the out-of-memory issue (Please refer to [1] for details how it addresses memory issues.), it does not convincingly demonstrate whether Mamba can achieve superior performance. \n2. Even if the performance of Mamba-based models outperform Transformer-based models in metrics, it remains unclear whether Mamba effectively models long-term dependencies. The authors could strengthen their claim by presenting activation maps or receptive fields. In VMamba [3], Figure 5 provides a feasible activation map for individual scanning path of Mamba. The reason may be a strong inductive bias on medical image segmentation tasks. [2] indicates Transformer-based models can not achieve better performance comparied with CNN-based models.\n3. Although the authors replace the original 1D DWConv with a multi-scale 3D DWConv and design a Tri-scan, the authors do not clarify why we need Mamba for medical images. Is it necessary to use Mamba for medical images. The authors could provide how SSM helps modeling 3D medical images, such as activation maps. \n4. For AMOS22,  it is advisable to evaluate your models using the testing dataset on the official challenge website. Can the authors provide the results for the testing dataset?\n5. Lack of comparisons with existing Mamba-based methods, such as UMamba, SegMamba, and VM-UNet. \n6. While the author provide Figure 4 about the comparison with existing SOTA methods, it is better if the author can provide a quantitative table to compare with exsiting SOTA methods. Meanwhile, can the authors provide more recent methods, such as nnUNet_ResEnc [2], nnFormer [8], 3D UX-Net [7] and MedNeXt [9].\n\n\n[1] Zhou, Hong-Yu, et al. \"nnformer: Interleaved transformer for volumetric segmentation.\" arXiv preprint arXiv:2109.03201 (2021).\n\n[2] Isensee, Fabian, et al. \"nnu-net revisited: A call for rigorous validation in 3d medical image segmentation.\" International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2024.\n\n[3] Liu, Yue, et al. \"Vmamba: Visual state space model.\" arXiv preprint arXiv:2401.10166 (2024)\n\n[4] Ma, Jun, Feifei Li, and Bo Wang. \"U-mamba: Enhancing long-range dependency for biomedical image segmentation.\" arXiv preprint arXiv:2401.04722 (2024).\n\n[5] Xing, Zhaohu, et al. \"Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation.\" International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2024.\n\n[6] Ruan, Jiacheng, and Suncheng Xiang. \"Vm-unet: Vision mamba unet for medical image segmentation.\" arXiv preprint arXiv:2402.02491 (2024).\n\n[7] Lee, Ho Hin, et al. \"3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation.\" arXiv preprint arXiv:2209.15076 (2022).\n\n[8] Zhou, Hong-Yu, et al. \"nnformer: Interleaved transformer for volumetric segmentation.\" arXiv preprint arXiv:2109.03201 (2021).\n\n[9] Roy, Saikat, et al. \"Mednext: transformer-driven scaling of convnets for medical image segmentation.\" International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023."}},"nonreaders":[],"tmdate":1731428021908,"tcdate":1730498750696,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4048/Reviewer_Njbp"],"signatures":["ICLR.cc/2025/Conference/Submission4048/Reviewer_Njbp"],"forum":"m3cKeqvC7z","number":1,"license":"CC BY 4.0","cdate":1730498750696,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4048/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428021908,"domain":"ICLR.cc/2025/Conference","replyto":"m3cKeqvC7z","id":"6lL78mwPtP","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["3D Volumetric Medical Image Segmentation","Mamba","Transformer"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba, with its State Space Model (SSM), offers a more computationally efficient solution than Transformers for long-range dependency modeling. However, there is still a debate about its effectiveness in \\textit{high-resolution 3D medical image} segmentation. \nIn this study, we present a comprehensive investigation into Mamba's capabilities in 3D medical image segmentation by tackling three pivotal questions: Can Mamba replace Transformers? Can it elevate multi-scale representation learning? Is complex scanning necessary to unlock its full potential? We evaluate Mamba’s performance across three large public benchmarks—AMOS, TotalSegmentator, and BraTS. Our findings reveal that UlikeMamba, a U-shape Mamba-based network, consistently surpasses UlikeTrans, a U-shape Transformer-based network, particularly when enhanced with custom-designed 3D depthwise convolutions, boosting accuracy and computational efficiency. Further, our proposed multi-scale Mamba block demonstrates superior performance in capturing both fine-grained details and global context, especially in complex segmentation tasks, surpassing Transformer-based counterparts. We also critically assess complex scanning strategies, finding that simpler methods often suffice, while our Tri-scan approach delivers notable advantages in the most challenging scenarios. By integrating these advancements, we introduce a new network for 3D medical image segmentation, positioning Mamba as a transformative force that outperforms leading models such as nnUNet, CoTr, and U-Mamba, offering competitive accuracy with superior computational efficiency. This study provides key insights into Mamba's unique advantages, paving the way for more efficient and accurate approaches to 3D medical imaging. All code used in the experiments will be made publicly available."},"_bibtex":{"value":"@misc{\nwang2025what,\ntitle={What can Mamba do for 3D Volumetric Medical Image Segmentation?},\nauthor={Chaohan Wang and Qi Chen and Yuyin Zhou and Qi Wu and Yutong Xie},\nyear={2025},\nurl={https://openreview.net/forum?id=m3cKeqvC7z}\n}"},"title":{"value":"What can Mamba do for 3D Volumetric Medical Image Segmentation?"},"pdf":{"value":"/pdf/55e8279499c3fd9aa3b9b4feaef96011a3755337.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"wang|what_can_mamba_do_for_3d_volumetric_medical_image_segmentation"},"authorids":{"value":["~Chaohan_Wang1","~Qi_Chen4","~Yuyin_Zhou1","~Qi_Wu3","~Yutong_Xie4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chaohan Wang","Qi Chen","Yuyin Zhou","Qi Wu","Yutong Xie"]}},"version":2},{"content":{"summary":{"value":"This paper investigates memory in deep state-space models. However, the analysis is limited to linear state-space models, which presents a significant gap compared to models commonly used in practice. This opens up a potential research direction to explore whether the phenomena observed in this paper are specific to linear models or hold more generally for nonlinear cases."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. It is difficult to discern the key contributions of this paper in relation to the existing literature, such as Recurrent Neural Networks: Vanishing and Exploding Gradients Are Not the End of the Story (https://arxiv.org/abs/2405.21064). There are a lot of similarity in the key message. It is important to clearly demonstrate how your work advances the recent literature, as the key contributions of this paper are currently hard to identify. \n2. Another issue is the significant duplication of content in the appendix for Lemma 1 (from the arxiv paper Recurrent Neural Networks: Vanishing and Exploding Gradients Are Not the End of the Story). If the proof has been adapted from previous work, it may be more appropriate to cite the original source instead.\n3. In appendix Figure 5, the current argument is hard to show adding depth allows a deep SSMs to capture long-term dependencies with smaller eigenvalues across layers. Maybe there should be a bit more discussion."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"This paper provides a detailed analysis of deep state-space models, using the theory of linear state-space models to demonstrate the performance of recurrent models. It includes a thorough review of related work, and the topic of identifying the advantages of state-space models in learning long-term memory is indeed important. **However, I would expect the paper to engage more deeply with prior research rather than focusing primarily on synthetic examples, which may not offer much inspiration for theoretical advancements or practical applications**."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Lack of Reference to Relevant Literature**: The paper’s calculation of the autocorrelation function of linear state-space models should reference time series literature (e.g., ARMA models). It doesn’t add new insight and lacks generalization to nonlinear models, leaving questions on how the results extend.\n2. **Limited Scale of Phenomenon Studied**: The scope of the paper is too narrow, focusing on a small-scale phenomenon, making comparisons with larger, well-established models (like S5 and Mamba) less meaningful.\n3. **Insufficient Depth in Experiments (Figure 4)**: The experiments only use 1-3 layers of LRU, which is far from practical applications. More layers (up to 12) should be tested to understand the true behavior of the model as depth increases. Given the layers used in Figure 2, it is believed that extending the layers up to 12 is feasible within the available computational budget.\n4. **Unclear Key Message for Different Audiences**: The paper doesn’t clearly separate the key takeaways for theoretical researchers and practitioners, which weakens its impact for both groups."}},"nonreaders":[],"tmdate":1731428330700,"tcdate":1729151218598,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13020/Reviewer_m1Wn"],"signatures":["ICLR.cc/2025/Conference/Submission13020/Reviewer_m1Wn"],"forum":"hgjpO0H0id","number":1,"license":"CC BY 4.0","cdate":1729151218598,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13020/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428330700,"domain":"ICLR.cc/2025/Conference","replyto":"hgjpO0H0id","id":"WGEZBtHfaT","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["state space models","long-term dependencies","sequence modeling","linear time-invariant systems","theory","memory"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Deep state-space models (SSMs) have emerged as a powerful deep learning architecture for sequence modeling, but the theory of how these models learn long-term dependencies lags the practice. To explain how parameterization and the number of layers affect a model's expressiveness, we study the properties of deep $\\textit{linear}$ SSMs, i.e., linearly coupled stacks of linear time-invariant systems. We show that such systems share timescales across layers, and we provide novel analysis on the role of linear feedforward connections in regularizing these temporal dependencies. In practice, SSMs can struggle with an explosion of the hidden state variance when learning long-term dependencies. We expand our theoretical understanding of this problem for deep SSMs and provide new intuitions on how this problem may be resolved by increasing the number of layers. Finally, we confirm our theoretical results in a teacher-student framework and show the effects of model parameterization on learning convergence."},"_bibtex":{"value":"@misc{\nsmekal2025on,\ntitle={On the interplay between learning and memory in deep state space models},\nauthor={Jakub Smekal and Nicolas Zucchet and Dan Biderman and E. Kelly Buchanan and Jimmy T.H. Smith and Scott Linderman},\nyear={2025},\nurl={https://openreview.net/forum?id=hgjpO0H0id}\n}"},"title":{"value":"On the interplay between learning and memory in deep state space models"},"pdf":{"value":"/pdf/30bec6d5f6ada03d5951b57893c0b74592bfb6ea.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"smekal|on_the_interplay_between_learning_and_memory_in_deep_state_space_models"},"authorids":{"value":["~Jakub_Smekal1","~Nicolas_Zucchet1","~Dan_Biderman1","~E._Kelly_Buchanan1","~Jimmy_T.H._Smith1","~Scott_Linderman1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jakub Smekal","Nicolas Zucchet","Dan Biderman","E. Kelly Buchanan","Jimmy T.H. Smith","Scott Linderman"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2401.13660v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"wang|mambabyte_tokenfree_selective_state_space_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Junxiong_Wang:","~Tushaar_Gangavarapu1","https://dblp.org/search/pid/api?q=author:Jing_Nathan_Yan:","~Alexander_M._Rush1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2401.13660"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2401-13660,\n  publtype={informal},\n  author={Junxiong Wang and Tushaar Gangavarapu and Jing Nathan Yan and Alexander M. Rush},\n  title={MambaByte: Token-free Selective State Space Model},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2401.13660},\n  url={https://doi.org/10.48550/arXiv.2401.13660}\n}\n"},"abstract":{"value":"Token-free language models learn directly from raw bytes and remove the inductive bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences. In this setting, standard autoregressive Transformers scale poorly as the effective memory required grows with sequence length. The recent development of the Mamba state space model (SSM) offers an appealing alternative approach with a fixed-sized memory state and efficient decoding. We propose MambaByte, a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences. In terms of modeling, we show MambaByte to be competitive with, and even to outperform, state-of-the-art subword Transformers on language modeling tasks while maintaining the benefits of token-free language models, such as robustness to noise. In terms of efficiency, we develop an adaptation of speculative decoding with tokenized drafting and byte-level verification. This results in a $2.6\\times$ inference speedup to the standard MambaByte implementation, showing similar decoding efficiency as the subword Mamba. These findings establish the viability of SSMs in enabling token-free language modeling."},"title":{"value":"MambaByte: Token-free Selective State Space Model"},"authors":{"value":["Junxiong Wang","Tushaar Gangavarapu","Jing Nathan Yan","Alexander M. Rush"]}},"tmdate":1769967439156,"pdate":1704067200000,"tcdate":1721747245674,"writers":["~"],"signatures":["~Tushaar_Gangavarapu1"],"forum":"civGsIcVex","license":"CC BY-SA 4.0","number":45945,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769967439156,"domain":"DBLP.org","id":"civGsIcVex","version":2},{"content":{"TLDR":{"value":"A token-free selective state space model."},"venue":{"value":"COLM"},"code_of_ethics":{"value":["I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"]},"keywords":{"value":"byte-level language model, state space model"},"abstract":{"value":"Token-free language models learn directly from raw bytes and remove the inductive bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences. In this setting, standard autoregressive Transformers scale poorly as the effective memory required grows with sequence length. The recent development of the Mamba state space model (SSM) offers an appealing alternative approach with a fixed-sized memory state and efficient decoding. We propose MambaByte, a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences. In terms of modeling, we show MambaByte to be competitive with, and even to outperform, state-of-the-art subword Transformers on language modeling tasks while maintaining the benefits of token-free language models, such as robustness to noise. In terms of efficiency, we develop an adaptation of speculative decoding with tokenized drafting and byte-level verification. This results in a $2.6\\times$ inference speedup to the standard MambaByte implementation, showing similar decoding efficiency as the subword Mamba. These findings establish the viability of SSMs in enabling token-free language modeling."},"_bibtex":{"value":"@inproceedings{\nwang2024mambabyte,\ntitle={MambaByte: Token-free Selective State Space Model},\nauthor={Junxiong Wang and Tushaar Gangavarapu and Jing Nathan Yan and Alexander M Rush},\nbooktitle={First Conference on Language Modeling},\nyear={2024},\nurl={https://openreview.net/forum?id=X1xNsuKssb}\n}"},"title":{"value":"MambaByte: Token-free Selective State Space Model"},"research_area":{"value":["Compute efficient LMs","Learning algorithms for LMs","LMs on diverse modalities and novel applications"]},"pdf":{"value":"/pdf/4e0d96a750783ca5755105830b0b48af3fbca06c.pdf"},"venueid":{"value":"colmweb.org/COLM/2024/Conference"},"paperhash":{"value":"wang|mambabyte_tokenfree_selective_state_space_model"},"authorids":{"value":["~Junxiong_Wang1","~Tushaar_Gangavarapu1","~Jing_Nathan_Yan1","~Alexander_M_Rush1"]},"author_guide":{"value":["I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"]},"authors":{"value":["Junxiong Wang","Tushaar Gangavarapu","Jing Nathan Yan","Alexander M Rush"]}},"tmdate":1724633533836,"pdate":1720608913082,"tcdate":1711078917282,"writers":["colmweb.org/COLM/2024/Conference","colmweb.org/COLM/2024/Conference/Submission572/Authors"],"signatures":["colmweb.org/COLM/2024/Conference/Submission572/Authors"],"forum":"X1xNsuKssb","license":"CC BY 4.0","number":572,"cdate":1711078917282,"readers":["everyone"],"invitations":["colmweb.org/COLM/2024/Conference/-/Submission","colmweb.org/COLM/2024/Conference/-/Post_Submission","colmweb.org/COLM/2024/Conference/Submission572/-/Revision","colmweb.org/COLM/2024/Conference/-/Edit","colmweb.org/COLM/2024/Conference/Submission572/-/Camera_Ready"],"mdate":1724633533836,"odate":1724633533820,"domain":"colmweb.org/COLM/2024/Conference","id":"X1xNsuKssb","version":2},{"content":{"summary":{"value":"This paper introduces Linear Oscillatory State-Space models (LinOSS), a novel approach to sequence modeling based on forced harmonic oscillators. The model comes in two variants: LinOSS-IM (implicit) and LinOSS-IMEX (implicit-explicit). The key innovation is using second-order ODEs with diagonal state matrices, offering stable dynamics while only requiring non-negative diagonal elements. The paper provides theoretical guarantees for stability and universal approximation, along with empirical validation showing significant improvements over state-of-the-art models like Mamba and LRU on long sequences."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Could the model be extended to incorporate coupled oscillations while maintaining stability guarantees?\n\n2. What is the impact of the time step parameter Δt on model performance and stability?\n\n3. How does the choice between IM and IMEX variants affect training dynamics and convergence?"},"rating":{"value":8},"details_of_ethics_concerns":{"value":"NA"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The paper demonstrates strong theoretical foundations by providing rigorous mathematical analysis of stability conditions, proving universal approximation capability, and establishing clear connections to Hamiltonian systems and symplectic integration. Implementation-wise, it offers a remarkably simple parameterization requiring only non-negative diagonal elements, achieves efficient computation through parallel scans, and presents two complementary variants with different preservation properties. The empirical results are great, showing strong performance on diverse tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Limited Analysis of Model Interpretability:\n\nWhile based on oscillatory dynamics, lacks discussion of learned frequencies\n\nNo analysis of how the model captures different timescales\n\n2. Experiment:\n\nNo ablation studies on the impact of different initialization schemes\n\nThe implementation details are unclear. For instance, how does it compare to Mamba or S5 in terms of speed, training time, FLOPs, and memory usage? Discussing these aspects could enhance its practical utility."}},"nonreaders":[],"tmdate":1732691059838,"tcdate":1730694619461,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10203/Reviewer_Wn4t"],"signatures":["ICLR.cc/2025/Conference/Submission10203/Reviewer_Wn4t"],"forum":"GRMfXcAAFh","number":2,"license":"CC BY 4.0","cdate":1730694619461,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10203/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732691059838,"domain":"ICLR.cc/2025/Conference","replyto":"GRMfXcAAFh","id":"J0S9oYEz5i","forumContent":{"venue":{"value":"ICLR 2025 Oral"},"TLDR":{"value":"Oscillatory state-space models are provably able to learn long-range interactions of arbitrary length, are universal, and achieve state-of-the-art performance in practice."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["state-space models","sequence models","oscillators","long-range interactions","time-series"]},"supplementary_material":{"value":"/attachment/3164107c21760e5affe8c52a94d5b512c586b277.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learning on long sequences. Inspired by cortical dynamics of biological neural networks, we base our proposed LinOSS model on a system of forced harmonic oscillators. A stable discretization, integrated over time using fast associative parallel scans, yields the proposed state-space model. We prove that LinOSS produces stable dynamics only requiring nonnegative diagonal state matrix. This is in stark contrast to many previous state-space models relying heavily on restrictive parameterizations. Moreover, we rigorously show that LinOSS is universal, i.e., it can approximate any continuous and causal operator mapping between time-varying functions, to desired accuracy. In addition, we show that an implicit-explicit discretization of LinOSS perfectly conserves the symmetry of time reversibility of the underlying dynamics. Together, these properties enable efficient modeling of long-range interactions, while ensuring stable and accurate long-horizon forecasting. Finally, our empirical results, spanning a wide range of time-series tasks from mid-range to very long-range classification and regression, as well as long-horizon forecasting, demonstrate that our proposed LinOSS model consistently outperforms state-of-the-art sequence models. Notably, LinOSS outperforms Mamba and LRU by nearly 2x on a sequence modeling task with sequences of length 50k."},"_bibtex":{"value":"@inproceedings{\nrusch2025oscillatory,\ntitle={Oscillatory State-Space Models},\nauthor={T. Konstantin Rusch and Daniela Rus},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=GRMfXcAAFh}\n}"},"title":{"value":"Oscillatory State-Space Models"},"pdf":{"value":"/pdf/bf6dc1854b6692b28f47ffd7cc59efd8001dffad.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"rusch|oscillatory_statespace_models"},"authorids":{"value":["~T._Konstantin_Rusch1","~Daniela_Rus1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["T. Konstantin Rusch","Daniela Rus"]}},"version":2},{"content":{"summary":{"value":"This paper addresses the challenge of establishing an interpretable and explicit mapping for multivariate time series forecasting. It proposes a channel-time dual unmixing network built upon the Mamba framework, which decomposes the time series into bases and coefficients across both temporal and channel dimensions. Extensive experiments on seven benchmark datasets demonstrate that the proposed model consistently outperforms ten state-of-the-art baselines."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- In line 105, 'Ummixing' appears to be a typo."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The paper presents a novel dual-mixing mechanism that effectively mitigates noise and redundancy in multivariate time series data.\n- A component-sharing strategy is adopted to enhance the model's conciseness and simplicity.\n- Comprehensive experimental evaluations demonstrate the superior performance of the proposed method."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- It remains unclear what specific advantages the proposed method provides in contrast to decomposition-based approaches (e.g., Autoformer, FEDformer, DLinear) and state-space models (e.g., Mamba). This weakens the overall novelty of the paper. The author should provide a detailed comparative analysis in the Introduction section between the proposed model and these representative methods to better highlight the unique contributions of the proposed method.\n- Several key arguments in this paper require further clarification. For instance, the rationale behind performing decomposition along the channel dimension is unclear. There are multiple existing strategies for modeling channel correlations, including the channel-independent strategy [1], the channel-dependent strategy [2], and several trade-off strategies [3, 4]. The authors should include a comparative analysis between the proposed channel unmixing strategy and these established channel modeling methods to justify their design choice. In addition, the definition of $S$ in lines 165 and 170 appears inconsistent, and the meaning of $t$ is ambiguous. Moreover, in Eq. 7,  it remains unclear why the authors choose to share the coefficient matrix along the channel dimension while sharing the base matrix along the temporal dimension. A detailed explanation or ablation study should be provided to clarify why this asymmetric sharing strategy is preferred over alternative designs, such as sharing either the coefficient or the base matrix across both dimensions. It is also unclear why the authors chose to employ bidirectional Mamba blocks instead of graph neural networks or Transformer-based architectures for modeling channel dependencies. Besides, the original order in the channel dimension mentioned in line 268 is not well explained and requires further clarification.\n- In line 128, the authors claim that the proposed model enhances physical interpretability and prediction reliability. However, the paper does not provide any visual analysis of the decomposed components along either the temporal or channel dimensions.\n- The authors should include state-space models, such as Mamba, as baseline methods to provide a more comprehensive performance comparison.\n\n[1] Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, \"A time series is worth 64 words: Long-term forecasting with transformers,\" in International Conference on Learning Representations (ICLR), 2023.\n\n[2] Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, and M. Long,  \"itransformer: Inverted transformers are effective for time series forecasting,\" in International Conference on Learning Representations (ICLR), 2024.\n\n[3] L. Han, H.-J. Ye, and D.-C. Zhan,  \"The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting,\" IEEE Transactions on Knowledge and Data Engineering, 2024.\n\n[4] J. Chen, J. E. Lenssen, A. Feng, W. Hu, M. Fey, L. Tassiulas, J. Leskovec, and R. Ying, \"From similarity to superiority: Channel clustering for time series forecasting,\" in Advances in Neural Information Processing Systems, 2024."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917506255,"tcdate":1761986749946,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4673/Reviewer_pueq"],"signatures":["ICLR.cc/2026/Conference/Submission4673/Reviewer_pueq"],"forum":"PiwfjPYEUI","number":3,"license":"CC BY 4.0","cdate":1761986749946,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4673/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917506255,"domain":"ICLR.cc/2026/Conference","replyto":"PiwfjPYEUI","id":"G3RW6QclNv","forumContent":{"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"TLDR":{"value":"MTS-UNMixers disentangles temporal and channel patterns via dual-path unmixing for accurate multivariate time series forecasting."},"keywords":{"value":["Time series forecasting","Unmixing","Mamba network","Shared mechanism."]},"supplementary_material":{"value":"/attachment/91adfca16fea8337a66f57973fd6ba60089f8d3e.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Multivariate time series data provide a robust framework for future predictions by leveraging information across multiple dimensions, ensuring broad applicability in practical scenarios. However, their high dimensionality and mixing patterns pose significant challenges in establishing an interpretable and explicit mapping between historical and future series, as well as extracting long-range feature dependencies. To address these challenges, we propose a channel-time dual unmixing network for multivariate time series forecasting (named MTS-UNMixers), which decomposes the entire series into critical bases and coefficients across both the time and channel dimensions. This approach establishes a robust sharing mechanism between historical and future series, enabling accurate representation and enhancing physical interpretability. Specifically, MTS-UNMixers represent sequences over time as a mixture of multiple trends and cycles, with the time-correlated representation coefficients shared across both historical and future time periods. In contrast, sequence over channels can be decomposed into multiple tick-wise bases, which characterize the channel correlations and are shared across the whole series. To estimate the shared time-dependent coefficients, a vanilla Mamba network is employed, leveraging its alignment with directional causality. Correspondlingly, a bidirectional Mamba network is utilized to model the shared channel-correlated bases, accommodating noncausal relationships. Experimental results show that MTS-UNMixers significantly outperform existing methods on multiple benchmark datasets."},"_bibtex":{"value":"@misc{\nanonymous2026mtsunmixers,\ntitle={{MTS}-{UNM}ixers: Multivariate Time Series Forecasting via Channel-Time Dual Unmixing},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=PiwfjPYEUI}\n}"},"title":{"value":"MTS-UNMixers: Multivariate Time Series Forecasting via Channel-Time Dual Unmixing"},"pdf":{"value":"/pdf/ebdc738547ddda3e7bd664663e54f64296439602.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"zhu|mtsunmixers_multivariate_time_series_forecasting_via_channeltime_dual_unmixing"},"authorids":{"value":["~Xuanbing_Zhu1","~Dunbin_Shen1","~Zhongwen_Rao1","~Huiyi_Ma1","~Yingguang_Hao1","~Hongyu_Wang1"]},"authors":{"value":["Xuanbing Zhu","Dunbin Shen","Zhongwen Rao","Huiyi Ma","Yingguang Hao","Hongyu Wang"]}},"version":2},{"content":{"summary":{"value":"The Open Megagenomic Corpus (OMG) and gLM2 model are important and novel contributions to biologic sequence modeling. Authors make use of the largest repositories of metagenomic data to construct a new database for training that is larger than exiting, popular biologic LM training datasets. Using both coding sequences and intergenic sequences, authors show that their modeling strategy captures long range interactions of coding and non-coding genomic elements at multiple scales of biologic sequence interaction.\n\nThe paper should be accepted given the technical and strategic innovation database creation and the novelty in modeling strategy that performs on par with an existing state-of-the-art model and captures important biology across sequence modalities."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"Authors claim their modeling approach achieves \"more efficient scaling\" performance with flops on a set of benchmark tasks than a prior top performing model; however, they only show two training flops. When looking at Appendix J there are some tasks where scaling is observed and some where it does not. It would be a stronger claim if there were additional data points for gLM2. On a per-task basis, it appears that gLM2 has similar scaling to ESM2 even if at the aggregated score level the scaling looks slightly better (Figure 3).\n\nMinor comments:\n1. \"start a new contig\" should be \"start a new element\" (l.255, 266-267)\n2. DGEB is not defined (l.333)\n3. Figure 5 has two \"n\" that are mentioned but not present in the figure"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":4},"strengths":{"value":"The work addresses the need for larger databases of sequence data that can be used to train language models for biologic applications by making use of rapidly growing repositories of metagenomic sequence data. Authors utilize knowledge and methods from recent insights in NLP to construct the large database used for training and develop their own genomic deduplication procedure that captures biologic reality. They also utilize deep domain knowledge about metagenomic sequencing to filter terabytes of raw data for database construction. The new curated database is made available to the community. The authors utilize a novel modeling approach, constituting of which includes multi-modal input in the form of amino acid and nucleotide data."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The language model trained does not explicitly include eukaryotic genomic sequences and it is unclear gLM2 is meant to be used to model eukaryotic sequences. Authors should be more explicit about the application for eukaryotic sequence modeling. Authors do mention contamination and likely inclusion of eukaryotic sequences in metagenomic samples but this in in the context of challenges and mitigation thereof with metagenomic sequence data.\n\nApplications of the gLM2 model are not explored. It would be interesting to see performance of gLM2 in areas where more and diverse training data is of importance, such as remote homology, or where the multi-modal training strategy is helpful, such as contig-level prediction of taxonomy."}},"nonreaders":[],"tmdate":1731428250424,"tcdate":1730497195188,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10586/Reviewer_Sobx"],"signatures":["ICLR.cc/2025/Conference/Submission10586/Reviewer_Sobx"],"forum":"jlzNb1iWs3","number":2,"license":"CC BY 4.0","cdate":1730497195188,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10586/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428250424,"domain":"ICLR.cc/2025/Conference","replyto":"jlzNb1iWs3","id":"7hCKejKDoH","forumContent":{"TLDR":{"value":"We introduce the OMG dataset and gLM2. OMG is a massive Open MetaGenomic corpus totaling 3.1T base-pairs, and we train gLM2, the first mixed-modality genomic language model which learns zero-shot protein-protein interfaces."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["metagenomics","pretraining dataset","genomic language model"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Biological language model performance depends heavily on pretraining data quality, diversity, and size. While metagenomic datasets feature enormous biological diversity, their utilization as pretraining data has been limited due to challenges in data accessibility, quality filtering and deduplication. Here, we present the Open MetaGenomic (OMG) corpus, a genomic pretraining dataset totalling 3.1T base pairs and 3.3B protein coding sequences, obtained by combining two largest metagenomic dataset repositories (JGI's IMG and EMBL's MGnify). We first document the composition of the dataset and describe the quality filtering steps taken to remove poor quality data. We make the OMG corpus available as a mixed-modality genomic sequence dataset that represents multi-gene encoding genomic sequences with translated amino acids for protein coding sequences, and nucleic acids for intergenic sequences. We train the first mixed-modality genomic language model (gLM2) that leverages genomic context information to learn robust functional representations, as well as coevolutionary signals in protein-protein interfaces and genomic regulatory syntax. Furthermore, we show that deduplication in embedding space can be used to balance the corpus, demonstrating improved performance on downstream tasks. The OMG dataset is publicly hosted on the Hugging Face Hub at https://huggingface.co/datasets/tattabio/OMG and gLM2 is available at https://huggingface.co/tattabio/gLM2_650M."},"_bibtex":{"value":"@inproceedings{\ncornman2025the,\ntitle={The {OMG} dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling},\nauthor={Andre Cornman and Jacob West-Roberts and Antonio Pedro Camargo and Simon Roux and Martin Beracochea and Milot Mirdita and Sergey Ovchinnikov and Yunha Hwang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=jlzNb1iWs3}\n}"},"title":{"value":"The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling"},"pdf":{"value":"/pdf/d8c876824681e1d9200328089d58f388b9a11124.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"cornman|the_omg_dataset_an_open_metagenomic_corpus_for_mixedmodality_genomic_language_modeling"},"authorids":{"value":["~Andre_Cornman2","~Jacob_West-Roberts1","~Antonio_Pedro_Camargo1","~Simon_Roux1","~Martin_Beracochea1","~Milot_Mirdita1","~Sergey_Ovchinnikov2","~Yunha_Hwang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Andre Cornman","Jacob West-Roberts","Antonio Pedro Camargo","Simon Roux","Martin Beracochea","Milot Mirdita","Sergey Ovchinnikov","Yunha Hwang"]}},"version":2},{"content":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Rejected by TMLR"},"abstract":{"value":"Transformer self-attention is powerful, but its quadratic dependence on sequence length limits\nefficiency at long context. Biological nervous systems, by contrast, appear to rely ki on sparse,\nlocal, and hierarchical processing rather than all-to-all pairwise comparison. Motivated by this\ncontrast, we introduce EvoRimNet, a sequence model with two parallel pathways:\n(Inhibitory Rim Attention, a local competitive operator implemented with multi-scale\ncausal depthwise convolutions initialized with Mexican-hat-like profiles, and Content-Addressable Thalamic Relay, a hierarchical sparse memory that supports content-based write and read operations.\nOn WikiText-2 word-level language modeling, EvoRimNet achieves 162.6 test perplexity,\nversus 167.3 for a Modern Transformer (RoPE + SwiGLU + RMSNorm) and 188.9 for GPT, improving over both baselines at matched parameter counts across three random seeds. At sequence length 4096, our implementation is\n14.8 times faster and uses 67 times less peak memory than the GPT baseline in our measurements. Ablations confirm that each biological component; inhibition, thalamic relay and hierarchical routing contributes independently and measurably to performance. These results demonstrate that local competitive inhibition combined with sparse content-addressable relay can match or exceed transformer-based attention on language modeling benchmarks, while scaling linearly in both time and memory."},"_bibtex":{"value":"@article{\nanonymous2026evolutionary,\ntitle={Evolutionary Rim Attention for Linear-Time Sequence Modeling},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2026},\nurl={https://openreview.net/forum?id=IOYlVchL1e},\nnote={Rejected}\n}"},"title":{"value":"Evolutionary Rim Attention for Linear-Time Sequence Modeling"},"pdf":{"value":"/pdf/1612b84a4d6f7c3429a6e008c100a52e3685e388.pdf"},"venueid":{"value":"TMLR/Rejected"},"assigned_action_editor":{"value":"~Surbhi_Goel1"}},"tmdate":1789950433589,"tcdate":1773005641956,"writers":["TMLR","TMLR/Paper7840/Authors"],"signatures":["TMLR/Paper7840/Authors"],"forum":"IOYlVchL1e","license":"CC BY 4.0","number":7840,"cdate":1773005641956,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/Paper7840/-/Revision","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/-/Rejected"],"mdate":1789950433589,"odate":1775655971525,"domain":"TMLR","id":"IOYlVchL1e","version":2},{"content":{"summary":{"value":"This paper theoretically proves that:\n- single-layer Mamba-2 can simulate one step of gradient descent for in-context learning (ICL).\n- hybrid Mamba-2 and linear self-attention(LSA) can simulate stochastic gradient descent for ICL.\n- The error of simulating SGD is well bounded, and the theoretical results are verified by experiments."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. Is the experiments follow the theoretical construction? I think it should be clarified in the paper.\n\n2. This paper theoretically claims that hybrid model simulates SGD for ICL, but the experimental results only compare the squared error between models and (S)GD. Comparing their error is not enough to show that hybrid model actually simulates SGD. I recommend the authors to present more evidence to show that."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This paper presents an interesting theoretical result on ICL for hybrid Mamba-2 and LSA. i.e., simulating SGD for ICL.\n2. This paper provides experimental results to support the theoretical results.\n3. Paper is well structured."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The claimed connection between SSMs and SGD is not particularly novel, and similar ideas have been discussed in previous works.\n- Yang, Songlin, et al. Gated Delta Networks: Improving Mamba2 with Delta Rule. At ICLR 2025.\n- Behrouz, Ali, et al. Atlas: Learning to optimally memorize the context at test time. arXiv preprint arXiv:2505.23735 (2025).\n- Jiang, Jiarui, et al. Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression. At NeurIPS 2025.\n- Li, Yingcong, et al. Gating is weighting: Understanding gated linear attention through in-context learning. At COLM 2025\n\nThus, the **novelty should be better clarified**.\n\n2. The assumption seems too strong: The SGD pattern relies on the construction of the model's weights, and this paper focuses on how these weights work (expresivity), but it does not guarantee that the model can be trained to such weights (trainability). The difficulty of proof is not high; therefore, there is a **lack of technical contribution**.\n\n3. Some experiments are redundant: For example, 10 runs vs. 1000 runs, they share similar meanings. I also recommend that the authors plot the error bars in the experiments.\n\nMinors and Typos:\n\nI suggest that vectors and matrices should be **bolded**."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931365129,"tcdate":1760967203302,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19444/Reviewer_mZkm"],"signatures":["ICLR.cc/2026/Conference/Submission19444/Reviewer_mZkm"],"forum":"lhvdbCiiE6","number":1,"license":"CC BY 4.0","cdate":1760967203302,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19444/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931365129,"domain":"ICLR.cc/2026/Conference","replyto":"lhvdbCiiE6","id":"4YDGagn16k","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mamba-2","Stochastic Gradient Descent","In-context Learning"]},"primary_area":{"value":"interpretability and explainable AI"},"abstract":{"value":"State space models (SSMs) have gained popularity as an alternative to Transformers by mitigating the quadratic computational cost associated with self-attention. However, despite their widespread adoption, the theoretical principles underlying their ability to perform in-context learning (ICL) remain poorly understood. In this work, we theoretically analyze the widely used Mamba-2 model (Dao et al. 2024) and demonstrate that a single-layer Mamba-2 can simulate one step of gradient descent, while a hybrid architecture combining Mamba with a Transformer (Mamba $\\circ$ TF) can perform mini-batch stochastic gradient descent. Our experimental results support these theoretical findings."},"_bibtex":{"value":"@misc{\nwang2026incontext,\ntitle={In-Context Stochastic Gradient Descent with Hybrid Mamba-2 and Linear Self-Attention Model},\nauthor={Zhijie Wang and Shuai Li},\nyear={2026},\nurl={https://openreview.net/forum?id=lhvdbCiiE6}\n}"},"title":{"value":"In-Context Stochastic Gradient Descent with Hybrid Mamba-2 and Linear Self-Attention Model"},"pdf":{"value":"/pdf/b29d3c388b906a2dc7dd59de9cc6bccbba82a3f8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wang|incontext_stochastic_gradient_descent_with_hybrid_mamba2_and_linear_selfattention_model"},"authorids":{"value":["~Zhijie_Wang7","~Shuai_Li3"]},"authors":{"value":["Zhijie Wang","Shuai Li"]}},"version":2},{"content":{"summary":{"value":"The author introduces UlikeMamba_3dMT, a Mamba-based model for 3D medical image segmentation tasks, comparing it with UlikeTransformer, a transformer-based counterpart. UlikeMamba_3dMT incorporates multi-scale depthwise convolutions and a Tri-scan strategy to capture spatial dependencies in 3D data. Through evaluations on AMOS, TotalSegmentator, and BraTS datasets, the authors claim computational efficiency and competitive segmentation accuracy for UlikeMamba_3dMT as a substitute for UlikeTransformer models."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. How would UlikeMamba perform if other commonly used transformer blocks, such as the SwinTransformer block, were used instead of SRA attention? Testing with a popular transformer design could clarify the suitability of Mamba as an alternative.\n\n2. Can the authors provide visual or qualitative results to validate the claim that UlikeMamba effectively captures fine-grained details and larger anatomical structures? A comparison of multi-scale and single-scale configurations could justify the use of the multi-scale approach.\n\n3. Why do author use tri-scan strategy, why not other tri-scan (e.g., right-left, down-up, and back-front) or hexa-scan strategy (e.g., left-right, right-left, up-down, down-up, front-back, and back-front)? A comparative study of these alternatives could help justify the use of Tri-scan in the model."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This paper provides a direct comparison between UlikeMamba and its transformer-based counterpart, UlikeTransformer. This head-to-head evaluation highlights both the strengths and limitations of UlikeMamba.\n2. Mamba’s linear complexity with sequence length potentially offers a promising approach for processing high-resolution 3D images without high computational resources."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The overall novelty of the proposed method is limited. Key components of UlikeMamba, such as multi-scale depthwise convolutions and multi-way scanning, lack originality. The multi-scale depthwise convolutions was previously introduced in EMCAD [1], while multi-way scanning strategies have been explored in VisionMamba [2] and VMamba [3]. While the Tri-scan offers only incremental improvements, the paper does not clearly differentiate how UlikeMamba's method significantly advances these existing strategies. The potential disadvantages of large-kernel (e.g., 3x3 or 5x5, 7x7) depthwise convolutions, such as the loss of inter-channel relationships, are also not addressed.\n\n[1] Rahman, M.M., Munir, M. and Marculescu, R., 2024. Emcad: Efficient multi-scale convolutional attention decoding for medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11769-11779).\n[2] Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W. and Wang, X., 2024. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417.\n[3] Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q. and Liu, Y., 2024. VMamba: Visual State Space Model. arXiv preprint arXiv:2401.10166.\n\n2. Essential baseline comparisons are missing, especially with state-of-the-art models like 3D UX-NET [4], which uses large-kernel depthwise convolutions, and other Mamba-based models such as SegMamba [5] and SwinUMamba [6]. The incorporation of these missing methods in comparison will strengthen UlikeMamba_3dMT's performance in a broader context. Additionally, final evaluations completely missing on TotalSegmentor dataset with UlikeMamba_3dMT, the incorporation of which will strengthen the contribution of the paper.\n\n[4] Lee, H.H., Bao, S., Huo, Y. and Landman, B.A., 2022. 3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation. arXiv preprint arXiv:2209.15076.\n\n[5] Xing, Z., Ye, T., Yang, Y., Liu, G. and Zhu, L., 2024, October. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 578-588). Cham: Springer Nature Switzerland.\n\n[6] Liu, J., Yang, H., Zhou, H.Y., Xi, Y., Yu, L., Li, C., Liang, Y., Shi, G., Yu, Y., Zhang, S. and Zheng, H., 2024, October. Swin-umamba: Mamba-based unet with imagenet-based pretraining. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 615-625). Cham: Springer Nature Switzerland.\n\n3. The paper lacks a theoretical explanation for Mamba’s effectiveness over transformers in 3D segmentation, thus relying solely on experimental results for validation. Without an analytical foundation or insight into why Mamba performs well in 3D segmentation, this work feels similar to existing Mamba-based methods [5,6] that only present experimental validation.\n\n4. Important computational metrics, such as memory usage and latency, are missing from the analysis. Furthermore, the paper is hard to follow due to its organization, with experimental results scattered throughout the methodology. A clearer structure and a table summarizing parameters, FLOPs, latency, and dataset performance would improve readability and transparency.\n\n5. The Tri-scan strategy results in only little improvements in Dice scores, while significantly increasing computational costs. The minimal accuracy gains may not justify the added complexity, particularly without evidence that demonstrates the necessity of these trade-offs in practical scenarios.\n\n6. The multi-scale convolutions contributes minimally to UlikeMamba’s performance, especially on smaller tasks (e.g., BraTS, where MSv4 shows reduced performance). The authors should consider a few other 3D binary segmentation datasets to validate that this is not a common limitation of UlikeMamba in case of smaller tasks.\n\n7. The authors claim that UlikeMamba captures long-range dependencies effectively, yet there is no evidence supporting this beyond the reported segmentation accuracy. This improvement in accuracy may not directly result from enhanced long-range dependency modeling, and further clarification on this aspect would strengthen the paper."}},"nonreaders":[],"tmdate":1731428021254,"tcdate":1730665076116,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4048/Reviewer_e7hy"],"signatures":["ICLR.cc/2025/Conference/Submission4048/Reviewer_e7hy"],"forum":"m3cKeqvC7z","number":3,"license":"CC BY 4.0","cdate":1730665076116,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4048/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428021254,"domain":"ICLR.cc/2025/Conference","replyto":"m3cKeqvC7z","id":"Llm6femsYh","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["3D Volumetric Medical Image Segmentation","Mamba","Transformer"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba, with its State Space Model (SSM), offers a more computationally efficient solution than Transformers for long-range dependency modeling. However, there is still a debate about its effectiveness in \\textit{high-resolution 3D medical image} segmentation. \nIn this study, we present a comprehensive investigation into Mamba's capabilities in 3D medical image segmentation by tackling three pivotal questions: Can Mamba replace Transformers? Can it elevate multi-scale representation learning? Is complex scanning necessary to unlock its full potential? We evaluate Mamba’s performance across three large public benchmarks—AMOS, TotalSegmentator, and BraTS. Our findings reveal that UlikeMamba, a U-shape Mamba-based network, consistently surpasses UlikeTrans, a U-shape Transformer-based network, particularly when enhanced with custom-designed 3D depthwise convolutions, boosting accuracy and computational efficiency. Further, our proposed multi-scale Mamba block demonstrates superior performance in capturing both fine-grained details and global context, especially in complex segmentation tasks, surpassing Transformer-based counterparts. We also critically assess complex scanning strategies, finding that simpler methods often suffice, while our Tri-scan approach delivers notable advantages in the most challenging scenarios. By integrating these advancements, we introduce a new network for 3D medical image segmentation, positioning Mamba as a transformative force that outperforms leading models such as nnUNet, CoTr, and U-Mamba, offering competitive accuracy with superior computational efficiency. This study provides key insights into Mamba's unique advantages, paving the way for more efficient and accurate approaches to 3D medical imaging. All code used in the experiments will be made publicly available."},"_bibtex":{"value":"@misc{\nwang2025what,\ntitle={What can Mamba do for 3D Volumetric Medical Image Segmentation?},\nauthor={Chaohan Wang and Qi Chen and Yuyin Zhou and Qi Wu and Yutong Xie},\nyear={2025},\nurl={https://openreview.net/forum?id=m3cKeqvC7z}\n}"},"title":{"value":"What can Mamba do for 3D Volumetric Medical Image Segmentation?"},"pdf":{"value":"/pdf/55e8279499c3fd9aa3b9b4feaef96011a3755337.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"wang|what_can_mamba_do_for_3d_volumetric_medical_image_segmentation"},"authorids":{"value":["~Chaohan_Wang1","~Qi_Chen4","~Yuyin_Zhou1","~Qi_Wu3","~Yutong_Xie4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chaohan Wang","Qi Chen","Yuyin Zhou","Qi Wu","Yutong Xie"]}},"version":2},{"content":{"summary":{"value":"This paper proposes 3DET-Mamba, the first attempt to exploit the State Space Model for end-to-end 3D object detection. It introduces a local-to-global scanning technique, including an Inner Mamba block to capture local geometry and Dual Mamba blocks to extract scene features in a global view. Furthermore, it proposes a query-aware Mamba block to effectively decode scene context information into object sets with the guidance of box queries. Experiments show promising results compared to 3DETR and validate the effectiveness of the proposed modules with detailed ablation studies."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Sec. 4.2 title: Obejct -> Object\nline 241: Transforme -> Transformer"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- The basic idea is easy to follow, and the key challenges are clearly presented.\n- The methodology and implementation details are also clear, with good illustrations and mathematical / algorithm presentations.\n- The proposed framework is the first to successfully adopt Mamba in end-to-end 3D object detection.\n- The experimental results and ablation studies are solid and can support the method convincingly."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- One of the motivations is the computational costs of transformers compared to Mamba, but there are no analyses regarding memory costs, latency, etc. I am curious about more advantages of the Mamba framework in this task, such as computational efficiency and scaling up performance, except for the current traditional benchmarks.\n- Although the paper proposes several methods to deal with the unordered point clouds in the ordered manner that is inherently designed in the State Space Model, even though the performance also seems good, the mechanism is still not most suitable for this task or handling point clouds. I would expect a more matching mechanism to deal with point clouds with the fundamental idea of the State Space Model instead of converting the problem with a non-perfect workaround.\n- There are several typos and I just provide a few examples in Questions. The paper needs more check on such details."},"limitations":{"value":"None."}},"nonreaders":[],"tmdate":1730878625591,"tcdate":1720680384773,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission112/Reviewer_eAYh"],"signatures":["NeurIPS.cc/2024/Conference/Submission112/Reviewer_eAYh"],"forum":"iOleSlC80F","number":1,"license":"CC BY 4.0","cdate":1720680384773,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission112/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878625591,"domain":"NeurIPS.cc/2024/Conference","replyto":"iOleSlC80F","id":"1Y88xMnwxF","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["3D Object Detection","Mamba"]},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Transformer-based architectures have been proven successful in detecting 3D objects from point clouds. However, the quadratic complexity of the attention mechanism struggles to encode rich information as point cloud resolution increases. Recently, state space models (SSM) such as Mamba have gained great attention due to their linear complexity and long sequence modeling ability for language understanding. To exploit the potential of Mamba on 3D scene-level perception, for the first time, we propose 3DET-Mamba, which is a novel SSM-based model designed for indoor 3d object detection. Specifically, we divide the point cloud into different patches and use a lightweight yet effective Inner Mamba to capture local geometric information. To observe the scene from a global perspective, we introduce a novel Dual Mamba module that models the point cloud in terms of spatial distribution and continuity. Additionally, we design a Query-aware Mamba module that decodes context features into object sets under the guidance of learnable queries. Extensive experiments demonstrate that 3DET-Mamba surpasses previous 3DETR on indoor 3D detection benchmarks such as ScanNet, improving AP25/AP50 from 65.0\\%/47.0\\% to 70.4\\%/54.4\\%, respectively."},"_bibtex":{"value":"@inproceedings{\nli2024detmamba,\ntitle={3{DET}-Mamba: Causal Sequence Modelling for End-to-End 3D Object Detection},\nauthor={Mingsheng Li and Jiakang Yuan and Sijin Chen and Lin Zhang and Anyu Zhu and Xin Chen and Tao Chen},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=iOleSlC80F}\n}"},"title":{"value":"3DET-Mamba: Causal Sequence Modelling for End-to-End 3D Object Detection"},"pdf":{"value":"/pdf/e562d505611831135151fdb8395d9283e0291da6.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"li|3detmamba_causal_sequence_modelling_for_endtoend_3d_object_detection"},"authorids":{"value":["~Mingsheng_Li2","~Jiakang_Yuan1","~Sijin_Chen1","~Lin_Zhang15","~Anyu_Zhu1","~Xin_Chen16","~Tao_Chen6"]},"authors":{"value":["Mingsheng Li","Jiakang Yuan","Sijin Chen","Lin Zhang","Anyu Zhu","Xin Chen","Tao Chen"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Numerion, which achieves multi-frequency decomposition and prediction by mapping time series to multiple hypercomplex spaces. The core innovation lies in generalizing linear layers and Tanh activation functions to hypercomplex spaces of arbitrary power-of-two dimensions and designing the RHR-MLP architecture. The paper also provides extensive ablation experiments, visualizations, and frequency analyses to validate the effectiveness of the Numerion model."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please refer to Weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The authors' idea of achieving decomposition by mapping time series data to hypercomplex spaces of different dimensions is very inspiring.\n2. The authors provide a mathematical formulation of the Numerion framework based on the Cayley-Dickson algebra system. They use the Cayley-Dickson construction and random matrix theory principles of spectral bias as the theoretical foundation of the model. This is very novel and interesting.\n3. The paper provides extensive comparative and ablation experiments, visualizations, and frequency-domain analyses to support their claims. The experimental design is relatively complete, and the results are clear."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. There are multiple instances of missing spaces throughout the manuscript, such as after model names.\n2. The mathematical formula formatting needs revision, as punctuation marks (commas or periods) are missing at the end of equations.\n3. While the authors claim that experimental results are averaged over multiple runs, standard deviations are not reported.\n4. The authors only analyze the training time of the model but lack analysis of time complexity.\n5. It is suggested that the authors move critical theoretical analyses to the main text to improve readability.\n6. The authors acknowledge that the hypercomplex linear layers do not satisfy the hypercomplex differentiability conditions, and the gradients are not strictly hypercomplex. Does this imply that the backpropagation process is actually performed in the real domain? What are the implications of this approach for model optimization and convergence?\n7. The authors acknowledge that hypercomplex networks consume significantly more memory than real-valued layers. Have the authors considered approximation methods (e.g., low-rank decomposition, sparsification) to reduce computational costs?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925587582,"tcdate":1761961617755,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15291/Reviewer_auNm"],"signatures":["ICLR.cc/2026/Conference/Submission15291/Reviewer_auNm"],"forum":"2hCwLqx5gm","number":3,"license":"CC BY 4.0","cdate":1761961617755,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15291/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925587582,"domain":"ICLR.cc/2026/Conference","replyto":"2hCwLqx5gm","id":"BzmIZmwccV","forumContent":{"TLDR":{"value":"We propose Numerion, a hypercomplex space-based model, decomposes and forecasts time series using multi-dimensional RHR-MLPs, achieving state-of-the-art results."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Time Series Forecasting","Hypercomplex Numbers","Hypercomplex Time Series Models","Multi-Hypercomplex Space"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Many methods aim to enhance time series forecasting by decomposing the series through intricate model structures and prior knowledge, yet they are inevitably limited by computational complexity and the robustness of the assumptions. Our research uncovers that in the complex domain and higher-order hypercomplex spaces, the characteristic frequencies of time series naturally decrease. Leveraging this insight, we propose Numerion, a time series forecasting model based on multiple hypercomplex spaces. Specifically, grounded in theoretical support, we generalize linear layers and activation functions to hypercomplex spaces of arbitrary power-of-two dimensions and introduce a novel Real-Hypercomplex-Real Domain Multi-Layer Perceptron (RHR-MLP) architecture. Numerion utilizes multiple RHR-MLPs to map time series into hypercomplex spaces of varying dimensions, naturally decomposing and independently modeling the series, and adaptively fuses the latent patterns exhibited in different spaces through a dynamic fusion mechanism. Experiments validate the model’s performance, achieving state-of-the-art results on multiple public datasets. Visualizations and quantitative analyses comprehensively demonstrate the ability of multi-dimensional RHR-MLPs to naturally decompose time series and reveal the tendency of higher-dimensional hypercomplex spaces to capture lower-frequency features."},"_bibtex":{"value":"@inproceedings{\ncao2026numerion,\ntitle={Numerion: A Multi-Hypercomplex Model for Time Series Forecasting},\nauthor={Hanzhong Cao and WenBo Yan and Ying Tan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=2hCwLqx5gm}\n}"},"title":{"value":"Numerion: A Multi-Hypercomplex Model for Time Series Forecasting"},"pdf":{"value":"/pdf/0b40c1391b1d3dadb27f73e92392b8e4eb6c195b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"cao|numerion_a_multihypercomplex_model_for_time_series_forecasting"},"authorids":{"value":["~Hanzhong_Cao1","~WenBo_Yan1","~Ying_Tan1"]},"authors":{"value":["Hanzhong Cao","WenBo Yan","Ying Tan"]}},"version":2},{"content":{"summary":{"value":"The authors propose a zero-shot text-to-speech (TTS) system, called L3Speech, aming for real-time speech synthesis from continuous text chunks. Specifically, they 1) adopt a Mamba-based autoregressive decoder for linear-time generation, integrating cross-attention mechanisms utilize enrollment speech and transcript; 2) employ rotary positional embeddings within the cross-attention mechanisms to handle an infinite text stream by sliding a window; 3) apply semantic guidance to mitigate the misalignment between text and speech. The models are trained on LibriLight dataset and evaluated on the test-clean set of LibriTTS. Experimental results demonstrate that the proposed system achieves performance comparable to state-of-the-art TTS models while providing flexibility to balance latency and content accuracy in streaming scenarios."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- For line 034-035, the citations are not included in parentheses.  \n- For line 190-191, \"The speech encoder is a transformer-based encoder that converts enrollment speech of arbitrary length into a fixed-length sequence of embeddings.\" Could the authors provide more details on how to obtain a fixed-length sequence of embeddings? \n- For line 241-242, should $\\tau_{i+1}$ be $\\tau_{i}+1$? \n- For line 245-246, I can't understand the formulas of $p(t)$ and $f(t)$. Are they correct? \n- For line 268-270, as the author mentioned \"we replace each blank token with the first non-blank tokens to the right of the sequence\", but in the given example, why the last two tokens in the processed sequence are still blank tokens? Shouldn't they be \"c\"? \n- For line 270-272, could the authors provide more details about the upsampling process (from 50 tokens per second to 75 acoustic tokens per second)? \n- In Algorithm 1, for line 310-311, should $p_t^{(g)}$ be the modified (scaled) one? Why is $T_{top-k}$ is not used? Steps in line 306-312 seem not fully consistent with descriptions in line 279-281. \n- For Equaltion 1, the attention formula is not correct. I think the value tensors should be placed out of the softmax function."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- The proposed system accesses past and future text contexts in chunk-wise decoding, which improves the performance of steaming TTS. By incoperating semantic tokens (i.e. graphemes) alongside acoustic tokens, the authors address the problem of misalignment in autoregressive generation. These are validated by the ablation studies.  \n- The writing style is good, and the English is clear and concise. The provided discussions and analysis are insightful."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The submitted maniscript does not include the Section of References, which hampers the reading and reviewing process.\n- While the authors emphasize the advantage of linear-time decoding with the Mamba-based decoder, they don't provide ablation experiments that compare against a counterpart with transformer-based decoder. I would suggest conducting comparisons in terms of inference time or quality-speed trade-offs at different sequence lengths.\n- The authors propose to use rotary positional embeddings in the cross-attention modules where the positional indices are based on arrial time, but don't provide ablation experiments about this. I would suggest comparing to using fixed positional encodings and no positional encodings."}},"nonreaders":[],"tmdate":1731427414078,"tcdate":1730430866433,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1359/Reviewer_QLwQ"],"signatures":["ICLR.cc/2025/Conference/Submission1359/Reviewer_QLwQ"],"forum":"RK3Gj9J5my","number":2,"license":"CC BY 4.0","cdate":1730430866433,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1359/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427414078,"domain":"ICLR.cc/2025/Conference","replyto":"RK3Gj9J5my","id":"z8j89JJiaF","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We introduce a zero-shot text-to-speech model that is capable of generating speech from a continuous text stream, with support for on-device streaming."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["zero-shot text-to-speech","streaming","state-space models"]},"supplementary_material":{"value":"/attachment/eb3ee0f6f9fc423a83e128f51cb138fde2d018cc.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Existing zero-shot text-to-speech (TTS) systems are typically designed to process complete sentences and are constrained by the maximum duration for which they have been trained. However, in many streaming applications, texts arrive continuously in short chunks, necessitating instant responses from the system. We identify the essential capabilities required for chunk-level streaming and introduce L3Speech, a stream-aware model that supports non-stop speech generation, text-audio stream synchronization, and seamless transitions between short speech chunks. To achieve these, we propose (1) adopting Mamba, a class of sequence modeling distinguished by linear-time decoding, which is augmented by cross-attention mechanisms for conditioning, (2) utilizing rotary positional embeddings in the computation of cross-attention, enabling the model to process a non-stop text stream by sliding a window, and (3) decoding with semantic guidance, a technique that aligns speech with the transcript during inference with minimal overhead. Experimental results demonstrate that our models are competitive with state-of-the-art language model-based zero-shot TTS models, while also providing flexibility to support a wide range of streaming scenarios."},"_bibtex":{"value":"@misc{\ndang2025zeroshot,\ntitle={Zero-Shot Text-to-Speech from Continuous Text Streams},\nauthor={Trung Dang and David Aponte and Dung Tran and Tianyi Chen and Kazuhito Koishida},\nyear={2025},\nurl={https://openreview.net/forum?id=RK3Gj9J5my}\n}"},"title":{"value":"Zero-Shot Text-to-Speech from Continuous Text Streams"},"pdf":{"value":"/pdf/ae6d24286703bb7fbfcd629543b4f9e8d221ff52.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"dang|zeroshot_texttospeech_from_continuous_text_streams"},"authorids":{"value":["~Trung_Dang1","~David_Aponte1","~Dung_Tran2","~Tianyi_Chen3","~Kazuhito_Koishida1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Trung Dang","David Aponte","Dung Tran","Tianyi Chen","Kazuhito Koishida"]}},"version":2},{"content":{"summary":{"value":"This paper introduces HGMN, the first to apply selective state space models (SSSMs) to heterogeneous graph learning. HGMN captures long-range dependencies and adapts SSSMs for heterogeneous data. It outperforms 19 state-of-the-art methods in both accuracy and efficiency."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see Weaknesses."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The paper's description of the model is relatively clear. The language is fluent."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The problem statement is unclear. The introduction does not effectively clarify whether the focus is on learning heterogeneous graph representations or capturing long-range dependencies. I suggest the authors clearly delineate the specific focus and differentiate between these two aspects in the introduction.\n\n2. Lack of sufficient motivation. The paper introduces challenges related to selective state space models (SSSMs) but fails to explain the necessity of integrating SSSMs. I recommend the authors elaborate on the limitations of existing approaches and provide examples where SSSMs would be particularly beneficial to enhance the motivation.\n\n3. The experimental results are not convincing:\n\n(1) The paper mentions two recent baselines—LMSPS (Li et al., 2024a) and Graph-Mamba (Wang et al., 2024)—but LMSPS is not included in the experiments. The authors should explain why LMSPS was omitted and discuss its relevance for comparison.\n\n(2) Graph-Mamba focuses on long-range graph sequence modeling rather than heterogeneous graph representation learning. The authors should justify the appropriateness of using it as a baseline.\n\n(3) The paper claims that HGMN performs well on large heterogeneous graphs, yet the four datasets used are all standard datasets, not large-scale ones. I suggest the authors consider using specific large-scale heterogeneous graph datasets to better support their claims regarding HGMN's performance."}},"nonreaders":[],"tmdate":1731428142199,"tcdate":1729511307393,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5526/Reviewer_Byh9"],"signatures":["ICLR.cc/2025/Conference/Submission5526/Reviewer_Byh9"],"forum":"mn61GWpEiK","number":1,"license":"CC BY 4.0","cdate":1729511307393,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5526/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428142199,"domain":"ICLR.cc/2025/Conference","replyto":"mn61GWpEiK","id":"RBC9gHAYhM","forumContent":{"TLDR":{"value":"a heterogeneous graph mamba network (HGMN) as the first exploration in leveraging the powerful selective state space models (SSSMs) for heterogeneous graph learning."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["graph neural network","state space model"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We propose a heterogeneous graph mamba network (HGMN) as the first exploration in leveraging the selective state space models (SSSMs) for heterogeneous graph learning. Compared with the literature, our HGMN overcomes two major challenges: (i) capturing long-range dependencies among heterogeneous nodes and (ii) adapting SSSMs to heterogeneous graph data. Our key contribution is a general graph architecture that can solve heterogeneous nodes in real-world scenarios, followed an efficient flow. Methodologically, we introduce a two-level efficient tokenization approach that first captures long-range dependencies within identical node types, and subsequently across all node types. Empirically, we conduct comparisons between our framework and 19 state-of-the-art methods on the heterogeneous benchmarks. The extensive comparisons demonstrate that our framework outperforms other methods in both the accuracy and efficiency dimensions."},"_bibtex":{"value":"@misc{\npan2024hetegraphmamba,\ntitle={HeteGraph-Mamba: Heterogeneous Graph Learning via Selective State Space Model},\nauthor={Zhenyu Pan and Yoonsung Jeong and Xiaoda Liu and Han Liu},\nyear={2024},\nurl={https://openreview.net/forum?id=mn61GWpEiK}\n}"},"title":{"value":"HeteGraph-Mamba: Heterogeneous Graph Learning via Selective State Space Model"},"pdf":{"value":"/pdf/f3e375dedad64b3fb6a91ac3fc520e533a0243a1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"pan|hetegraphmamba_heterogeneous_graph_learning_via_selective_state_space_model"},"authorids":{"value":["~Zhenyu_Pan1","~Yoonsung_Jeong1","~Xiaoda_Liu1","~Han_Liu4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zhenyu Pan","Yoonsung Jeong","Xiaoda Liu","Han Liu"]}},"version":2},{"content":{"summary":{"value":"The paper introduces TFMAudio, a Mamba-based text-to-audio (T2A) model that integrates a Time-Frequency Mamba (TFMamba) backbone and Energy-Aware Guidance (EAG) for long-form generation. The model combines linear-complexity state-space modeling with flow matching to improve efficiency and stability over transformer-based diffusion models. The authors claim that TFMAudio achieves state-of-the-art performance on AudioCaps and WavCaps benchmarks and can generate up to 30 minutes of 44.1kHz audio with temporal consistency.\n\nWhile the motivation—to overcome the quadratic complexity of transformers for long-form generation—is reasonable, the experimental evidence and claims are weak and overinterpreted. The model is only trained on 10-second clips, and therefore the claimed ability to generate consistent 30-minute audio is not supported by data. The long-form generation experiment effectively tests extrapolation far outside the training distribution and yields repetitive, semantically meaningless output. Overall, the paper reads more as a well-written engineering demonstration than a solid scientific contribution."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. why the demo pages donot show the full 30 mins audio?\n\n2. The Mamba structure can bring better performance than transformer?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper is well-organized and technically clear. Mathematical derivations (flow matching, EAG) are properly explained, and the figures are informative.\n\n2. Using Mamba for efficient long-range modeling is a timely and interesting idea. The dual-scan mechanism (TimeMamba + FrequencyMamba) provides a coherent architecture for time–frequency modeling."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The entire training uses 10-second AudioCaps/WavCaps clips, yet the main claim of the paper is about ultra-long (30-minute) generation. This makes the core contribution unverifiable—the model has never seen long-form data, so the “30-minute consistency” claim lacks credibility. The long audio is almost certainly repetitive or degenerate, as suggested by the flat metric curves and absence of qualitative human evaluation.\n\n\n2. Only four metrics (CLAP, IS, FAD, KL) are reported—no subjective MOS tests, no human preference studies. The improvement margins over baselines are modest and not statistically validated. \n\n3."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918252908,"tcdate":1761841187224,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5772/Reviewer_ipzw"],"signatures":["ICLR.cc/2026/Conference/Submission5772/Reviewer_ipzw"],"forum":"WFgtMF6vk6","number":3,"license":"CC BY 4.0","cdate":1761841187224,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5772/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918252908,"domain":"ICLR.cc/2026/Conference","replyto":"WFgtMF6vk6","id":"Qx6w1sXwzv","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Text-to-Audio","Flow-Matching","Mamba","SSM","Diffusion"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Recent advancements in audio generation have been dominated by transformer-based diffusion models, which face challenges in extrapolating positional encodings and exhibit quadratic complexity in self-attention, limiting their consistency and efficiency for long-form generation.\nTo address these limitations, we propose TFMAudio, a novel latent audio generation model that integrates the strengths of Flow Matching and a custom-designed TFMamba backbone.\nTFMamba employs a dual-scan mechanism: TimeMamba captures long-range causal dependencies with linear complexity, while FrequencyMamba models spectral correlations such as harmonic structures. To enhance stability, we further introduce Energy-Aware Guidance (EAG), which mitigates state drift by adaptively regularizing classifier-free guidance. Experiments demonstrate that TFMAudio achieves state-of-the-art performance on text-to-audio benchmarks and exhibits robust extrapolation to ultra-long sequences. Remarkably, our model generates 30-minute high-fidelity audio while preserving temporal consistency and semantic alignment, significantly advancing the scalability and usability of text-to-audio models.\n  Demo:https://huggingface.co/spaces/tfmaudio/TFMAudio"},"_bibtex":{"value":"@misc{\ndai2026tfmaudio,\ntitle={{TFMA}udio: High-Fidelity Long-Form Text-to-Audio via Mamba-based Flow Matching},\nauthor={Hao Dai and Jagmohan Chauhan},\nyear={2026},\nurl={https://openreview.net/forum?id=WFgtMF6vk6}\n}"},"title":{"value":"TFMAudio: High-Fidelity Long-Form Text-to-Audio via Mamba-based Flow Matching"},"pdf":{"value":"/pdf/5833cc8c9f598257ed7359368750dd201f830de2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"dai|tfmaudio_highfidelity_longform_texttoaudio_via_mambabased_flow_matching"},"authorids":{"value":["~Hao_Dai1","~Jagmohan_Chauhan1"]},"authors":{"value":["Hao Dai","Jagmohan Chauhan"]}},"version":2},{"content":{"summary":{"value":"The paper proposes a novel layer aggregation method S6LA that treats the outputs from layers as states of a continuous process and utilizes selective state space models architecture for layer aggregation. The paper reformulates two different layer aggregation processes with the proposed S6LA for ConvNet and Transformer-based backbones. The experiments are conducted on two tasks image classification\nand detection on ImageNet, MS COCO 2017 datasets."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"I have a question about transformer-based S6LA integration.\n\nIn the paper, equations 15, 18, 19, and Figure 3, $X^{t−1}_{input}$ go through the attention block, add & norm, and split into two parts: the cls embedding goes through the S6LA, image embeddings add with the product of itself with hidden state, and the final output is the concatenated cls embedding and image embedding.\n\n1. Can you provide a clarification that with the S6LA integration, are the MLP blocks excluded from the original Transformer block?\n\n2. If yes, can you clarify the reason for excluding?\n\n3. If no, how it fits into the pipeline detailed in equations 15, 18, 19, and Figure 3? \n\n4. (I am assuming Linear Projection in Figure 3 means MLP block in the Not Exlcuded case) Since in the traditional transformer-based methods,  after attention and add & norm, cls embedding is also fed into the MLP layer along with image embeddings. From Figure 3, the proposed method seems to exclude cls embedding from MLP layers. Can you clarify the reason and potential impact on layer aggregation if this is the case?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper is well-written and well-organized.\n2. The idea of utilizing a selective state space-based model for seamless layer aggregation with continuous layer output is novel and interesting. The different integration formulation using the proposed S6LA for different backbones(ConvNet and Tranformer) is reasonable.\n3. The quantitative results in the experiment sections show the proposed method's effectiveness over baseline methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The pipeline figure (Figure 3) for the Transformer-based approach can be improved so it will be easier for the reader to follow. For example, the naming labels in Figure 3 should be more aligned with what's in the paper along with some pipeline walkthrough captions.\n2. Selective State Space models are known for slow training due to their nature of backpropagation through time (even with parallel scans). Will this affect the transformer and ConvNet backbone's parallelization training compared to the baseline methods that are Transformer or ConvNet-based?"}},"nonreaders":[],"tmdate":1733171770131,"tcdate":1730238380806,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2351/Reviewer_1DRX"],"signatures":["ICLR.cc/2025/Conference/Submission2351/Reviewer_1DRX"],"forum":"msD4DHZzFg","number":3,"license":"CC BY 4.0","cdate":1730238380806,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2351/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733171770131,"domain":"ICLR.cc/2025/Conference","replyto":"msD4DHZzFg","id":"fEkuEtstlL","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["deep neural network","sequential model","state space model","statistical model"]},"supplementary_material":{"value":"/attachment/ca8e1f039c9a544b65764a56a07189b73607378e.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The depth of neural networks is a critical factor for their capability, with deeper models often demonstrating superior performance. Motivated by this, significant efforts have been made to enhance layer aggregation - reusing information from previous layers to better extract features at the current layer, to improve the representational power of deep neural networks. However, previous works have primarily addressed this problem from a discrete-state perspective which is not suitable as the number of network layers grows. This paper novelly treats the outputs from layers as states of a continuous process and considers leveraging the state space model (SSM) to design the aggregation of layers in very deep neural networks. Moreover, inspired by its advancements in modeling long sequences,  the Selective State Space Models (S6) is employed to design a new module called Selective State Space Model Layer Aggregation (S6LA). This module aims to combine traditional CNN or transformer architectures within a sequential framework, enhancing the representational capabilities of state-of-the-art vision networks. Extensive experiments show that S6LA delivers substantial improvements in both image classification and detection tasks, highlighting the potential of integrating SSMs with contemporary deep learning techniques."},"_bibtex":{"value":"@inproceedings{\nliu2025from,\ntitle={From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics},\nauthor={Qinshuo Liu and Weiqin Zhao and Wei Huang and Yanwen Fang and Lequan Yu and Guodong Li},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=msD4DHZzFg}\n}"},"title":{"value":"From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics"},"pdf":{"value":"/pdf/0eb34c1e14f3708d34cf6fcd2cb15565ff25ade7.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"liu|from_layers_to_states_a_state_space_model_perspective_to_deep_neural_network_layer_dynamics"},"authorids":{"value":["~Qinshuo_Liu2","~Weiqin_Zhao1","~Wei_Huang36","~Yanwen_Fang1","~Lequan_Yu1","~Guodong_Li1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Qinshuo Liu","Weiqin Zhao","Wei Huang","Yanwen Fang","Lequan Yu","Guodong Li"]}},"version":2},{"content":{"summary":{"value":"The paper presents MRconv, a novel type of global convolution layer designed for long 1-D sequence modeling. MRconv is built on an efficient and effective parameterization that produces normalized multi-resolution convolutions. The authors conduct comprehensive empirical evaluations on several benchmarks, including ImageNet-1K, LRA, sCIFAR, and Speech Commands, demonstrating that MRconv achieves near SoTA performance on several tasks and modalities. Finally, several ablations are conducted to justify the design choices and explore additional variants of the layer."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please review the weaknesses, particularly those highlighted in W.1, W.2, W.4, and W.5."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The authors demonstrate a **comprehensive empirical evaluation** of the method across several benchmarks and domains, achieving near SOTA results on real-world tasks such as image and small-scale speech classification, highlighting the robustness and versatility of the approach.\n\n2. The **novel and efficient** parametrization, particularly advantageous during inference, yields promising results, as evidenced by Figure 3 (Left) and Table 4.\n\n3. **Simplicity:** The layer is relatively simple and can be explained with just a few equations, making it easier for the community to adopt (as described in Algorithms 1,2,3)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Empirical evaluation should be improved:**\n\n- 1.a. **Results on NLP**: Global conv layers such as S4, Hyena, and Gated state-space are effective in language modeling. Thus, assessing the language modeling capabilities of MRconv (including both positive or negative results), can enhance the manuscript. Since MRconv is implemented within the S4 repository (which includes WikiText-103), conducting such experiments requires minimal additional effort. \n\n- 1.b. Conducting experiments with **small-scale synthetic tasks** and controlled environments can provide valuable insights into the properties, weaknesses, and strengths of the layer compared to alternatives. Examples of such tasks include atomic tasks [1], copy, selective copy [2], associative recall [3], and others [4]. Even negative results are important, as they offer valuable information about the limitations and failure cases of the layer.\n\n- 1.c.  **Gated convolutions** have proven to be highly effective in various domains. Understanding their critical role in each domain and how much they can enhance MRconv is both important and informative. This knowledge can improve the usability of these layers.\n\n2. **Efficiency benchmarking should be improved:**\n\n- 2.a. **FLOPS comparison:** The provided comparison of FLOPs focuses solely on ImageNet, utilizing the ConvNeXt backbone. However, this comparison may be less informative as most of the FLOPs in these backbones are not located in the tested layers (SGconv, Conv2D, MRconv). Instead, the majority of the FLOPs are found in the MLP (or 1x1 Conv) layers. Therefore, I find these comparisons less relevant. Am I overlooking something here? If not, please perform a FLOPs comparison in other, more relevant regimes. Additionally, I recommend the authors include a figure showing the FLOPs (and/or latency, and throughput) on the y-axis for various sequence lengths (x-axis) across several layers (MRconv, SGconv, SSM variant, Conv1D, and others) to fully describe the empirical complexity of the method.\n\n- 2.b. **A normalized amount of parameters:** In some of the tables there is no number of parameters (Table 3), and in others tables the proposed method has a higher amount of parameters (Table 12), this is even more important for Table 2. At least part of the difference between the first two rows (Dense kernel vs Multiresolution) can be explain by additional parameters. Can the authors conduct the experiments in Table 2 where the number of parameters is normalized across rows? \n\n3. Measuring the **impact of hyperparameters and ablation studies**: Conducting additional ablation studies could provide more insights into the robustness and design principles of the method. Examples include ablating the number of resolutions, the initial resolution, the decay factor, and conducting experiments with LayerNorm instead of BatchNorm, among others.\nThere is no theoretical justification provided, which would greatly enhance the paper. I suggest the authors explore the expressiveness, initialization, generalization, and inductive bias of MRconv in comparison to other convolutional layers such as SSMs, SGconv, and others.\n\n4. **Insights on MRconv:**  It would be beneficial if the authors provided more insights gained during their research on MRconv. For instance, a subsection discussing what makes MRconv successful would be valuable. Potential reasons could include suitable inductive bias towards multi-resolution and better optimization properties arising from stable parameterization (e.g., Batch Normalization). Directly analyzing these factors could offer informative perspectives to the community and pave the way for further improvements.\n\n5. **The Focus of the paper:** The authors focus on long sequence modeling, probably due to the results over LRA benchmark and the sub-quadratic dependencies in sequence length. However, I am not certain this is the strongest aspect of the method in practice. For instance, achieving SOTA results on the LRA benchmark could be due to a strong inductive bias toward locality and other properties, as explored by [5] and [6]. Perhaps the strongest aspect of the method is its ability to learn high and low frequencies through a stable and efficient parameterization, resulting in a very natural inductive bias.\n\n6. **Extensions:** exploring extensions such as multi-axis variants (2D, 3D..), bidirectional design, gated convolutions, and the inner block design could enhance the paper.\n\n7. **Novelty:** The method is relatively simple and can be described with just a few equations (as described in Algorithms 1,2,3). Although there are no particularly surprising ideas, the design choices are sound, the empirical evaluations are thorough, and the performance is nearly SOTA across several modalities. Therefore, I do not consider this limitation to be significant.\n\nI am willing to raise my score if the concerns specified in this section will be sufficiently addressed during the rebuttal period.\n\n[1] Simplifying and Understanding State Space Models with Diagonal Linear RNNs. Gupta et al.\n\n[2] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Gu et al.\n\n[3] Hungry Hungry Hippos: Towards Language Modeling with State Space Models. Fu et al.\n\n[4] Mechanistic Design and Scaling of Hybrid Architectures. Poli et al.\n\n[5] Viewing Transformers Through the Lens of Long Convolutions Layers. Zimerman et al.\n\n[6] Never Train from Scratch: FAIR COMPARISON OF LONGSEQUENCE MODELS REQUIRES DATA-DRIVEN PRIORS. Amos et al."},"limitations":{"value":"The limitations section can be improved. Please refer to W.1.b for details on the failure cases of the layer."}},"nonreaders":[],"tmdate":1730879322360,"tcdate":1720794144990,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission9552/Reviewer_JNPq"],"signatures":["NeurIPS.cc/2024/Conference/Submission9552/Reviewer_JNPq"],"forum":"RwgNbIpCpk","number":2,"license":"CC BY 4.0","cdate":1720794144990,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission9552/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879322360,"domain":"NeurIPS.cc/2024/Conference","replyto":"RwgNbIpCpk","id":"xqTLFZecV3","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Sequence Modelling","Convolutions","Structural Reparameterization","Self Attention","State Space Models"]},"supplementary_material":{"value":"/attachment/b31a082bbd7b4c4b945895ba10a0c90591d33abb.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ($\\texttt{MRConv}$), a novel approach to parameterizing global convolutional kernels for long-sequence modeling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, $\\texttt{MRConv}$ learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D $\\texttt{MRConv}$ layers."},"_bibtex":{"value":"@inproceedings{\ncunningham2024reparameterized,\ntitle={Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling},\nauthor={Harry Jake Cunningham and Giorgio Giannone and Mingtian Zhang and Marc Peter Deisenroth},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=RwgNbIpCpk}\n}"},"title":{"value":"Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling"},"pdf":{"value":"/pdf/99fdd8c02bfd174bfdb9bf7e36f2b9016b370ad4.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"cunningham|reparameterized_multiresolution_convolutions_for_long_sequence_modelling"},"authorids":{"value":["~Harry_Jake_Cunningham1","~Giorgio_Giannone1","~Mingtian_Zhang1","~Marc_Peter_Deisenroth1"]},"authors":{"value":["Harry Jake Cunningham","Giorgio Giannone","Mingtian Zhang","Marc Peter Deisenroth"]}},"version":2},{"content":{"TLDR":{"value":"We introduce MS-SSM which enhances traditional SSMs by modeling sequence dynamics at multiple resolutions using independent SSMs, scale-dependent initialization, and an input-dependent scale-mixer."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["MS-SSM: Sequence Models","Language models","State Space Model","Multi-Scale","Multi-Resolution"]},"abstract":{"value":"State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurrences to integrate information over time, enabling fast inference, parallelizable training, and control over recurrence stability.  However, traditional SSMs often suffer from limited effective memory, requiring larger state sizes for improved recall. Moreover, existing SSMs struggle to capture multi-scale dependencies, which are essential for modeling complex structures in time series, images, and natural language. This paper introduces a multi-scale SSM framework that addresses these limitations by representing sequence dynamics across multiple resolution and processing each resolution with specialized state-space dynamics.  By capturing both fine-grained, high-frequency patterns and coarse, global trends, MS-SSM enhances memory efficiency and long-range modeling. We further introduce an input-dependent scale-mixer, enabling dynamic information fusion across resolutions.\nThe proposed approach significantly improves sequence modeling, particularly in long-range and hierarchical tasks, while maintaining computational efficiency. Extensive experiments on benchmarks, including Long Range Arena, hierarchical reasoning, time series classification, and image recognition, demonstrate that MS-SSM consistently outperforms prior SSM-based models, highlighting the benefits of multi-resolution processing in state-space architectures."},"_bibtex":{"value":"@inproceedings{\nkarami2025msssm,\ntitle={{MS}-{SSM}: A Multi-Scale State Space Model for Efficient Sequence Modeling},\nauthor={Mahdi Karami and Ali Behrouz and Peilin Zhong and Razvan Pascanu and Vahab Mirrokni},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=cCYWeCzAv0}\n}"},"title":{"value":"MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling"},"pdf":{"value":"/pdf/483868b1b70acb79914fb7e065ea71d3d5a37968.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"karami|msssm_a_multiscale_state_space_model_for_efficient_sequence_modeling"},"authorids":{"value":["~Mahdi_Karami2","~Ali_Behrouz1","~Peilin_Zhong1","~Razvan_Pascanu1","~Vahab_Mirrokni2"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Mahdi Karami","Ali Behrouz","Peilin Zhong","Razvan Pascanu","Vahab Mirrokni"]}},"tmdate":1756169227819,"pdate":1751944476606,"tcdate":1742511610296,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Submission1131/Authors"],"signatures":["colmweb.org/COLM/2025/Conference/Submission1131/Authors"],"forum":"cCYWeCzAv0","license":"CC BY 4.0","number":1131,"cdate":1742511610296,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/-/Submission","colmweb.org/COLM/2025/Conference/-/Post_Submission","colmweb.org/COLM/2025/Conference/Submission1131/-/Full_Submission","colmweb.org/COLM/2025/Conference/-/Edit","colmweb.org/COLM/2025/Conference/Submission1131/-/Camera_Ready"],"mdate":1756169227819,"odate":1756169227792,"domain":"colmweb.org/COLM/2025/Conference","id":"cCYWeCzAv0","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2507.20128v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"yuan|diffusionbased_symbolic_music_generation_with_structured_state_space_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Shenghua_Yuan:","","https://dblp.org/search/pid/api?q=author:Jiatao_Chen:","https://dblp.org/search/pid/api?q=author:Tianming_Xie:","https://dblp.org/search/pid/api?q=author:Jing_Wang_0063:","https://dblp.org/search/pid/api?q=author:Bing_Shi_0002:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2507.20128"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2507-20128,\n  publtype={informal},\n  author={Shenghua Yuan and Xing Tang and Jiatao Chen and Tianming Xie and Jing Wang and Bing Shi},\n  title={Diffusion-based Symbolic Music Generation with Structured State Space Models},\n  year={2025},\n  month={July},\n  cdate={1751328000000},\n  journal={CoRR},\n  volume={abs/2507.20128},\n  url={https://doi.org/10.48550/arXiv.2507.20128}\n}\n"},"abstract":{"value":"Recent advancements in diffusion models have significantly improved symbolic music generation. However, most approaches rely on transformer-based architectures with self-attention mechanisms, which are constrained by quadratic computational complexity, limiting scalability for long sequences. To address this, we propose Symbolic Music Diffusion with Mamba (SMDIM), a novel diffusion-based architecture integrating Structured State Space Models (SSMs) for efficient global context modeling and the Mamba-FeedForward-Attention Block (MFA) for precise local detail preservation. The MFA Block combines the linear complexity of Mamba layers, the non-linear refinement of FeedForward layers, and the fine-grained precision of self-attention mechanisms, achieving a balance between scalability and musical expressiveness. SMDIM achieves near-linear complexity, making it highly efficient for long-sequence tasks. Evaluated on diverse datasets, including FolkDB, a collection of traditional Chinese folk music that represents an underexplored domain in symbolic music generation, SMDIM outperforms state-of-the-art models in both generation quality and computational efficiency. Beyond symbolic music, SMDIM's architectural design demonstrates adaptability to a broad range of long-sequence generation tasks, offering a scalable and efficient solution for coherent sequence modeling."},"title":{"value":"Diffusion-based Symbolic Music Generation with Structured State Space Models"},"authors":{"value":["Shenghua Yuan","Xing Tang","Jiatao Chen","Tianming Xie","Jing Wang","Bing Shi"]}},"tmdate":1775179610931,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2507-20128"],"tcdate":1758764979110,"writers":["~"],"signatures":["~Xing_Tang4"],"forum":"kTYynyzKXo","license":"CC BY-SA 4.0","number":628152,"cdate":1751328000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1775179610931,"domain":"DBLP.org","id":"kTYynyzKXo","version":2},{"content":{"venue":{"value":"TIME 2002"},"pdf":{"value":"https://ieeexplore.ieee.org/iel5/7975/22066/01027486.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2002"},"paperhash":{"value":"combi|a_logical_approach_to_represent_and_reason_about_calendars"},"authorids":{"value":["~Carlo_Combi1","https://dblp.org/search/pid/api?q=author:Massimo_Franceschet:","https://dblp.org/search/pid/api?q=author:Adriano_Peron:"]},"html":{"value":"https://doi.org/10.1109/TIME.2002.1027486"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/CombiFP02,\n  author={Carlo Combi and Massimo Franceschet and Adriano Peron},\n  title={A Logical Approach to Represent and Reason about Calendars},\n  year={2002},\n  cdate={1009843200000},\n  pages={134-140},\n  url={https://doi.org/10.1109/TIME.2002.1027486},\n  booktitle={TIME},\n  crossref={conf/time/2002}\n}\n"},"abstract":{"value":"We propose a logical approach to represent and reason about different time granularities. We identify a time granularity as a discrete infinite sequence of time points properly labelled with proposition symbols marking the starting and ending points of the corresponding granules, and we intensively model sets of granularities with linear time logic formulas. Some real-world granularities are provided to motivate and exemplify our approach. The proposed framework permits to algorithmically solve the consistency, the equivalence, and the classification problems in a uniform way, by reducing them to the validity problem for the considered linear time logic."},"title":{"value":"A Logical Approach to Represent and Reason about Calendars"},"authors":{"value":["Carlo Combi","Massimo Franceschet","Adriano Peron"]}},"tmdate":1747652546607,"pdate":1009843200000,"tcdate":1747652519431,"writers":["~"],"signatures":["~Carlo_Combi1"],"forum":"68WsmGoVek","license":"CC BY-SA 4.0","number":526623,"cdate":1009843200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747652546607,"domain":"DBLP.org","id":"68WsmGoVek","version":2},{"content":{"summary":{"value":"This paper invetigate the linear attention in the Vision task"},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. The source code is not avaliable, the reproductability is unclear at this time.\n\n2. how this linear attetnion compare with Vision Mamba since Mamba is efficient.\n\n3. The experimental results are mainly on CV domain, is this algorithm adaptive to NLP or time series domain?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper is well written, the motivation is clear.\n\n2. The findings is this work is 1) inear attention is not injective, which is prone to assign identical attention weights to different query vector. 2) effective local modeling is essential for the success of Softmax attention, which linear attention deos not have. Thoses findings may be improtant for design efficient linear transformer.\n\n3. The experimental results are good."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The source code is not avaliable, the reproductability is unclear at this time.\n\n2. how this linear attetnion compare with Vision Mamba since Mamba is efficient.\n\n3. The experimental results are mainly on CV domain, is this algorithm adaptive to NLP or time series domain?"},"limitations":{"value":"The experimental results are mainly on CV domain, is this algorithm adaptive to NLP or time series domain?"}},"nonreaders":[],"tmdate":1730878808850,"tcdate":1720763553865,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2854/Reviewer_3cj2"],"signatures":["NeurIPS.cc/2024/Conference/Submission2854/Reviewer_3cj2"],"forum":"RSiGFzQapl","number":2,"license":"CC BY 4.0","cdate":1720763553865,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2854/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878808850,"domain":"NeurIPS.cc/2024/Conference","replyto":"RSiGFzQapl","id":"MJwfxuSNER","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Softmax attention","linear attention"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Widely adopted in modern Vision Transformer designs, Softmax attention can effectively capture long-range visual information; however, it incurs excessive computational cost when dealing with high-resolution inputs. In contrast, linear attention naturally enjoys linear complexity and has great potential to scale up to higher-resolution images. Nonetheless, the unsatisfactory performance of linear attention greatly limits its practical application in various scenarios. In this paper, we take a step forward to close the gap between the linear and Softmax attention with novel theoretical analyses, which demystify the core factors behind the performance deviations. Specifically, we present two key perspectives to understand and alleviate the limitations of linear attention: the injective property and the local modeling ability. Firstly, we prove that linear attention is not injective, which is prone to assign identical attention weights to different query vectors, thus adding to severe semantic confusion since different queries correspond to the same outputs. Secondly, we confirm that effective local modeling is essential for the success of Softmax attention, in which linear attention falls short. The aforementioned two fundamental differences significantly contribute to the disparities between these two attention paradigms, which is demonstrated by our substantial empirical validation in the paper. In addition, more experiment results indicate that linear attention, as long as endowed with these two properties, can outperform Softmax attention across various tasks while maintaining lower computation complexity. Code is available at https://github.com/LeapLabTHU/InLine."},"_bibtex":{"value":"@inproceedings{\nhan2024bridging,\ntitle={Bridging the Divide: Reconsidering Softmax and Linear Attention},\nauthor={Dongchen Han and Yifan Pu and Zhuofan Xia and Yizeng Han and Xuran Pan and Xiu Li and Jiwen Lu and Shiji Song and Gao Huang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=RSiGFzQapl}\n}"},"title":{"value":"Bridging the Divide: Reconsidering Softmax and Linear Attention"},"pdf":{"value":"/pdf/31bad536df26e6977656df403fe28b991643b271.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"han|bridging_the_divide_reconsidering_softmax_and_linear_attention"},"authorids":{"value":["~Dongchen_Han1","~Yifan_Pu1","~Zhuofan_Xia2","~Yizeng_Han1","~Xuran_Pan1","~Xiu_Li1","~Jiwen_Lu1","~Shiji_Song1","~Gao_Huang1"]},"authors":{"value":["Dongchen Han","Yifan Pu","Zhuofan Xia","Yizeng Han","Xuran Pan","Xiu Li","Jiwen Lu","Shiji Song","Gao Huang"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Long-sequence modeling","Mamba","Memory mechanisms"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the explosive growth of data, long-sequence modeling has become increasingly important in tasks such as natural language processing and bioinformatics. However, existing methods face inherent trade-offs between efficiency and memory. Recurrent neural networks suffer from gradient vanishing and explosion, making them hard to scale. Transformers can model global dependencies but are constrained by quadratic complexity. Recently, selective state-space models such as Mamba have demonstrated high efficiency with $O(n)$ time and $O(1)$ recurrent inference, yet their long-range memory decays exponentially. In this work, we conduct mathematical derivations and information-theoretic analysis to systematically uncover the memory decay mechanism of Mamba, answering a fundamental question: what is the nature of Mamba’s long-range memory and how does it retain information? To quantify key information loss, we further introduce horizontal–vertical memory fidelity metrics that capture degradation both within and across layers. Inspired by how humans distill and retain salient information when reading long documents, we propose MemMamba, a novel architectural framework that integrates state summarization mechanism together with cross-layer and cross-token attention, which alleviates long-range forgetting while preserving linear complexity. MemMamba achieves significant improvements over existing Mamba variants and Transformers on long-sequence benchmarks such as PG19-PPL and Passkey Retrieval, while delivering a 48\\% speedup in inference efficiency. Both theoretical analysis and empirical results demonstrate that MemMamba achieves a breakthrough in the complexity–memory trade-off, offering a new paradigm for ultra-long sequence modeling. The code and the pre-trained models will be released upon acceptance."},"_bibtex":{"value":"@misc{\nwang2026memmamba,\ntitle={MemMamba: Rethinking Memory Patterns in State Space Model},\nauthor={Youjin Wang and Yangjingyi Chen and Jiahao yan and Jiaxuan Lu and Xiao Sun},\nyear={2026},\nurl={https://openreview.net/forum?id=jPYJuvzof2}\n}"},"title":{"value":"MemMamba: Rethinking Memory Patterns in State Space Model"},"pdf":{"value":"/pdf/26376d032f685fb35659e4ab241730a5eb60a8ff.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wang|memmamba_rethinking_memory_patterns_in_state_space_model"},"authorids":{"value":["~Youjin_Wang1","~Yangjingyi_Chen1","~Jiahao_yan1","~Jiaxuan_Lu1","~Xiao_Sun8"]},"authors":{"value":["Youjin Wang","Yangjingyi Chen","Jiahao yan","Jiaxuan Lu","Xiao Sun"]}},"tmdate":1770804899466,"tcdate":1758029377655,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7614/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7614/Authors"],"forum":"jPYJuvzof2","license":"CC BY 4.0","number":7614,"cdate":1758029377655,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission7614/-/Full_Submission","ICLR.cc/2026/Conference/Submission7614/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804899466,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"jPYJuvzof2","version":2},{"content":{"summary":{"value":"This paper introduces LION, a novel sequence-to-sequence framework that combines the bidirectionality and parallelized training of Transformers with fast inference of recurrent neural networks. \nMore specifically this paper at first shows how Linear Attention and Gated Linear attention variants (including Selective State Space Models, such as Mamba-2) can be reformulated as bi-directional RNNs, such that they can use a parallel masked matrix multiply formulation (e.g. Y=scale(QK^T * M) V ) for training similar to transformers while still using the recurrent formulation for inference.\n\nThen, the paper proposes LION-S a novel Gated or Selective Linear Attention variant with a specific parameterization gate or selectivity parameter inspired by zero-order hold discretization and LION-LIT a bi-directional non-gated Linear Attention variant, where the selectivity parameter is set to 1. \n\nThrough extensive experiments on the Long Range Arena Benchmark, Masked-Language Modeling, Image classification and Causal-Language Modeling the paper demonstrates that their proposed recurrences LION-S and LION-LIT achieve competitive performance to state of the art Transformer and State Space Models."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"- It seems that the paper often uses the terms “attention”, “attention mechanism”, “full-attention” or “attention-formulas” interchangeably, which makes it sometimes hard for reader to disambiguate from the context whether softmax-attention, linear attention or causal (linear) attention is meant (e.g. L. 187, or L. 923). It would improve the readability if the terminology is unified in the paper. \n\n- L. 190 The terminology forward / backward path might be misleading, as the term backward path is usually used for computing the derivatives / delta errors. I would suggest to clarify this distinction at some point or rather use forward / backward recurrence always.\nOut of curiosity: Why is it called LION?\n\n- In Table 8, you apply a Non-linearity (elu) to q and k in addition to the LION-S mask. What is the default for LION-S? Only Mask and no non-linearity or mask and non-linearity?\n\n- In equation 85, 88 and 101 is there the mask M missing?\n\n- The exponential gating in the mLSTM (xLSTM) requires a stabilizer (or “max”) state in addition to S and z. See (eq. 15, 16, 17 and  74, 75, 80 in [2]). This is not explicitly addressed in 94 to 96, but the LION bidirectional reformulation should still be applicable.\n\n- Have you tried using just the forward recurrence of LION-S, but reverse the order of the input sequence in consecutive blocks as it was done for example in “Vision-LSTM” [1] ? It would be interesting how this compares to the forward/backward LION-S. The same experiment could be also applicable to masked language modeling in Section 5.2.\n\n[1] Alkin, B., Beck, M., Pöppel, K., Hochreiter, S., & Brandstetter, J. (2024). Vision-LSTM: xLSTM as Generic Vision Backbone. arXiv preprint arXiv:2406.04303\n\n[2] Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., ... & Hochreiter, S. (2024). xLSTM: Extended Long Short-Term Memory. arXiv preprint arXiv:2405.04517."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":4},"strengths":{"value":"This paper does an exceptionally good job in presenting the complex formulas and transformations in a very readable and understandable way. It also provides a very good overview of related work and provides lots of insights in relations between those works. \nTheir experiments are very extensive and the paper also provides careful ablation studies on different design decisions of their LION-S recurrence (e.g. Table 6, Table 7). The authors also provide inference efficiency experiments in the appendix.\n\nI enjoyed reading this paper and believe that it is a good contribution, therefore I recommend to accept this paper and hope the authors can clarify the questions and unclarities below:"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- There are only minor weaknesses such that the Causal Language Modeling and Vision experiments are rather small scale. Nevertheless, the paper demonstrates with these experiments the effectiveness of LION-S also on these compute-intense domains.\n\n- The connection to the continuous domain of LION-S is rather loose and could be elaborated on further."}},"nonreaders":[],"tmdate":1731428746799,"tcdate":1730321536103,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13771/Reviewer_3WDt"],"signatures":["ICLR.cc/2025/Conference/Submission13771/Reviewer_3WDt"],"forum":"VxJjw52dZu","number":1,"license":"CC BY 4.0","cdate":1730321536103,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13771/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428746799,"domain":"ICLR.cc/2025/Conference","replyto":"VxJjw52dZu","id":"fIccgsI2Ij","forumContent":{"TLDR":{"value":"We provide a bidirectional selective recurrent form of full kernelized attention with learnable mask allowing scalability in context length and superior inference efficiency"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Self-attention","transformer","bidirectionality","mamba","ssm","rnn","inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We introduce LION, a novel sequence-to-sequence framework that unifies the bidirectionality and parallelized training of Transformers with the fast inference of recurrent neural networks. LION is built upon a mathematical formulation where full kernelized attention with a learnable mask is efficiently computed using a bidirectional selective recurrent model, matching the effectiveness of softmax-based attention with constant-time inference. Our framework naturally accounts for spatial and temporal relationships within input sequences, reducing reliance on heuristic positional embeddings and facilitating straightforward scalability in context length and resolution. Using our framework and inspired by the recent state-space models, we propose three main running examples LIOn-LIT, LION-RETNET, and LION-S, a transformer with selective mask and recurrent inference. Numerical evaluations on tasks such as language modeling, the Long-Range Arena, and image classification show that LION framework achieves performance on par with state-of-the-art models while delivering fast training and inference efficiency."},"_bibtex":{"value":"@misc{\nafzal2025lion,\ntitle={{LION}: A bidirectional framework that trains like a Transformer and infers like an {RNN}},\nauthor={Arshia Afzal and Elias Abad Rocamora and Leyla Naz Candogan and Pol Puigdemont and Francesco Tonin and Yongtao Wu and Mahsa Shoaran and Volkan Cevher},\nyear={2025},\nurl={https://openreview.net/forum?id=VxJjw52dZu}\n}"},"title":{"value":"LION: A bidirectional framework that trains like a Transformer and infers like an RNN"},"pdf":{"value":"/pdf/6e8b8585752d6f27da24ffe876859459a5b56efe.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"afzal|lion_a_bidirectional_framework_that_trains_like_a_transformer_and_infers_like_an_rnn"},"authorids":{"value":["~Arshia_Afzal1","~Elias_Abad_Rocamora1","~Leyla_Naz_Candogan1","~Pol_Puigdemont1","~Francesco_Tonin1","~Yongtao_Wu1","~Mahsa_Shoaran1","~Volkan_Cevher1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Arshia Afzal","Elias Abad Rocamora","Leyla Naz Candogan","Pol Puigdemont","Francesco Tonin","Yongtao Wu","Mahsa Shoaran","Volkan Cevher"]}},"version":2},{"content":{"venue":{"value":"NeurIPS 2024"},"pdf":{"value":"https://proceedings.neurips.cc/paper_files/paper/2024/file/5ce377d14a21ef1fea0400049ad324b4-Paper-Conference.pdf"},"venueid":{"value":"dblp.org/conf/NIPS/2024"},"paperhash":{"value":"hu|vision_mamba_mender"},"authorids":{"value":["~Jiacong_Hu1","~Anda_Cao1","https://dblp.org/search/pid/api?q=author:Zunlei_Feng:","https://dblp.org/search/pid/api?q=author:Shengxuming_Zhang:","https://dblp.org/search/pid/api?q=author:Yi_Wang:","https://dblp.org/search/pid/api?q=author:Lingxiang_Jia:","https://dblp.org/search/pid/api?q=author:Mingli_Song:"]},"html":{"value":"http://papers.nips.cc/paper_files/paper/2024/hash/5ce377d14a21ef1fea0400049ad324b4-Abstract-Conference.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/nips/HuCFZWJS24,\n  author={Jiacong Hu and Anda Cao and Zunlei Feng and Shengxuming Zhang and Yi Wang and Lingxiang Jia and Mingli Song},\n  title={Vision Mamba Mender},\n  year={2024},\n  cdate={1704067200000},\n  url={http://papers.nips.cc/paper_files/paper/2024/hash/5ce377d14a21ef1fea0400049ad324b4-Abstract-Conference.html},\n  booktitle={NeurIPS},\n  crossref={conf/nips/2024}\n}\n"},"abstract":{"value":"Mamba, a state-space model with selective mechanisms and hardware-aware architecture, has demonstrated outstanding performance in long sequence modeling tasks, particularly garnering widespread exploration and application in the field of computer vision. While existing works have mixed opinions of its application in visual tasks, the exploration of its internal workings and the optimization of its performance remain urgent and worthy research questions given its status as a novel model. Existing optimizations of the Mamba model, especially when applied in the visual domain, have primarily relied on predefined methods such as improving scanning mechanisms or integrating other architectures, often requiring strong priors and extensive trial and error. In contrast to these approaches, this paper proposes the Vision Mamba Mender, a systematic approach for understanding the workings of Mamba, identifying flaws within, and subsequently optimizing model performance. Specifically, we present methods for predictive correlation analysis of Mamba's hidden states from both internal and external perspectives, along with corresponding definitions of correlation scores, aimed at understanding the workings of Mamba in visual recognition tasks and identifying flaws therein. Additionally, tailored repair methods are proposed for identified external and internal state flaws to eliminate them and optimize model performance. Extensive experiments validate the efficacy of the proposed methods on prevalent Mamba architectures, significantly enhancing Mamba's performance."},"title":{"value":"Vision Mamba Mender"},"authors":{"value":["Jiacong Hu","Anda Cao","Zunlei Feng","Shengxuming Zhang","Yi Wang","Lingxiang Jia","Mingli Song"]}},"tmdate":1747314801806,"pdate":1704067200000,"tcdate":1744716548516,"writers":["~"],"signatures":["~Jiacong_Hu1"],"forum":"qKKmdJqSKZ","license":"CC BY-SA 4.0","number":397468,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747314801806,"domain":"DBLP.org","id":"qKKmdJqSKZ","version":2},{"content":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","State Space Models","Computer Vision"]},"supplementary_material":{"value":"/attachment/231f16a7aedf14ace21ad17af99f0f84a5a06ee6.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Mamba, a state-space model with selective mechanisms and hardware-aware architecture, has demonstrated outstanding performance in long sequence modeling tasks, particularly garnering widespread exploration and application in the field of computer vision. While existing works have mixed opinions of its application in visual tasks, the exploration of its internal workings and the optimization of its performance remain urgent and worthy research questions given its status as a novel model. Existing optimizations of the Mamba model, especially when applied in the visual domain, have primarily relied on predefined methods such as improving scanning mechanisms or integrating other architectures, often requiring strong priors and extensive trial and error. In contrast to these approaches, this paper proposes the Vision Mamba Mender, a systematic approach for understanding the workings of Mamba, identifying flaws within, and subsequently optimizing model performance. Specifically, we present methods for predictive correlation analysis of Mamba's hidden states from both internal and external perspectives, along with corresponding definitions of correlation scores, aimed at understanding the workings of Mamba in visual recognition tasks and identifying flaws therein. Additionally, tailored repair methods are proposed for identified external and internal state flaws to eliminate them and optimize model performance. Extensive experiments validate the efficacy of the proposed methods on prevalent Mamba architectures, significantly enhancing Mamba's performance."},"_bibtex":{"value":"@inproceedings{\nhu2024vision,\ntitle={Vision Mamba Mender},\nauthor={Jiacong Hu and Anda Cao and Zunlei Feng and Shengxuming Zhang and Yi Wang and Lingxiang Jia and Mingli Song},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=9VnevS2YoR}\n}"},"title":{"value":"Vision Mamba Mender"},"pdf":{"value":"/pdf/8a5b7efdabc5e5544e1b8f3b86095c84a05b91e1.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"hu|vision_mamba_mender"},"authorids":{"value":["~Jiacong_Hu1","~Anda_Cao1","~Zunlei_Feng1","~Shengxuming_Zhang1","~Yi_Wang56","~Lingxiang_Jia1","~Mingli_Song1"]},"authors":{"value":["Jiacong Hu","Anda Cao","Zunlei Feng","Shengxuming Zhang","Yi Wang","Lingxiang Jia","Mingli Song"]}},"tmdate":1736941790038,"pdate":1727287841671,"tcdate":1715616693556,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission7275/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission7275/Authors"],"forum":"9VnevS2YoR","license":"CC BY 4.0","number":7275,"cdate":1715616693556,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/-/Submission","NeurIPS.cc/2024/Conference/-/Edit","NeurIPS.cc/2024/Conference/-/Post_Submission","NeurIPS.cc/2024/Conference/Submission7275/-/Revision","NeurIPS.cc/2024/Conference/Submission7275/-/Camera_Ready_Revision"],"mdate":1736941790038,"odate":1730873901333,"domain":"NeurIPS.cc/2024/Conference","id":"9VnevS2YoR","version":2},{"content":{"venue":{"value":"CoLoRAI Poster"},"pdf":{"value":"/pdf/34721b3d34ae8a523f4f8806207b713c437f2ef0.pdf"},"keywords":{"value":["LRNN","structured architectures","block-diagonal","state mixing","expressivity and efficiency trade-off","inductive biases"]},"venueid":{"value":"ICML.cc/2026/Workshop/CoLoRAI"},"paperhash":{"value":"dubinin|improved_state_mixing_in_higherorder_and_block_diagonal_linear_recurrent_networks"},"authorids":{"value":["~Igor_Dubinin1","~Antonio_Orvieto3","~Felix_Effenberger1"]},"abstract":{"value":"Linear recurrent networks (LRNNs) and linear state space models (SSMs) promise computational and memory efficiency on sequence modeling tasks, yet their diagonal state transitions limit expressivity. Dense and/or nonlinear architectures (e.g., LSTMs) on the other hand are provably more expressive, but computationally costly.  Here, we explore how expressivity in LRNNs can be increased via richer state mixing across time and channels while maintaining competitive efficiency. Specifically, we introduce two structured LRNN architectures: (i) Higher-order Linear Recurrent Units (H-LRU), which generalize recurrences to arbitrary order, mixing multiple past states, and (ii) Block-Diagonal LRUs (BD-LRU), which enable dense intra-block channel mixing. To ensure stable training, we introduce a selective gate normalization scheme that allows for scalable window and block sizes. To maintain efficiency, we utilize a parallel-scan implementation that keeps the throughput competitive with diagonal LRNNs for moderate orders (H-LRU) and block sizes (BD-LRU). \nConsistent with prior theoretical studies on the limitations of diagonal models, we empirically demonstrate in both synthetic sequence modeling and language modeling that our architectures significantly benefit from the increased expressivity of structured state mixing. Our results show that the structure of state mixing is a critical driver of performance in LRNNs, offering a practical pathway to closing the efficiency–expressivity gap in linear sequence models."},"_bibtex":{"value":"@inproceedings{\ndubinin2026improved,\ntitle={Improved State Mixing in Higher-order and Block Diagonal Linear Recurrent Networks},\nauthor={Igor Dubinin and Antonio Orvieto and Felix Effenberger},\nbooktitle={ICML'26 workshop on CoLoRAI - The 2nd Workshop on Connecting Low-rank Representations in AI},\nyear={2026},\nurl={https://openreview.net/forum?id=uJucghuZf3}\n}"},"title":{"value":"Improved State Mixing in Higher-order and Block Diagonal Linear Recurrent Networks"},"authors":{"value":["Igor Dubinin","Antonio Orvieto","Felix Effenberger"]}},"tmdate":1784004858807,"pdate":1780064776718,"tcdate":1778233268403,"writers":["ICML.cc/2026/Workshop/CoLoRAI","ICML.cc/2026/Workshop/CoLoRAI/Submission121/Authors"],"signatures":["ICML.cc/2026/Workshop/CoLoRAI/Submission121/Authors"],"forum":"uJucghuZf3","license":"CC BY 4.0","number":121,"cdate":1778233268403,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/CoLoRAI/-/Submission","ICML.cc/2026/Workshop/CoLoRAI/-/Post_Submission","ICML.cc/2026/Workshop/CoLoRAI/-/Edit","ICML.cc/2026/Workshop/CoLoRAI/Submission121/-/Camera_Ready"],"mdate":1784004858807,"odate":1778329857539,"domain":"ICML.cc/2026/Workshop/CoLoRAI","id":"uJucghuZf3","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2601.20518v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"chen|ccmamba_selective_statespace_models_for_higherorder_graph_learning_on_combinatorial_complexes"},"html":{"value":"https://doi.org/10.48550/arXiv.2601.20518"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2601-20518,\n  publtype={informal},\n  author={Jiawen Chen and Qi Shao and Mingtong Zhou and Duxin Chen and Wenwu Yu},\n  title={CCMamba: Selective State-Space Models for Higher-Order Graph Learning on Combinatorial Complexes},\n  year={2026},\n  month={January},\n  cdate={1767225600000},\n  journal={CoRR},\n  volume={abs/2601.20518},\n  url={https://doi.org/10.48550/arXiv.2601.20518}\n}\n"},"abstract":{"value":"Topological deep learning has emerged as a powerful paradigm for modeling higher-order relational structures beyond pairwise interactions that standard graph neural networks fail to capture. While combinatorial complexes (CCs) offer a unified topological foundation for the higher-order graph learning, existing topological deep learning methods rely heavily on local message passing and attention mechanisms. These suffer from quadratic complexity and local neighborhood constraints, limiting their scalability and capacity for rank-aware, long-range dependency modeling. To overcome these challenges, we propose Combinatorial Complex Mamba (CCMamba), the first unified Mamba-based neural framework for learning on combinatorial complexes. CCMamba reformulates higher-order message passing as a selective state-space modeling problem by linearizing multi-rank incidence relations into structured, rank-aware sequences. This architecture enables adaptive, directional, and long-range information propagation in linear time bypassing the scalability bottlenecks of self-attention. Theoretically, we further establish that the expressive power of CCMamba is upper-bounded by the 1-dimensional combinatorial complex Weisfeiler-Lehman (1-CCWL) test. Extensive experiments across graph, hypergraph, and simplicial benchmarks demonstrate that CCMamba consistently outperforms existing methods while exhibiting superior scalability and remarkable robustness against over-smoothing in deep architectures."},"title":{"value":"CCMamba: Selective State-Space Models for Higher-Order Graph Learning on Combinatorial Complexes"},"authors":{"value":[{"fullname":"Jiawen Chen","username":""},{"fullname":"Qi Shao","username":""},{"fullname":"Mingtong Zhou","username":""},{"fullname":"Duxin Chen","username":"~Duxin_Chen1"},{"fullname":"Wenwu Yu","username":""}]}},"tmdate":1784821657255,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2601-20518"],"tcdate":1784821644529,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Duxin_Chen1"],"forum":"kQn8jNNUfT","license":"CC BY-SA 4.0","number":97200,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784821657255,"domain":"OpenReview.net/Public_Article","id":"kQn8jNNUfT","version":2},{"content":{"comment":{"value":"We thank the reviewer for their comments and for valuing the contributions. We address the main concern about real world datasets below by highlighting the extensive evaluations with a 1.5B parameter language model.\n\n**“There is limited evidence on the performance of looped models in real-world applications or datasets…  Authors could scale up the experiments, and validate the effectiveness of the looped model in real-world tasks.”**\n\nA: We would like to direct the reviewer’s attention the results in sections 3 and 4 wherein we trained a 1.5B parameter language model on a real-world language pretraining dataset (Pile), and performed evaluations on 15 real-world benchmark datasets like TriviQA, Squad, and mathematical reasoning datasets like SVAMP and also 4 additional reasoning primitive datasets. Perhaps we misunderstood what the reviewer meant by “real-world applications or datasets” and we are happy to provide a more detailed answer.\nRegarding scaling up experiments further beyond 1.5B, we would like to point out that pre-training experiments are heavily compute hungry (unlike fine-tuning setups which can be more easily performed with 7B sized models) and other leading publications such as “Mamba: Linear Time Sequence Modeling with Selective State Spaces” indeed perform pre-training experiments with similar sized models (up to 2.8B parameters) and similar sized pre-training datasets. We strongly believe that our qualitative insights will extend to larger scale models as well but that would require a significantly larger amount of resources."},"title":{"value":"Author response"}},"tmdate":1732345008747,"tcdate":1732345008747,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12582/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission12582/Authors"],"forum":"din0lGfZFd","number":2,"license":"CC BY 4.0","cdate":1732345008747,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12582/-/Official_Comment"],"mdate":1732345008747,"domain":"ICLR.cc/2025/Conference","replyto":"qcPV95oxE5","id":"pVp5mSAvG3","forumContent":{"TLDR":{"value":"Looped models can solve many reasoning problems and have an inductive bias towards improving reasoning of language models"},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["looped models","reasoning","language model","iterative algorithm","inductive bias","scaling"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Large language models have shown remarkable reasoning abilities and scaling laws suggest that large parameter count, especially along the depth axis, is the primary driver. In this work, we make a stronger claim --- many reasoning problems require a large depth but not necessarily many parameters. This unlocks a novel application of looped models for reasoning. Firstly, we show that for many synthetic reasoning problems like addition, $p$-hop induction, and math problems, a $k$-layer transformer looped $L$ times nearly matches the performance of a $kL$-layer non-looped model, and is significantly better than a $k$-layer model. This is further corroborated by theoretical results showing that many such reasoning problems can be solved via iterative algorithms, and thus, can be solved effectively using looped models with nearly optimal depth. Perhaps surprisingly, these benefits also translate to practical settings of language modeling --- on many downstream reasoning tasks, a language model with $k$-layers looped $L$ times can be competitive to, if not better than, a $kL$-layer language model. In fact, our empirical analysis reveals an intriguing phenomenon: looped and non-looped models exhibit scaling behavior that depends on their effective depth, akin to the inference-time scaling of chain-of-thought (CoT) reasoning. We further elucidate the connection to CoT reasoning by proving that looped models implicitly generate latent thoughts and can simulate $T$ steps of CoT with $T$ loops. Inspired by these findings, we also present an interesting dichotomy between reasoning and memorization, and design a looping-based regularization that is effective on both fronts."},"_bibtex":{"value":"@inproceedings{\nsaunshi2025reasoning,\ntitle={Reasoning with Latent Thoughts: On the Power of Looped Transformers},\nauthor={Nikunj Saunshi and Nishanth Dikkala and Zhiyuan Li and Sanjiv Kumar and Sashank J. Reddi},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=din0lGfZFd}\n}"},"title":{"value":"Reasoning with Latent Thoughts: On the Power of Looped Transformers"},"pdf":{"value":"/pdf/3b6624b631e0aa8b60e9db8890eda637165a01fa.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"saunshi|reasoning_with_latent_thoughts_on_the_power_of_looped_transformers"},"authorids":{"value":["~Nikunj_Saunshi1","~Nishanth_Dikkala1","~Zhiyuan_Li2","~Sanjiv_Kumar1","~Sashank_J._Reddi1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Nikunj Saunshi","Nishanth Dikkala","Zhiyuan Li","Sanjiv Kumar","Sashank J. Reddi"]}},"version":2},{"content":{"comment":{"value":"**[Reply 1/3]**\n\nThank you for your thoughtful review and valuable feedback. Below we address your concerns.\n\n-----\n\n\n\n**[W1&Q1]. A more comprehensive exploration of higher-dimensional token dynamics should be included. [...] Additionally, comparisons with more sophisticated state-space approaches (such as diagonal vs. non-diagonal state space models) could show how the proposed ideas generalize.**\n\n**Answer:** **Regarding the high-dimensional case:** We conjecture that our theorems still hold in the high-dimensional case. In the high-dimensional case, the input-output matrix $S_C^{\\top}S_B$ is a square matrix of size $D \\times D$ and it may have complex eigenvalues. In this case, we consider the symmetric matrix $\\mu:=\\frac{1}{2}(S_C^{\\top}S_B)^{\\top} + \\frac{1}{2}S_C^{\\top}S_B$, which is the symmetric part of the input-output projection matrix $S_C^{\\top}S_B$. This term is motivated by the observation that: the term $x_l^{\\top} \\cdot (S_C^{\\top}S_B) \\cdot x_l$ in the considered dynamical system is a scalar and it can be rewritten as\n$$x_l^{\\top} \\cdot (S_C^{\\top}S_B) \\cdot x_l = \\frac{1}{2}[x_l^{\\top} \\cdot (S_C^{\\top}S_B) \\cdot x_l]^{\\top} + \\frac{1}{2}[x_l^{\\top} \\cdot (S_C^{\\top}S_B) \\cdot x_l] = x_l^{\\top} \\cdot \\mu \\cdot x_l.$$\nSince $\\mu$ is a symmetric matrix, it always has only real eigenvalues. We particularly conjecture that, in the high-dimensional case, all tokens will converge to zero if $\\mu$ has only negative eigenvalues, while all tokens will diverge to infinity if $\\mu$ has at least one positive eigenvalue.\n\nTo illustrate this conjecture, we run simulations in two-dimensional cases and draw the trajectories of tokens in Figures 6 in Appendix B of the revised version. We observe the following:\n\n- In Figure 6A and 6B, the matrix $\\mu$ has at least one positive eigenvalue, and the divergent behavior of the tokens is also observed to persist.\n- In Figure 6C, the matrix $\\mu$ has only negative eigenvalues, and the convergent behavior of the tokens is observed to persist.\n\n**Regarding comparisons with more sophisticated state-space approaches:** As described in Algorithms 1 and 2 on page 6 of [Gu2023], S4 and its variants (diagonal, diagonal plus low-rank, HIPPO) can be regarded as special cases of Mamba when the input matrix $B$, output matrix $C$, and step sizes $\\Delta$ are input-independent. Consequently, our theoretical analysis of Mamba encompasses those for S4 and its variants as special cases.\n\n**References**\n\n[Gu2023] Gu, A., & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752.\n\n**[W2&Q2]. Expanding the experiments to include tasks like **language modeling** (e.g., WIKITEXT or other sequence-based datasets) or **sequence-based tasks** in reinforcement learning would provide stronger evidence that the proposed refinements generalize **beyond vision** tasks. This would substantiate the claim that these refinements are broadly beneficial across domains.**\n\n**Answer:** Thanks for your suggestion. We have expanded our experiment to provide evidence that our proposed method can benefits in a more wide range of tasks. In particular, we have benchmarked on two additional tasks, *language modeling* and *text classification*. \n\nFor language modeling task, we benchmarked on WikiText103 dataset and achieved perplexity reduction of 1.49, as shown in Table 2.\n\n*Table 2: Test Perplexity ($\\downarrow$) on Wikitext-103*\n|**Model**|**Test Perplexity**|\n|-|-|\n|Mamba (baseline)| 16.46|\n|Mamba + reordering| **14.97**|\n\nFor text classification, we benchmark on two datasets, IMDB [I] and AGNews [Ag]. The results in Table 3 indicate that our re-ordering technique further improves the baseline model's performance on these tasks.\n\n*Table 3: Test accuracy ($\\uparrow$) on IMDB and AGNews datasets*\n|**Model**|**IMDB**| **AGNews**|\n|-|-|-|\n|Mamba (baseline)| 88.46|92.08|\n|Mamba + reordering| **88.66**|**92.37**|\n\nThese results suggest that our refinement has the potential to generalize effectively beyond vision-related tasks."}},"tmdate":1732161190755,"tcdate":1732161190755,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5608/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5608/Authors"],"forum":"qtTIP5Gjc5","number":4,"license":"CC BY 4.0","cdate":1732161190755,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5608/-/Official_Comment"],"mdate":1732161190755,"domain":"ICLR.cc/2025/Conference","replyto":"JrrOsu3fFK","id":"nZiE94E7uZ","forumContent":{"TLDR":{"value":"We describe the dynamical properties of tokens in a deep selective state-space model, and based on that, we suggest improvements to the model by excluding negatively impactful scenarios and reordering tokens using importance scores."},"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Selective state-space model","continuous-time limit","dynamical system","asymptotic behavior","token reordering"]},"supplementary_material":{"value":"/attachment/f01b2b77a31c531086f0fefd1143d89ad8415a39.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity.  We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance.  For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training.  Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance.  Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications."},"_bibtex":{"value":"@inproceedings{\nvo2025demystifying,\ntitle={Demystifying the Token Dynamics of Deep Selective State Space Models},\nauthor={Thieu Vo and Duy-Tung Pham and Xin T. Tong and Tan Minh Nguyen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=qtTIP5Gjc5}\n}"},"title":{"value":"Demystifying the Token Dynamics of Deep Selective State Space Models"},"pdf":{"value":"/pdf/abc1efe1f910ee48b743f7ba26000b1d5e622f85.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"vo|demystifying_the_token_dynamics_of_deep_selective_state_space_models"},"authorids":{"value":["~Thieu_Vo1","~Duy-Tung_Pham1","~Xin_T._Tong2","~Tan_Minh_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Thieu Vo","Duy-Tung Pham","Xin T. Tong","Tan Minh Nguyen"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their constructive points, and appreciate the reviewer recognizing the novelty of our perspective on SSMs as gradient accumulators, the soundness of our mathematical theory, and the clarity of our step-by-step theoretical exposition.\n\nResponses to weaknesses:\n\n> While the paper does a good job of explaining the theory and formulation of GD-SSM and empirically validating GD-SSM on regression tasks, I didn't feel like it shed that much insight into the currently used SSM variants used in practice.\n\nWe agree that the constructed approach doesn’t exactly match existing methods, although the use of sliding window attention is becoming more common on highly-performant models [1,2] and 2-D state and input-dependent output computation is pretty standard through gating [3].\nTo explore why Griffin and Mamba do so badly, we explored their performance a bit more (see Table 1 in appendix). We found that adding additional layers (our earlier experiments were with 1-layer), increasing the model dimension, and increasing the number of heads improves performance . We hypothesize that this is due to the fact that the architecture doesn’t exactly match GD, and additional layers are required to play the role of the sliding window attention to perform time-mixing (access adjacent elements in the sequence). \n\nOur construction distils out the exact inductive bias required for ICL.\n\n> Related to the above, the experimental section is light on architectural details, making it hard to determine what exactly is being compared empirically and what conclusions can be drawn. Please include more experimental details including the architectures and hyperparameters.\n\nWe’ve added more details in the Appendix about the details of the experiments and architectural details. Please let us know if any specific aspects still remain unclear.\n\n> The paper is often unclear on the terminology of local self-attention and local linear self-attention. The formulation in Section 3.2 appears to only require local linear self-attention, yet other times in the paper local self-attention is used. In the related works section the two are contrasted. I would recommend being very explicit and consistent on this point as the two are very different.\n\nWe agree with the reviewer, and have changed all references to “Local Self-attention” into \"Sliding Window Attention” which is more explicit.\n\n> The paper is limited to regression style in-context learning. This is interesting and amenable to theoretical analysis, but also limits the impact of the investigation. See https://arxiv.org/abs/2401.12973 and https://arxiv.org/abs/2402.04248 for other papers that have empirically investigated other types of in-context learning in different architectures.\n\nWe agree with the reviewer, and plan to look at other ICL tasks in future work. \n\n[1] S. De et al., ‘Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models’, Feb. 29, 2024, arXiv:2402.19427. doi: 10.48550/arXiv.2402.19427.\n\n[2] L. Ren, Y. Liu, Y. Lu, Y. Shen, C. Liang, and W. Chen, ‘Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling’, Jun. 11, 2024, arXiv:2406.07522. doi: 10.48550/arXiv.2406.07522.\n\n[3] A. Gu and T. Dao, ‘Mamba: Linear-Time Sequence Modeling with Selective State Spaces’, Dec. 01, 2023, arXiv: arXiv:2312.00752. \n\n[4] T. Dao and A. Gu, ‘Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality’, in Proceedings of the 41st International Conference on Machine Learning, PMLR, Jul. 2024, pp. 10041–10071. \n\n[5] J. Park et al., ‘Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks’, Feb. 06, 2024, arXiv: arXiv:2402.04248."},"title":{"value":"thank you for a constructive review"}},"tmdate":1732373863687,"tcdate":1732373863687,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10588/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10588/Authors"],"forum":"52XG8eexal","number":4,"license":"CC BY 4.0","cdate":1732373863687,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10588/-/Official_Comment"],"mdate":1732373863687,"domain":"ICLR.cc/2025/Conference","replyto":"sbOdtDO2zh","id":"meg4z50DJb","forumContent":{"TLDR":{"value":"the mechanism of in-context learning in state-space models corresponds to gradient descent"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["state-space models","in-context learning","linear recurrent networks","mesa-learning"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Deep state-space models (Deep SSMs) have shown capabilities for in-context learning on autoregressive tasks, similar to transformers. \nHowever, the architectural requirements and mechanisms enabling this in recurrent networks remain unclear. \nThis study demonstrates that state-space model architectures can perform gradient-based learning and use it for in-context learning.\nWe prove that a single structured state-space model layer, augmented with local self-attention, can reproduce the outputs of an implicit linear model with least squares loss after one step of gradient descent.\nOur key insight is that the diagonal linear recurrent layer can act as a gradient accumulator, which can be `applied' to the parameters of the implicit regression model.\nWe validate our construction by training randomly initialized augmented SSMs on simple linear regression tasks. The empirically optimized parameters match the theoretical ones, obtained analytically from the implicit model construction. \nExtensions to multi-step linear and non-linear regression yield consistent results.\nThe constructed SSM encompasses features of modern deep state-space models, with the potential for scalable training and effectiveness even in general tasks. \nThe theoretical construction elucidates the role of local self-attention and multiplicative interactions in recurrent architectures as the key ingredients for enabling the expressive power typical of foundation models."},"_bibtex":{"value":"@misc{\nsushma2025statespace,\ntitle={State-space models can learn in-context by gradient descent},\nauthor={Neeraj Mohan Sushma and Yudou Tian and Harshvardhan Mestha and Nicol{\\`o} Colombo and David Kappel and Anand Subramoney},\nyear={2025},\nurl={https://openreview.net/forum?id=52XG8eexal}\n}"},"title":{"value":"State-space models can learn in-context by gradient descent"},"pdf":{"value":"/pdf/79782d0adb25d06310acbb304699a8b0ae8c34ab.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sushma|statespace_models_can_learn_incontext_by_gradient_descent"},"authorids":{"value":["~Neeraj_Mohan_Sushma1","~Yudou_Tian1","~Harshvardhan_Mestha1","~Nicolò_Colombo1","~David_Kappel2","~Anand_Subramoney2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Neeraj Mohan Sushma","Yudou Tian","Harshvardhan Mestha","Nicolò Colombo","David Kappel","Anand Subramoney"]}},"version":2},{"content":{"summary":{"value":"The paper proposes FLAMES, a long-term time-series forecasting model. The FLAMES augments Mamba with three components. (i) multi-scale feature extraction, (ii) Adaptive Fourier Filter Mamba that performs FFT-domain masking and global/local learnable frequency mixing to denoise and enhance periodic structure, and (iii) Laplace transform to capture transient dynamics. Experiments on multiple benchmarks report consistent gains over baselines. The paper also conducts ablations for each component, robustness to synthetic noise, and a scalability analysis to show the superiority of FLAMES."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See the weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"+ The time series forecasting, particularly long-term time series forecasting is an important and practical problem. \n\n+ The evaluation is comprehensive. The experimental results comprehensively cover multiple datasets and horizons with ablations, look-back sensitivity, robustness to noise, and scalability analysis."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The method is a careful combination of known tools (such as Mamba, Fourier filter, and Laplace transform) and rather than a fundamentally new learning principle. The paper would benefit from a more formal analysis combing these modules together.\n\n- The paper does not compare many Mamba-based time series forecaster despite mainly built on Mamba backbone. The only relevant baseline is DTMamba. It is not clear why the authors ignore many other Mamba-based time series forecasting papers, even if they are earlier or have been accepted by decent conferences. Just list some examples as below:\n\n[1] Ahamed et al. TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting. ECAI. 2024\n\n[2] Xu et al. SST: Multi-Scale Hybrid Mamba-Transformer Experts for Long-Short Range Time Series Forecasting. CIKM. 2025\n\n[3] Patro et al. SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series. arxiv. 2024.\n\n[4] Hu et al. Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting. arxiv, 2024."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917359530,"tcdate":1761976212912,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4431/Reviewer_A1tm"],"signatures":["ICLR.cc/2026/Conference/Submission4431/Reviewer_A1tm"],"forum":"eEmTxqxLtU","number":3,"license":"CC BY 4.0","cdate":1761976212912,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4431/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917359530,"domain":"ICLR.cc/2026/Conference","replyto":"eEmTxqxLtU","id":"RG6q8yoZZW","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Time Series Forecasting","Adaptive Fourier Filter","Mamba","Multi- Scale","Laplace Transform"]},"supplementary_material":{"value":"/attachment/c0b5728e9c85ba05bbc40d733bfdf8d5f1b474c9.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Time series data usually exhibit intricate characteristics such as non-stationarity, noise, multi-scale periodicity, and transient dynamics, posing significant challenges to long-term time series forecasting (LTSF). While transformer-based models effectively capture long-range dependencies, their practical applications are hindered by high computational cost with quadratic complexity, noise sensitivity, and overfitting on small datasets. Moreover, time series present distinct patterns at different temporal resolutions, containing both fine-grained (micro) and coarse-grained (macro) information. To address these issues, we propose a novel framework, Flames (multi-scale Fourier Filter Mamba with Laplace), designed for efficient and robust LTSF. Specifically: (i) We introduce an adaptive Fourier filter with a selection module embedded into Mamba. At each scale, the neural operator uses Fourier analysis to refine feature representations, applies learnable thresholds for noise reduction, and captures inter-frequency interactions via global-local semantic filters through element multiplication. (ii) We incorporate the Laplace transform to capture transient dynamics. Extensive experiments on multiple benchmarks demonstrate that Flames consistently outperforms SOTA methods, achieving superior accuracy–efficiency trade-offs. Results highlight its strong robustness and scalability, particularly in noisy or transient settings."},"_bibtex":{"value":"@misc{\nwu2026flames,\ntitle={Flames: Multi-Scale Mamba with Adaptive Fourier Filters and Laplace Transform for Time Series Forecasting},\nauthor={Yuhan Wu and Xiyu Meng and Peiheng Li and Xin Zhang and Peiliang Gong and Canran Xiao and Yabo Dong and Dongming Lu},\nyear={2026},\nurl={https://openreview.net/forum?id=eEmTxqxLtU}\n}"},"title":{"value":"Flames: Multi-Scale Mamba with Adaptive Fourier Filters and Laplace Transform for Time Series Forecasting"},"pdf":{"value":"/pdf/307b599472799e0765bbdf39d7638180867a1b17.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wu|flames_multiscale_mamba_with_adaptive_fourier_filters_and_laplace_transform_for_time_series_forecasting"},"authorids":{"value":["~Yuhan_Wu6","~Xiyu_Meng1","~Peiheng_Li2","~Xin_Zhang33","~Peiliang_Gong1","~Canran_Xiao1","~Yabo_Dong1","~Dongming_Lu1"]},"authors":{"value":["Yuhan Wu","Xiyu Meng","Peiheng Li","Xin Zhang","Peiliang Gong","Canran Xiao","Yabo Dong","Dongming Lu"]}},"version":2},{"content":{"comment":{"value":"Similar to Mamba, we have chosen the selectivity of the proposed bidirectional model based on the connection to ZOH discretization, and our results demonstrate that it works for bidirectional sequence modeling as well. Our claim is not that we invented this approach, but rather that it is a traditional method for sampling continuous signals, which has been successfully used in models like Mamba and other SSMs for causal sequence modeling. We are therefore showing that using LION with this approach also works for well-known bidirectional sequence modeling tasks, such as vision and masked language modeling.\n\n---\n\n**3: No Position Encodings required is a consequence of using SSMs**\n\nWe would like to emphasize again that the primary contribution of LION is extending causal Linear Transformers and their selective variants into a bidirectional setting. In this setting, the model trains like a Transformer using attention and infers like an RNN/SSM. As a result, LION leverages the advantages of SSMs/RNNs during inference, such as the elimination of the need for positional embeddings. While we do not claim this as the novelty of our work, we demonstrate that because LION is equivalent to an RNN during inference, it inherently possesses these advantages.\n\nLION train using attention parallelization (which is faster than scan algorithm, please see our response to question 5) while benefiting from the memory and computational efficiency of RNNs/SSMs during inference. Our experimental results provide evidence that, if the model is equivalent to an RNN during inference, it should inherit the advantages of RNNs/SSMs, such as not requiring positional embeddings and the ability to extrapolate. \n\nWe demonstrate that LION and its selective variant LION-S (inspired by Mamba) indeed exhibit these features. Aside from positional embeddings, details about efficiency of LION are also provided at our response to concern 3 of reviewer sped.\n\n---\n\n**4: Practicality of Recurrent Inference**\n\nThanks for the comment. We would like to clarify that techniques like chunking can also be applied to LION. Since LION uses a scalar-valued state parameter, it is $d$ times more memory-efficient than Mamba and comparable to Mamba-2 with a scalar value $a_i$. This efficiency allows LION to achieve $O(Ld)$ memory requirements without needing chunking. However, chunking can still be applied for parallelization and further acceleration during inference when memory demands are high.\n\nAs you can see, LION enables Linear Transformers to be extended into a bidirectional setting, *while retaining all the advantages of linear attention during inference*, such as chunk parallelization, and allowing for parallelized training using attention.\n\nThe key advantage of LION over other SSMs that use parallel scan is that LION leverages attention parallelism during training, which is significantly faster than parallel scan, as shown in the table in our response to Question 5 of your review. This results in LION being much faster to train compared to parallel scan-based approaches.\n\n\n---"},"title":{"value":"Authors Rebuttal to Reviewer tj5X (2/3)"}},"tmdate":1731945417153,"tcdate":1731878576122,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13771/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13771/Authors"],"forum":"VxJjw52dZu","number":3,"license":"CC BY 4.0","cdate":1731878576122,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13771/-/Official_Comment"],"mdate":1731945417153,"domain":"ICLR.cc/2025/Conference","replyto":"JuT21G6yPS","id":"JJBOiX0WrH","forumContent":{"TLDR":{"value":"We provide a bidirectional selective recurrent form of full kernelized attention with learnable mask allowing scalability in context length and superior inference efficiency"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Self-attention","transformer","bidirectionality","mamba","ssm","rnn","inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We introduce LION, a novel sequence-to-sequence framework that unifies the bidirectionality and parallelized training of Transformers with the fast inference of recurrent neural networks. LION is built upon a mathematical formulation where full kernelized attention with a learnable mask is efficiently computed using a bidirectional selective recurrent model, matching the effectiveness of softmax-based attention with constant-time inference. Our framework naturally accounts for spatial and temporal relationships within input sequences, reducing reliance on heuristic positional embeddings and facilitating straightforward scalability in context length and resolution. Using our framework and inspired by the recent state-space models, we propose three main running examples LIOn-LIT, LION-RETNET, and LION-S, a transformer with selective mask and recurrent inference. Numerical evaluations on tasks such as language modeling, the Long-Range Arena, and image classification show that LION framework achieves performance on par with state-of-the-art models while delivering fast training and inference efficiency."},"_bibtex":{"value":"@misc{\nafzal2025lion,\ntitle={{LION}: A bidirectional framework that trains like a Transformer and infers like an {RNN}},\nauthor={Arshia Afzal and Elias Abad Rocamora and Leyla Naz Candogan and Pol Puigdemont and Francesco Tonin and Yongtao Wu and Mahsa Shoaran and Volkan Cevher},\nyear={2025},\nurl={https://openreview.net/forum?id=VxJjw52dZu}\n}"},"title":{"value":"LION: A bidirectional framework that trains like a Transformer and infers like an RNN"},"pdf":{"value":"/pdf/6e8b8585752d6f27da24ffe876859459a5b56efe.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"afzal|lion_a_bidirectional_framework_that_trains_like_a_transformer_and_infers_like_an_rnn"},"authorids":{"value":["~Arshia_Afzal1","~Elias_Abad_Rocamora1","~Leyla_Naz_Candogan1","~Pol_Puigdemont1","~Francesco_Tonin1","~Yongtao_Wu1","~Mahsa_Shoaran1","~Volkan_Cevher1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Arshia Afzal","Elias Abad Rocamora","Leyla Naz Candogan","Pol Puigdemont","Francesco Tonin","Yongtao Wu","Mahsa Shoaran","Volkan Cevher"]}},"version":2},{"content":{"summary":{"value":"The paper introduces **Masked Autoregressive Pre-training (MAP)**, a pre-training method designed to improve the performance of a hybrid **Mamba-Transformer** vision backbone. While **Mamba** offers advantages in long-context modeling for vision tasks, it struggles with scalability in large parameter settings. Traditional pre-training techniques like **Masked Auto Encoder (MAE)** work well for Transformers but show limited success with Mamba. The proposed MAP technique combines MAE and autoregressive pre-training to improve both Mamba and Transformer components' performance in a unified framework. Experimental results show that MAP-pretrained hybrid models outperform other techniques in both 2D and 3D tasks across various datasets, indicating that MAP is highly effective in enhancing the Mamba-Transformer hybrid model’s performance."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please refer to the weakness part."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. **Innovative Hybrid Pre-training Strategy**: The proposed MAP approach effectively combines MAE and autoregressive pre-training, which promisingly addresses the limitations of existing pre-training methods for Mamba-based backbones.\n2. **Comprehensive Experimental Validation**: The authors validate the effectiveness of different pre-training paradigm through extensive experiments on 2D and 3D tasks (e.g., ImageNet-1K, ModelNet40, and ShapeNetPart), showing competitive results.\n3. **Scalability and Adaptability**: MAP works for hybrid architectures (Mamba and Transformer) and is promising to be applied across multiple domains.\n4. **Detailed Analysis and Ablation Studies**: The paper includes in-depth studies on various factors like scanning orders, masking ratios, and layer arrangements, providing insights into the specific impacts of these design choices on model performance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Limited Other Tasks Validation**: Most 2D vision experiments are conducted on standard IN1K datasets, I wonder if the downstream dense prediction tasks, i.e., classification, detection, could benefit from the proposed pre-training techniques.\n2. **Out-of-data Citation**: The paper references preprints and arXiv versions of significant works, such as Mamba (COLM), Vision Mamba (ICML), and VMamba (NeurIPS). The authors should update these citations to their final published versions to reflect the current state of the literature."}},"nonreaders":[],"tmdate":1731427595254,"tcdate":1730991933574,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2428/Reviewer_8rdL"],"signatures":["ICLR.cc/2025/Conference/Submission2428/Reviewer_8rdL"],"forum":"hBVywPrw6d","number":5,"license":"CC BY 4.0","cdate":1730991933574,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2428/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427595254,"domain":"ICLR.cc/2025/Conference","replyto":"hBVywPrw6d","id":"NCuFD4SlFB","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"TLDR":{"value":"We propose a new pre-training strategy tailored for the Mamba-Transformer backbone."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Hybrid Vision Backbone","Masked Autoregressive","Self-supervised Learning"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has achieved significant advantages in long-context modeling and autoregressive tasks, but its scalability with large parameters remains a major limitation in vision applications. pretraining is a widely used strategy to enhance backbone model performance. Although the success of Masked Autoencoder in Transformer pretraining is well recognized, it does not significantly improve Mamba's visual learning performance. We found that using the correct autoregressive pretraining can significantly boost the performance of the Mamba architecture. Based on this analysis, we propose Masked Autoregressive Pretraining(MAP) to pretrain a hybrid Mamba-Transformer vision backbone network. This strategy combines the strengths of both MAE and Autoregressive pretraining, improving the performance of Mamba and Transformer modules within a unified paradigm. Experimental results show that both the pure Mamba architecture and the hybrid Mamba-Transformer vision backbone network pretrained with MAP significantly outperform other pretraining strategies, achieving state-of-the-art performance. We validate the effectiveness of the method on both 2D and 3D datasets and provide detailed ablation studies to support the design choices for each component."},"_bibtex":{"value":"@misc{\nliu2024map,\ntitle={{MAP}: Unleashing Hybrid Mamba-Transformer Vision Backbone{\\textquoteright}s Potential with Masked Autoregressive Pretraining},\nauthor={Yunze Liu and Li Yi},\nyear={2024},\nurl={https://openreview.net/forum?id=hBVywPrw6d}\n}"},"title":{"value":"MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone’s Potential with Masked Autoregressive Pretraining"},"pdf":{"value":"/pdf/10243949177ada96d6426f1e3d852f7611be953b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"liu|map_unleashing_hybrid_mambatransformer_vision_backbones_potential_with_masked_autoregressive_pretraining"},"authorids":{"value":["~Yunze_Liu2","~Li_Yi2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yunze Liu","Li Yi"]}},"version":2},{"content":{"summary":{"value":"This paper proposes DriveMamba, a Task-Centric Scalable State Space Model for efficient end-to-end autonomous driving. The core innovation lies in replacing the traditional attention-based Transformer architecture with a Unified Mamba Decoder, which jointly models perception, prediction, and planning in a single-stage pipeline with linear complexity. The authors further introduce several technical components — Hybrid Spatiotemporal Scan (HSS), task-centric tokenization, 3D sensor token localization, and long-term memory fusion — aiming to enhance scalability, efficiency, and task-level relational modeling. Experiments on nuScenes and Bench2Drive demonstrate consistent performance improvements in L2 and collision metrics, with notable inference speed."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. How is the trajectory prior in Trajectory-L2G obtained? Any risk of future information leakage?\n2. How sensitive is the method to depth noise or calibration errors?\n3. Could the authors clarify the HSS layer configuration and its effect on stability?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Unified Mamba Decoder: Achieves linear complexity while jointly processing perception, map, and planning queries, showing clear scalability advantages on high-resolution multi-camera inputs.\n2. Hybrid Spatiotemporal Scan (HSS): Cleverly alternates between spatial and ego-centric scanning to balance locality preservation and long-range temporal consistency.\n3. Task-centric tokenization: Structured query design (ego/map/agent) improves modular interpretability and relational learning.\n4. 3D sensor token localization: Replacing uniform ray sampling with depth-predicted projection enhances geometric accuracy and spatial reasoning.\n5. Comprehensive experiments: Covers multiple benchmarks and provides results with and without ego-status, supporting generalizability and robustness claims."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Insufficient HSS details: The paper lacks explicit layer-wise configurations or stability studies when varying scan order; the contribution of each H/V-first and L2G layer is not isolated.\n2. FPS reporting: Experimental FPS comparisons are unclear due to missing details on resolution, camera count, and hardware setup.\n3. Depth branch robustness: No analysis of depth estimation noise, calibration error, or trade-offs between uniform-ray and learned-depth methods.\n4. Trajectory prior ambiguity: The source of trajectory guidance (e.g., ego-pose history vs. future leakage) is not clarified, raising potential fairness concerns.\n5. Limited interpretability of task relations: The “shared Task Query B-Mamba” strategy lacks ablation or visualization, leaving its contribution unclear.\n6. Efficiency gap: The Large model exhibits a steep FPS drop, suggesting scalability bottlenecks at high model capacity."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922095133,"tcdate":1761936893239,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10886/Reviewer_pyiN"],"signatures":["ICLR.cc/2026/Conference/Submission10886/Reviewer_pyiN"],"forum":"MY0NHvqzi2","number":3,"license":"CC BY 4.0","cdate":1761936893239,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10886/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922095133,"domain":"ICLR.cc/2026/Conference","replyto":"MY0NHvqzi2","id":"kwezr7VL7Y","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Autonomous Driving","Task-Centric Paradigm","Scalable State Space Model"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Recent advances towards End-to-End Autonomous Driving (E2E-AD) focus on integrating modular designs into a unified framework for joint optimization. Most of these advances follow a sequential paradigm (i.e., perception-prediction-planning) based on separable Transformer decoders and rely on dense BEV features to encode scene representations. However, such manual ordering design can inevitably cause information loss and cumulative errors, lacking flexible and diverse relation modeling among different modules and sensors. Meanwhile, insufficient training of image backbone and quadratic-complexity of attention mechanism also hinder the scalability and efficiency of E2E-AD system to handle spatiotemporal input. To this end, we propose DriveMamba, a Task-Centric Scalable paradigm for efficient E2E-AD, which integrates dynamic task relation modeling, implicit view correspondence learning and long-term temporal fusion into a single-stage Unified Mamba decoder. Specifically, both extracted image features and expected task outputs are converted into token-level sparse representations in advance, which are then sorted by their instantiated positions in 3D space. The linear-complexity operator enables efficient long-context sequential token modeling to capture task-related inter-dependencies simultaneously. Additionally, a bidirectional trajectory-guided \"local-to-global\" scan method is designed to preserve spatial locality from ego-perspective, thus facilitating the ego-planning. Extensive experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superiority, generalizability and great efficiency of DriveMamba."},"_bibtex":{"value":"@inproceedings{\nsu2026drivemamba,\ntitle={DriveMamba: Task-Centric Scalable State Space Model for Efficient End-to-End Autonomous Driving},\nauthor={Haisheng Su and Wei Wu and Feixiang Song and Junjie Zhang and Zhenjie Yang and Junchi Yan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=MY0NHvqzi2}\n}"},"title":{"value":"DriveMamba: Task-Centric Scalable State Space Model for Efficient End-to-End Autonomous Driving"},"pdf":{"value":"/pdf/b3b5c4558ca37cf63ba4e2cf32a91db9148cf95a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"su|drivemamba_taskcentric_scalable_state_space_model_for_efficient_endtoend_autonomous_driving"},"authorids":{"value":["~Haisheng_Su1","~Wei_Wu14","~Feixiang_Song1","~Junjie_Zhang9","~Zhenjie_Yang1","~Junchi_Yan2"]},"authors":{"value":["Haisheng Su","Wei Wu","Feixiang Song","Junjie Zhang","Zhenjie Yang","Junchi Yan"]}},"version":2},{"content":{"summary":{"value":"This paper proposes **MAVE**, a hybrid architecture leveraging a Mamba-based state-space sequence model enhanced with cross-attention for efficient, high-fidelity, and context-aware speech editing and zero-shot text-to-speech (TTS) generation. MAVE models long-range dependencies in audio tokens via Mamba, while dynamically aligning text and audio through cross-attention on phoneme embeddings, enabling bidirectional and precise speech modifications. The empirical evaluation on standard benchmarks demonstrates performance gains in naturalness, intelligibility, and speaker consistency compared to contemporary baselines such as VoiceCraft and FluentSpeech."},"soundness":{"value":1},"confidence":{"value":5},"questions":{"value":"N/A"},"rating":{"value":0},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"1. The integration of Mamba SSMs with a cross-attention mechanism tailored for audio-text alignment addresses the quadratic inefficiency of transformer-based decoders in long speech generation."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Lack of novelty. This work is essentially a copy of VoiceCraft in every aspect — design, writing, and overall structure — with the only difference being that the model architecture is changed from Transformer to Mamba.\n\n2. A severe lack of baselines, such as F5-TTS and MaskGCT, which both support TTS and editing."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923385952,"tcdate":1761539480901,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12518/Reviewer_zYYp"],"signatures":["ICLR.cc/2026/Conference/Submission12518/Reviewer_zYYp"],"forum":"yEKyIwJvvl","number":1,"license":"CC BY 4.0","cdate":1761539480901,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12518/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923385952,"domain":"ICLR.cc/2026/Conference","replyto":"yEKyIwJvvl","id":"VrGZHpi1PU","forumContent":{"TLDR":{"value":"MAVE is a cross-attentive Mamba model for voice editing and zero-shot text-to-speech that produces more natural, high-fidelity speech while being faster and more memory-efficient than prior methods."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mamba","cross-attention","voice editing","zero-shot TTS","autoregressive model","structured state-space model","speech synthesis"]},"supplementary_material":{"value":"/attachment/4d48284223f8b139bb3b786a23dbae41a4b9efce.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"We introduce $\\textbf{MAVE}$ ($\\textbf{M}$amba with Cross-$\\textbf{A}$ttention for $\\textbf{V}$oice $\\textbf{E}$diting and Synthesis), a novel autoregressive architecture for text-conditioned voice editing and high-fidelity text-to-speech (TTS) synthesis, built on a cross-attentive Mamba backbone. MAVE achieves state-of-the-art performance in speech editing and very competitive results in zero-shot TTS, while not being explicitly trained on the latter task, outperforming leading autoregressive and diffusion models on diverse, real-world audio. By integrating Mamba for efficient audio sequence modeling with cross-attention for precise text-acoustic alignment, MAVE enables context-aware voice editing with exceptional naturalness and speaker consistency. In pairwise human evaluations on a random 40-sample subset of the RealEdit benchmark (400 judgments), 57.2\\% of listeners rated MAVE-edited speech as perceptually equal to the original, while 24.8\\% prefered the original and 18.0\\% MAVE- demonstrating that in the majority of cases edits are indistinguishable from the source. MAVE compares favorably with VoiceCraft and FluentSpeech both on pairwise comparisons and standalone mean opinion score (MOS) evaluations. For zero-shot TTS, MAVE exceeds VoiceCraft in both speaker similarity and naturalness, without requiring multiple inference runs or post-processing. Remarkably, these quality gains come with a significantly lower memory cost and approximately the same latency: MAVE requires $\\sim6\\times$ less memory than VoiceCraft during inference on utterances from the RealEdit database (mean duration: 6.21s, A100, FP16, batch size 1). Our results demonstrate that MAVE establishes a new standard for flexible, high-fidelity voice editing and synthesis through the synergistic integration of structured state-space modeling and cross-modal attention."},"_bibtex":{"value":"@misc{\nmohammad2026speak,\ntitle={Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot {TTS} with Cross-Attentive Mamba},\nauthor={Baher Mohammad and Magauiya Zhussip and Stamatios Lefkimmiatis},\nyear={2026},\nurl={https://openreview.net/forum?id=yEKyIwJvvl}\n}"},"title":{"value":"Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba"},"pdf":{"value":"/pdf/f8c081b3d8aeb597edeb63bf7efb90df2bbd6909.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"mohammad|speak_edit_repeat_highfidelity_voice_editing_and_zeroshot_tts_with_crossattentive_mamba"},"authorids":{"value":["~Baher_Mohammad2","~Magauiya_Zhussip1","~Stamatios_Lefkimmiatis1"]},"authors":{"value":["Baher Mohammad","Magauiya Zhussip","Stamatios Lefkimmiatis"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a novel medical image segmentation framework called “Uncertainty-Driven Mamba.” The authors propose an Uncertainty-Driven Selective Scanning Block and an Uncertainty-Driven Selective Scanning Optimization Strategy to enhance Mamba-based segmentation methods. Experimental results imply that this framework achieves good performance."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See weakness."},"rating":{"value":5},"details_of_ethics_concerns":{"value":"None"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The writing is well and easy to follow.\n2. The idea of Uncertainty-Driven Mamba is interesting.\n3. The code is available (on supplementary)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The compared methods used in this paper are outdated. With the exception of Mamba-UNet on arXiv, all other methods were **originally** proposed in 2021 or earlier. At least the authors should include comparisons with some more recent, well-cited medical segmentation methods, such as SS-Former [*1], H2Former [*2], PVT-CASCADE [*3], Medsegdiff-v2 [*4].\n2. The results appear to lack reproducibility. For each dataset, it seems that the authors use their own **random** dataset splits instead of following the widely adopted standard benchmarks. For instance, in skin lesion segmentation, most studies (such as [*5, *6]) employ a 7:3 split for training and testing data, whereas this paper uses a customised 7:1:2 split. Given that these standard splits are publicly available, there seems to be no need to deviate from widely used benchmarks.\n3. As the title refers to “medical image segmentation,” the readers would like to see results on datasets that have also been widely evaluated in other studies [*1, *2, *3]. Please consider using the polyp segmentation benchmark proposed in [*7].\n4. Efficiency is important in medical image segmentation. Please provide details on the parameters, FLOPs, and FPS of the different methods for a comprehensive comparison.\n5. In the last column of Figure 5, the results of UD-Mamba is also not good (although a little better than other methods).\n\n[*1] Stepwise Feature Fusion: Local Guides Global, MICCAI 2022\n\n[*2] H2Former: An Efficient Hierarchical Hybrid Transformer for Medical Image Segmentation, IEEE TMI 2023\n\n[*3] Medical Image Segmentation via Cascaded Attention Decoding, WACV 2023\n\n[*4] Medsegdiff-v2: Diffusion-based Medical image Segmentation with Transformer, AAAI 2024\n\n[*5] MALUNet: A Multi-Attention and Light-weight UNet for Skin Lesion Segmentation, IEEE BIBM 2022\n\n[*6] EGE-UNet: An Efficient Group Enhanced UNet for Skin Lesion Segmentation, MICCAI 2023\n\n[*7] Pranet: Parallel Reverse Attention Network for Polyp Segmentation, MICCAI 2020"}},"nonreaders":[],"tmdate":1731428031847,"tcdate":1730552478524,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4085/Reviewer_iHXH"],"signatures":["ICLR.cc/2025/Conference/Submission4085/Reviewer_iHXH"],"forum":"OuUKXhV2Uz","number":3,"license":"CC BY 4.0","cdate":1730552478524,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4085/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428031847,"domain":"ICLR.cc/2025/Conference","replyto":"OuUKXhV2Uz","id":"Cp7qWMET8w","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Medical Image Segmentation","State Space Models","Mamba","Uncertainty"]},"supplementary_material":{"value":"/attachment/f82417d94f029ddea0474ec6f42029895c698965.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent advancements have highlighted the Mamba framework, a state-space models (SSMs) known for its efficiency in capturing long-range dependencies with linear computational complexity. While Mamba has shown competitive performance in medical image segmentation, it encounters difficulties in modeling local features due to the sporadic nature of traditional location-based scanning methods and the complex, ambiguous boundaries often present in medical images. To overcome these challenges, we propose Uncertainty-Driven Mamba (UD-Mamba), which redefines the pixel-order scanning process by incorporating channel uncertainty into the scanning mechanism. UD-Mamba introduces two key scanning techniques: sequential scanning, which prioritizes regions with high uncertainty by scanning in a row-by-row fashion, and skip scanning, which processes columns vertically, moving from high-to-low or low-to-high uncertainty at fixed intervals. Sequential scanning efficiently clusters high-uncertainty regions, such as boundaries and foreground objects, to improve segmentation precision, while skip scanning enhances the interaction between background and foreground regions, allowing for timely integration of background information to support more accurate foreground inference. Recognizing the advantages of scanning from certain to uncertain areas, we introduce four learnable parameters to balance the importance of features extracted from different scanning methods. Additionally, a cosine consistency loss is employed to mitigate the drawbacks of transitioning between uncertain and certain regions during the scanning process. Our method demonstrates robust segmentation performance, validated across three distinct medical imaging datasets involving pathology, dermatological lesions, and cardiac tasks."},"_bibtex":{"value":"@misc{\nzhao2024udmamba,\ntitle={{UD}-Mamba: A pixel-level uncertainty-driven mamba model for medical image segmentation},\nauthor={Weiren zhao and Feng Wang and Yutong Xie and Qi Wu and Yuyin Zhou},\nyear={2024},\nurl={https://openreview.net/forum?id=OuUKXhV2Uz}\n}"},"title":{"value":"UD-Mamba: A pixel-level uncertainty-driven mamba model for medical image segmentation"},"pdf":{"value":"/pdf/b9e806a7c0308814b02666e4d924df4fcf6e4b3a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhao|udmamba_a_pixellevel_uncertaintydriven_mamba_model_for_medical_image_segmentation"},"authorids":{"value":["~Weiren_zhao1","~Feng_Wang14","~Yutong_Xie4","~Qi_Wu3","~Yuyin_Zhou1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Weiren zhao","Feng Wang","Yutong Xie","Qi Wu","Yuyin Zhou"]}},"version":2},{"content":{"summary":{"value":"This paper develops a new parameterization scheme to help improve training of state space models (SSMs) in sequence modeling. The new parameterization method is based on Hankel operator theory, which resolves the low-rank and numerical instability issues of an LTI system, and the parameterized SSMs could enjoy non-decaying memory. Numerical experiments are conducted on the long range arena benchmark."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"* Why do the authors choose to show the Hankel singular values of SSMs trained after 10 epochs instead of after the whole training process (e.g. Figure 3 & 5)?\n\n* How do the authors validate the assumption on the distribution of the diagonal entry of $\\bar{A}$ in Theorem 1? For linear recurrent unit [1], $\\bar{A}$ is initialized such that $a_j$ is uniformly distributed in a disc $[r_{\\min}, r_{\\max}]$ where $r_{\\min}, r_{\\max}$ are both close to $1$.\n\n* Also for Theorem 1, the theoretical bound does not indicate $\\beta < 1$, but $\\beta \\leq 1$ instead. Is there a way to mitigate the logarithmic term in theory?\n\n\n\n\n\n[1] Orvieto, Antonio, et al. \"Resurrecting recurrent neural networks for long sequences.\" International Conference on Machine Learning. PMLR, 2023."},"rating":{"value":6},"details_of_ethics_concerns":{"value":"NA"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"* The paper is written clearly and the ideas have been presented in the proper order.\n\n* The idea using Hankel operator theory to parameterize state space models is novel, which can provide more insights for other recurrent architectures on sequence modeling."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The connections of the Hankel rank/stability and the SSM performance seem not that positively correlated in this paper. For example, from Figure we see that the numerical rank of HOPE-SSM is even lower than init3 before and after training, but these two initializations produce similar numerical performance on the sCIFAR-10 task. Does it mean that even slightly low rank Hankel matrix can achieve good results? \n\n* From Algorithm1, the Hankel operator method looks very similar to the convolution method, it would be better if the authors could add more discussion on these two methods. For example, [1] can be a good reference. \n\n* The notations for $A$ and $\\bar{A}$ are not consistently used in this paper. From my understanding, $A$ is the hidden matrix for the continuous system and $\\bar{A}$ is the hidden matrix for the discrete system. The HiPPO-Legs initialization is for $A$ matrix, and $\\bar{A}$ contains the information of the timescale $\\Delta t$. It would be better to distinguish these two notations when making statements. For example, in Figure 4, the perturbation is added to the continuous system while the Hankel singular values are calculated  for the discrete system, and the discrete system contains the information for $\\Delta t$. It seems that $\\Delta t$ is set to be $1$ in this paper, but in practice, the timescale is scaled based on the sequence length. So what is the reason to take $\\Delta t = 1$?\nAlso, it would be better if the authors could add more details on how the perturbation is added here.\n\n[1] Li, Yuhong, et al. \"What Makes Convolutional Models Great on Long Sequence Modeling?.\" The Eleventh International Conference on Learning Representations."}},"nonreaders":[],"tmdate":1732550795621,"tcdate":1730382193396,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8704/Reviewer_1q8g"],"signatures":["ICLR.cc/2025/Conference/Submission8704/Reviewer_1q8g"],"forum":"RZwtbg3qYD","number":2,"license":"CC BY 4.0","cdate":1730382193396,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8704/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732550795621,"domain":"ICLR.cc/2025/Conference","replyto":"RZwtbg3qYD","id":"YE5ABwHAHR","forumContent":{"TLDR":{"value":"We introduce a novel parameterization for state-space models (SSMs) based on Hankel operator theory, which enhances the initialization and training stability and significantly improves long-term memory retention in our models."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["state space model","sequence modeling","Long-Range Arena","long memory"]},"supplementary_material":{"value":"/attachment/4c41f7c31373b806742d9bcee1461399d9611102.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State-space models (SSMs) that utilize linear, time-invariant (LTI) systems are known for their effectiveness in learning long sequences. To achieve state-of-the-art performance, an SSM often needs a specifically designed initialization, and the training of state matrices is on a logarithmic scale with a very small learning rate. To understand these choices from a unified perspective, we view SSMs through the lens of Hankel operator theory. Building upon it, we develop a new parameterization scheme, called HOPE, for LTI systems that utilizes Markov parameters within Hankel operators. Our approach helps improve the initialization and training stability, leading to a more robust parameterization. We efficiently implement these innovations by nonuniformly sampling the transfer functions of LTI systems, and they require fewer parameters compared to canonical SSMs. When benchmarked against HiPPO-initialized models such as S4 and S4D, an SSM parameterized by Hankel operators demonstrates improved performance on Long-Range Arena (LRA) tasks. Moreover, our new parameterization endows the SSM with non-decaying memory within a fixed time window, which is empirically corroborated by a sequential CIFAR-10 task with padded noise."},"_bibtex":{"value":"@inproceedings{\nyu2025hope,\ntitle={{HOPE} for a Robust Parameterization of Long-memory State Space Models},\nauthor={Annan Yu and Michael W. Mahoney and N. Benjamin Erichson},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=RZwtbg3qYD}\n}"},"title":{"value":"HOPE for a Robust Parameterization of Long-memory State Space Models"},"pdf":{"value":"/pdf/edcb58513f269aca339bd4aeca68f4610487f0f0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"yu|hope_for_a_robust_parameterization_of_longmemory_state_space_models"},"authorids":{"value":["~Annan_Yu1","~Michael_W._Mahoney1","~N._Benjamin_Erichson1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Annan Yu","Michael W. Mahoney","N. Benjamin Erichson"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Linear Attention Sequence Parallelism (LASP), a sequence parallelism (SP) approach tailored for linear attention-based transformer models. By exploiting the right-product kernel trick of linear attention, LASP aims to reduce communication overheads associated with traditional SP approaches. The authors propose a point-to-point ring-style communication mechanism, kernel fusion, and KV state caching to make LASP efficient and hardware-friendly, particularly on GPUs. Their approach is compatible with various batch-level data parallel methods, allowing for scalability in distributed training on long sequences across large clusters. \n\nThe paper presents a potentially useful approach to sequence parallelism for linear attention models, targeting an important aspect of long-sequence modeling. However, the limited novelty, lack of clarity on specific contributions, and insufficient comparative analysis reduce the impact of this work. To strengthen its case, the authors could clarify the unique contributions of LASP, elaborate on its motivation, and provide a more thorough comparison with alternative solutions. A stronger focus on the limitations of existing SP methods and LASP’s distinctive benefits would help clarify its position and contribution within the broader landscape of sequence parallelism techniques."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- Could the authors clarify the changes made to TNL and Linear Transformer models to integrate LASP? When compared with other methods, such as Ring Attention, DeepSpeed-Ulysses, and Megatron-SP, the models used lack native support in those frameworks. What would be required to add right-product support in these methods, and how would this impact the performance?\n- The paper claims that LASP can be seamlessly integrated into other linear attention models. Could the authors expand on the practical challenges of integrating LASP into different models? Understanding these integration requirements would help clarify LASP’s unique advantages and challenges.\n- In scenarios where LASP is employed, what specific limitations of traditional SP methods are being overcome? For example, does the ring-style communication offer benefits that cannot be achieved by alternative SP configurations with minor modifications? Detailing these trade-offs would better motivate LASP’s design decisions."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The paper seeks to address an important problem in handling long sequence lengths in transformer models, which is a practical need in scaling LLMs for extended contexts.\n- The paper presents a solid system design and implementation recipe, which is beneficial for practitioners working on distributed training with long sequences."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- While the paper introduces LASP as a specialized SP method for linear attention, it lacks clarity on what are its unique contributions. The proposed approach largely integrates known techniques (e.g., ring-style communication, kernel fusion, and KV cache), but it remains unclear if LASP introduces novel elements beyond combining these strategies. The synergy of linear attention sequence parallelism is not fully explained, leaving it open to interpretation as a simple integration of existing techniques. This lack of clarity on the novelty makes it challenging to gauge the paper’s contribution to the field, especially in comparison with alternative methods.\n- The motivation behind LASP’s specific design choices is not well-developed. It remains unclear why existing SP methods are insufficient for linear attention or why LASP’s approach is inherently better. Additionally, the paper lacks evidence on how significantly LASP improves over existing approaches in real-world scenarios.\n- The experiments primarily demonstrate LASP’s effectiveness with linear attention-based models like TNL and Linear Transformers, yet these models are not natively supported in the alternative SP methods (e.g., Ring Attention, DeepSpeed-Ulysses). A thorough comparison with alternative SP techniques like Ring Attention, DeepSpeed-Ulysses, and Megatron-SP would be helpful, given that these methods, although not natively designed for linear attention, may be modified to incorporate the right-product kernel trick and still achieve competitive results. The paper does not consider these alternatives, which raises questions about the practical advantage of LASP over simpler modifications to existing methods."}},"nonreaders":[],"tmdate":1731428086141,"tcdate":1730655989642,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5885/Reviewer_a2Nu"],"signatures":["ICLR.cc/2025/Conference/Submission5885/Reviewer_a2Nu"],"forum":"oVnfVnwh6y","number":4,"license":"CC BY 4.0","cdate":1730655989642,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5885/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428086141,"domain":"ICLR.cc/2025/Conference","replyto":"oVnfVnwh6y","id":"4DsqfYURcV","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["sequence parallelism","distributed training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Sequence parallelism (SP) serves as a prevalent strategy to handle long sequences that exceed the memory limit of a single device. However, for linear sequence modeling methods like linear attention, existing SP approaches do not take advantage of their right-product-first feature, resulting in sub-optimal communication efficiency and usability. In this paper, we introduce Linear Attention Sequence Parallelism (LASP), an efficient SP approach designed for linear attention-based transformer models. Specifically, we design an efficient point-to-point ring-style communication mechanism to leverage the right-product kernel trick of linear attention, which sharply decreases the communication overhead, comparing with existing SP methods. We enhance the computation efficiency of LASP by performing kernel fusion and intermediate state caching, making the implementation of LASP hardware-friendly on GPUs. Furthermore, we meticulously ensure the compatibility of sequence-level LASP with all types of batch-level data parallel methods, which is vital for distributed training on large clusters with very-long sequences. We also discuss the generalization of LASP on other linear sequence modeling methods. Extensive experiments on linear attention-based models are conducted with varying sequence lengths from 2K to 4096K. LASP scales sequence length up to 4096K on 128 GPUs, which is 8$\\times$ longer than existing SP methods."},"_bibtex":{"value":"@misc{\nsun2025linear,\ntitle={Linear Attention Sequence Parallelism},\nauthor={Weigao Sun and Zhen Qin and Dong Li and Xuyang Shen and Yu Qiao and Yiran Zhong},\nyear={2025},\nurl={https://openreview.net/forum?id=oVnfVnwh6y}\n}"},"title":{"value":"Linear Attention Sequence Parallelism"},"pdf":{"value":"/pdf/91f0bb8ec553fe88b189a2550d301b39cef48faa.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"sun|linear_attention_sequence_parallelism"},"authorids":{"value":["~Weigao_Sun1","~Zhen_Qin6","~Dong_Li11","~Xuyang_Shen1","~Yu_Qiao1","~Yiran_Zhong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Weigao Sun","Zhen Qin","Dong Li","Xuyang Shen","Yu Qiao","Yiran Zhong"]}},"version":2},{"content":{"TLDR":{"value":"We tailor Mamba for multivariate time series forecasting, equipping it with stronger cross-time and cross-channel modeling capabilities, which achieves the SOTA in real-world datasets."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forecasting","Mamba","Channel-Dependent"]},"supplementary_material":{"value":"/attachment/4810e3ea813ec5d058c32afc9ff6a129c09b060a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent advancements in multivariate time series forecasting have been propelled by Linear-based, Transformer-based, and Convolution-based models, with Transformer-based architectures gaining prominence for their efficacy in temporal and cross-channel mixing.\nMore recently, Mamba, a state space model, has emerged with robust sequence and feature mixing capabilities.\nHowever, the suitability of the vanilla Mamba design for time series forecasting remains an open question, particularly due to its inadequate handling of cross-channel dependencies.\nCapturing cross-channel dependencies is critical in enhancing the performance of multivariate time series prediction.\nRecent findings show that self-attention excels in capturing cross-channel dependencies, whereas other simpler mechanisms, such as MLP, may degrade model performance.\nThis is counterintuitive, as MLP, being a learnable architecture, should theoretically capture both correlations and irrelevances, potentially leading to neutral or improved performance.\nDiving into the self-attention mechanism, we attribute the observed degradation in MLP performance to its lack of data dependence and global receptive field, which result in MLP's lack of generalization ability.\nConsidering the powerful sequence modeling capabilities of Mamba and the high efficiency of MLP, the combination of the two is an effective strategy for solving multivariate time series prediction.\nBased on the above insights, we introduce a refined Mamba variant tailored for time series forecasting.\nOur proposed model, \\textbf{CMamba}, incorporates a modified Mamba (M-Mamba) module for temporal dependencies modeling, a global data-dependent MLP (GDD-MLP) to effectively capture cross-channel dependencies, and a Channel Mixup mechanism to mitigate overfitting.\nComprehensive experiments conducted on seven real-world datasets demonstrate the efficacy of our model in improving forecasting performance."},"_bibtex":{"value":"@misc{\nzeng2025cmamba,\ntitle={{CM}amba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting},\nauthor={Chaolv Zeng and Zhanyu Liu and Guanjie Zheng and Linghe Kong},\nyear={2025},\nurl={https://openreview.net/forum?id=OclHGmt2ZM}\n}"},"title":{"value":"CMamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting"},"pdf":{"value":"/pdf/c6642767e189d0d9628d810af94b4cdc57554e7a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zeng|cmamba_channel_correlation_enhanced_state_space_models_for_multivariate_time_series_forecasting"},"authorids":{"value":["~Chaolv_Zeng2","~Zhanyu_Liu1","~Guanjie_Zheng1","~Linghe_Kong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chaolv Zeng","Zhanyu Liu","Guanjie Zheng","Linghe Kong"]}},"tmdate":1738735717967,"tcdate":1727323646686,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5801/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5801/Authors"],"forum":"OclHGmt2ZM","license":"CC BY 4.0","number":5801,"cdate":1727323646686,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/Submission5801/-/Full_Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission5801/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735717967,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"OclHGmt2ZM","version":2},{"content":{"summary":{"value":"This paper aims to solve the computational challenges of Softmax attention in vision tasks due to its quadratic complexity with respect to sequence length. Linear attention as an alternative, reduces complexity to linear time by altering the similarity function from Softmax to kernel functions. However, the authors argue linear attention’s poor expressive power and non-injective nature can lead to semantic confusion. The authors propose two methods to enhance linear attention: enforcing injective properties and improving local modeling capabilities. Using the Swin Transformer architecture, they validate these methods, showing that linear attention can match or exceed Softmax attention’s performance while maintaining lower computational costs. The main contributions are highlighting the importance of injectivity and local modeling in attention mechanisms and demonstrating that linear attention, with these enhancements, can outperform traditional Softmax attention."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Overall the reviewer is quite concerned about the performance of the proposed method, especially given the fact it loses the direct comparison to a very related and comparable linear attention model."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1.\tThis paper thoroughly analyzes the shortcomings of linear attention in vision tasks compared to Softmax attention, identifying non-injective properties and attention confusion as potential root causes. The authors validate these issues through quantitative and qualitative experiments, demonstrating that they contribute to performance drops. The claims seem well-founded, and the verification process appears robust.\n\n2.\tTo address these issues, the authors propose a simple yet effective modification: using subtraction in the normalization of linear attention instead of division, creating a method they call injective linear attention (InLine).\n\n3.\tThe proposed InLine method achieves competitive performance on ImageNet 1k classification and various downstream tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Although the authors' hypothesis and claims seem reasonable, the performance of the proposed method is not remarkable. This paper also lacks the comparison to some related works. For example, another linear attention based method VVT [A] achieves the Top-1 Acc(%) of 84.1 on ImageNet1k with 61.8M Param and 10.8 GFLOPs. The proposed method, InLine-CSwin-B has a Top-1 Acc(%) of 84.5 on ImageNet1k with 73M Param and 14.9G FLOPs. Although InLine-CSwin-B is higher regarding accuracy by 0.4%, it uses 20% more Params and 40% more GFLOPs. This largely weakens the authors' claims. \n\n2. The analysis of local modeling capability (L264-L275) indicates correlation rather than causation. The authors gradually increase the window size and find it does not lead to better performance. Based on this observation, they claim “the insufficient local modeling capability: a small window size restricts the receptive field but introduces strong local bias, enhancing local modeling, while a large window size enlarges the receptive field but further diminishes local modeling ability”. They find that adding a residual connection can solve the problem and thus claim that the local modeling capability of linear attention is problematic. However, as a common practice, a larger receptive field usually requires a different learning rate to ensure the network converges sufficiently. The effect of the residual connection here may not be as the authors claim, but rather just stabilizing the gradient.\n\n\n[A] Sun, W., Qin, Z., Deng, H., Wang, J., Zhang, Y., Zhang, K., Barnes, N., Birchfield, S., Kong, L. and Zhong, Y., 2023. Vicinity vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), pp.12635-12649."},"limitations":{"value":"Yes."}},"nonreaders":[],"tmdate":1730878808726,"tcdate":1720990738114,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2854/Reviewer_My8Q"],"signatures":["NeurIPS.cc/2024/Conference/Submission2854/Reviewer_My8Q"],"forum":"RSiGFzQapl","number":3,"license":"CC BY 4.0","cdate":1720990738114,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2854/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878808726,"domain":"NeurIPS.cc/2024/Conference","replyto":"RSiGFzQapl","id":"t3M8za2gM5","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Softmax attention","linear attention"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Widely adopted in modern Vision Transformer designs, Softmax attention can effectively capture long-range visual information; however, it incurs excessive computational cost when dealing with high-resolution inputs. In contrast, linear attention naturally enjoys linear complexity and has great potential to scale up to higher-resolution images. Nonetheless, the unsatisfactory performance of linear attention greatly limits its practical application in various scenarios. In this paper, we take a step forward to close the gap between the linear and Softmax attention with novel theoretical analyses, which demystify the core factors behind the performance deviations. Specifically, we present two key perspectives to understand and alleviate the limitations of linear attention: the injective property and the local modeling ability. Firstly, we prove that linear attention is not injective, which is prone to assign identical attention weights to different query vectors, thus adding to severe semantic confusion since different queries correspond to the same outputs. Secondly, we confirm that effective local modeling is essential for the success of Softmax attention, in which linear attention falls short. The aforementioned two fundamental differences significantly contribute to the disparities between these two attention paradigms, which is demonstrated by our substantial empirical validation in the paper. In addition, more experiment results indicate that linear attention, as long as endowed with these two properties, can outperform Softmax attention across various tasks while maintaining lower computation complexity. Code is available at https://github.com/LeapLabTHU/InLine."},"_bibtex":{"value":"@inproceedings{\nhan2024bridging,\ntitle={Bridging the Divide: Reconsidering Softmax and Linear Attention},\nauthor={Dongchen Han and Yifan Pu and Zhuofan Xia and Yizeng Han and Xuran Pan and Xiu Li and Jiwen Lu and Shiji Song and Gao Huang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=RSiGFzQapl}\n}"},"title":{"value":"Bridging the Divide: Reconsidering Softmax and Linear Attention"},"pdf":{"value":"/pdf/31bad536df26e6977656df403fe28b991643b271.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"han|bridging_the_divide_reconsidering_softmax_and_linear_attention"},"authorids":{"value":["~Dongchen_Han1","~Yifan_Pu1","~Zhuofan_Xia2","~Yizeng_Han1","~Xuran_Pan1","~Xiu_Li1","~Jiwen_Lu1","~Shiji_Song1","~Gao_Huang1"]},"authors":{"value":["Dongchen Han","Yifan Pu","Zhuofan Xia","Yizeng Han","Xuran Pan","Xiu Li","Jiwen Lu","Shiji Song","Gao Huang"]}},"version":2},{"content":{"summary":{"value":"This paper studies exploration under general neural networks. They propose the method kind of like \"generative Thompson Sampling\", which does not explicitly model the unknown latent variables, but aims at modeling the missing rewards. The procedure of the algorithm starts with implicitly learns the Bayesian model via pre-training by minimizing a sequence prediction loss on successive reward using historical data, then at inference time, it uses the both the observed and imputed reward to make decisions. This basically reduces \n the sequential decision-making problem to a sequence prediction training with low loss. Experiments are done on synthetic and news recommendation dataset."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"See weakness."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- This paper tackles the important issue of exploration in general neural networks, a challenge with broad implications for real-world applications like recommendation systems, education, and even LLM generation. Informed exploration are critical for these complex decision-making systems.\n- The authors effectively bridge the gap between sequence modeling and Bayesian inference, introducing the concept of \"generative Thompson sampling\". This provides a strong theoretical justification for their approach and nicely clarifies the role of pre-training as a form of empirical Bayes.\n- The proposed method is easy to implement, and the experimental results on recommendation tasks demonstrate the effectiveness of the proposed approach, in terms of both reducing regret and providing valid confidence intervals."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Lack of Clarity and Potential Inconsistencies: The paper would benefit from increased clarity in its methodological details and problem formulation. Specifically:\n\n1.  While the motivating example of news recommendation emphasizes no overlap in the action set across time steps, this constraint seems not to be the case  in the formal problem definition and Algorithm 1.\n2. The role of different users in Equation 2 and Algorithm 1 remains unclear. It appears to estimate a population-level mean reward for each action, over all users. This then seems a multi-arm bandit problem than the contextual bandit one? The authors should explicitly state whether they are addressing a multi-arm bandit or a contextual bandit problem and revise the relevant sections accordingly. It is hard to estimate the context-dependent reward based on the current alg.\n\n- Computational Cost: The proposed method's computational complexity scales linearly with both the action set size and the \"overlap\" between new and historical actions, and time-step T. This could become prohibitive in applications with large action spaces or significant small overlap, even disjoint support. The authors should discuss more on the scaling of the method.\n\n- Limited Novelty: The proposed approach is very similar to reward models [1] employed in the offline bandit and RL literature. While subtle differences may exist, the authors need to provide a more thorough comparison with existing reward modeling techniques, explicitly highlighting their novel contributions here.\n\nRef: https://arxiv.org/pdf/1103.4601"}},"nonreaders":[],"tmdate":1731429184538,"tcdate":1730571361063,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12226/Reviewer_vLMe"],"signatures":["ICLR.cc/2025/Conference/Submission12226/Reviewer_vLMe"],"forum":"JNhU9NeOFr","number":1,"license":"CC BY 4.0","cdate":1730571361063,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12226/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429184538,"domain":"ICLR.cc/2025/Conference","replyto":"JNhU9NeOFr","id":"MdpkKKfBdR","forumContent":{"TLDR":{"value":"We use generative sequence modeling to implement a principled version of Thompson Sampling with neural networks in a meta-learning bandit setting; our approach forms approximate posterior draws by generating plausible sequences of future rewards."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["bandit algorithms","Thompson sampling","bayesian inference","generative models"]},"supplementary_material":{"value":"/attachment/847cf65e9a6e3d2a0b4faaeb6ebaf676d05dace9.zip"},"primary_area":{"value":"reinforcement learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Conventionally trained neural networks excel at prediction but often struggle to model uncertainty in their own predictions. We explore this challenge in a meta-learning bandit decision-making problem for news recommendations; this setting require decision-making algorithms to incorporate pretrained language models to process text data for the best performance. We present a scalable approach to Bayesian uncertainty quantification by posing it as a problem of autoregressive generative modeling of future rewards. First, we use historical data on previously released news articles to pre-train a generative model to predict sequences of future potential rewards. At inference time, our algorithm makes decisions based on limited previous rewards and autoregressively generated future rewards. Far from a heuristic, we synthesize insights from the literature to show our method is a novel implementation of Thompson (posterior) sampling, a prominent bandit algorithm. We prove our pretraining loss directly controls online decision-making performance, and we demonstrate our framework on a news recommendation task where we integrate end-to-end fine-tuning of a pretrained language model to process news article headline text to improve performance."},"_bibtex":{"value":"@misc{\nzhang2024informed,\ntitle={Informed Exploration via Generative Modeling},\nauthor={Kelly W. Zhang and Tiffany Cai and Hongseok Namkoong and Daniel Russo},\nyear={2024},\nurl={https://openreview.net/forum?id=JNhU9NeOFr}\n}"},"title":{"value":"Informed Exploration via Generative Modeling"},"pdf":{"value":"/pdf/cb06135fb26af4e36b7e403eee7807fc3af1c23c.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhang|informed_exploration_via_generative_modeling"},"authorids":{"value":["~Kelly_W._Zhang2","~Tiffany_Cai1","~Hongseok_Namkoong2","~Daniel_Russo1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Kelly W. Zhang","Tiffany Cai","Hongseok Namkoong","Daniel Russo"]}},"version":2},{"content":{"TLDR":{"value":"Hybrid architecture that can operate with either self-attention or linear attention mechanisms across the sequence while sharing majority of parameters."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["hybrid architecture","attention","linear attention"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Softmax attention is the cornerstone of modern large language models.\nHowever, its memory requirements scale linearly with sequence length, and its compute requirements scale quadratically.\nLinear recurrent models, such as linear attention and state space models, have become widely studied as alternatives to softmax attention due to their linear compute and constant memory requirements.\nWhile these sub-quadratic token mixing methods (mixers) achieve promising efficiency gains and competitive results on a wide range of benchmarks, current linear recurrent models still lag behind on tasks that require long-context retrieval or in-context learning.\nA growing body of work studies hybrid architectures that attempt to mitigate these trade-offs by statically interleaving or merging attention and recurrent blocks. \nIn this work, we explore a new axis of developing hybrid models: across the token sequence. \nWe propose \\antelope, a hybrid model that can flexibly switch between different mixers throughout a sequence, e.g., quadratic attention for rich context utilization and linear recurrences to reduce quadratic computation.\nOryx ties at least 90% of its parameters across mixers, enabling attention and recurrent modes to operate over shared internal representations.\nWe validate our design with Mamba-2 and Gated DeltaNet variants, up to 1.4B models.\nUnder fixed token budgets and a mixed-training strategy, Oryx achieves comparable or better performance than its single-mixer baselines.\nAt the 1.4B scale, all instances of Oryx outperform their respective baselines by at least 0.7 percentage points on averaged language modeling tasks. On retrieval tasks, Oryx achieves performance comparable to the Transformer baseline even when processing only a tiny fraction (<10%) of the tokens in the attention mode.\nThese results suggest that attention and linear recurrent models can share internal representations, and motivate sequence-axis hybridization as a promising direction."},"_bibtex":{"value":"@inproceedings{\nanonymous2026multimixer,\ntitle={Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=eaaNvYpDNd},\nnote={under review}\n}"},"title":{"value":"Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations"},"pdf":{"value":"/pdf/eaca065817d6d40f7d10c609ab19e4e1221047e5.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231751506,"tcdate":1789671169078,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission33960/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission33960/Authors"],"forum":"eaaNvYpDNd","license":"CC BY 4.0","number":33960,"cdate":1789671169078,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission33960/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791231751506,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"eaaNvYpDNd","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"Hybrid architecture that can operate with either self-attention or linear attention mechanisms across the sequence while sharing majority of parameters. Abstract:"},"venue":{"value":"ODI 2026 Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["hybrid architecture","attention","linear attention"]},"abstract":{"value":"Softmax attention is the cornerstone of modern large language models.\nHowever, its memory requirements scale linearly with sequence length, and its compute requirements scale quadratically.\nLinear recurrent models, such as linear attention and state space models, have become widely studied as alternatives to softmax attention due to their linear compute and constant memory requirements.\nWhile these sub-quadratic token mixing methods (mixers) achieve promising efficiency gains and competitive results on a wide range of benchmarks, current linear recurrent models still lag behind on tasks that require long-context retrieval or in-context learning.\nA growing body of work studies hybrid architectures that attempt to mitigate these trade-offs by statically interleaving or merging attention and recurrent blocks. \nIn this work, we explore a new axis of developing hybrid models: across the token sequence.\nWe propose Oryx, a hybrid model that can flexibly switch between different mixers throughout a sequence, e.g., quadratic attention for rich context utilization and linear recurrences to reduce quadratic computation.\nOryx ties at least 90% of its parameters across mixers, enabling attention and recurrent modes to operate over shared internal representations.\nWe validate our design with Mamba-2 and Gated DeltaNet variants, up to 1.4B models.\nUnder fixed token budgets and a mixed-training strategy, Oryx achieves comparable or better performance than its single-mixer baselines.\nAt the 1.4B scale, all instances of Oryx outperform their respective baselines by at least 0.7 percentage points on averaged language modeling tasks.\nOn real-world retrieval tasks, Oryx remains comparable to the Transformer baseline even when processing less than 10% of the tokens in the attention mode.\nThese results demonstrate that attention and linear recurrent models can share internal representations, and motivate sequence-axis hybridization as a promising direction."},"_bibtex":{"value":"@inproceedings{\nli2026multimixer,\ntitle={Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations},\nauthor={Kevin Li and Asher Trockman and Ananda Theertha Suresh and Ziteng Sun},\nbooktitle={On-Device Intelligence: Foundation Models under Real-World Constraints},\nyear={2026},\nurl={https://openreview.net/forum?id=JvKMM3X2WJ}\n}"},"title":{"value":"Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations"},"pdf":{"value":"/pdf/94dd85a3463c4aa82aabf5ee5ed1fade1b62b1b4.pdf"},"venueid":{"value":"NeurIPS.cc/2026/Workshop/ODI"},"paperhash":{"value":"li|multimixer_models_flexible_sequence_modeling_with_shared_representations"},"authors":{"value":[{"username":"~Kevin_Li3","fullname":"Kevin Li","institutions":[{"name":"Carnegie Mellon University","domain":"cmu.edu","country":"US"}]},{"username":"~Asher_Trockman1","fullname":"Asher Trockman","institutions":[{"name":"Google Research","domain":"google.com","country":"US"}]},{"username":"~Ananda_Theertha_Suresh1","fullname":"Ananda Theertha Suresh","institutions":[{"name":"Google","domain":"google.com","country":"US"}]},{"username":"~Ziteng_Sun1","fullname":"Ziteng Sun","institutions":[{"name":"Google","domain":"google.com","country":"US"}]}]}},"tmdate":1791108156867,"pdate":1791108153957,"tcdate":1788513467843,"writers":["NeurIPS.cc/2026/Workshop/ODI","NeurIPS.cc/2026/Workshop/ODI/Submission83/Authors"],"signatures":["NeurIPS.cc/2026/Workshop/ODI/Submission83/Authors"],"forum":"JvKMM3X2WJ","license":"CC BY 4.0","number":83,"cdate":1788513467843,"readers":["everyone"],"invitations":["NeurIPS.cc/2026/Workshop/ODI/-/Submission","NeurIPS.cc/2026/Workshop/ODI/-/Submission_Change_Before_Reviewing","NeurIPS.cc/2026/Workshop/ODI/Submission83/-/Camera_Ready_Revision","NeurIPS.cc/2026/Workshop/ODI/-/Submission_Release"],"mdate":1791108156867,"odate":1791108153957,"domain":"NeurIPS.cc/2026/Workshop/ODI","id":"JvKMM3X2WJ","version":2},{"content":{"summary":{"value":"This paper tries to tackle the training challenge of long convolutions with reparameterized multi-resolution convolutions (MRConv), which parameterizes global convolutional kernels for long-sequence modeling. The authors introduce learnable kernel decay to learn expressive long-range kernels that perform well across various data modalities. Extensive experiments on the Long Range Arena verify the effectiveness of the proposed MRConv across various tasks in comparison to state-space-based and attention-based models. Moreover, the proposed 1D MRConv can replace 2D convolutions for ImageNet classification tasks and yield better performances."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"(Q1) Are there any rules for selecting the multi-resolution kernels across different tasks? As detailed in D.2.1, the hyper-parameters of multiple kernels sweep for all experiments, and it can be important for the practical usage of MRConv. As shown in D.5.2, the learned kernels on ImageNet-1K are visualized in the Fourier domain, and I suggest the authors conduct a similar analysis with more tasks, which might provide some interesting findings.\n\n(Q2) Are there any training tricks for long convolution kernels in addition to the proposed MRConv? I found the authors utilize the different learning rates for long kernels and other parameters. The authors can provide more ablations of training stabilities (or convergence speeds).\n\n(Q3) As the proposed MRConv requires BatchNorm, does it conflict with some LayerNorm when plugging MRConv into existing architectures (e.g., ConvNeXt variants)?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"(S1) This paper improves the training challenge of SSMs with an intuitive and efficient design. The proposed three types of low-rank kernel parameterization methods are suitable and effective for long-sequence modeling scenarios.\n\n(S2) Compared to popular long-sequence modeling works, the proposed MRConv can achive state-of-the-art performances on some LRA benchmarks. Additional experiments on ImageNet-1K also verify that MRConv can be a general design and further benefit vision communities to some extent.\n\n(S3) The overall presentation is easy to follow, and the structure is clear and easy to read."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"(W1) The proposed MRConv is not novel compared to. The idea and techniques of structural reparameterization of convolution kernels are widely used in modern vision architectures (like RepLKNet variants) and have been recently adopted to long-sequence scenarios in CHELA [1]. As for the three proposed low-rank kernel parameterization strategies, the dilated kernels and sparse kernels have been adopted in vision networks VAN variants [2, 3] and SlaK variants [4, 5], and the Fourier kernels are somewhat novel (but intuitive according to S4 backgrounds).\n\n(W2) The authors should propose a final practical version among three designed versions of SGConv/MRConv, considering the performances, parameters & FLOPs, and generalization across tasks. It can be comprehensive to readers by providing all results (including ablations) of three versions in all comparison tables. However, I am confused about choosing a general and efficient implementation based on these results because I cannot find a version that consistently outperforms others. Therefore, I suggest the authors provide some summed conclusions (like take-home messages) and a final version.\n\n(W3) Drawbacks in experiments. Firstly, some recently proposed works like [1, 6] are missing in the comparison experiments, which weakens the comprehensive evaluations. Secondly, MRConv variants can only achieve competitive results as previous works on LRA benchmarks, and the performance gains are not significant. Moreover, this work is motivated to improve the training instabilities, and some verifications on this aspect are missing (e.g., convergence speeds).\n\n### Reference\n\n[1] Zicheng Liu, et al. \"Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences.\" ICML, 2024.\n\n[2] Menghao Guo, et al. “Visual Attention Network.” CVMJ, 2023.\n\n[3] Siyuan Li, et al. “MogaNet: Efficient Multi-order Gated Aggregation Network.” ICLR, 2024.\n\n[4] Shiwei Liu, et al. “More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity.” ICLR, 2023.\n\n[5] Honghao Chen, et al. “PeLK: Parameter-efficient Large Kernel ConvNets with Peripheral Convolution.” CVPR, 2024.\n\n[6] Songlin Yang, et al. “Gated Linear Attention Transformers with Hardware-Efficient Training.” ICML, 2024.\n\n================== Post-rebuttal Feedback ==================\n\nConsidering the authors' rebuttal and other reviews during the rebuttal period, my concerns have been well addressed and I believe this work meets the standard of NeurIPS. I suggest the authors further polish the manuscript with these valuable explanations and additional results in the main text or the appendix to make it an interesting and solid work."},"limitations":{"value":"Some limitations have been considered in the conclusions section, but more issues of practical usage can be discussed based on the weaknesses I mentioned.\n\nOverall, despite this manuscript having some merits, I appreciate that the weaknesses and questions I mentioned prevent it from accepting at the current stage. I am looking forwards to the authors’ feedback. I will consider raising my score if all these concerns are well addressed or perswading me."}},"nonreaders":[],"tmdate":1730879321940,"tcdate":1721343679796,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission9552/Reviewer_8iUW"],"signatures":["NeurIPS.cc/2024/Conference/Submission9552/Reviewer_8iUW"],"forum":"RwgNbIpCpk","number":5,"license":"CC BY 4.0","cdate":1721343679796,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission9552/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879321940,"domain":"NeurIPS.cc/2024/Conference","replyto":"RwgNbIpCpk","id":"rfyBf3qBaU","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Sequence Modelling","Convolutions","Structural Reparameterization","Self Attention","State Space Models"]},"supplementary_material":{"value":"/attachment/b31a082bbd7b4c4b945895ba10a0c90591d33abb.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ($\\texttt{MRConv}$), a novel approach to parameterizing global convolutional kernels for long-sequence modeling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, $\\texttt{MRConv}$ learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D $\\texttt{MRConv}$ layers."},"_bibtex":{"value":"@inproceedings{\ncunningham2024reparameterized,\ntitle={Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling},\nauthor={Harry Jake Cunningham and Giorgio Giannone and Mingtian Zhang and Marc Peter Deisenroth},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=RwgNbIpCpk}\n}"},"title":{"value":"Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling"},"pdf":{"value":"/pdf/99fdd8c02bfd174bfdb9bf7e36f2b9016b370ad4.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"cunningham|reparameterized_multiresolution_convolutions_for_long_sequence_modelling"},"authorids":{"value":["~Harry_Jake_Cunningham1","~Giorgio_Giannone1","~Mingtian_Zhang1","~Marc_Peter_Deisenroth1"]},"authors":{"value":["Harry Jake Cunningham","Giorgio Giannone","Mingtian Zhang","Marc Peter Deisenroth"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.19888v4"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"behrouz|mambamixer_efficient_selective_state_space_models_with_dual_token_and_channel_selection"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Ali_Behrouz:","~Michele_Santacatterina2","https://dblp.org/search/pid/api?q=author:Ramin_Zabih:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.19888"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-19888,\n  publtype={informal},\n  author={Ali Behrouz and Michele Santacatterina and Ramin Zabih},\n  title={MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.19888},\n  url={https://doi.org/10.48550/arXiv.2403.19888}\n}\n"},"abstract":{"value":"Recent advances in deep learning have mainly relied on Transformers due to their data dependency and ability to learn at scale. The attention module in these architectures, however, exhibits quadratic time and space in input size, limiting their scalability for long-sequence modeling. Despite recent attempts to design efficient and effective architecture backbone for multi-dimensional data, such as images and multivariate time series, existing models are either data independent, or fail to allow inter- and intra-dimension communication. Recently, State Space Models (SSMs), and more specifically Selective State Space Models, with efficient hardware-aware implementation, have shown promising potential for long sequence modeling. Motivated by the success of SSMs, we present MambaMixer, a new architecture with data-dependent weights that uses a dual selection mechanism across tokens and channels, called Selective Token and Channel Mixer. MambaMixer connects selective mixers using a weighted averaging mechanism, allowing layers to have direct access to early features. As a proof of concept, we design Vision MambaMixer (ViM2) and Time Series MambaMixer (TSM2) architectures based on the MambaMixer block and explore their performance in various vision and time series forecasting tasks. Our results underline the importance of selective mixing across both tokens and channels. In ImageNet classification, object detection, and semantic segmentation tasks, ViM2 achieves competitive performance with well-established vision models and outperforms SSM-based vision models. In time series forecasting, TSM2 achieves outstanding performance compared to state-of-the-art methods while demonstrating significantly improved computational cost. These results show that while Transformers, cross-channel attention, and MLPs are sufficient for good performance in time series forecasting, neither is necessary."},"title":{"value":"MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection"},"authors":{"value":["Ali Behrouz","Michele Santacatterina","Ramin Zabih"]}},"tmdate":1738781836938,"pdate":1704067200000,"tcdate":1738781833906,"writers":["~"],"signatures":["~Michele_Santacatterina2"],"forum":"GbZYGD7yMC","license":"CC BY-SA 4.0","number":296425,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1738781836938,"domain":"DBLP.org","id":"GbZYGD7yMC","version":2},{"content":{"summary":{"value":"This paper introduces a domain generalization approach named DGMS, specifically designed for Mamba-based super-resolution networks. The method aims to enhance model generalization capability on images with unknown degradation distributions by proposing two regularization techniques: hidden state update regularization (HUR) and parameter consistency regularization (PCR). This paper identify key variables governing domain shifts through a dedicated metric and apply explicit constraints to mitigate performance degradation caused by distribution discrepancies between source and target domains."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1、Please focus primarily on answering the two questions in the \"Weaknesses\" section.  \n2、If possible, please also address the impact of adding these two loss functions on training time. For example, how much would the training time increase for MambaIR when these loss functions are added?  \n3、Since this paper is related to low-level generalization, it would be more convincing if results from other low-level vision domains (e.g., denoising, deblurring) could be demonstrated.  \n4、The paper does not clarify whether different weights are assigned to the proposed multiple loss functions."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1、First application of domain generalization to Mamba-based low-level vision tasks, achieving expressive results.  \n2、Novelty:The proposed Hidden state Update Regularization (HUR) and Parameter Consistency Regularization (PCR) significantly improve model generalization performance.  \n3、Expression is good: Rigorous theoretical derivation and well-designed experiments enhance readability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1、Potential limitation in long-range modeling:\nIn the Hidden State Update Regularization section (lines 236-238), the constraint on discretization matrix A (Eq. 9) aims to reduce the domain shift term. However, in low-level vision tasks (e.g., super-resolution), pixel-level interactions and long-sequence modeling are critical. Since matrix A in state-space models primarily prevents catastrophic forgetting in long sequences, explicitly constraining it may impair Mamba’s ability to capture long-range dependencies, potentially degrading pixel-level perception. The authors should clarify whether this trade-off between domain generalization and spatial modeling capability is justified.  \n2、Potential interdependence between regularization terms:\nThe proposed HUR and PCR modules independently constrain the red term and blue term derived from the domain shift formulation. However, the interaction between these terms is not addressed—applying constraints to one term may inadvertently affect the other (e.g., altering the red term via matrix A regularization could propagate changes to the blue term). This decoupled approach lacks justification for its independence assumption and may lead to suboptimal or unstable optimization. The authors should analyze the coupling effects between these terms or provide empirical/theoretical evidence supporting their separability.  \n3、"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762921442645,"tcdate":1761857541619,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10039/Reviewer_bxW8"],"signatures":["ICLR.cc/2026/Conference/Submission10039/Reviewer_bxW8"],"forum":"9T1agMpZ8i","number":3,"license":"CC BY 4.0","cdate":1761857541619,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10039/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762921442645,"domain":"ICLR.cc/2026/Conference","replyto":"9T1agMpZ8i","id":"XFq51a3ZQF","forumContent":{"TLDR":{"value":"We propose a domain generalization method for Mamba-based SR networks (DGMS), which effectively enhances the performance of Mamba-based SR networks on target domain samples with unknown distributions after training with source domain samples."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Domain generalization","Super resolution"]},"primary_area":{"value":"transfer learning, meta learning, and lifelong learning"},"abstract":{"value":"Mamba-based domain generalization methods can effectively improve the performance of Mamba networks on samples with unknown distributions. However, existing methods target high-level  vision tasks like image and point cloud classification, with limited research on low-level vision tasks such as image super-resolution (SR). To bridge this gap, we propose a Domain Generalization method for Mamba-based Super-Resolution networks (DGMS), which introduces a domain shift metric for SR tasks and identifies key variables governing domain shifts. Subsequently, based on the identified key variables, we propose hidden state update regularization and parameter consistency regularization. Through explicit supervisory constraints on these key variables, the method effectively enhances the network’s generalization performance across images with different degradation models. Extensive experiments across diverse data distributions and network architectures demonstrate the effectiveness of the proposed method on low-level vision tasks, where it outperforms existing state-of-the-art Mamba-based domain generalization methods.\nOur code is available at ***."},"_bibtex":{"value":"@misc{\ntang2025dgms,\ntitle={{DGMS}: Domain Generalization Method for Mamba-based Super-Resolution Networks},\nauthor={Zaizuo Tang and Yu-Bin Yang},\nyear={2025},\nurl={https://openreview.net/forum?id=9T1agMpZ8i}\n}"},"title":{"value":"DGMS: Domain Generalization Method for Mamba-based Super-Resolution Networks"},"pdf":{"value":"/pdf/08a3f8639de5f9a3426efaff8fe365f6fefcf210.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"tang|dgms_domain_generalization_method_for_mambabased_superresolution_networks"},"authorids":{"value":["~Zaizuo_Tang1","~Yu-Bin_Yang3"]},"authors":{"value":["Zaizuo Tang","Yu-Bin Yang"]}},"version":2},{"content":{"summary":{"value":"The authors attempt to solve the problem of learning state-space models of multivariate time series data with non-linear interactions amongst the state variables in the observation space. This is done by using orthogonal multivariate Legendre polynomials to capture time-varying correlations amongst the state variables. To mitigate model size increase due to combinatorial explosion, they impose heuristic constraints on the interaction matrix. The model parameters are optimized via a transformer architecture. The authors conclude that their algorithm Poly-Mamba outperforms state of the are methods on, especially when there are correlations amongst the variables."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"n/a"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"The problem of modeling high-dimensional, multivariate time series data with dynamic correlations is an important, interesting, and long-standing problem with many, many prior works."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Unfortunately, due to several issues outlined below, this paper is not acceptable for publications in its current form. \n\nMajor issues:\n\t-Most critically, while there may be a preponderance of ‘better performance’ of their models compared to other models, the results are so small as to make the difference practically meaningless. Take, for example, the comparisons made in Table 1 (main results) on the Weather data, which the authors themselves highlight in the results as an example of particular relevance. Here, we indeed see that the author's model performs ‘better’ in the vast majority of cases. However, when one looks at the magnitude of the effect in the third digit, amounting to a 2% -0.2% improvement. I simply do not believe this is a real effect. The insignificance of this result is further highlighted by the plots in Figure 4 (which are never actually discussed), in which many of the model fits to the data are visually indistinguishable from each other, and all seem to make the same quality of errors.\n\t-I am very concerned about the lack of novelty of the model/method, which seems to be a fairly straightforward improvement of the prior work of Gu, et al., 2020, 2021, 2023. As far as I can tell, it's a very similar model/optimization, now with the utilization of orthogonal Legendre polynomials to represent interactions. \n\t-Related to my concern about novelty, the author exhibits a general lack of awareness of state-space models in general. For example, in the first line, they cite Gu et al, 2020 for state-space models; however, state-space models have been around for at least 60 years (since before Kalman!). \n-Furthermore, to side-step (I don’t think they actually deal with the issue) the curse of dimensionality in terms of high-dimensional interactions amongst variables, they introduce several ad hoc heuristics to simplify the interaction matrix. However, the heuristics they use have little-to-no mathematical support, and it is not discussed what failure modes they may induce. It is not at all clear (at least to me) that the model can support arbitrary interactions of large numbers of variables, thus limiting its utilization. The issue of scaling to large numbers of variables (e.g., 1000s) is never discussed. \n\n\nMinor issues:\n\t-the variable “L” is used in many different ways throughout the manuscript, which is confusing.\n\t-the ablation studies result in similarly small magnitude effects (~2%), making the results unremarkable.\n\t-writing needs a bunch of work; e.g., capitalization is odd at times, word choice can be confusing, there are omitted words, there are a lot of typos (too many to list here).\n\t-Several methods that address this problem are not discussed or compared against. For example, the orthogonal stochastic linear mixing model. \n\t-The most relevant comparison for efficiency seems to be with MambaT5, for which the current algorithm does modestly better but at ~3X memory cost and 50% speed reduction."}},"nonreaders":[],"tmdate":1731428117725,"tcdate":1730383714301,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9204/Reviewer_BrqH"],"signatures":["ICLR.cc/2025/Conference/Submission9204/Reviewer_BrqH"],"forum":"nclyFUZpX9","number":3,"license":"CC BY 4.0","cdate":1730383714301,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9204/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428117725,"domain":"ICLR.cc/2025/Conference","replyto":"nclyFUZpX9","id":"KZpVkBCU6I","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forecasting；Deep Learning； State Space Model"]},"supplementary_material":{"value":"/attachment/f064e3aed57d3d1812e4a9ee5e1f4ea5b44219ea.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"For multivariate time series (MTS) tasks, previous state space models (SSMs) followed the modeling paradigm of Transformer-based methods. However, none of them explicitly model the complex dependencies of MTS: the Channel Dependency variations with Time (CDT). In view of this, we delve into the derivation of SSM, which involves approximating continuously updated functions by orthogonal function basis. We then develop Poly-Mamba, a novel method for MTS forecasting. Its core concept is to expand the original orthogonal function basis space into a multivariate orthogonal function space containing variable mixing terms, and make a projection on this space so as to explicitly describe the CDT by weighted coefficients. In Poly-Mamba, we propose the Multivariate Orthogonal Polynomial Approximation (MOPA) as a simplified implementation of this concept. For the simple linear relationship between channels, we propose Linear Channel Mixing (LCM) and generate CDT patterns adaptively for different channels through a proposed Order Combining method. Experiments on six real-world datasets demonstrate that Poly-Mamba outperforms the SOTA methods, especially when dealing with datasets having a large number of channels and complex correlations. The codes and log files are in the supplementary."},"_bibtex":{"value":"@misc{\nwu2025a,\ntitle={A {SSM} is Polymerized from Multivariate Time Series},\nauthor={Haixiang Wu},\nyear={2025},\nurl={https://openreview.net/forum?id=nclyFUZpX9}\n}"},"title":{"value":"A SSM is Polymerized from Multivariate Time Series"},"pdf":{"value":"/pdf/56e1fd4acec2afe967a5ef8777c48a18b660772d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"wu|a_ssm_is_polymerized_from_multivariate_time_series"},"authorids":{"value":["~Haixiang_Wu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Haixiang Wu"]}},"version":2},{"content":{"comment":{"value":"Thank you for your review. We appreciate your positive feedback on the clarity of our presentation and the comprehensiveness of our related work section. Below, we address each of your requested changes:\n\n**[RC1] Attention-type baselines**\n\nRegarding the utilization of attention-based models as baselines, we note that several of our baseline models (including Mamba and the diffusion-based models in the SST section) already incorporate attention mechanisms in their architectures. While we did implement attention components within our Alternator (as mentioned in our experimental setup: \"2-layer attention models\"), we didn't explicitly compare against standalone Transformer or Conformer architectures.\n\nPrior work, particularly the Mamba [1] paper, has demonstrated substantial improvements over vanilla attention models in sequence modeling tasks.\n\n[1] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n\n**[RC2] Hyperparameter selection methodology**\n\nYou're right that we should clarify our hyperparameter selection process. We performed a grid search over key hyperparameters (σ_z, σ_x, α_t) for the Alternator, and manually tuned other hyperparameters like learning rate and architecture depth based on validation performance. For the baselines, we initially used recommended configurations from their original papers, then fine-tuned them on validation data. We'll add a detailed section in the appendix describing this process for all models to ensure fair comparison.\n\n**[RC3] Derivation for equation (9)**\n\nThank you for pointing this out. You're correct that equation (8) uses the definition of conditional probability rather than Bayes' rule. We'll clarify this and provide a detailed derivation of equation (9) in the appendix of our revised manuscript.\n\n**[RC4] Limitations discussion**\n\nFor the imputation task, we used missing-at-random sampling, which allows missing values to occur anywhere in the sequence, including the middle. In this scenario, the latent state z functions as a dynamic memory that helps generate the missing observations.\nWe agree that bidirectional architectures can enhance performance when both the beginning and end of a sequence are available. In our revised manuscript, we will discuss how bidirectional structures could be incorporated into both the OTN and FTN to smooth predictions through backward information flow.\n\nWe will add a dedicated limitations section discussing constraints of our approach, computational trade-offs, sensitivity to hyperparameter choices, and challenges in extremely high-dimensional settings.\n\n**[RC5] Include neural stochastic differential equations**\n\nWe agree that neural stochastic differential equations (NSDEs) deserve mention when discussing limitations of NODEs. We'll expand our related work section to include NSDEs, citing the papers you suggested and discussing how they address the deterministic limitations of NODEs when modeling noisy observations.\n\nAlternators differ fundamentally from neural SDEs in several important ways:\n\nAlternators use direct neural network mappings rather than expensive numerical solvers, making them significantly faster, as demonstrated in our sea-surface temperature forecasting experiments.\n\nThey explicitly model low-dimensional latent variables (D_z << D_x), unlike neural SDEs, which typically maintain high-dimensional state representations.\n\nThe alternating OTN/FTN architecture allows for specialized network designs tailored to observation and feature spaces, which is not possible in neural SDE formulations.\n\nOur cross-entropy minimization approach differs fundamentally from the training objectives used in neural SDEs, leading to better trajectory modeling as shown in our Lorenz attractor experiments.\n\nAlternators can be easily applied to sequence-to-sequence prediction, imputation, and forecasting with minimal modifications, while neural SDEs require substantial adaptation for these tasks.\n\n**Minor Changes**\n\nWe will define all acronyms upon first use, including SSR (Spread-Skill Ratio), SST (Sea-Surface Temperature), MAE (Mean Absolute Error), MSE (Mean Squared Error), and CC (Correlation Coefficient).\nThank you again for your constructive feedback, which will help us improve our manuscript."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1744817848842,"tcdate":1744817848842,"writers":["TMLR","TMLR/Paper4579/Authors"],"signatures":["TMLR/Paper4579/Authors"],"forum":"Q70C1HQ0VO","number":3,"license":"CC BY 4.0","cdate":1744817848842,"readers":["everyone"],"invitations":["TMLR/Paper4579/-/Official_Comment"],"mdate":1744817848842,"domain":"TMLR","replyto":"7o1q2m39Ly","id":"4I7arMroj9","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"abstract":{"value":"This paper introduces alternators, a novel family of non-Markovian dynamical models for sequences. An alternator features two neural networks: the observation trajectory network (OTN) and the feature trajectory network (FTN). The OTN and the FTN work in conjunction, alternating between outputting samples in the observation space and some feature space, respectively. The parameters of the OTN and the FTN are not time-dependent and are learned via a minimum cross-entropy criterion over the trajectories. Alternators are versatile. They can be used as dynamical latent-variable generative models or as sequence-to-sequence predictors. Alternators can uncover the latent dynamics underlying complex sequential data, accurately forecast and impute missing data, and sample new trajectories. We showcase the capabilities of alternators in three applications. We first used alternators to model the Lorenz equations, often used to describe chaotic behavior. We then applied alternators to Neuroscience to map brain activity to physical activity. Finally, we applied alternators to Climate Science, focusing on sea-surface temperature forecasting. In all our experiments, we found alternators are stable to train, fast to sample from, yield high-quality generated samples and latent variables, and often outperform strong baselines such as Mambas, neural ODEs, and diffusion models in the domains we studied."},"_bibtex":{"value":"@article{\nrezaei2025alternators,\ntitle={Alternators For Sequence Modeling},\nauthor={Mohammad Reza Rezaei and Adji Bousso Dieng},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=Q70C1HQ0VO},\nnote={}\n}"},"title":{"value":"Alternators For Sequence Modeling"},"changes_since_last_submission":{"value":"We had the wrong style files and the submission was desk-rejected"},"pdf":{"value":"/pdf/3b78be26e3e20f87deafef1d6b5ca461b14bb894.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"rezaei|alternators_for_sequence_modeling"},"authorids":{"value":["~Mohammad_Reza_Rezaei1","~Adji_Bousso_Dieng1"]},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=rdhRm2yTAu&nesting=2&sort=date-desc"},"assigned_action_editor":{"value":"~Adam_Arany1"},"authors":{"value":["Mohammad Reza Rezaei","Adji Bousso Dieng"]}},"version":2},{"content":{"summary":{"value":"This work presents a parameter-efficient fine-tuning (PEFT) framework for unified single-object tracking across multiple modalities (RGB, RGB-T, RGB-D, RGB-E, RGB-Language). The method introduces two novel components: a Memory-Aware Compression Prompt (MCP) module and Dynamic State Fusion (DSF) modules. MCP maintains a fixed set of learnable query tokens that dynamically compress the historical memory bank into a small set of prompt tokens. These tokens are concatenated to the input sequence, acting as additional prompts that inject long-term contextual memory into the backbone in a parameter-efficient way. DSF uses a selective state-space model (SSM) with gating to capture continuous target dynamics across layers. The experiments and ablations confirm efficiency and generalizability."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. Could the authors provide the inference time (FPS) and runtime complexity comparisons with existing baselines (e.g., SUTrack, HIPTrack)? This would better substantiate the efficiency claims of Uni-MDTrack.\n2. The DSF module seems structurally similar to Mamba’s selective state-space formulation. Could the authors clarify what specific modifications or adaptations were made beyond directly applying the Mamba equations?\n3. The paper already covers various benchmarks (LaSOT, LasHeR, VisEvent, DepthTrack). Could the authors further discuss or analyze how Uni-MDTrack behaves in long-term tracking situations (e.g., target reappearance, heavy occlusion) to demonstrate temporal robustness?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Novel architecture that merges memory compression and state-space modeling for efficient and unified multimodal tracking across diverse modalities.\n2. Parameter efficiency: The base version (Uni-MDTrack-B) fine-tunes only 30 % of its total parameters while achieving comparable or better accuracy than other full-tuning baselines.\n3. Comprehensive experiments and ablations demonstrating component effectiveness and scalability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Although the paper reports reduced GFLOPs and parameter counts, it does not provide the actual inference speed (FPS). Given that the backbone is HiViT and additional modules (MCP and DSF) involve memory querying and state updates, the real-time performance might be limited.\n2. The DSF module employs an SSM formulation that is similar to the Mamba architecture, including the gating and state update mechanisms. While this integration into a tracking framework is reasonable, the design itself does not introduce substantial architectural novelty."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919219706,"tcdate":1761899852280,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7007/Reviewer_aS26"],"signatures":["ICLR.cc/2026/Conference/Submission7007/Reviewer_aS26"],"forum":"FMOn4cjJKG","number":1,"license":"CC BY 4.0","cdate":1761899852280,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7007/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919219706,"domain":"ICLR.cc/2026/Conference","replyto":"FMOn4cjJKG","id":"toKfbF4UUC","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Unified Single Object Tracking","Memory","Prompt","Parameter-Efficient Fine-Tuning"]},"supplementary_material":{"value":"/attachment/0954a65cda366bf1ea2b8504886ce91a8db04a08.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"In this paper, we propose a simple but powerful parameter-efficient fine-tuning (PEFT) framework designed for unified single object trackers. Our framework is built upon two novel components: a Memory-Aware Compression Prompt (MCP) module and Dynamic State Fusion (DSF) modules. MCP effectively compresses memory features into memory-aware prompt tokens, which are deeply interacted with the input sequence throughout the entire backbone, significantly enhancing model performance while maintaining a stable computational load. DSF complements the discrete memory features by capturing the continuous dynamic state of the target, progressively introducing the updated dynamic state features from shallow to deep layers of the tracker, while also preserving high operational efficiency. MCP effectively overcomes the limitations of previous trackers that rely on only a few frames when introducing memory, which significantly increases input length and computational cost. It also addresses the insufficient fusion problem in existing memory-prompting methods. DSF remedies the lack of dynamic feature about continuous target variation in prior PEFT methods. Based on the MCP and DSF modules, we propose Uni-MDTrack, a tracker that supports tracking across five modalities. Experimental results across 10 datasets spanning five modalities demonstrate that Uni-MDTrack achieves state-of-the-art performance, with only 30\\% of parameters requiring training. Furthermore, both MCP and DSF exhibit excellent generality, functioning as plug-and-play components that can boost the performance of various trackers. Code will be released for further research."},"_bibtex":{"value":"@misc{\ncai2026unimdtrack,\ntitle={Uni-{MDT}rack: Prompt Unified Single Object Tracking with Deep Fusion of Memory and Dynamic State},\nauthor={Wenrui Cai and Zhenyi Lu and Yongchao Feng and Jinqing Zhang and Qingjie Liu},\nyear={2026},\nurl={https://openreview.net/forum?id=FMOn4cjJKG}\n}"},"title":{"value":"Uni-MDTrack: Prompt Unified Single Object Tracking with Deep Fusion of Memory and Dynamic State"},"pdf":{"value":"/pdf/2cbeb1e30f805c2d772ae40510801773a97ff838.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cai|unimdtrack_prompt_unified_single_object_tracking_with_deep_fusion_of_memory_and_dynamic_state"},"authorids":{"value":["~Wenrui_Cai1","~Zhenyi_Lu2","~Yongchao_Feng1","~Jinqing_Zhang2","~Qingjie_Liu2"]},"authors":{"value":["Wenrui Cai","Zhenyi Lu","Yongchao Feng","Jinqing Zhang","Qingjie Liu"]}},"version":2},{"content":{"summary":{"value":"The paper presents LC-PLM, a protein language model based on a bidirectional selective structured state-space models with shared projection layers (BiMamba-S) for learning protein representations. Additionally, it introduces a graph-based variant, LC-PLM-G, incorporating protein-protein interaction (PPI) graphs for training and show learning paradigm with extended context beyond individual proteins. The authors claim that (1) LC-PLM demonstrates better length extrapolation capabilities and outperforms Transformer-based models like ESM-2 on various essential protein-related tasks; (2) LC-PLM-G shows promise in protein structure and function prediction tasks by integrating biological interaction contexts."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- In RQ1, how do the authors hold out the test 250k sequence? Do the authors re-train the ESM2 for plotting the Figure 4? Moreover, in Figure 4/5, what is in specific the “Evaluation Loss” for the label of the y-axis (vertical direction)?\n- When reporting folding evaluation, how do the structures obtained from ESM-2?  My concern is, for evaluation fairness, one should align the folding trunk while on the other hand, the distribution over the residue-level embeddings from (1) LC-PLM and (2) ESM-2 can be different  in the vector space. Please elaborate or point out some relevant context to address this question. \n- A related questions from above, i understand the folding (RQ4) task for the authors is just a “proof-of-concept” such that the authors adopt downsampling for the openfold training data. However, 1.5% of protein single chains can have large variance for the model to show enough performance bias. Could the authors test the robustness of this downsampling strategy and show error bar on it?\n- Question regarding the results incorporating the protein-protein interaction PPI graph saying LC-PLM-G.  In specific, in table 3 of the two tasks from TAPE benchmark, the LC-PLM-G/ESM-2-G shows very similar results compared to their vanilla version (LC-PLM/ESM-2). The performance improvement margin seems to be within the reported std range and I do not see significant improvement of doing this; in table 4/ProteinGym benchmark, for ESM-2-G, the incorporating of PPI graph drastically hurt the performance on top of ESM-2 while the performance gap between LC-PLM-G/LC-PLM is also hard to tell. These results can weaken the “necessity” claim of the main motivation of this paper: modeling based on multiple sequence input (PPI in this case). Could the author further justify the benefit for inference with additional PPI graph? \n\n- Could the authors mentioned some concrete modeling examples with biological significance that we have to turn to Mamba because the attention/Transformer sucks or even fails to handle the problem at all? Can the authors provide further evidence that the extended context window length improves the “related” downstream protein tasks beyond the results presented? At present, it is still unclear to me why should I buy the idea of using S4/mamba instead of attention?\n\nMISC:\n- In figure 6, why using “value x 2^1” as the ticks? That looks a bit weird"},"rating":{"value":3},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The paper presents a range of experimental evaluations across downstream tasks, showing better performance compared to one standard Transformer-based protein language model ESM-2. The evaluation tasks are diverse and comprehensive.\n- The introduction of LC-PLM-G, incorporating PPI graphs, is a novel direction for protein language models. Using relational information between proteins has potential for realistic applications where protein interactions play a crucial role."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The paper does not adequately justify the selective structured state-space (S4) models over existing Transformers-based PLMs (eg. ESM-2), which are widely adopted for protein sequence modeling. Protein sequence understanding does not usually face the same latency/bandwidth constraints as real-time LLM deployments such as ChatGPT, where SSMs could possibly bring more practical value with linear inference time. Without clear advantages or innovations in the application of SSMs, this choice lacks strong motivation to stand convincing.\n- The paper assumes that longer context modeling is inherently beneficial for protein sequences. However, the biological need for extremely long contexts remains unclear in many protein-related applications, especially since functional motifs often reside within local regions of a protein rather than requiring entire sequence context. Without a clear biological or empirical rationale, the assumption that long contexts are “essential” for protein understanding is questionable to me. \n- The authors highlight LC-PLM’s computational efficiency over Transformers due to the quadratic complexity of Transformers. However, given that training/inference-efficient Transformers exist (flash attention for example), the emphasis on efficiency without practical demand diminishes the paper’s significance. \n- Across the evaluation tasks (yes i agree they are many), only ESM-2 is there for comparison while for many of the task ESM-2 (alone) is not the real state-of-the-art (SOTA). Note that there are many interesting works built based on the ESM-2 embeddings and give very promising SOTA results for function prediction or mutation effect modeling. I wonder how the LC-PLM embedding accommodate these existing model and perform better than ESM-2."}},"nonreaders":[],"tmdate":1731429172650,"tcdate":1730615934884,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8832/Reviewer_8QMc"],"signatures":["ICLR.cc/2025/Conference/Submission8832/Reviewer_8QMc"],"forum":"Et0SIGDpP5","number":2,"license":"CC BY 4.0","cdate":1730615934884,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8832/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429172650,"domain":"ICLR.cc/2025/Conference","replyto":"Et0SIGDpP5","id":"FOcfyJ6QMG","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["protein language model"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. \nMost protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. Such protein LMs cannot extrapolate to longer proteins and protein complexes well. They also fail to account for the underlying biological mechanisms carried out by biomolecular interactions and dynamics i.e., proteins often interact with other proteins, molecules, and pathways in complex biological systems. \nIn this work, we propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built off selective structured state-space models, to learn high-quality universal protein representations at the amino acid token level using masked language modeling. We also introduce its graph-contextual variant, LC-PLM-G, which contextualizes protein-protein interaction (PPI) graphs for a second stage of training. LC-PLM demonstrates favorable neural scaling laws, better length extrapolation capability, and a 7\\% to 34\\% improvement on protein downstream tasks than Transformer-based ESM-2. LC-PLM-G further trained within the context of PPI graphs shows promising results on protein structure and function prediction tasks. Our study demonstrates the benefit of increasing the context size with computationally efficient LM architecture (e.g. structured SSMs) in learning universal protein representations and incorporating molecular interaction contexts contained in biological graphs."},"_bibtex":{"value":"@misc{\nwang2025longcontext,\ntitle={Long-context Protein Language Model},\nauthor={Yingheng Wang and Zichen Wang and Gil Sadeh and Luca Zancato and Alessandro Achille and George Karypis and Huzefa Rangwala},\nyear={2025},\nurl={https://openreview.net/forum?id=Et0SIGDpP5}\n}"},"title":{"value":"Long-context Protein Language Model"},"pdf":{"value":"/pdf/3dea956cc09965e552e48b137c346e62db12eb35.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"wang|longcontext_protein_language_model"},"authorids":{"value":["~Yingheng_Wang1","~Zichen_Wang3","~Gil_Sadeh2","~Luca_Zancato1","~Alessandro_Achille1","~George_Karypis1","~Huzefa_Rangwala2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yingheng Wang","Zichen Wang","Gil Sadeh","Luca Zancato","Alessandro Achille","George Karypis","Huzefa Rangwala"]}},"version":2},{"content":{"venue":{"value":"IEEE Transactions on Geoscience and Remote Sensing"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/36/10354519/10756788.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"he|igroupssmamba_interval_group_spatialspectral_mamba_for_hyperspectral_image_classification"},"html":{"value":"https://doi.org/10.1109/TGRS.2024.3502055"},"abstract":{"value":"Hyperspectral image (HSI) classification has garnered substantial attention in remote sensing fields. Recent mamba architectures built upon the selective state-space models (S6) have demonstrated enormous potential in long-range sequence modeling. However, the high dimensionality of hyperspectral data and information redundancy pose challenges to the application of S6 in HSI classification, suffering from suboptimal performance and computational efficiency. In light of this, this article investigates a lightweight interval group spatial-spectral mamba framework (IGroupSS-Mamba) for HSI classification, which allows for multidirectional and multiscale global spatial-spectral information extraction in a grouping and hierarchical manner. Technically, an interval group S6 mechanism (IGSM) is developed as the core component, which partitions high-dimensional features into multiple nonoverlapping groups at intervals, and then integrates a unidirectional S6 for each group with a specific scanning direction to achieve nonredundant sequence modeling. Compared with conventional applying multidirectional scanning to all bands, this grouping strategy leverages the complementary strengths of different scanning directions while decreasing computational costs. To adequately capture the spatial-spectral contextual information, an interval group spatial-spectral block (IGSSB) is introduced, in which two IGSM-based spatial and spectral operators are cascaded to characterize the global spatial-spectral relationship along the spatial and spectral dimensions, respectively. IGroupSS-Mamba is constructed as a hierarchical structure stacked by multiple IGSSB blocks, integrating a pixel aggregation-based downsampling strategy for multiscale spatial-spectral semantic learning from shallow to deep stages. Extensive experiments demonstrate that IGroupSS-Mamba significantly outperforms the state-of-the-art methods in classification accuracy and achieves lower model parameters and floating point operations (FLOPs). The code is available at https://github.com/IIP-Team/ IGroupSS-Mamba."},"title":{"value":"IGroupSS-Mamba: Interval Group Spatial–Spectral Mamba for Hyperspectral Image Classification"},"authors":{"value":[{"fullname":"Yan He"},{"fullname":"Bing Tu","username":"~Bing_Tu1"},{"fullname":"Puzhao Jiang"},{"fullname":"Bo Liu"},{"fullname":"Jun Li"},{"fullname":"Antonio Plaza"}]}},"tmdate":1789090580898,"pdate":1704067200000,"externalIds":["doi:10.1109/tgrs.2024.3502055"],"tcdate":1762941874526,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Bing_Tu1"],"forum":"kUPX0zTIQ0","license":"CC BY-SA 4.0","number":14814,"cdate":1760462998638,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/-/Edit"],"mdate":1789090580898,"domain":"OpenReview.net/Public_Article","id":"kUPX0zTIQ0","version":2},{"content":{"summary":{"value":"The paper introduces ProtMamba, a state-space protein language model that is trained on sets of homologous sequences concatenated together. The model is trained to both generate sequences from scratch and to infill sequences using a fill-in-the-middle objective. For fitness prediction on ProteinGym, the method is shown to perform on-par with much larger models that explicitly use a multiple sequence alignment. For a narrower dataset of chorismate mutase activities, they demonstrate that they can apply prompt engineering and the FIM objective to improve fitness prediction. Finally, they perform a limited evaluation of the model’s autoregressive generation capabilities and show that the top 10% of generated sequences have some properties similar to natural proteins."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"ProteinGym’s performance metrics are computed by averaging together the Spearman correlations for all assays with the same (UniProt ID, Function) pair, computing the average-of-averages for each function, and then averaging over functions. When computing the depth-based (and other) averages, I believe the UniProt IDs are averaged first as well, though not the functions. Can the authors confirm that they use the appropriate hierarchical averages to compute results for ProtMamba in Table 1?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"Prompting protein language models with sequences of homologous proteins (rather than an MSA) is an exciting direction for retrieval-augmented models. Given the inefficiencies of long-context transformers, using a state-space model for this objective is a natural idea to explore.\n\nThe choice to couple standard autoregressive language modeling with a fill-in-the-middle objective is an interesting one that has been relatively unexplored for protein language modeling, and the authors show its value for a fitness prediction task involving chorismate mutases.\n\nThe authors also provide a limited demonstration that prompting the model with high-activity sequences can improve its ability to perform fitness prediction."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Major points:\nGeneral clarity: I have a hard time following all the details of the paper because of the copious references to supplementary figures to support central claims in the main text.\n\nTable 1: Given that the code & model parameters are publicly available, I would like to see the authors reproduce results for PoET [1], both without retrieval and with retrieval using the same prompt that is provided to ProtMamba. Given how similar the two approaches are, I cannot accept this paper without seeing this baseline.\n\nTable 1: I would like to see how ProtMamba performs when one uses the autoregressive log likelihood of an unmasked sequence, rather than the FIM objective. Without this comparison, the value of FIM vs. autoregressive language modeling is less clear to me, since the scope of the experiment in Section 3.3 is much more limited.\n\nLines 208-212: More recent work [2, 3, 4] suggests that it is beneficial to mask as much as 50% of a sequence. I would like to see ablations that evaluate different masking fractions, rather than just results for the somewhat arbitrary choice of 20%.\n\nFigure 4: There are major loss spikes and periods where the training loss actually increases. The authors should comment on the overall training stability of ProtMamba with some analysis of the gradient norms during training. This is important for a reader to decide whether they would choose to adopt Mamba over a transformer.\n\nMinor points\nLines 89-95: Modern attention implementations like FlashAttention have linear memory complexity, though they still have quadratic time complexity [5]. The authors should update the text to reflect this fact.\n\nTable 1: Indicating the top performers for each evaluation in bold would improve the readability of the table.\n\nFigures 6-7: It is unclear to me what L denotes, and why the 150 < L < 250 line extends so much further than the other 2 in Figure 6.\n\n[1] Truong Jr. & Bepler. PoET: A generative model of protein families as sequences-of-sequences. NeurIPS, 2023.\n[2] Wettig et al. Should You Mask 15% in Masked Language Modeling? EACL, 2023.\n[3] Tay et al. UL2: Unifying Language Learning Paradigms. arXiv, 2022.\n[4] Hayes et al. Simulating 500 million years of evolution with a language model. bioRxiv, 2024.\n[5] Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR, 2023."}},"nonreaders":[],"tmdate":1732593506150,"tcdate":1730688438971,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6891/Reviewer_tE4Y"],"signatures":["ICLR.cc/2025/Conference/Submission6891/Reviewer_tE4Y"],"forum":"BMfHO2lXGe","number":4,"license":"CC BY 4.0","cdate":1730688438971,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6891/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732593506150,"domain":"ICLR.cc/2025/Conference","replyto":"BMfHO2lXGe","id":"Kl6bIbEVQD","forumContent":{"TLDR":{"value":"ProtMamba is a protein state space model that can handle very long context and enables autoregressive generation and fitness prediction conditioned on a set of homologous sequences."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["proteins","protein sequence","protein language model","computational biology","generative model","protein engineering","protein fitness prediction","protein design"]},"supplementary_material":{"value":"/attachment/dae9053c91a055be470f2724557660f77a9bb545.zip"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Protein design has important implications for drug discovery, personalized medicine, and biotechnology. Models based on multiple sequence alignments efficiently capture the evolutionary information in homologous protein sequences, but multiple sequence alignment construction is imperfect. We present ProtMamba, a homology-aware but alignment-free protein language model based on the Mamba architecture. In contrast with attention-based models, ProtMamba efficiently handles very long context, comprising hundreds of protein sequences. We train ProtMamba on a large dataset of concatenated homologous sequences, using two GPUs. We combine autoregressive modeling and masked language modeling through a fill-in-the-middle training objective. This makes the model adapted to various protein design applications. We demonstrate ProtMamba's usefulness for the generation of novel sequences and for fitness prediction. ProtMamba reaches competitive performance with other protein language models despite its smaller size, which sheds light on the importance of long-context conditioning."},"_bibtex":{"value":"@misc{\nsgarbossa2025protmamba,\ntitle={ProtMamba: a homology-aware but alignment-free protein state space model},\nauthor={Damiano Sgarbossa and Cyril Malbranke and Anne-Florence Bitbol},\nyear={2025},\nurl={https://openreview.net/forum?id=BMfHO2lXGe}\n}"},"title":{"value":"ProtMamba: a homology-aware but alignment-free protein state space model"},"pdf":{"value":"/pdf/b0e6d7eb3e3ff7a0d5496fa74b52528e6ef7a954.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sgarbossa|protmamba_a_homologyaware_but_alignmentfree_protein_state_space_model"},"authorids":{"value":["~Damiano_Sgarbossa1","~Cyril_Malbranke1","~Anne-Florence_Bitbol1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Damiano Sgarbossa","Cyril Malbranke","Anne-Florence Bitbol"]}},"version":2},{"content":{"summary":{"value":"The paper presents a MLLM that leverages a hybrid transformer-MAMBA architecture to efficiently process long input contexts. The contributions of this study are: it introduces the MMJAMBA model optimized for lengthy contexts from high-resolution inputs, a \"train-on-short-infer-on-long\" strategy for improved training and inference efficiency. Experimental validation demonstrates stronger performance against both open-source and proprietary models."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Do the authors intend to open-source the code and meta-data?\n\n2. In Figure 2, why did the authors only compare with LLAVA-NEXT? Have other models been compared as well?\n\n3. How is the resolution on the X-axis of Figure 2 calculated?\n\n4. Have the authors compared efficiency in video modeling? It seems that the reviewer has only seen a comparison of efficiency in image modeling."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The use of a SSM architecture to enhance the modeling efficiency for long contexts and tokens in MLLM is intuitively motivated.\n\nOverall, the experimental section of the study appears to be thorough and solid, both in terms of content and effort. It also confirms that the proposed model indeed improves both efficiency and performance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The primary concern of the reviewer is the possible lack of technical innovation by the authors. Although the idea of leveraging a SSM architecture to enhance the modeling efficiency for long contexts and tokens in MLLM is interesting, the reviewer notes (in sec. 3) that the authors have simply applied the existing SSM framework to the modeling and processing of images and videos without any additional technical improvements or innovation (since this is basically explored in existing research [1-5]). While the experimental section does validate some effectiveness, the real contribution from the authors might be largely limited. And the reviewer suggests clarifying these aspects, especially distinguishing them from existing related work.\n\n- In the Related Work section, the reviewer found that the authors did not provide a necessary survey on SSM-related technical architectures and research, which might be regrettable and inappropriate, given that the backbone of this work is exactly SSM and Mamba. Have other works already explored SSM in addressing long context modeling in MLLMs, cf. [1-5]? How do their methods and technologies differ from the one proposed in this paper? What is the unique advantage of this work? These aspects should be covered in the Related Work, which the reviewer hopes the authors will complete in the revision. Therefore, the reviewer suggests that the authors promptly supplement the discussion with SSM-related research and clarify the differences.\n\n- Overall, the content presented in detail in this paper might be insufficient. For instance, the details of the training of the model proposed (Sec. 3.3) are not adequately addressed for the reviewer. It would be more informative to see such as the training resources consumed at each training step, the time taken, GPU hours, memory usage, or convergence rates and a comparison of training efficiency between non-SSM architectures and the model proposed in this work. Please provide further detailed explanations.\n\n- The authors should conduct direct experimental validation for \"low-resolution image modeling to high-resolution image modeling\", and from \"low frame rate video to high frame rate video\", since these scenarios have been repeatedly emphasized in the introduction as their strong motivation. The reviewer expects to see experimental validation in these areas as direct evidence:\n  1) Please compare the efficiency of the proposed model at both low and high resolutions.\n  2) For fairness and clarity, the authors are advised to also display the output image resolution and video frame rate of the baseline MLLM in Tables 1, 2, and 3.\n\n- Also, the authors have not directly compared the impact of token length. The reviewer suggests adding experiments regarding the end task performance and efficiency under different lengths of visual context (both language and visual tokens). \n\n- One potential concern is that the scalability of SSM-based architecture in MLLM has not been fully validated yet, i.e., whether SSM can exhibit emergent phenomena similar to the Transformer architecture. Of course, this point is a discussion regarding the entire SSM architecture and not just specific to this paper. However, the reviewer would like to see a discussion on this from the authors.\n\nOverall, the reviewer is open-minded. If the authors can actively and effectively address these concerns, the reviewer would consider raising the rating.\n\n-------\n\n[1] VL-Mamba: Exploring State Space Models for Multimodal Learning  \n[2] Efficient Classification of Long Documents via State-Space Models  \n[3] Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models  \n[4] QMambaBSR: Burst Image Super-Resolution with Query State Space Model  \n[5] LongSSM: On the Length Extension of State-space Models in Language Modelling"}},"nonreaders":[],"tmdate":1731428512043,"tcdate":1730341125640,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8139/Reviewer_zoqs"],"signatures":["ICLR.cc/2025/Conference/Submission8139/Reviewer_zoqs"],"forum":"cagNCwQEEN","number":3,"license":"CC BY 4.0","cdate":1730341125640,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8139/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428512043,"domain":"ICLR.cc/2025/Conference","replyto":"cagNCwQEEN","id":"RknQRquKMO","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["multimodal","hybrid SSM","Jamba"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Handling lengthy context is crucial for enhancing the recognition and understanding capabilities of multimodal large language models (MLLMs) in applications such as processing high-resolution images or high frame rate videos. The rise in image resolution and frame rate substantially increases computational demands due to the increased number of input tokens. This challenge is further exacerbated by the quadratic complexity with respect to sequence length of the self-attention mechanism. Most prior works either pre-train models with long contexts, overlooking the efficiency problem, or attempt to reduce the context length via downsampling (e.g., identify the key image patches or frames) to decrease the context length, which may result in information loss. To circumvent this issue while keeping the remarkable effectiveness of MLLMs, we propose a novel approach using a hybrid transformer-MAMBA model to efficiently handle long contexts in multimodal applications. Our multimodal model can effectively process long context input exceeding 100k tokens, outperforming existing models across various benchmarks. Remarkably, our model enhances inference efficiency for high-resolution images and high-frame-rate videos by about 4 times compared to current models, with efficiency gains increasing as image resolution or video frames rise. Furthermore, our model is the first to be trained on low-resolution images or low-frame-rate videos while being capable of inference on high-resolution images and high-frame-rate videos, offering flexibility for inference in diverse scenarios."},"_bibtex":{"value":"@misc{\nzhou2025multimodal,\ntitle={Multimodal Instruction Tuning with Hybrid State Space Models},\nauthor={Jianing Zhou and Han Li and Shuai Zhang and Ning Xie and Ruijie Wang and Xiaohan Nie and Sheng Liu and Suma Bhat and Lingyun Wang},\nyear={2025},\nurl={https://openreview.net/forum?id=cagNCwQEEN}\n}"},"title":{"value":"Multimodal Instruction Tuning with Hybrid State Space Models"},"pdf":{"value":"/pdf/51731f3d0f7f40bf97378c6902da6362f9a13395.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhou|multimodal_instruction_tuning_with_hybrid_state_space_models"},"authorids":{"value":["~Jianing_Zhou1","~Han_Li5","~Shuai_Zhang7","~Ning_Xie6","~Ruijie_Wang2","~Xiaohan_Nie2","~Sheng_Liu1","~Suma_Bhat1","~Lingyun_Wang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jianing Zhou","Han Li","Shuai Zhang","Ning Xie","Ruijie Wang","Xiaohan Nie","Sheng Liu","Suma Bhat","Lingyun Wang"]}},"version":2},{"content":{"summary":{"value":"This paper introduces an RNN architecture called Factorization Memory (FM). Its core innovation is that FM maintains multiple RNN hidden states (memories) and routes input tokens to the 'top-k' most relevant hidden states for updates. This approach employs sparse memory activation, and memories that are not selected remain unchanged. FM utilizes Mamba-2 as its memory update rule. The authors claim that FM performs well while being more efficient than dense activation. However, this concept has been introduced in former works."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. Could the authors add a discussion to the manuscript clarifying the relationship between FM and MoM, and detailing the specific novel contributions of this work?\n2. The paper claims the model 'factorizes' inputs into topics. What empirical evidence supports this claim beyond simple sparse routing?\n3. The speed comparison is conducted on sequences longer than 32k; what are the results on short-length sequences?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This paper aims to solve the problem that linear models often struggle to compress long sequences into a fixed-size memory. Their performance on long-context tasks, such as NIAH and recall-intensive tasks, typically falls behind Transformers. This is a well-known and valuable problem.\n2. FM shows better performance on long-context tasks, verifying the benefits of sparse memory activation and increased memory capacity.\n3. FM can be more efficient than dense activation of memories."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The key point of this paper highly overlaps with MoM [1]. Both methods adopt sparse activation of memories with a router to direct different input tokens to the top-k memories. The theoretical framework was introduced in MoM, and FM seems like a special case that adopts Mamba-2 as the memory update rule.\n2. The experiments are conducted using Mamba-2, but while FM could be applied to other linear models to verify its effect, this paper lacks such validation on other architectures (e.g., GLA, DeltaNet).\n3. This paper claims that different memories are factorized, but the experiments are all focused on task performance. I think it would be better to include an analysis of how these memories are factorized and what the role of specific memories is.\n\n[1] MoM: Linear Sequence Modeling with Mixture-of-Memories"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926941703,"tcdate":1761718775853,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16914/Reviewer_XEe4"],"signatures":["ICLR.cc/2026/Conference/Submission16914/Reviewer_XEe4"],"forum":"LPqjfqw2OI","number":2,"license":"CC BY 4.0","cdate":1761718775853,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16914/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926941703,"domain":"ICLR.cc/2026/Conference","replyto":"LPqjfqw2OI","id":"MzlwfybxXb","forumContent":{"TLDR":{"value":"We propose Factorization Memory (FM), a novel and efficient recurrent neural network (RNN) architecture that achieves performance comparable to Transformer models."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["llm","rnn","factorization memory"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"We propose Factorization Memory, an efficient recurrent neural network (RNN) architecture that achieves performance comparable to Transformer models on short-context language modeling tasks while also demonstrating superior generalization in long-context scenarios. Our model builds upon Mamba-2, enabling Factorization Memory to exploit parallel computations during training while preserving constant computational and memory complexity during inference. To further optimize model efficiency and representational capacity, we develop a sparse formulation of Factorization Memory that updates only a subset of recurrent states at each step while preserving the strong performance of its dense counterpart. To our knowledge, this represents the first RNN architecture that successfully combines sparse memory activation with competitive performance across both short and long-context settings. This work provides a systematic empirical analysis of Factorization Memory in comparison to Transformer and Mamba-2 architectures."},"_bibtex":{"value":"@misc{\nxiong2026language,\ntitle={Language Modeling With Factorization Memory},\nauthor={Lee Xiong and Maksim Tkachenko and Johanes Effendi and Ting Cai},\nyear={2026},\nurl={https://openreview.net/forum?id=LPqjfqw2OI}\n}"},"title":{"value":"Language Modeling With Factorization Memory"},"pdf":{"value":"/pdf/08058ce05792fde07379de57f8529a6f25058ca1.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"xiong|language_modeling_with_factorization_memory"},"authorids":{"value":["~Lee_Xiong1","~Maksim_Tkachenko1","~Johanes_Effendi1","~Ting_Cai2"]},"authors":{"value":["Lee Xiong","Maksim Tkachenko","Johanes Effendi","Ting Cai"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their valuable feedback. We are pleased that they found the presentation clear, the experimental evaluation thorough, and the proposed approach novel. While we were somewhat surprised by the final score given the largely positive assessment, we appreciate the reviewer's engagement with our work. In the following, we address the main questions raised and hope that these clarifications will allow the reviewer to reassess their evaluation.\n\n**1. \\& 2. Questions on expressivity.** In this paper, we study block-diagonal linear time-variant RNNs with input-dependent transition matrices. It is well known [1,2] that diagonal input-dependent linear RNNs have reduced expressivity~(we develop on what we mean below) compared to dense input-dependent linear RNNs. The reviewer correctly states that dense real matrices are diagonalizeable with probability 1. This is correct, however (as discussed thoroughly in [2]), this is not enough in the time-variant case: All $A_t$s need to be simultaneously diagonalizable for the system to enjoy the diagonal-dense equivalence the reviewer is claiming. Note that if $A_t$ is constant, as for instance in [3], then this is indeed true, but crucially is not the case for the input-dependent ``selective'' setting of all recently proposed efficient RNNs such as Mamba.\n\nTo detail more on expressivity in this context: it is easy to prove, see e.g. [4], that linear dense or diagonal RNNs with constant gates can only approximate linear functionals (path-to-point maps), even as the width increases to infinity. Instead, [2] shows theoretically that dense linear RNNs with input-dependent dense (i.e. non-constant and non-simultaneously-diagonalizable) gates can approximate arbitrary non-linear path-to-point maps as the width increases. Crucially, [2] also shows that the diagonal setting is distinct: one can approximate only specific input features that do not capture global non-linear interactions. [1] goes one step further and provides a concrete example task: state-tracking, which is not solvable with selective linear diagonal RNNs but can be solved with dense variants (see theory theorem 4.4). \n\nOverall, the studies discussed above, together with the classical results on universal approximation theorems, motivate our characterization of block-diagonal linear time-variant RNNs as forming a *hierarchy* of expressive power. We hope that this answers the reviewer's questions and are happy to discuss this further should this not be the case. \n\n**3. \\& 4. Questions on optimal choice of $m$.**\nWe thank the reviewer for raising the important question regarding the optimal choice of $m$ and we acknowledge that our initial submission did not sufficiently address this aspect. However, including this discussion can significantly enhance the practical value of our work. Accordingly, we have added an extended discussion on this topic to the Conclusion section (Appendix A). \n\n**4. On parameterization of models.**\nOur architecture uses and benefits from exponential reparameterization (softmax), one can find the exact implementation of our reparameterization in Appendix I. Addressing feedback from multiple reviewers, we have updated Figure 1 to enhance clarity. The figure now explicitly includes results for diagonal LRUs (BD-LRU and H-LRU with $m=1$) alongside Mamba2 [5,6], the state-of-the-art diagonal baseline that improves upon earlier SSMs like S4.\n\n**5. On the limitations of H-LRU.** In our discussion, we explicitly acknowledge the limitations of H-LRU, noting that its primary advantage lies in parameter efficiency rather than  peak performance. Additionally, Appendix G highlights the potential utility of non-selective H-LRU variants in heavily parameter-constrained settings, such as efficient embedding layers.\n\nReferences:\n\n[1] Merrill, W., Petty, J., \\& Sabharwal, A. (2024). The illusion of state in state-space models. arXiv preprint arXiv:2404.08819.\n\n[2] Muca Cirone, N., Orvieto, A., Walker, B., Salvi, C., \\& Lyons, T. (2024). Theoretical foundations of deep selective state-space models. Advances in Neural Information Processing Systems, 37, 127226-127272.\n\n[3] Orvieto, A., Smith, S. L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., \\& De, S. (2023, July). Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning (pp. 26670-26698). PMLR.\n\n[4] Li, Z., Han, J., \\& Li, Q. (2022). Approximation and optimization theory for linear continuous-time recurrent neural networks. Journal of Machine Learning Research, 23(42), 1-85.\n\n[5] Gu, A., \\& Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint arXiv:2312.00752.\n\n[6] Dao, T., \\& Gu, A. (2024). Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764189249306,"tcdate":1764188568086,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17598/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission17598/Authors"],"forum":"fF72TYOKjZ","number":8,"license":"CC BY 4.0","cdate":1764188568086,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17598/-/Official_Comment"],"mdate":1764189249306,"domain":"ICLR.cc/2026/Conference","replyto":"PLtJKqGO6a","id":"AcKpHUCnlC","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["linear RNNs","higher-order recurrence","block diagonal recurrence","normalization","synthetic tasks"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Linear recurrent networks (LRNNs) and linear state space models (SSMs) promise computational and memory efficiency on long-sequence modeling tasks, yet their diagonal state transitions limit expressivity. Dense and/or nonlinear architectures (e.g., LSTMs) on the other hand are provably more expressive, but computationally costly. \nHere, we explore how expressivity in LRNNs can be increased via richer state mixing across time and channels while maintaining competitive efficiency. Specifically, we introduce two structured LRNN architectures: (i) Higher-order Linear Recurrent Units (H-LRU), which generalize first-order recurrence to $m$-th order, mixing multiple past states, and (ii) Block-Diagonal LRUs (BD-LRU), which enable dense intra-block channel mixing. Per-channel (H-LRU) / per-row (BD-LRU) L1-normalization of selective gates stabilizes training and allows for scaling window/block sizes. In synthetic sequence-modeling benchmarks (compression, selective copying, associative recall), H-LRU is found to be the most parameter-efficient in compression, while the performance of BD-LRU matches or exceeds those of linear SSMs (Mamba), low-rank LRNNs (DeltaNet) and LSTM baselines. In permutation composition tasks ($S_3$-$S_5$), BD-LRU is found to efficiently solve these tasks at moderate block sizes, outperforming both linear and non-linear baselines. A parallel-scan implementation of the proposed architectures keeps the throughput competitive with diagonal LRNNs for moderate orders (H-LRU) and block sizes (BD-LRU), while preserving the efficiency that motivated LRNNs. \nThese results indicate that the structure of state mixing rather than width alone shapes expressivity of LRNNs, offering a practical route to closing the efficiency–expressivity gap in linear sequence models."},"_bibtex":{"value":"@misc{\ndubinin2026improved,\ntitle={Improved state mixing in higher-order and block diagonal linear recurrent networks},\nauthor={Igor Dubinin and Antonio Orvieto and Felix Effenberger},\nyear={2026},\nurl={https://openreview.net/forum?id=fF72TYOKjZ}\n}"},"title":{"value":"Improved state mixing in higher-order and block diagonal linear recurrent networks"},"pdf":{"value":"/pdf/7a4d7bd95df82a2db1bd82c38287ef7ad0d06fc5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"dubinin|improved_state_mixing_in_higherorder_and_block_diagonal_linear_recurrent_networks"},"authorids":{"value":["~Igor_Dubinin1","~Antonio_Orvieto3","~Felix_Effenberger1"]},"authors":{"value":["Igor Dubinin","Antonio Orvieto","Felix Effenberger"]}},"version":2},{"content":{"summary":{"value":"This paper introduces LiNo, a Recursive Residual Decomposition framework for time series forecasting. LiNo addresses the limitations of existing deep learning models by extracting both linear and nonlinear patterns. The framework employs a Li block to capture linear patterns and a No block to model nonlinear patterns. This recursive and deeper-level decomposition enhances the extraction of complex, multi-granular patterns in real-world time series data. Experimental results on thirteen real-world benchmarks show that LiNo outperforms state-of-the-art models in both univariate and multivariate forecasting scenarios, providing robust and precise predictions."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please see the weaknesses above."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. In experimental evaluations, the proposed method surpasses state-of-the-art models in both univariate and multivariate forecasting scenarios.\n2. The method is easy to understand.\n3. This paper proposes an efficient method to capture the linear and nonlinear patterns for time series forecasting."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The components of the method, while effective, are not entirely novel. Recursive residual decomposition has been demonstrated to be effective in N-BEATS [1], and the utility of frequency-domain MLPs is highlighted in FreMLP [2]. Additionally, the channel mixing technique used in LiNo is similar to that used in SOFTS [3], and it would be beneficial to include a more detailed comparison and discussion of these similarities.\n2. The authors argue that the primary distribution involves linear and nonlinear decomposition, with ablation studies primarily focused on multivariate time series forecasting. In line 428, they state, \"Extracting nonlinear patterns, such as inter-series dependencies, temporal variations, and frequency information, is crucial for accurate predictions.\" The classification of inter-series dependencies as nonlinear patterns may be debatable, as methods like the Multivariate Autoregressive Model, which are inherently linear, can also capture such dependencies. It would be valuable to separately study the effects of the channel mixing method and the linear-nonlinear decomposition. The claimed superiority of the No block ablation, using iTransformer and TSMixer, which mainly attributes its performance to the channel mixing part, could benefit from further evidence (Table 5, Figure 4).\n3. The paper lacks an error bar analysis. To demonstrate the robustness of the proposed method, the experiments should be conducted multiple times using different random seeds.\n4. The paper contains a few minor typographical errors. For instance, on line 103, \"TSMxier\" should be corrected to \"TSMixer,\" and on line 252, \"ReVIN\" should be changed to \"RevIN.\"\n\n[1] Oreshkin B N, Carpov D, Chapados N, et al. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting[J]. ICLR, 2020.\n\n[2] Yi K, Zhang Q, Fan W, et al. Frequency-domain MLPs are more effective learners in time series forecasting[J]. NeurIPS, 2023.\n\n[3] Lu H, XY Chen, et al. SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion. ArXiv, 2024."}},"nonreaders":[],"tmdate":1731427452709,"tcdate":1730366598085,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1614/Reviewer_dLTN"],"signatures":["ICLR.cc/2025/Conference/Submission1614/Reviewer_dLTN"],"forum":"JeZS4jQF77","number":2,"license":"CC BY 4.0","cdate":1730366598085,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1614/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427452709,"domain":"ICLR.cc/2025/Conference","replyto":"JeZS4jQF77","id":"yrTWiTeTcX","forumContent":{"TLDR":{"value":"We propose LiNo, a framework offering better linear and nonlinear patterns decomposition in time series data to enhance forecasting accuracy and robustness."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time Series Forecasting","Deep learning."]},"supplementary_material":{"value":"/attachment/f36dc2f069b68d2e524f94233db2602455035352.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Forecasting models are pivotal in a data-driven world with vast volumes of time series data that appear as a compound of vast $\\textbf{Li}$near and $\\textbf{No}$nlinear patterns. \nRecent deep time series forecasting models struggle to utilize seasonal and trend decomposition to separate the entangled components. Such a strategy only explicitly extracts simple linear patterns like trends, leaving the other linear modes and vast unexplored nonlinear patterns to the residual. Their flawed linear and nonlinear feature extraction models and shallow-level decomposition limit their adaptation to the diverse patterns present in real-world scenarios.\nGiven this, we innovate Recursive Residual Decomposition by introducing explicit extraction of both linear and nonlinear patterns. This deeper-level decomposition framework, which is named $\\textbf{LiNo}$, captures linear patterns using a Li block which can be a moving average kernel, and models nonlinear patterns using a No block which can be a Transformer encoder. The extraction of these two patterns is performed alternatively and recursively. To achieve the full potential of LiNo, we develop the current simple linear pattern extractor to a general learnable autoregressive model, and design a novel No block that can handle all essential nonlinear patterns.\nRemarkably, the proposed LiNo achieves state-of-the-art on thirteen real-world benchmarks under univariate and multivariate forecasting scenarios. Experiments show that current forecasting models can deliver more robust and precise results through this advanced Recursive Residual Decomposition. We hope this work could offer insight into designing more effective forecasting models. Code is available at this anonymous repository: https://anonymous.4open.science/r/LiNo-8225/."},"_bibtex":{"value":"@misc{\nyu2025lino,\ntitle={LiNo: Advancing Recursive Residual Decomposition of Linear and Nonlinear Patterns for Robust Time Series Forecasting},\nauthor={Guoqi Yu and Yaoming Li and Xiaoyu Guo and Shujun Wang and Zirui Liu and Dayu Wang and Tong Yang},\nyear={2025},\nurl={https://openreview.net/forum?id=JeZS4jQF77}\n}"},"title":{"value":"LiNo: Advancing Recursive Residual Decomposition of Linear and Nonlinear Patterns for Robust Time Series Forecasting"},"pdf":{"value":"/pdf/fa10c828099387707e229fc9c4a29cb300051efb.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"yu|lino_advancing_recursive_residual_decomposition_of_linear_and_nonlinear_patterns_for_robust_time_series_forecasting"},"authorids":{"value":["~Guoqi_Yu1","~Yaoming_Li1","~Xiaoyu_Guo6","~Shujun_Wang1","~Zirui_Liu2","~Dayu_Wang1","~Tong_Yang6"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Guoqi Yu","Yaoming Li","Xiaoyu Guo","Shujun Wang","Zirui Liu","Dayu Wang","Tong Yang"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We introduce an OBS–based pruning methodology tailored to SSM-based architectures such as Mamba, effectively reducing redundancy while preserving model performance."},"keywords":{"value":["Neural network pruning","Mamba","Hessian-based pruning"]},"supplementary_material":{"value":"/attachment/6b5f630754dd0367304e126456137bc23a85f5bf.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"State-space language models such as Mamba match Transformer quality while permitting linear complexity inference, yet still comprise billions of parameters that hinder deployment. Existing one-shot pruning methods are tailored to attention blocks and fail to account for the time-shared and discretized state-transition matrix at the heart of the selective state-space module (SSM). In this paper, we introduce SparseSSM, the first training-free pruning framework that extends the classic optimal brain surgeon (OBS) framework to state space architectures. Our layer-wise algorithm (i) derives an approximate second-order saliency score that aggregates Hessian-trace information across time steps, (ii) incorporates a component sensitivity analysis to guide feed-forward network (FFN) pruning, which also sheds light on where redundancy resides in mamba architecture, (iii) can be easily extended to semi-structured and structured sparsity. Empirically, we prune 50% of SSM weights without fine-tuning and observe no zero-shot accuracy loss, achieving the current state-of-the-art pruning algorithm for Mamba-based LLMs."},"_bibtex":{"value":"@misc{\ntuo2026sparsessm,\ntitle={Sparse{SSM}: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot},\nauthor={Kaiwen Tuo and Huan Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=bzb9OJVfCP}\n}"},"title":{"value":"SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot"},"pdf":{"value":"/pdf/8e219971d001aa872a5a0e48ae02be58f8abb56b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"tuo|sparsessm_efficient_selective_structured_state_space_models_can_be_pruned_in_oneshot"},"authorids":{"value":["~Kaiwen_Tuo1","~Huan_Wang3"]},"authors":{"value":["Kaiwen Tuo","Huan Wang"]}},"tmdate":1770804807823,"tcdate":1756877961298,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1381/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission1381/Authors"],"forum":"bzb9OJVfCP","license":"CC BY 4.0","number":1381,"cdate":1756877961298,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission1381/-/Full_Submission","ICLR.cc/2026/Conference/Submission1381/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804807823,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"bzb9OJVfCP","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2412.07481v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"huang|manta_enhancing_mamba_for_fewshot_action_recognition_of_long_subsequence"},"authorids":{"value":["~Wenbo_Huang1","~Jinghui_Zhang2","~Guang_Li2","https://dblp.org/search/pid/api?q=author:Lei_Zhang_0130:","https://dblp.org/search/pid/api?q=author:Shuoyuan_Wang:","https://dblp.org/search/pid/api?q=author:Fang_Dong:","https://dblp.org/search/pid/api?q=author:Jiahui_Jin:","~Takahiro_Ogawa1","~Miki_Haseyama1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2412.07481"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-07481,\n  publtype={informal},\n  author={Wenbo Huang and Jinghui Zhang and Guang Li and Lei Zhang and Shuoyuan Wang and Fang Dong and Jiahui Jin and Takahiro Ogawa and Miki Haseyama},\n  title={Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2412.07481},\n  url={https://doi.org/10.48550/arXiv.2412.07481}\n}\n"},"abstract":{"value":"In few-shot action recognition (FSAR), long sub-sequences of video naturally express entire actions more effectively. However, the high computational complexity of mainstream Transformer-based methods limits their application. Recent Mamba demonstrates efficiency in modeling long sequences, but directly applying Mamba to FSAR overlooks the importance of local feature modeling and alignment. Moreover, long sub-sequences within the same class accumulate intra-class variance, which adversely impacts FSAR performance. To solve these challenges, we propose a Matryoshka MAmba and CoNtrasTive LeArning framework (Manta). Firstly, the Matryoshka Mamba introduces multiple Inner Modules to enhance local feature representation, rather than directly modeling global features. An Outer Module captures dependencies of timeline between these local features for implicit temporal alignment. Secondly, a hybrid contrastive learning paradigm, combining both supervised and unsupervised methods, is designed to mitigate the negative effects of intra-class variance accumulation. The Matryoshka Mamba and the hybrid contrastive learning paradigm operate in two parallel branches within Manta, enhancing Mamba for FSAR of long sub-sequence. Manta achieves new state-of-the-art performance on prominent benchmarks, including SSv2, Kinetics, UCF101, and HMDB51. Extensive empirical studies prove that Manta significantly improves FSAR of long sub-sequence from multiple perspectives."},"title":{"value":"Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence"},"authors":{"value":["Wenbo Huang","Jinghui Zhang","Guang Li","Lei Zhang","Shuoyuan Wang","Fang Dong","Jiahui Jin","Takahiro Ogawa","Miki Haseyama"]}},"tmdate":1768461295647,"pdate":1704067200000,"tcdate":1737610756546,"writers":["~"],"signatures":["~Wenbo_Huang1"],"forum":"N15W6t7uyf","license":"CC BY-SA 4.0","number":270262,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1768461295647,"domain":"DBLP.org","id":"N15W6t7uyf","version":2},{"content":{"venue":{"value":"AAAI 2025"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/32391/34546"},"venueid":{"value":"dblp.org/conf/AAAI/2025"},"paperhash":{"value":"huang|manta_enhancing_mamba_for_fewshot_action_recognition_of_long_subsequence"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Wenbo_Huang_0001:","~Jinghui_Zhang2","~Guang_Li2","https://dblp.org/search/pid/api?q=author:Lei_Zhang_0130:","https://dblp.org/search/pid/api?q=author:Shuoyuan_Wang:","https://dblp.org/search/pid/api?q=author:Fang_Dong:","~Jiahui_Jin1","~Takahiro_Ogawa1","https://dblp.org/search/pid/api?q=author:Miki_Haseyama:"]},"html":{"value":"https://doi.org/10.1609/aaai.v39i4.32391"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/0001Z00WDJ0H25,\n  author={Wenbo Huang and Jinghui Zhang and Guang Li and Lei Zhang and Shuoyuan Wang and Fang Dong and Jiahui Jin and Takahiro Ogawa and Miki Haseyama},\n  title={Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence},\n  year={2025},\n  cdate={1735689600000},\n  pages={3751-3759},\n  url={https://doi.org/10.1609/aaai.v39i4.32391},\n  booktitle={AAAI},\n  crossref={conf/aaai/2025}\n}\n"},"abstract":{"value":"In few-shot action recognition (FSAR), long sub-sequences of video naturally express entire actions more effectively. However, the high computational complexity of mainstream Transformer-based methods limits their application. Recent Mamba demonstrates efficiency in modeling long sequences, but directly applying Mamba to FSAR overlooks the importance of local feature modeling and alignment. Moreover, long sub-sequences within the same class accumulate intra-class variance, which adversely impacts FSAR performance. To solve these challenges, we propose a Matryoshka MAmba and CoNtrasTive LeArning framework (Manta). Firstly, the Matryoshka Mamba introduces multiple Inner Modules to enhance local feature representation, rather than directly modeling global features. An Outer Module captures dependencies of timeline between these local features for implicit temporal alignment. Secondly, a hybrid contrastive learning paradigm, combining both supervised and unsupervised methods, is designed to mitigate the negative effects of intra-class variance accumulation. The Matryoshka Mamba and the hybrid contrastive learning paradigm operate in two parallel branches within Manta, enhancing Mamba for FSAR of long sub-sequence. Manta achieves new state-of-the-art performance on prominent benchmarks, including SSv2, Kinetics, UCF101, and HMDB51. Extensive empirical studies prove that Manta significantly improves FSAR of long sub-sequence from multiple perspectives."},"title":{"value":"Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence"},"authors":{"value":["Wenbo Huang","Jinghui Zhang","Guang Li","Lei Zhang","Shuoyuan Wang","Fang Dong","Jiahui Jin","Takahiro Ogawa","Miki Haseyama"]}},"tmdate":1768461279102,"pdate":1735689600000,"externalIds":["dblp:conf/aaai/0001Z00WDJ0H25"],"tcdate":1754013811219,"writers":["~"],"signatures":["~Jinghui_Zhang2"],"forum":"IUTLgzs1ov","license":"CC BY-SA 4.0","number":607167,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1768461279102,"domain":"DBLP.org","id":"IUTLgzs1ov","version":2},{"content":{"summary":{"value":"This paper proposes Gated DeltaNet, a novel model architecture combing gated recurrent mechanism and Delta update rule. Gated DeltaNet support chunk-parallel computation as efficiency optimization. Besides simple Gated DeltaNet, the paper also discuses the hybrid design with Sliding-Window Attention or Mamba layers.\n\nIn experiments, the paper shows strong modeling capability of Gated DeltaNet. First, Gated DeltaNet outperforms Transformer and other linear-complexity models in language modeling and zero-shot downstream tasks. Second, Gated DeltaNet shows a strong length-extrapolation ability in language modeling. In ablation studies, the paper makes comparisons with other design choices."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"How do you think the application scenario of linear-complexity models? Since it is going to be common sense that linear complexity makes a sacrifice on long-context capability."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper is well written, showing the connection and improvement with previous works.\n\n2. Gated DeltaNet makes kernel optimization to achieve better training and inference throughput, which is essential for downstream application and community reproduction.\n\n3. Gated DeltaNet shows strong performance than other linear-complexity models, which is a good academic contribution in model architecture area."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. This paper does not discuss the performance on long-context tasks. Since the quadratic complexity only matters when the sequence is long, long-context performance is the most important metric for linear-complexity models.\n\n2. This paper does not compare Gated DeltaNet with other linear-complexity models on training and inference efficiency. Since Delta-rule is harder for parallel computation, there may be a trade-off between performance and computation cost."}},"nonreaders":[],"tmdate":1731427194807,"tcdate":1730215435032,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission90/Reviewer_3Thw"],"signatures":["ICLR.cc/2025/Conference/Submission90/Reviewer_3Thw"],"forum":"r8H7xhYPwz","number":1,"license":"CC BY 4.0","cdate":1730215435032,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission90/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427194807,"domain":"ICLR.cc/2025/Conference","replyto":"r8H7xhYPwz","id":"Ja6vgbX0IK","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"We introduce Gated DeltaNet, which combines the gating mechanism from Mamba2 with the delta rule from DeltaNet, achieving superior performance compared to both models individually."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["linear RNN","state-space model","linear transformer","subquadractic model","linear attention","delta rule","mamba"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited.  To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications. We observe that these mechanisms are complementary—gating enables rapid memory erasure while the delta rule facilitates targeted updates. Building on this insight, we introduce the gated delta rule and develop a parallel training algorithm optimized for modern hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks, including language modeling, common-sense reasoning, in-context retrieval, length extrapolation, and long-context understanding. We further enhance performance by developing hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance."},"_bibtex":{"value":"@inproceedings{\nyang2025gated,\ntitle={Gated Delta Networks: Improving Mamba2 with Delta Rule},\nauthor={Songlin Yang and Jan Kautz and Ali Hatamizadeh},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=r8H7xhYPwz}\n}"},"title":{"value":"Gated Delta Networks: Improving Mamba2 with Delta Rule"},"pdf":{"value":"/pdf/3b2c0abfb1a7e49a0aec0aacedcce4ef8fbd620d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"yang|gated_delta_networks_improving_mamba2_with_delta_rule"},"authorids":{"value":["~Songlin_Yang1","~Jan_Kautz1","~Ali_Hatamizadeh1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Songlin Yang","Jan Kautz","Ali Hatamizadeh"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2407.06841v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"shen|htdmamba_efficient_hyperspectral_target_detection_with_pyramid_state_space_model"},"authorids":{"value":["~Dunbin_Shen1","https://dblp.org/search/pid/api?q=author:Xuanbing_Zhu:","https://dblp.org/search/pid/api?q=author:Jiacheng_Tian:","https://dblp.org/search/pid/api?q=author:Jianjun_Liu:","~Zhenrong_Du1","https://dblp.org/search/pid/api?q=author:Hongyu_Wang_0001:","https://dblp.org/search/pid/api?q=author:Xiaorui_Ma:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2407.06841"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2407-06841,\n  publtype={informal},\n  author={Dunbin Shen and Xuanbing Zhu and Jiacheng Tian and Jianjun Liu and Zhenrong Du and Hongyu Wang and Xiaorui Ma},\n  title={HTD-Mamba: Efficient Hyperspectral Target Detection with Pyramid State Space Model},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2407.06841},\n  url={https://doi.org/10.48550/arXiv.2407.06841}\n}\n"},"abstract":{"value":"Hyperspectral target detection (HTD) identifies objects of interest from complex backgrounds at the pixel level, playing a vital role in Earth observation. However, HTD faces challenges due to limited prior knowledge and spectral variation, leading to underfitting models and unreliable performance. To address these challenges, this paper proposes an efficient self-supervised HTD method with a pyramid state space model (SSM), named HTD-Mamba, which employs spectrally contrastive learning to distinguish between target and background based on the similarity measurement of intrinsic features. Specifically, to obtain sufficient training samples and leverage spatial contextual information, we propose a spatial-encoded spectral augmentation technique that encodes all surrounding pixels within a patch into a transformed view of the center pixel. Additionally, to explore global band correlations, we divide pixels into continuous group-wise spectral embeddings and introduce Mamba to HTD for the first time to model long-range dependencies of the spectral sequence with linear complexity. Furthermore, to alleviate spectral variation and enhance robust representation, we propose a pyramid SSM as a backbone to capture and fuse multiresolution spectral-wise intrinsic features. Extensive experiments conducted on four public datasets demonstrate that the proposed method outperforms state-of-the-art methods in both quantitative and qualitative evaluations. Code is available at \\url{https://github.com/shendb2022/HTD-Mamba}."},"title":{"value":"HTD-Mamba: Efficient Hyperspectral Target Detection with Pyramid State Space Model"},"authors":{"value":["Dunbin Shen","Xuanbing Zhu","Jiacheng Tian","Jianjun Liu","Zhenrong Du","Hongyu Wang","Xiaorui Ma"]}},"tmdate":1785462823411,"pdate":1704067200000,"tcdate":1738843933125,"writers":["~"],"signatures":["~Dunbin_Shen1"],"forum":"Xy3rR2fkdi","license":"CC BY-SA 4.0","number":307043,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1785462823411,"domain":"DBLP.org","id":"Xy3rR2fkdi","version":2},{"content":{"summary":{"value":"This paper is mostly a theoretical study on the computational expressiveness of the Mamba model. \nIt studied Mamba's ability on a self-defined, non-standard \"COPY\" task and the computational cost needed to solve dynamic programming (DP) problems.\nUnder several important yet *opaque* assumptions, the paper draws several contingent conclusions ---\n\n* Mamba *\"may\"* have trouble performing the \"COPY\" task reliably when the context to be copied can have arbitrary length (while the model is of a constant size)\n* If the model size is allowed to scale linearly with context length to be copied, it can perform the \"COPY\" task perfectly\n* Mamba requires a computational cost to Transformer to solve DP problems when equipped with CoT\n* In DP problems with locality properties, Mamba can save computation\n\nIn addition to theoretical analysis, which comprises most of the paper, the authors also conduct empirical experiments that further sheds light on the realistic behaviors of Mamba on the COPY task."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"As written, variables in definitions/theorems/assumptions are so entangled that, as a reader, it is quite hard to understand their logical relationships, e.g.\n\n* the task definition depends on $\\delta$, but then the theorem states that $\\delta$ observes some constraint; it would be helpful to discuss this circular dependence\n\n* the task definition depends on $b_i, c_i$ which depends on the Mamba block, but then the theorem shows the existence of a Mamba block given the task.  Again, it would be helpful to discuss this circular dependence\n\nAnother concrete way to improve these confusion points is to stay close to standard mathematical language, such as \"Given ___, there exists ___, s.t. ___ holds\" and ensure that the assumptions are actually made clear.  For example, Definition 1 probably should include \"Given a Mamba block and ...\"; Assumption 1 first part can be written as \"There exists $\\rho$ such that for any $i \\in [N]$, ... holds\"."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"* I appreciate the detailed yet concise background introduction to Mamba in Section 3 and 4.1.  These sections are well-written, set up good notation and summarize the literature on Mamba's architecture and its connection to linear attention well.\n\n* Despite the existing theoretical work on SSM's ability to perform COPY task, the authors attempted to formulate a different kind of COPY task and studies Mamba's ability to solve it.  This formulation goes beyond the discretized formulation of the COPY in prior work and study the numerical approximation aspect (however, to make this possible, the authors introduces several *significant* caveats that I'll detail in Weakness section). \n\n* Studying the computational complexity of Mamba using the connection between SSM and linear attention is novel.  This angle allows the authors to adapt the proofs in Feng et al. (2024) that were intended for the softmax-attention in the standard Transformer.\n\n* To the extent of the portions that I have checked, the mathematical derivations seem correct (this does not mean that the results are significant)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"My main issue is that the theoretical results presented in the paper not only comes with significant caveats (in the form of assumptions) that are very hard to interpret, but also have conclusions that are hard to interpret and may be too weak to draw conclusions from.  These issues are somewhat swept under the rug by the authors since they did not discuss the implications. \n\nStarting with Definition 1, it appears that the set $S_i$ is actually dependent on a specific instance of Mamba model since the definition of the COPY task involves the intermediate hidden states $c_i$ and $b_i$.  Furthermore, the COPY task is limited to the cases where the positions to copy must be in the set that's defined *with respect to* a specific instance of Mamba model.  The fact the task is dependent on the specific model parameters seems circular and hard to interpret, but the authors made no effort in discussing or disclosing this caveat.  \n\nAside from the previous hidden assumption, the actual Assumption 1 is also opaque.  It's very unclear when an actual Mamba instance will satisfy this assumption, and without such discussion, the theorems derived under this assumption should not be used to draw even more general conclusions, such as \"Mamba models may encounter bottlenecks when handling COPY task\".  The assumption itself is also written in rather confusing language.  I spent a long time trying to figure out the dependence of the variables.\n\nFinally, theorem 1 states that there is a specific Mamba model that can perform COPY but its lower bound $\\rho$ can blow up.  What it *doesn't* say, is that any Mamba model that can perform COPY (up to $\\epsilon$) will suffer from this exponential blow up.  The statement of the theorem seems too weak that the later \"conclusion\" seems misleading.  I interpreted this as -- Given the task and $L$, one can find a Mamba block that approximates the COPY task well; but the theorem says nothing about the rest of the Mamba blocks in the hypothesis space.  This is in contrast to previous work, such as Jelassi et al. (2024), who provided universal error bounds for SSMs on the COPY task.\n\nOverall, the dependence between variables should be made clear (See concrete confusion points in the Questions section).\nThe authors can perhaps think deeper about the effects of the caveats and incorporate such thinking into the writing. \nMoreover, the authors should more explicitly frame the limitations of their conclusions, so that readers won't think their time is wasted after they decided to dig deeper into paper based on the abstract or introduction.  I'd be glad to update my scores if these concerns are addressed."}},"nonreaders":[],"tmdate":1731428782819,"tcdate":1731235557660,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6956/Reviewer_PWf8"],"signatures":["ICLR.cc/2025/Conference/Submission6956/Reviewer_PWf8"],"forum":"UYarAv7rUx","number":5,"license":"CC BY 4.0","cdate":1731235557660,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6956/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428782819,"domain":"ICLR.cc/2025/Conference","replyto":"UYarAv7rUx","id":"aCEEr6WVpH","forumContent":{"TLDR":{"value":"This paper explores the expressive power of Mamba."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Expressive Power","Chain of Thought"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Transformers have been the cornerstone of current Large Language Models (LLMs); however, its linear growth in overhead during inference with respect to sequence length poses challenges for modeling long sequences. In this context, Mamba has gradually attracted attention due to its constant-level size during inference and existing empirical results have shown that it can perform comparably to Transformers in sequence modeling while offering significant savings. However, one may ask that, can Mamba always enjoy the ``free lunch\"? In this paper, we focus on analyzing the expressive ability of Mamba from a theoretical standpoint. First, inspired by the connection between Mamba and linear attention, we investigate potential shortcomings of the Mamba when performing the COPY operation. Our results indicate that Mamba with constant size may encounter bottlenecks when handling COPY, while it can achieve perfect performance when the size scales linearly with sequence length. Based on this observation, we analyze Mamba's ability to tackle DP problems when equipped with Chain of Thought (CoT). Our findings suggest that to solve arbitrary DP problems, the total cost of Mamba is comparable to standard and efficient Transformers. However, similar to efficient Transformers, when facing DP problems with favorable properties such as locality, Mamba can provide savings in overhead. Our results contribute to a deeper understanding of Mamba."},"_bibtex":{"value":"@misc{\nren2025can,\ntitle={Can Mamba Always Enjoy the ''Free Lunch''?},\nauthor={Ruifeng Ren and Zhicong Li and Yong Liu},\nyear={2025},\nurl={https://openreview.net/forum?id=UYarAv7rUx}\n}"},"title":{"value":"Can Mamba Always Enjoy the \"Free Lunch\"?"},"pdf":{"value":"/pdf/b4552a0f966e40cf480dfcacf744f021f4b8b521.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ren|can_mamba_always_enjoy_the_free_lunch"},"authorids":{"value":["~Ruifeng_Ren3","~Zhicong_Li1","~Yong_Liu7"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Ruifeng Ren","Zhicong Li","Yong Liu"]}},"version":2},{"content":{"summary":{"value":"This paper investigates Mamba's length extrapolation capabilities through a systematic analysis approach. The authors introduce the Mamba Mean Distance metric as a novel way to quantify and analyze Mamba's ability to model long-range dependencies. Their findings reveal inherent limitations in Mamba's capacity to handle longer sequences effectively. To address this challenge, they propose a selective token processing mechanism leveraging the delta t parameter from the Mamba formula. This approach intelligently filters tokens by retaining those with larger delta t values, effectively aligning the input complexity with Mamba's modeling capacity. The method is specifically designed for the prefilling phase and demonstrates promising improvements in Mamba's long-text processing capabilities."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- The relationship between distance and performance shown in Fig 1 appears counterintuitive. If Mamba struggles with long-distance relationships, why does the performance degradation manifest primarily in middle lengths rather than showing a clear diagonal boundary from top-left to bottom-right between the red and green regions? Could this suggest a more complex underlying mechanism?\n- Given the increasing importance of long-text generation in applications like complex reasoning and creative writing, have the authors explored possibilities of extending this method beyond prefilling to support long-text generation scenarios? It would be benefitial to discuss any challenges the authors foresee in extending the approach to generation tasks.\n- The experimental setup shows variations in model selection across different figures and analyses (e.g. Fig 3 and Fig 4). Could the authors provide more detailed justification for these choices and discuss how they might impact the generalizability of the findings?\n- Regarding the Mamba Mean Distance results in Fig 4, the findings don't definitively prove that Mamba's long-text capability decreases with length, as this metric wouldn't necessarily scale linearly with sequence length. Are there similar studies for attention-based architectures?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The investigation of token importance scoring in the context of Mamba represents a valuable contribution to the field, especially given the growing interest in alternatives to attention-based mechanisms\n- The method achieves substantial improvements while maintaining implementation simplicity, making it readily applicable in practical scenarios"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Limited scope of application: The method's restriction to the prefilling phase significantly limits its practical impact, especially considering the increasing demand for both long-text prefilling and generation in modern applications\n- Potential information loss: The token discarding approach may have unintended consequences in scenarios requiring comprehensive context understanding. This is particularly problematic in tasks like document question-answering, where discarded tokens during prefilling might be crucial for subsequent generation phases\n- Incomplete solution to fundamental limitations: While the approach provides a practical workaround, it doesn't address the underlying limitations of Mamba in processing long sequences. A more thorough analysis of the Mamba Mean Distance metric could potentially lead to more fundamental solutions"}},"nonreaders":[],"tmdate":1731428295555,"tcdate":1730643312551,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5066/Reviewer_gREq"],"signatures":["ICLR.cc/2025/Conference/Submission5066/Reviewer_gREq"],"forum":"iWSl5Zyjjw","number":3,"license":"CC BY 4.0","cdate":1730643312551,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5066/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428295555,"domain":"ICLR.cc/2025/Conference","replyto":"iWSl5Zyjjw","id":"xPKah23ymI","forumContent":{"TLDR":{"value":"We explore Mamba's length extrapolation potential, identifying its limitation due to a restricted effective receptive field, and introduce DeciMamba, the first context-extension method specifically designed for Mamba."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Long context","Context extension","Long-range language modeling"]},"supplementary_material":{"value":"/attachment/7b340eb64654c7225cf43f5c6138ec972b453804.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrates high performance and achieves Transformer-level capabilities while requiring substantially fewer computational resources. In this paper we explore the length-generalization capabilities of Mamba, which we find to be relatively limited. Through a series of visualizations and analyses we identify that the limitations arise from a restricted effective receptive field, dictated by the sequence length used during training. To address this constraint, we introduce DeciMamba, a context-extension method specifically designed for Mamba. This mechanism, built on top of a hidden filtering mechanism embedded within the S6 layer, enables the trained model to extrapolate well even without additional training. Empirical experiments over real-world long-range NLP tasks show that DeciMamba can extrapolate to context lengths that are significantly longer than the ones seen during training, while enjoying faster inference. We will release our code and models."},"_bibtex":{"value":"@inproceedings{\nben-kish2025decimamba,\ntitle={DeciMamba: Exploring the Length Extrapolation Potential of Mamba},\nauthor={Assaf Ben-Kish and Itamar Zimerman and Shady Abu-Hussein and Nadav Cohen and Amir Globerson and Lior Wolf and Raja Giryes},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=iWSl5Zyjjw}\n}"},"title":{"value":"DeciMamba: Exploring the Length Extrapolation Potential of Mamba"},"pdf":{"value":"/pdf/b2e18164fc1679d810d80cd5fdb09a2cc1a6003f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"benkish|decimamba_exploring_the_length_extrapolation_potential_of_mamba"},"authorids":{"value":["~Assaf_Ben-Kish1","~Itamar_Zimerman1","~Shady_Abu-Hussein1","~Nadav_Cohen1","~Amir_Globerson1","~Lior_Wolf1","~Raja_Giryes1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Assaf Ben-Kish","Itamar Zimerman","Shady Abu-Hussein","Nadav Cohen","Amir Globerson","Lior Wolf","Raja Giryes"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2505.08299v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"shihab|efficient_unstructured_pruning_of_mamba_statespace_models_for_resourceconstrained_environments"},"authorids":{"value":["~Ibne_Farabi_Shihab1","~Sanjeda_Akter1","~Anuj_Sharma1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2505.08299"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2505-08299,\n  publtype={informal},\n  author={Ibne Farabi Shihab and Sanjeda Akter and Anuj Sharma},\n  title={Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments},\n  year={2025},\n  month={May},\n  cdate={1746057600000},\n  journal={CoRR},\n  volume={abs/2505.08299},\n  url={https://doi.org/10.48550/arXiv.2505.08299}\n}\n"},"abstract":{"value":"State-space models (SSMs), particularly the Mamba architecture, have emerged as powerful alternatives to Transformers for sequence modeling, offering linear-time complexity and competitive performance across diverse tasks. However, their large parameter counts pose significant challenges for deployment in resource-constrained environments. We propose a novel unstructured pruning framework tailored for Mamba models that achieves up to 70\\% parameter reduction while retaining over 95\\% of the original performance. Our approach integrates three key innovations: (1) a gradient-aware magnitude pruning technique that combines weight magnitude and gradient information to identify less critical parameters, (2) an iterative pruning schedule that gradually increases sparsity to maintain model stability, and (3) a global pruning strategy that optimizes parameter allocation across the entire model. Through extensive experiments on WikiText-103, Long Range Arena, and ETT time-series benchmarks, we demonstrate significant efficiency gains with minimal performance degradation. Our analysis of pruning effects on Mamba's components reveals critical insights into the architecture's redundancy and robustness, enabling practical deployment in resource-constrained settings while broadening Mamba's applicability."},"title":{"value":"Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments"},"authors":{"value":["Ibne Farabi Shihab","Sanjeda Akter","Anuj Sharma"]}},"tmdate":1791746066505,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2505-08299"],"tcdate":1756244323429,"writers":["~"],"signatures":["~Ibne_Farabi_Shihab1"],"forum":"PAYtPd0QLz","license":"CC BY-SA 4.0","number":618469,"cdate":1746057600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1791746066505,"domain":"DBLP.org","id":"PAYtPd0QLz","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.04419v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"ahamed|tscmamba_mamba_meets_multiview_learning_for_time_series_classification"},"authorids":{"value":["~Md_Atik_Ahamed1","https://dblp.org/search/pid/api?q=author:Qiang_Cheng_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.04419"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-04419,\n  publtype={informal},\n  author={Md Atik Ahamed and Qiang Cheng},\n  title={TSCMamba: Mamba Meets Multi-View Learning for Time Series Classification},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.04419},\n  url={https://doi.org/10.48550/arXiv.2406.04419}\n}\n"},"abstract":{"value":"Time series classification (TSC) on multivariate time series is a critical problem. We propose a novel multi-view approach integrating frequency-domain and time-domain features to provide complementary contexts for TSC. Our method fuses continuous wavelet transform spectral features with temporal convolutional or multilayer perceptron features. We leverage the Mamba state space model for efficient and scalable sequence modeling. We also introduce a novel tango scanning scheme to better model sequence relationships. Experiments on 10 standard benchmark datasets demonstrate our approach achieves an average 6.45% accuracy improvement over state-of-the-art TSC models."},"title":{"value":"TSCMamba: Mamba Meets Multi-View Learning for Time Series Classification"},"authors":{"value":["Md Atik Ahamed","Qiang Cheng"]}},"tmdate":1738766907452,"pdate":1704067200000,"tcdate":1738766904956,"writers":["~"],"signatures":["~Md._Atik_Ahamed1"],"forum":"UrnP7SSxrx","license":"CC BY-SA 4.0","number":293423,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1738766907452,"domain":"DBLP.org","id":"UrnP7SSxrx","version":2},{"content":{"summary":{"value":"This paper introduces OUFlow, a unified framework that innovatively combines normalizing flows with Ornstein-Uhlenbeck SDEs for time series generation, forecasting, and imputation. Its key strengths include enabling exact likelihood computation, continuous-time modeling, and scalable inference by leveraging analytic solutions to SDEs, thus avoiding numerical solvers.\n\nHowever, the work has significant limitations. The motivation is unclear, lacking a definitive problem statement or justification for why existing models are insufficient. The core assumption of a linear SDE process is potentially restrictive and not well-justified for complex, real-world dynamics. Empirically, the study is limited to a few clean, periodic datasets, raising concerns about generalizability to noisy or high-dimensional real-world data. Furthermore, it lacks a theoretical analysis of the model's expressiveness or convergence guarantees."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Please see Weaknesses"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"S1. The paper addresses time series generation, forecasting, and imputation under a unified framework, using appropriate metrics like energy distance and TAES, demonstrating a comprehensive and rigorous evaluation across multiple tasks.\n\nS2. OUFlow creatively combines normalizing flows with Ornstein-Uhlenbeck-based SDEs, enabling exact likelihood computation and continuous-time modeling, offering a theoretically grounded alternative to discrete diffusion models.\n\n\nS3. By leveraging analytic solutions to linear SDEs, the model avoids numerical solvers, ensuring stable training and scalable inference—particularly beneficial for long or irregularly sampled time series."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. The paper introduces OUFlow as a normalizing flow model for time series, but it fails to clearly define the core problem it aims to solve. The motivation is scattered. There is no discussion of why existing SDE-based or flow models are insufficient for the tasks, nor a compelling real-world use case that necessitates the proposed approach. This weakens the paper's narrative and perceived impact.\n\nW2. The model assumes the latent process follows a linear SDE with block-diagonal drift matrix (Eq. 53). While this enables analytic solutions (Eq. 55), it is a strong and potentially restrictive assumption. Real-world time series often exhibit nonlinear, non-stationary, or high-dimensional dynamics. The paper does not justify why a linear OU process is sufficient or how the model would generalize to more complex dynamics. This raises concerns about expressiveness and modeling capacity.\n\nW3. The experiments use only four datasets. While Lorenz63 is a classic chaotic system, the others are standard but relatively clean and periodic. There is no evaluation on high-dimensional, irregularly sampled, or real-world noisy data (e.g., medical, financial, or sensor data with missing values). This limits the generalizability and practical relevance of the results. Moreover, the absence of comparison to strong baselines weakens the empirical contribution.\n\nW4. Despite being a flow-based model, the paper does not provide any theoretical analysis of the model’s expressiveness (e.g., universal approximation, density coverage), convergence, or consistency. For instance, under what conditions does the learned flow converge to the true data distribution? How does the OU prior affect the posterior? The derivation focuses on computational tricks (e.g., Woodbury) but lacks deeper theoretical insights that would elevate the work beyond an engineering contribution."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918893053,"tcdate":1761951300818,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6547/Reviewer_9d8d"],"signatures":["ICLR.cc/2026/Conference/Submission6547/Reviewer_9d8d"],"forum":"I66uArUBJM","number":4,"license":"CC BY 4.0","cdate":1761951300818,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6547/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918893053,"domain":"ICLR.cc/2026/Conference","replyto":"I66uArUBJM","id":"Yon2PVVq2d","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Generative model","Time series analysis","Deep learning","Normalizing flow","State space model"]},"supplementary_material":{"value":"/attachment/b6fdbd4be78ec183bb844eed1f039d38ddca8278.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"We propose OUFlow, a general-purpose time-series generative model that robustly handles irregular sampling and generates sequences at arbitrary time points. OUFlow integrates latent dynamics governed by a mixture of Ornstein-Uhlenbeck processes with expressive target distributions via normalizing flows. Leveraging our analytically derived, efficiently computable likelihoods and posteriors for high-dimensional time series, OUFlow supports unconditional time-series generation, probabilistic forecasting, and imputation from partial observations within a unified model after a single training phase. It also enables explicit likelihood evaluation (e.g., for anomaly detection), clustering via modes of the latent OU process, and, in some cases, denoising under noisy supervision. By exploiting parallelization through the scan algorithm, OUFlow attains logarithmic runtime scaling in the number of generated points, while maintaining high accuracy in all three tasks. Comprehensive experiments on both synthetic and real-world datasets demonstrate that OUFlow consistently outperforms other models capable of all three tasks, in both generation quality and computational efficiency."},"_bibtex":{"value":"@misc{\nmorinaga2026fast,\ntitle={Fast Generation, Forecasting, and Imputation of Multivariate Irregular Time Series with {OUF}low},\nauthor={Taiki Morinaga},\nyear={2026},\nurl={https://openreview.net/forum?id=I66uArUBJM}\n}"},"title":{"value":"Fast Generation, Forecasting, and Imputation of Multivariate Irregular Time Series with OUFlow"},"pdf":{"value":"/pdf/20fbd240f3dda2a10554b65cfb52f545a736cd53.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"morinaga|fast_generation_forecasting_and_imputation_of_multivariate_irregular_time_series_with_ouflow"},"authorids":{"value":["~Taiki_Morinaga1"]},"authors":{"value":["Taiki Morinaga"]}},"version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"Hybrid architecture that can operate with either self-attention or linear attention mechanisms across the sequence while sharing majority of parameters."},"venue":{"value":"AdaptFM Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["hybrid architecture","attention","linear attention"]},"abstract":{"value":"Softmax attention is the cornerstone of modern large language models but is hindered by its linear memory and quadratic compute requirements. Linear recurrent models, such as linear attention and state space models, have become widely studied as alternatives due to their linear compute and constant memory requirements. While these sub-quadratic methods achieve promising efficiency gains and competitive results on a wide range of benchmarks, current linear recurrent models still lag behind on tasks that require long-context retrieval or in-context learning, resulting in a growing body of work on hybrid architectures.\nIn this work, we propose Oryx, which explores a new axis of developing hybrid models: across the sequence, the model has the ability to operate with different mixers, e.g., quadratic attention and linear recurrences, rather than fixing a single mixer throughout. Oryx ties more than 90% of its parameters across mixer types, enabling attention and recurrent modes to operate over shared internal representations. We validate our design with Mamba-2 and Gated DeltaNet variants up to the 1.4B model scale. Under fixed token budgets and a mixed-training strategy, all modes of the 1.4B Oryx outperform their respective baselines by at least 0.7 percentage points on average on downstream language modeling evaluations.\nOn retrieval tasks, Oryx achieves performance comparable to the Transformer baseline even when processing only a tiny fraction (<10%) of the tokens in the attention mode. These results suggest that attention and linear recurrent models can share internal representations, and motivate sequence-axis hybridization as a promising direction for efficient long-context language models."},"_bibtex":{"value":"@inproceedings{\nli2026multimixer,\ntitle={Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations},\nauthor={Kevin Li and Asher Trockman and Ananda Theertha Suresh and Ziteng Sun},\nbooktitle={AdaptFM: Resource-Adaptive Foundation Model Inference},\nyear={2026},\nurl={https://openreview.net/forum?id=NIO72rHhKj}\n}"},"title":{"value":"Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations"},"pdf":{"value":"/pdf/02704b327a3e8dc37f229ec2d39d068815b62403.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/AdaptFM"},"paperhash":{"value":"li|multimixer_models_flexible_sequence_modeling_with_shared_representations"},"authorids":{"value":["~Kevin_Li3","~Asher_Trockman1","~Ananda_Theertha_Suresh1","~Ziteng_Sun1"]},"authors":{"value":["Kevin Li","Asher Trockman","Ananda Theertha Suresh","Ziteng Sun"]}},"tmdate":1780353778513,"pdate":1780353776095,"tcdate":1776721165815,"writers":["ICML.cc/2026/Workshop/AdaptFM","ICML.cc/2026/Workshop/AdaptFM/Submission8/Authors"],"signatures":["ICML.cc/2026/Workshop/AdaptFM/Submission8/Authors"],"forum":"NIO72rHhKj","license":"CC BY 4.0","number":8,"cdate":1776721165815,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/AdaptFM/-/Submission","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Release"],"mdate":1780353778513,"odate":1780353776095,"domain":"ICML.cc/2026/Workshop/AdaptFM","id":"NIO72rHhKj","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2602.02918v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"dwarampudi|a_multiscale_lineartime_encoder_for_wholeslide_image_analysis"},"html":{"value":"https://doi.org/10.48550/arXiv.2602.02918"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2602-02918,\n  publtype={informal},\n  author={Jagan Mohan Reddy Dwarampudi and Joshua Wong and Hien Van Nguyen and Tania Banerjee},\n  title={A Multi-scale Linear-time Encoder for Whole-Slide Image Analysis},\n  year={2026},\n  month={February},\n  cdate={1769904000000},\n  journal={CoRR},\n  volume={abs/2602.02918},\n  url={https://doi.org/10.48550/arXiv.2602.02918}\n}\n"},"abstract":{"value":"We introduce Multi-scale Adaptive Recurrent Biomedical Linear-time Encoder (MARBLE), the first \\textit{purely Mamba-based} multi-state multiple instance learning (MIL) framework for whole-slide image (WSI) analysis. MARBLE processes multiple magnification levels in parallel and integrates coarse-to-fine reasoning within a linear-time state-space model, efficiently capturing cross-scale dependencies with minimal parameter overhead. WSI analysis remains challenging due to gigapixel resolutions and hierarchical magnifications, while existing MIL methods typically operate at a single scale and transformer-based approaches suffer from quadratic attention costs. By coupling parallel multi-scale processing with linear-time sequence modeling, MARBLE provides a scalable and modular alternative to attention-based architectures. Experiments on five public datasets show improvements of up to \\textbf{6.9\\%} in AUC, \\textbf{20.3\\%} in accuracy, and \\textbf{2.3\\%} in C-index, establishing MARBLE as an efficient and generalizable framework for multi-scale WSI analysis."},"title":{"value":"A Multi-scale Linear-time Encoder for Whole-Slide Image Analysis"},"authors":{"value":[{"fullname":"Jagan Mohan Reddy Dwarampudi","username":"~Jagan_Mohan_Reddy_Dwarampudi1"},{"fullname":"Joshua Wong","username":""},{"fullname":"Hien Van Nguyen","username":""},{"fullname":"Tania Banerjee","username":""}]}},"tmdate":1778045457777,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2602-02918"],"tcdate":1778045452846,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jagan_Mohan_Reddy_Dwarampudi1"],"forum":"qaeBRCluDq","license":"CC BY-SA 4.0","number":8895,"cdate":1769904000000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1778045457777,"domain":"OpenReview.net/Public_Article","id":"qaeBRCluDq","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2603.20074v1"},"venueid":{"value":"dblp.org/journals/CORR/2026"},"paperhash":{"value":"khadka|mfilmamba_multifilter_scanning_for_spatial_redundancyaware_visual_state_space_models"},"authorids":{"value":["","~KC_Santosh1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2603.20074"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2603-20074,\n  publtype={informal},\n  author={Puskal Khadka and KC Santosh},\n  title={MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models},\n  year={2026},\n  month={March},\n  cdate={1772323200000},\n  journal={CoRR},\n  volume={abs/2603.20074},\n  url={https://doi.org/10.48550/arXiv.2603.20074}\n}\n"},"abstract":{"value":"State Space Models (SSMs), especially recent Mamba architecture, have achieved remarkable success in sequence modeling tasks. However, extending SSMs to computer vision remains challenging due to the non-sequential structure of visual data and its complex 2D spatial dependencies. Although several early studies have explored adapting selective SSMs for vision applications, most approaches primarily depend on employing various traversal strategies over the same input. This introduces redundancy and distorts the intricate spatial relationships within images. To address these challenges, we propose MFil-Mamba, a novel visual state space architecture built on a multi-filter scanning backbone. Unlike fixed multi-directional traversal methods, our design enables each scan to capture unique and contextually relevant spatial information while minimizing redundancy. Furthermore, we incorporate an adaptive weighting mechanism to effectively fuse outputs from multiple scans in addition to architectural enhancements. MFil-Mamba achieves superior performance over existing state-of-the-art models across various benchmarks that include image classification, object detection, instance segmentation, and semantic segmentation. For example, our tiny variant attains 83.2% top-1 accuracy on ImageNet-1K, 47.3% box AP and 42.7% mask AP on MS COCO, and 48.5% mIoU on the ADE20K dataset. Code and models are available at https://github.com/puskal-khadka/MFil-Mamba."},"title":{"value":"MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models"},"authors":{"value":["Puskal Khadka","KC Santosh"]}},"tmdate":1776653735985,"pdate":1798675200000,"tcdate":1776653727183,"writers":["~"],"signatures":["~KC_Santosh1"],"forum":"UmfudKnKwa","license":"CC BY-SA 4.0","number":868237,"cdate":1772323200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1776653735985,"domain":"DBLP.org","id":"UmfudKnKwa","version":2},{"content":{"summary":{"value":"This paper proposes PDUNet, which formulates long-horizon multivariate time series forecasting as a structured optimization problem over future variables and solves it via proximal gradient methods combined with deep unrolling. Each iteration consists of a Gradient Descent Module (GDM) followed by a Proximal Mapping Module (PMM). The PMM contains a parallel Dual-SSD design—Temporal-SSD and Channel-SSD—that performs state-space modeling along the time and channel axes, yielding a progressively refined prediction trajectory. Training uses a combination of the primary forecasting loss and a stage-consistency regularizer. Across eight standard datasets and multiple strong baselines, PDUNet shows stable advantages for medium/long horizons, and ablations verify the necessity of the unrolling mechanism and dual-path state-space modeling."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Q1. Under channel permutation and channel-subset missing perturbations, do the factors/states of Dual-SSD remain similar?\n\nQ2. Under the same complexity budget (aligned parameters/memory/latency), how does PDUNet compare with S4/Mamba for long-horizon forecasting performance?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"S1. The PMM employs Dual-SSD (time/channel-parallel state-space units), shifting expensive nonlinear computation into a compressed state space, balancing expressiveness and efficiency.\n\nS2. The algorithmic pipeline, module boundaries, and data flow are clear.\n\nS3. Comparisons are fairly comprehensive across datasets and horizons; performance is robust for long-horizon forecasting."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. Early deep unrolling works have already networked “gradient steps + proximal mappings” (e.g., LISTA [1]); subsequent Plug-and-Play / proximal-unrolling lines further integrated learnable priors into iterative frameworks [2]. In the sequence domain, optimization- or recursion-based prediction/filtering (e.g., KalmanNet [3]) and continuous-dynamics modeling (Neural ODE [4]) are also representative. PDUNet’s contribution, explicitly optimizing future targets and realizing a structured “proximal” map via Dual-SSD, is interesting, but quantitative and mechanistic contrasts against these main lines (to clarify fundamental differences and irreplaceability) remain limited.\n\nW2. The Dual-SSD design is methodologically close to the SSM/S4 family (Gu et al., 2021) and Mamba (Gu & Dao, 2024) for efficient long-sequence modeling; the paper also compares to iTransformer (Liu et al., 2023), PatchTST (Nie et al., 2023), DLinear (Zeng et al., 2022), and TimeMixer (Chen et al., 2024). A more systematic contrast on complexity, spectral/phase stability, and long-horizon degradation would better position PDUNet’s strengths and limitations relative to these lines.\n\nW3. While the paper emphasizes progressive convergence, it lacks quantitative evidence such as iteration-wise error/step convergence curves and consistency of selected key lags across random seeds.\nW4. The PMM effectively learns a data-driven proximal operator, yet the corresponding prior/regularizer \nJ(⋅) is not sufficiently interpreted or visualized; instability in J could lead to pattern leakage.\n\nReferences\n[1] Gregor, K., & LeCun, Y. “Learning Fast Approximations of Sparse Coding.” ICML, 2010, pp. 399–406.\n[2] Venkatakrishnan, S. V., Bouman, C. A., & Wohlberg, B. “Plug-and-Play Priors for Model Based Reconstruction.” IEEE GlobalSIP, 2013, pp. 945–948.\n[3] Revach, G., Shlezinger, N., Ni, X., et al. “KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics.” IEEE Transactions on Signal Processing, 70:1532–1547, 2022.\n[4] Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. “Neural Ordinary Differential Equations.” NeurIPS, 2018."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918840296,"tcdate":1761634282796,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6447/Reviewer_nPrH"],"signatures":["ICLR.cc/2026/Conference/Submission6447/Reviewer_nPrH"],"forum":"0OENkI0UgR","number":2,"license":"CC BY 4.0","cdate":1761634282796,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6447/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918840296,"domain":"ICLR.cc/2026/Conference","replyto":"0OENkI0UgR","id":"0V0ZegygZd","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Time series forecasting","Unmixing","Mamba network","Shared mechanism."]},"supplementary_material":{"value":"/attachment/4558e02cf429b1b8dd397cc6e069744d97d0234a.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Recent time series forecasting methods have increasingly incorporated model-driven formulations and data-driven inference to leverage their complementary strengths in interpretability and pattern learning, achieving remarkable results in many practical tasks. However, in most existing methods, the forecasting target is only used as a training supervision signal rather than being explicitly incorporated into the modeling process, making it difficult for models to leverage future modality information for structural constraint and path guidance. As a result, the modeling structure and inference path remain isolated, lacking a clear optimization objective, and ultimately degenerate into an uncontrollable and uninterpretable static mapping process. To address these challenges, we propose Proximal Deep Unrolling Network (PDUNet), a coupled and closed-loop forecasting framework that starts from modeling, leverages data-driven mechanisms for solution, and progressively optimizes the inference path. Specifically, we formulate an optimizable forecasting equation based on the coupling between future variables and historical inputs, and adopt a proximal optimization algorithm to perform step-wise decoupling and update. This optimization process is further unfolded into a dual-branch state space structure, where the Temporal-SSD captures temporal evolution through causal modeling, while the Channel-SSD employs a non-causal mechanism to model global interactions among variables, jointly enabling progressive inference and dynamic prediction. Experiments on eight public benchmark datasets show that PDUNet outperforms existing state-of-the-art models in long-term forecasting tasks."},"_bibtex":{"value":"@misc{\nzhu2026pdunet,\ntitle={{PDUN}et: Proximal-Guided Deep Unrolling Network for Time Series Forecasting},\nauthor={Xuanbing Zhu and Dunbin Shen and Xiaorui Ma and Yingguang Hao and Hongyu Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=0OENkI0UgR}\n}"},"title":{"value":"PDUNet: Proximal-Guided Deep Unrolling Network for Time Series Forecasting"},"pdf":{"value":"/pdf/230954702f9f9c63191b48cc384aebe14f3ae3f2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhu|pdunet_proximalguided_deep_unrolling_network_for_time_series_forecasting"},"authorids":{"value":["~Xuanbing_Zhu1","~Dunbin_Shen1","~Xiaorui_Ma1","~Yingguang_Hao1","~Hongyu_Wang1"]},"authors":{"value":["Xuanbing Zhu","Dunbin Shen","Xiaorui Ma","Yingguang Hao","Hongyu Wang"]}},"version":2},{"content":{"comment":{"value":"### Weakness 4.1\nWe apologize for the incorrect dimensions. The dimension of $\\mathbf{B}$ and $\\mathbf{C}$ should be $\\mathbb{R}^{d_1 \\times 1}$ and $\\mathbb{R}^{1 \\times d_1}$, respectively. In this way $\\mathbf{A}\\mathbf{z}(\\tau)$ and $\\mathbf{B}\\mathbf{q}(\\tau)$ share the same dimension of $\\mathbb{R}^{d_1 \\times 1}$, and $r(\\tau)$ is a scalar. **We also notice this so it will be fixed in our revision.**\n\n### Weakness 4.2\n$d_2$ is the dimension of the input. In Eq. 1 and 2, $q(\\tau)\\in \\mathbb{R}$ and $q_\\tau\\in \\mathbb{R}$ are 1-dimensional inputs, so $d_2=1$. When $d_2 > 1$, $q(\\tau)\\in \\mathbb{R}^{d_2}$ and $q_\\tau\\in \\mathbb{R}^{d_2}$ become vectors. SISO means that given an input vector, an SSM will use the same set of parameters to process each element of this vector in parallel, and then collect the output of each dimension to form the processed vector. **We strictly follow [4] (its page 4 \"Structure and Dimensions.\") to introduce SISO in our paper. For better readability, we will better introduce $d_2$ and put a more detailed explanation of SISO into the appendix of our revision.**\n\n### Weakness 4.3\nThank you for spotting this! We will fix the typo.\n\n[1] Behrouz, Ali, and Farnoosh Hashemi. \"Graph mamba: Towards learning on graphs with state space models.\" Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024.\n\n[2] Yu, Le, et al. \"Towards better dynamic graph learning: New architecture and unified library.\" Advances in Neural Information Processing Systems 36 (2023): 67686-67700.\n\n[3] Tian, Y., Qi, Y., & Guo, F. (2023). FreeDyG: Frequency Enhanced Continuous-Time Dynamic Graph Model for Link Prediction. In The Twelfth International Conference on Learning Representations.\n\n[4] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023)."},"title":{"value":"Response to Reviewer 1QFu: Part B"}},"tmdate":1732019304549,"tcdate":1731945160302,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5146/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5146/Authors"],"forum":"f3gCs2a4ZD","number":14,"license":"CC BY 4.0","cdate":1731945160302,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5146/-/Official_Comment"],"mdate":1732019304549,"domain":"ICLR.cc/2025/Conference","replyto":"f4ptq7YFDu","id":"xoOyTUztu4","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["dynamic graph","state space model"]},"supplementary_material":{"value":"/attachment/2dc1a3502a1d53b635f3c846cad8277b987ac60b.zip"},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Learning useful representations for continuous-time dynamic graphs (CTDGs) is challenging, due to the concurrent need to span long node interaction histories and grasp nuanced temporal details. In particular, two problems emerge: (1) Encoding longer histories requires more computational resources, making it crucial for CTDG models to maintain low computational complexity to ensure efficiency; (2) Meanwhile, more powerful models are needed to identify and select the most critical temporal information within the extended context provided by longer histories. To address these problems, we propose a CTDG representation learning model named DyGMamba, originating from the popular Mamba state space model (SSM). DyGMamba first leverages a node-level SSM to encode the sequence of historical node interactions. Another time-level SSM is then employed to exploit the temporal patterns hidden in the historical graph, where its output is used to dynamically select the critical information from the interaction history. We validate DyGMamba experimentally on the dynamic link prediction task. The results show that our model achieves state-of-the-art in most cases. DyGMamba also maintains high efficiency in terms of computational resources, making it possible to capture long temporal dependencies with a limited computation budget."},"_bibtex":{"value":"@misc{\nding2025dygmamba,\ntitle={Dy{GM}amba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models},\nauthor={Zifeng Ding and Yifeng Li and Yuan He and Antonio Norelli and Jingcheng Wu and Volker Tresp and Yunpu Ma and Michael M. Bronstein},\nyear={2025},\nurl={https://openreview.net/forum?id=f3gCs2a4ZD}\n}"},"title":{"value":"DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models"},"pdf":{"value":"/pdf/0921c754d017d250e0d2f4b59a779766e47b30a6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ding|dygmamba_efficiently_modeling_longterm_temporal_dependency_on_continuoustime_dynamic_graphs_with_state_space_models"},"authorids":{"value":["~Zifeng_Ding1","~Yifeng_Li4","~Yuan_He5","~Antonio_Norelli2","~Jingcheng_Wu3","~Volker_Tresp1","~Yunpu_Ma1","~Michael_M._Bronstein1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zifeng Ding","Yifeng Li","Yuan He","Antonio Norelli","Jingcheng Wu","Volker Tresp","Yunpu Ma","Michael M. Bronstein"]}},"version":2},{"content":{"summary":{"value":"The paper studies the in-context learning of Mamba model in a theoretical way. It provides theoretical sample complexity bound on the number of pretraining tasks to achieve a fixed ICL test error via a two-stage gradient pretraining scheme. The paper also proves Mamba can achieve test time feature learning and conducts experiment to verify this."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Could the authors ellaborate on the fundamental difference between your paper and [1]? Though I haven't read [1] carefully, but it seems to me that your work inherits the target function $f_*$, the learning objective loss $L$, the two-stage pretraining algorithm and the diagonal structure of $W_B^\\top W_C$ (which in [1]'s case is $W_Q^\\top W_K$, and the $1$ in the right bottom corner could be ommitted because Mamba doesn't have value matrix $W_V$). The main difference is the replacement of the transformer model with the Mamba model, which mimics a linear self attention (admittedly the nonlinear gating may add some difficulty). Overall, I'm concerned about the techinical contribution of this paper and would be willing to reconsider my rating if the authors could convince me of it.\n\n\n[1] Nishikawa, N., Song, Y., Oko, K., Wu, D., & Suzuki, T. Nonlinear transformers can perform inference-time feature learning. In Forty-second International Conference on Machine Learning."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper theoretically studies the ICL of Mamba, which is an interesting topic.\n2. The paper is the first to provide a sample complexity result on the ICL of Mamba.\n3. The writing is clear which makes the paper easy to follow."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Since the authors listed proposition 4.1 as one of the core contributions, I think a formal version of it could benefit the paper, because currently it's hard to understand what $P_1$ and $P_2$ stands for without referring to the appendix (Lemma C.1).\n2. The experiment doesn't collaborate with the theoretical results well. It only shows that Mamba **can** achieve in-context feature learning, which is not that surprising given empirical results on the ICL of Mamba models [1,2]. Perhaps some trends toward how the number of pretraining tasks affects the final ICL test error could be more relative to the point.\n3. I'm currently not so sure about the siginificance of contribution compared with [3]. The two stage learning algorithm and the learning objective in this paper is the same as in [3], and the replacement of a transformer with a Mamba doesn't seem very fundamental.\n\n\n[1] Grazzi, R., Siems, J., Schrodi, S., Brox, T., & Hutter, F. (2024). Is mamba capable of in-context learning?. arXiv preprint arXiv:2402.03170.\n\n[2] Park, J., Park, J., Xiong, Z., Lee, N., Cho, J., Oymak, S., ... & Papailiopoulos, D. (2024). Can mamba learn how to learn? a comparative study on in-context learning tasks. arXiv preprint arXiv:2402.04248.\n\n[3] Nishikawa, N., Song, Y., Oko, K., Wu, D., & Suzuki, T. Nonlinear transformers can perform inference-time feature learning. In Forty-second International Conference on Machine Learning."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933992843,"tcdate":1761535205480,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20583/Reviewer_wSHm"],"signatures":["ICLR.cc/2026/Conference/Submission20583/Reviewer_wSHm"],"forum":"3KPsog4mpy","number":1,"license":"CC BY 4.0","cdate":1761535205480,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20583/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933992843,"domain":"ICLR.cc/2026/Conference","replyto":"3KPsog4mpy","id":"8gfzasd3um","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We study in-context learning of Mamba on single-index models"},"keywords":{"value":["Mamba","In-Context Learning","Feature Learning","Theory"]},"primary_area":{"value":"learning theory"},"abstract":{"value":"Mamba, a recently proposed linear-time sequence model, has attracted significant attention for its computational efficiency and strong empirical performance. However, a rigorous theoretical understanding of its underlying mechanisms remains limited. In this work, we provide a theoretical analysis of Mamba's in-context learning (ICL) capability by focusing on tasks defined by low-dimensional nonlinear target functions. Specifically, we study in-context learning of a single-index model $y \\approx g_*(\\langle \\boldsymbol{\\beta}, \\boldsymbol{x} \\rangle)$, which depends on only a single relevant direction $\\boldsymbol{\\beta}$, referred to as *feature*. We prove that Mamba, pretrained by gradient-based methods, can achieve efficient ICL via *test-time feature learning*, extracting the relevant direction directly from context examples. Consequently, we establish a test-time sample complexity that improves upon linear Transformers—analyzed to behave like kernel methods—and is comparable to nonlinear Transformers, which have been shown to surpass the Correlational Statistical Query (CSQ) lower bound and achieve near information-theoretically optimal rate in previous works. Our analysis reveals the crucial role of the *nonlinear gating* mechanism in Mamba for feature extraction, highlighting it as the fundamental driver behind Mamba’s ability to achieve both computational efficiency and high performance."},"_bibtex":{"value":"@misc{\noh2026mamba,\ntitle={Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning},\nauthor={Junsoo Oh and Wei Huang and Taiji Suzuki},\nyear={2026},\nurl={https://openreview.net/forum?id=3KPsog4mpy}\n}"},"title":{"value":"Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning"},"pdf":{"value":"/pdf/b872836f3f5ba22e086f394ffe799c2dd9c8c920.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"oh|mamba_can_learn_lowdimensional_targets_incontext_via_testtime_feature_learning"},"authorids":{"value":["~Junsoo_Oh1","~Wei_Huang6","~Taiji_Suzuki1"]},"authors":{"value":["Junsoo Oh","Wei Huang","Taiji Suzuki"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.18572v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"nguyen|taipan_efficient_and_expressive_state_space_language_models_with_selective_attention"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Chien_Van_Nguyen:","~Huy_Huu_Nguyen1","~Thang_M._Pham1","https://dblp.org/search/pid/api?q=author:Ruiyi_Zhang:","https://dblp.org/search/pid/api?q=author:Hanieh_Deilamsalehy:","https://dblp.org/search/pid/api?q=author:Puneet_Mathur:","https://dblp.org/search/pid/api?q=author:Ryan_A._Rossi:","https://dblp.org/search/pid/api?q=author:Trung_Bui:","https://dblp.org/search/pid/api?q=author:Viet_Dac_Lai:","https://dblp.org/search/pid/api?q=author:Franck_Dernoncourt:","https://dblp.org/search/pid/api?q=author:Thien_Huu_Nguyen:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.18572"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-18572,\n  publtype={informal},\n  author={Chien Van Nguyen and Huy Huu Nguyen and Thang M. Pham and Ruiyi Zhang and Hanieh Deilamsalehy and Puneet Mathur and Ryan A. Rossi and Trung Bui and Viet Dac Lai and Franck Dernoncourt and Thien Huu Nguyen},\n  title={Taipan: Efficient and Expressive State Space Language Models with Selective Attention},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.18572},\n  url={https://doi.org/10.48550/arXiv.2410.18572}\n}\n"},"abstract":{"value":"Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long sequences due to quadratic computational complexity in training and linearly scaling memory costs during inference. Recent State Space Models (SSMs) such as Mamba offer alternatives with constant memory usage, but they underperform in tasks requiring extensive in-context retrieval. We introduce Taipan, a novel hybrid architecture that combines Mamba-2 with Selective Attention Layers (SALs). These SALs identify tokens requiring long-range interactions, remove less important features, and then augment their representations using the attention module. This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. By constraining the attention budget, Taipan extends accurate predictions to context lengths of up to 1 million tokens while preserving computational efficiency. Our experiments demonstrate Taipan's superior performance across various scales and tasks, offering a promising solution for efficient long-context language modeling."},"title":{"value":"Taipan: Efficient and Expressive State Space Language Models with Selective Attention"},"authors":{"value":["Chien Van Nguyen","Huy Huu Nguyen","Thang M. Pham","Ruiyi Zhang","Hanieh Deilamsalehy","Puneet Mathur","Ryan A. Rossi","Trung Bui","Viet Dac Lai","Franck Dernoncourt","Thien Huu Nguyen"]}},"tmdate":1760566120200,"pdate":1704067200000,"tcdate":1733196417769,"writers":["~"],"signatures":["~Thang_Minh_Pham1"],"forum":"tFrVqDgV79","license":"CC BY-SA 4.0","number":252205,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1760566120200,"domain":"DBLP.org","id":"tFrVqDgV79","version":2},{"content":{"TLDR":{"value":"We introduce Taipan, a novel architecture that combines State Space Models with Selective Attention Layers, achieving superior performance in memory-intensive tasks while enabling efficient language modeling for sequences up to 1 million tokens."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Efficient Language Model","Model Architecture","Long-context Language Model","In-context Retrieval","Hybrid Architecture","Linear Complexity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long sequences due to quadratic computational complexity in training and linearly scaling memory costs during inference. Recent State Space Models (SSMs) such as Mamba offer alternatives with constant memory usage, but they underperform in tasks requiring extensive in-context retrieval. We introduce Taipan, a novel hybrid architecture that combines Mamba-2 with Selective Attention Layers (SALs). These SALs identify tokens requiring long-range interactions, remove less important features, and then augment their representations using the attention module. This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. By constraining the attention budget, Taipan extends accurate predictions to context lengths of up to 1 million tokens while preserving computational efficiency. Our experiments demonstrate Taipan's superior performance across various scales and tasks, offering a promising solution for efficient long-context language modeling."},"_bibtex":{"value":"@misc{\nnguyen2025taipan,\ntitle={Taipan: Efficient and Expressive State Space Language Models with Selective Attention},\nauthor={Chien Van Nguyen and Huy Huu Nguyen and Thang M. Pham and Ruiyi Zhang and Hanieh Deilamsalehy and Puneet Mathur and Ryan A. Rossi and Trung Bui and Viet Dac Lai and Franck Dernoncourt and Thien Huu Nguyen},\nyear={2025},\nurl={https://openreview.net/forum?id=9DnKZbOr4r}\n}"},"title":{"value":"Taipan: Efficient and Expressive State Space Language Models with Selective Attention"},"pdf":{"value":"/pdf/4c569111617737e5532042f859ac730be0ac18ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"nguyen|taipan_efficient_and_expressive_state_space_language_models_with_selective_attention"},"authorids":{"value":["~Chien_Van_Nguyen1","~Huy_Huu_Nguyen1","~Thang_M._Pham1","~Ruiyi_Zhang3","~Hanieh_Deilamsalehy1","~Puneet_Mathur2","~Ryan_A._Rossi2","~Trung_Bui1","~Viet_Dac_Lai1","~Franck_Dernoncourt1","~Thien_Huu_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chien Van Nguyen","Huy Huu Nguyen","Thang M. Pham","Ruiyi Zhang","Hanieh Deilamsalehy","Puneet Mathur","Ryan A. Rossi","Trung Bui","Viet Dac Lai","Franck Dernoncourt","Thien Huu Nguyen"]}},"tmdate":1738735791542,"tcdate":1727421562439,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9385/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission9385/Authors"],"forum":"9DnKZbOr4r","license":"CC BY 4.0","number":9385,"cdate":1727421562439,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission9385/-/Full_Submission","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735791542,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"9DnKZbOr4r","version":2},{"content":{"venue":{"value":"IEEE Signal Process. Lett. 2024"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/97/10380231/10674771.pdf"},"venueid":{"value":"dblp.org/journals/SPL/2024"},"paperhash":{"value":"zhou|kalmanssm_modeling_longterm_time_series_with_kalman_filter_structured_state_spaces"},"authorids":{"value":["","https://dblp.org/search/pid/api?q=author:Xu_Guo:","https://dblp.org/search/pid/api?q=author:Yu-Jie_Xiong:","~Chun-Ming_Xia1"]},"html":{"value":"https://doi.org/10.1109/LSP.2024.3457862"},"_bibtex":{"value":"@article{DBLP:journals/spl/ZhouGXX24,\n  author={Zheng Zhou and Xu Guo and Yu-Jie Xiong and Chun-Ming Xia},\n  title={Kalman-SSM: Modeling Long-Term Time Series With Kalman Filter Structured State Spaces},\n  year={2024},\n  cdate={1704067200000},\n  journal={IEEE Signal Process. Lett.},\n  volume={31},\n  pages={2470-2474},\n  url={https://doi.org/10.1109/LSP.2024.3457862}\n}\n"},"abstract":{"value":"In the field of time series forecasting, time series are often considered as linear time-varying systems, which facilitates the analysis and modeling of time series from a structural state perspective. Due to the non-stationary nature and noise interference in real-world data, existing models struggle to predict long-term time series effectively. To address this issue, we propose a novel model that integrates the Kalman filter with a state space model (SSM) approach to enhance the accuracy of long-term time series forecasting. The Kalman filter requires recursive computation, whereas the SSM approach reformulates the Kalman filtering process into a convolutional form, simplifying training and enhancing model efficiency. Our Kalman-SSM model estimates the future state of dynamic systems for forecasting by utilizing a series of time series data containing noise. In real-world datasets, the Kalman-SSM has demonstrated competitive performance and satisfactory efficiency in comparison to state-of-the-art (SOTA) models."},"title":{"value":"Kalman-SSM: Modeling Long-Term Time Series With Kalman Filter Structured State Spaces"},"authors":{"value":["Zheng Zhou","Xu Guo","Yu-Jie Xiong","Chun-Ming Xia"]}},"tmdate":1744101414714,"pdate":1704067200000,"tcdate":1744097379369,"writers":["~"],"signatures":["~ZHOU_ZHENG4"],"forum":"5WqFHaa1ra","license":"CC BY-SA 4.0","number":381320,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1744101414714,"domain":"DBLP.org","id":"5WqFHaa1ra","version":2},{"content":{"venue":{"value":"IEEE Signal Process. Lett. 2024"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/97/10380231/10674771.pdf"},"venueid":{"value":"dblp.org/journals/SPL/2024"},"paperhash":{"value":"zhou|kalmanssm_modeling_longterm_time_series_with_kalman_filter_structured_state_spaces"},"authorids":{"value":["","https://dblp.org/search/pid/api?q=author:Xu_Guo:","https://dblp.org/search/pid/api?q=author:Yu-Jie_Xiong:","https://dblp.org/search/pid/api?q=author:Chun-Ming_Xia:"]},"html":{"value":"https://doi.org/10.1109/LSP.2024.3457862"},"_bibtex":{"value":"@article{DBLP:journals/spl/ZhouGXX24,\n  author={Zheng Zhou and Xu Guo and Yu-Jie Xiong and Chun-Ming Xia},\n  title={Kalman-SSM: Modeling Long-Term Time Series With Kalman Filter Structured State Spaces},\n  year={2024},\n  cdate={1704067200000},\n  journal={IEEE Signal Process. Lett.},\n  volume={31},\n  pages={2470-2474},\n  url={https://doi.org/10.1109/LSP.2024.3457862}\n}\n"},"abstract":{"value":"In the field of time series forecasting, time series are often considered as linear time-varying systems, which facilitates the analysis and modeling of time series from a structural state perspective. Due to the non-stationary nature and noise interference in real-world data, existing models struggle to predict long-term time series effectively. To address this issue, we propose a novel model that integrates the Kalman filter with a state space model (SSM) approach to enhance the accuracy of long-term time series forecasting. The Kalman filter requires recursive computation, whereas the SSM approach reformulates the Kalman filtering process into a convolutional form, simplifying training and enhancing model efficiency. Our Kalman-SSM model estimates the future state of dynamic systems for forecasting by utilizing a series of time series data containing noise. In real-world datasets, the Kalman-SSM has demonstrated competitive performance and satisfactory efficiency in comparison to state-of-the-art (SOTA) models."},"title":{"value":"Kalman-SSM: Modeling Long-Term Time Series With Kalman Filter Structured State Spaces"},"authors":{"value":["Zheng Zhou","Xu Guo","Yu-Jie Xiong","Chun-Ming Xia"]}},"tmdate":1744097560872,"pdate":1704067200000,"tcdate":1744096869974,"writers":["~"],"signatures":["~ZHOU_ZHENG4"],"forum":"8jBqsT9Fo1","license":"CC BY-SA 4.0","number":381281,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1744097560872,"domain":"DBLP.org","id":"8jBqsT9Fo1","version":2},{"content":{"venue":{"value":"IEEE Signal Process. Lett. 2024"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/97/10380231/10674771.pdf"},"venueid":{"value":"dblp.org/journals/SPL/2024"},"paperhash":{"value":"zhou|kalmanssm_modeling_longterm_time_series_with_kalman_filter_structured_state_spaces"},"authorids":{"value":["","https://dblp.org/search/pid/api?q=author:Xu_Guo:","https://dblp.org/search/pid/api?q=author:Yu-Jie_Xiong:","https://dblp.org/search/pid/api?q=author:Chun-Ming_Xia:"]},"html":{"value":"https://doi.org/10.1109/LSP.2024.3457862"},"_bibtex":{"value":"@article{DBLP:journals/spl/ZhouGXX24,\n  author={Zheng Zhou and Xu Guo and Yu-Jie Xiong and Chun-Ming Xia},\n  title={Kalman-SSM: Modeling Long-Term Time Series With Kalman Filter Structured State Spaces},\n  year={2024},\n  cdate={1704067200000},\n  journal={IEEE Signal Process. Lett.},\n  volume={31},\n  pages={2470-2474},\n  url={https://doi.org/10.1109/LSP.2024.3457862}\n}\n"},"abstract":{"value":"In the field of time series forecasting, time series are often considered as linear time-varying systems, which facilitates the analysis and modeling of time series from a structural state perspective. Due to the non-stationary nature and noise interference in real-world data, existing models struggle to predict long-term time series effectively. To address this issue, we propose a novel model that integrates the Kalman filter with a state space model (SSM) approach to enhance the accuracy of long-term time series forecasting. The Kalman filter requires recursive computation, whereas the SSM approach reformulates the Kalman filtering process into a convolutional form, simplifying training and enhancing model efficiency. Our Kalman-SSM model estimates the future state of dynamic systems for forecasting by utilizing a series of time series data containing noise. In real-world datasets, the Kalman-SSM has demonstrated competitive performance and satisfactory efficiency in comparison to state-of-the-art (SOTA) models."},"title":{"value":"Kalman-SSM: Modeling Long-Term Time Series With Kalman Filter Structured State Spaces"},"authors":{"value":["Zheng Zhou","Xu Guo","Yu-Jie Xiong","Chun-Ming Xia"]}},"tmdate":1744097560827,"pdate":1704067200000,"tcdate":1744097361495,"writers":["~"],"signatures":["~ZHOU_ZHENG4"],"forum":"0KyKMEKBoL","license":"CC BY-SA 4.0","number":381319,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1744097560827,"domain":"DBLP.org","id":"0KyKMEKBoL","version":2},{"content":{"TLDR":{"value":"We propose GD-Mamba, an alternating graph diffusion model with two-level Mamba and wavelet attention to align discrete graph and continuous latent modalities."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Graph Generation","Diffusion Models","Deep Generative Models"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Graph generation is a critical task in deep learning, aimed at learning and modelling the structural distribution of graphs, with broad applications in areas such as social network analysis and molecular design. Current mainstream graph generation methods adopt diffusion models as the foundational architecture. Graph data typically consists of two modalities: the discrete graph modality based on topological structures, and the continuous latent space modality based on vector representations. Diffusion models for the graph modality focus on denoising discrete nodes and edges and lack unified control over the global structure, resulting in jump-like changes in the generation process. In contrast, diffusion models for the latent space modality perform denoising in a continuous space, resulting in gradual changes throughout the generation process. To incorporate global semantic guidance from the latent space modality into graph modality denoising, while injecting discrete priors from the graph modality into latent space denoising, we propose GD-Mamba, an Adaptive Cross-Domain Alternating Graph Diffusion Model with Selective State Space Model. GD-Mamba alternately performs denoising between the continuous and discrete modalities, namely the graph and latent space modalities. To stabilize the dual-modality generation process, a two-level Mamba structure (G-Mamba and S-Mamba) is designed to align features across both modalities. Additionally, a multiscale wavelet positional encoding attention (MSW-PE) mechanism is introduced in the cross-domain mapping between both modalities. This mechanism effectively extracts the core structure of graphs by locating neighbourhoods of nodes and edges at different frequency scales, improving the accuracy and robustness in the alternating denoising process. The experimental results demonstrate that the proposed GD-Mamba method has superior performance on both generic graph datasets and molecular graph datasets."},"_bibtex":{"value":"@inproceedings{\nanonymous2026gdmamba,\ntitle={{GD}-Mamba: Adaptive Cross-Domain Alternating Graph Diffusion Model with Selective State Space Model},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=Z88SQC1YML},\nnote={under review}\n}"},"title":{"value":"GD-Mamba: Adaptive Cross-Domain Alternating Graph Diffusion Model with Selective State Space Model"},"pdf":{"value":"/pdf/55f171509c7bee24d26d3537a2f1eaeb4efaba8f.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791228435788,"tcdate":1789384790520,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission17860/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission17860/Authors"],"forum":"Z88SQC1YML","license":"CC BY 4.0","number":17860,"cdate":1789384790520,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission17860/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791228435788,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"Z88SQC1YML","version":2},{"content":{"summary":{"value":"The paper argues that when decoding fMRI into the representation spaces of large foundation models, a simple linear mapping trained with a contrastive objective is consistently better than ridge regression and shallow MLPs. The authors evaluate three public datasets, and use retrieval in embedding space as the metric. A small architecture with subject-specific linear alignment followed by a linear projection yields Top-1 gains over ridge and also outperforms their non-linear MLP across all modalities."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"1. Please according to the soundness issues in Weaknesses part\n2. It is recommend that the author could explain more why linear model is better than other more complex methods."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The paper’s key strength is its evidence that a simple linear mapping is adequate for aligning fMRI signals with semantically rich embedding spaces (e.g., from multimodal foundation models). This is a surprising and consequential observation, suggesting that the geometry of brain-recorded activity admits an approximately linear relationship to these representations, thereby refining our understanding of both fMRI signal structure and neural encoding at the population level."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Although the conclusion—that an efficient linear mapping suffices for fMRI decoding—is intriguing, the experimental design supporting this claim appears insufficiently rigorous:\n\n1. The paper compares ridge regression (trained with an MSE objective and L2 regularization ?) to the proposed linear model trained with a contrastive (InfoNCE-style) objective. Because the target embedding (e.g., CLIP) itself is learned with a contrastive loss, this comparison is not fair: one would expect a method trained with a geometry-aligned contrastive loss to outperform MSE by construction. \n\n2. The paper states that an MLP underperforms the linear model (Table 1), yet Table 2 shows that **Identity** activation yields the best MLP performance. Theoretically, an MLP with identity activations collapses to a linear map; with sufficient training and comparable regularization, it should match (or closely approach) the linear model. So does the comparsion of MLP with identity activation and linear model make sense?\n\n3. The proposed linear model includes a subject-specific adjustment matrix (A_k). It is unclear whether the MLP baseline incorporates an analogous adjustment . The paper should document these details (main text or appendix).\n\nI am willing to increase my score if the authors address the issues above."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927788126,"tcdate":1761752343926,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17992/Reviewer_Rfki"],"signatures":["ICLR.cc/2026/Conference/Submission17992/Reviewer_Rfki"],"forum":"W8N4eZ9KSU","number":1,"license":"CC BY 4.0","cdate":1761752343926,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17992/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927788126,"domain":"ICLR.cc/2026/Conference","replyto":"W8N4eZ9KSU","id":"RgymXZiVn5","forumContent":{"TLDR":{"value":"Linear contrastive alignment is the most effective way to decode fMRI activity, by leveraging vectorial representations of concepts and the linearization inherent to fMRI measurements."},"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"keywords":{"value":["Computational Neuroscience","Cognitive Science","fMRI Decoding","Contrastive Learning"]},"supplementary_material":{"value":"/attachment/0bf8ea627737897f4f0a23da23cf794c024d0439.zip"},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"A prominent theory in cognitive science suggests that concepts in the brain are organized as high-dimensional vectors, with semantic meaning captured by directions and relative angles in this space. Brain decoding is the effort of reconstructing or retrieving stimuli (or their representations) from neural activity and involves finding a function that approximates how the brain represents concepts. This motivates the investigation of contrastive objectives as biologically plausible candidates to reverse the brain loss function. In this work, we study how functional MRI (fMRI) activity can generally be aligned with the embedding spaces of foundation models in vision, language, and audio. Although neural computations are highly non-linear at the microscale, fMRI measurements average signals across space and time, further smoothed by noise, effectively linearizing the observable representation. Consistent with these views, our experiments across multiple datasets demonstrate that linear contrastive decoders consistently outperform ridge regression and standard non-linear alternatives, and that these results generalize across images, text, and sound. These findings indicate that decoding gains arise more from the choice of training objective than from architectural complexity, pointing to contrastive-linear models as a principled strategy for brain decoding."},"_bibtex":{"value":"@misc{\nanonymous2026linear,\ntitle={Linear Maps, Contrastive Objectives: A Principled Strategy for f{MRI} Decoding Consistent Across Modalities},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=W8N4eZ9KSU}\n}"},"title":{"value":"Linear Maps, Contrastive Objectives: A Principled Strategy for fMRI Decoding Consistent Across Modalities"},"pdf":{"value":"/pdf/67979268e82863183912ab32fa35913bdd379b10.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"ciferri|linear_maps_contrastive_objectives_a_principled_strategy_for_fmri_decoding_consistent_across_modalities"},"authorids":{"value":["~Matteo_Ciferri1","~Matteo_Ferrante1","~Nicola_Toschi1"]},"authors":{"value":["Matteo Ciferri","Matteo Ferrante","Nicola Toschi"]}},"version":2},{"content":{"comment":{"value":"1. We based our implementation on the official Mamba codebase. In the original paper [1], the authors stated making the matrices input-dependent in Section 3.2 and shown also in Algorithm 2. In the codebase (which we import from in our code), the sequence input is passed to a Linear projection layer. In turn, $\\bar{B}(x)$ and $\\bar{C}(x)$ are derived from this output so we also call them learnable as their values depend on the input sequence and the Linear layer.\n\n2. Under *specific* initialization, the model has the form of a gated RNN [1]. The selection mechanism of Mamba was one of the original motivations of the Mamba paper to give a notion of *context* during sequence processing (like attention in transformers). Empirically across many domains, Mamba has achieved similar/better performance than transformers. Our motivation for using Mamba was to investigate its behaviour in the RL molecule paradigm and we found its distribution learning capabilities to be advantageous in the settings we tested.\n\nIn our above response about \"local sampling\" (again, we are sorry for the insufficient clarity which we hope we better conveyed now) which leads to high sample efficiency, it is very dependent on the ability to model the token sequences of all the augmented SMILES forms. A model with better capability to model sequences benefits this mechanism. This is what we show in the main text results.\n\n3. We define the generation as a Markov process so the probability of the next action only depends on the previous time-step's state. The generation process is then the token selected given the state so far and the expression in equation 1 takes the product of the probabilities of these steps from time-step 1 to T. \n\nThank you to the reviewer for discussing with us during this period as it was very helpful. We hope we were able to provide better clarity on certain aspects. \n\n\n\\\n\\\n[1] https://arxiv.org/abs/2312.00752\n\n[2] https://github.com/state-spaces/mamba/blob/main/mamba_ssm/modules/block.py"}},"tmdate":1733123933836,"tcdate":1733123933836,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1831/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission1831/Authors"],"forum":"Hbpzrh7JbN","number":39,"license":"CC BY 4.0","cdate":1733123933836,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1831/-/Official_Comment"],"mdate":1733123933836,"domain":"ICLR.cc/2025/Conference","replyto":"57HHDDMWOZ","id":"Y2H8aSPmDO","forumContent":{"TLDR":{"value":"state-of-the-art sample efficiency using mamba for generative molecular design"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["generative design","drug discovery","sample efficiency","language models","reinforcement learning","scaling"]},"supplementary_material":{"value":"/attachment/4735cc8666aadddbad349c1f2433393b9784f551.pdf"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Generative molecular design for drug discovery has very recently achieved a wave\nof experimental validation, with language-based backbones being the most common\narchitectures employed. The most important factor for downstream success is\nwhether an *in silico* oracle is well correlated with the desired end-point. To this end,\ncurrent methods use cheaper proxy oracles with higher throughput before evaluating\nthe most promising subset with high-fidelity oracles. The ability to directly optimize\nhigh-fidelity oracles would greatly enhance generative design and be expected to\nimprove hit rates. However, current models are not efficient enough to consider such\na prospect, exemplifying the sample efficiency problem. In this work, we introduce\n**Saturn**, which leverages the Augmented Memory algorithm and demonstrates the\nfirst application of the Mamba architecture for generative molecular design. We\nelucidate *how* experience replay with data augmentation improves sample efficiency\nand *how* Mamba synergistically exploits this mechanism. Saturn outperforms 22\nmodels on multi-parameter optimization tasks relevant to drug discovery and may\npossess sufficient sample efficiency to consider the prospect of directly optimizing\nhigh-fidelity oracles."},"_bibtex":{"value":"@misc{\nguo2025saturn,\ntitle={Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation},\nauthor={Jeff Guo and Philippe Schwaller},\nyear={2025},\nurl={https://openreview.net/forum?id=Hbpzrh7JbN}\n}"},"title":{"value":"Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation"},"pdf":{"value":"/pdf/c0929a667bd3f510fb8322a9d402f95c457e38e1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"guo|saturn_sampleefficient_generative_molecular_design_using_memory_manipulation"},"authorids":{"value":["~Jeff_Guo1","~Philippe_Schwaller1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jeff Guo","Philippe Schwaller"]}},"version":2},{"content":{"summary":{"value":"Most of sequence models include the token mixers and the channel mixers, and this paper provides a detailed summary. They also identify the matrix parameterization is crucial for recent SSMs. Therefore, they extend the Mamba model by adding bidirectional quasiseparable matrix mixer. The experiments on GLUE benchmark and ImageNet data verify their method."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. The methods to implement token mixer include not only structured matrix mixer, such as implicit reparameterization of CKConv. So the conclusion in Line 117 is inaccurate. What do you think are the advantages and disadvantages of implicit matrices over structured matrices?\n\n2. How to understand \"Sequence Aligned\" property in Table 1?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. This paper summarizes the previous relevant methods very well, as illustrated in Table 1. The authors also claim that the following two properties are important for sequence aligned matrices: data dependency and extendability. The former one is a well-known propoerty.\n\n2. The proposed quasiseparable matrix mixer is simple and easy to understand. It is implemented through two Semiseparable matrixes.\n\n3. The provided results show the advancement of their method."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The authors claim that the sequence model should be extended beyond their trained length, but they didn't provide the corresponding results.\n\n2. I am curious about the computational complexity of Hydra compared with Mamba. Because they use two SSD operations as shown in Figure 4. The ablation studies are also necessary for the different variants.\n\n3. Please compare with the most advanced methods, such as xLSTM. It also expands the hidden state into a matrix form."},"limitations":{"value":"N/A"}},"nonreaders":[],"tmdate":1730878826825,"tcdate":1721149279622,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission3122/Reviewer_9VST"],"signatures":["NeurIPS.cc/2024/Conference/Submission3122/Reviewer_9VST"],"forum":"preo49P1VY","number":4,"license":"CC BY 4.0","cdate":1721149279622,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission3122/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878826825,"domain":"NeurIPS.cc/2024/Conference","replyto":"preo49P1VY","id":"y5LexPAw2a","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Deep learning","sequence models","state space models","mamba"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"A wide array of sequence models are built on a framework modeled after Transformers, comprising alternating sequence mixer and channel mixer layers.  This paper studies a unifying *matrix mixer* view of sequence mixers that can be conceptualized as a linear map on the input sequence. This framework encompasses a broad range of well-known sequence models, including the self-attention of Transformers as well as recent strong alternatives such as structured state space models (SSMs), and allows understanding downstream characteristics such as efficiency and expressivity through properties of their structured matrix class. We identify a key axis of matrix parameterizations termed *sequence alignment*, which increases the flexibility and performance of matrix mixers, providing insights into the strong performance of Transformers and recent SSMs such as Mamba. Furthermore, the matrix mixer framework offers a systematic approach to developing sequence mixers with desired properties, allowing us to develop several new sub-quadratic sequence models. In particular, we propose a natural bidirectional extension of the Mamba model (**Hydra**), parameterized as a *quasiseparable matrix mixer*, which demonstrates superior performance over other sequence models including Transformers on non-causal tasks. As a drop-in replacement for attention layers, \\name outperforms BERT by 0.8 points on the GLUE benchmark and ViT by 2% Top-1 accuracy on ImageNet."},"_bibtex":{"value":"@inproceedings{\nhwang2024hydra,\ntitle={Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers},\nauthor={Sukjun Hwang and Aakash Lahoti and Ratish Puduppully and Tri Dao and Albert Gu},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=preo49P1VY}\n}"},"title":{"value":"Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers"},"pdf":{"value":"/pdf/975e174ed724f910ffb63bad51c2578e145b4c50.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"hwang|hydra_bidirectional_state_space_models_through_generalized_matrix_mixers"},"authorids":{"value":["~Sukjun_Hwang1","~Aakash_Lahoti1","~Ratish_Puduppully1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Sukjun Hwang","Aakash Lahoti","Ratish Puduppully","Tri Dao","Albert Gu"]}},"version":2},{"content":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/OpenNLPLab/LASP"},"supplementary_material":{"value":"/attachment/62220f7527cbb0080f8c523716a22198519b4d36.zip"},"abstract":{"value":"Sequence parallelism (SP) serves as a prevalent strategy to handle long sequences that exceed the memory limit of a single device. However, for linear sequence modeling methods like linear attention, existing SP approaches do not take advantage of their right-product-first feature, resulting in sub-optimal communication efficiency and usability. In this paper, we introduce Linear Attention Sequence Parallelism (LASP), an efficient SP approach designed for linear attention-based transformer models. Specifically, we design an efficient point-to-point ring-style communication mechanism to leverage the right-product kernel trick of linear attention, which sharply decreases the communication overhead, comparing with existing SP methods. We enhance the computation efficiency of LASP by performing kernel fusion and intermediate state caching, making the implementation of LASP hardware-friendly on GPUs. Furthermore, we meticulously ensure the compatibility of sequence-level LASP with all types of batch-level data parallel methods, which is vital for distributed training on large clusters with very-long sequences. We also discuss the generalization of LASP on other linear sequence modeling methods. Extensive experiments on linear attention-based models are conducted with varying sequence lengths from 2K to 4096K. LASP scales sequence length up to 4096K on 128 GPUs, which is 8$\\times$ longer than existing SP methods. Code is available at: \\url{https://github.com/OpenNLPLab/LASP}."},"_bibtex":{"value":"@article{\nsun2025lasp,\ntitle={{LASP}: Linear Attention Sequence Parallelism},\nauthor={Weigao Sun and Zhen Qin and Dong Li and Xuyang Shen and Yu Qiao and Yiran Zhong},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=gG8sQUUtN7},\nnote={}\n}"},"title":{"value":"LASP: Linear Attention Sequence Parallelism"},"pdf":{"value":"/pdf/156e20065604d197a500a879d09a0adb9ffe8bff.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"sun|lasp_linear_attention_sequence_parallelism"},"authorids":{"value":["~Weigao_Sun1","~Zhen_Qin6","~Dong_Li11","~Xuyang_Shen1","~Yu_Qiao1","~Yiran_Zhong1"]},"assigned_action_editor":{"value":"~Shuangfei_Zhai3"},"authors":{"value":["Weigao Sun","Zhen Qin","Dong Li","Xuyang Shen","Yu Qiao","Yiran Zhong"]}},"tmdate":1747874125001,"pdate":1747874124944,"tcdate":1736391363122,"writers":["TMLR"],"signatures":["TMLR/Paper3917/Authors"],"forum":"gG8sQUUtN7","license":"CC BY 4.0","number":3917,"cdate":1736391363122,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/Paper3917/-/Revision","TMLR/Paper3917/-/Camera_Ready_Revision","TMLR/-/Accepted"],"mdate":1747874125001,"odate":1736882766042,"domain":"TMLR","id":"gG8sQUUtN7","version":2},{"content":{"summary":{"value":"The paper proposes to sparsify the query of self-attention layers in the context of layerwise hybridization between state-space models and self-attention."},"soundness":{"value":1},"confidence":{"value":5},"questions":{"value":"1. Line 337: Jamba does not have positional embedding.\n\n2. How is the model performance sensitive to the $\\lambda$ of the constraint loss?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The proposed approach is simple and shows more stable extrapolation on perplexity compared to Jamba.\n2. The feature refinement mechanism, which uses the underlying probability distributino of the gumbel softmax to interporlate the residual branch output and the layer input, looks interesting."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The comparison between Jamba and Taipan is not fair: Taipan uses 1:6 for number of attention layers v.s. number of Mamba layers, while Jamba uses 1:7. Also, Taipan uses Mamba 2 while Jamba uses Mamba 1. The performance gain of Taipan in Table 1 can be from the fact that Taipan uses Mamba 2 and has more attention layers, and may have nothing to do with the proposed selective attention.\n2. Lack of novelty: Hybridization between state-space-models and dynamic selective SWA has been explored in SeqBoat [1], but the paper does not include any discussions or emipical study to compare different selction mechanisms. Also, Taipan does not select key-value pairs, which will limit its long context performance.\n3. Lack of important baselines: The paper should at least compare the performance of Taipan with a simple baseline that has 1:6 SWA-Mamba2 ratio to prove the effectivnees of the proposed selective attention. A more thorough comparisons should includes different sparse attention baselines as proposed in BigBird [2] and LongFormer, which are now well supported by FlexAttention [3].\n4. Lack of implementation details: The paper does not includes a detailed description of how hyperparameters are configurated, such as: the temperature of Gumbel softmax, and how the query selection is efficiently implemented so that the proposed selective attention can result in wall-time speed up.\n5. Taipan only shows non-exploading perplexity for long context extrapolation, which is trival for SWA based Mamba hybrid models, considering that Samba [4] already shows improving perplexity up to 1M context length. The paper can be strengthened with more evidiences on long context tasks such as Passkey Retrieval.\n\n\nMissing References: \n\n[1] Sparse Modular Activation for Efficient Sequence Modeling (NeurIPS 2023)\n\n[2] Big Bird: Transformers for Longer Sequences (NeuIPS 2020)\n\n[3] https://github.com/pytorch-labs/attention-gym\n\n[4] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling (arXiv 2023)"}},"nonreaders":[],"tmdate":1731429282855,"tcdate":1730611878617,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9385/Reviewer_BrUv"],"signatures":["ICLR.cc/2025/Conference/Submission9385/Reviewer_BrUv"],"forum":"9DnKZbOr4r","number":2,"license":"CC BY 4.0","cdate":1730611878617,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9385/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429282855,"domain":"ICLR.cc/2025/Conference","replyto":"9DnKZbOr4r","id":"3MM7AUDV2e","forumContent":{"TLDR":{"value":"We introduce Taipan, a novel architecture that combines State Space Models with Selective Attention Layers, achieving superior performance in memory-intensive tasks while enabling efficient language modeling for sequences up to 1 million tokens."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Efficient Language Model","Model Architecture","Long-context Language Model","In-context Retrieval","Hybrid Architecture","Linear Complexity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long sequences due to quadratic computational complexity in training and linearly scaling memory costs during inference. Recent State Space Models (SSMs) such as Mamba offer alternatives with constant memory usage, but they underperform in tasks requiring extensive in-context retrieval. We introduce Taipan, a novel hybrid architecture that combines Mamba-2 with Selective Attention Layers (SALs). These SALs identify tokens requiring long-range interactions, remove less important features, and then augment their representations using the attention module. This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. By constraining the attention budget, Taipan extends accurate predictions to context lengths of up to 1 million tokens while preserving computational efficiency. Our experiments demonstrate Taipan's superior performance across various scales and tasks, offering a promising solution for efficient long-context language modeling."},"_bibtex":{"value":"@misc{\nnguyen2025taipan,\ntitle={Taipan: Efficient and Expressive State Space Language Models with Selective Attention},\nauthor={Chien Van Nguyen and Huy Huu Nguyen and Thang M. Pham and Ruiyi Zhang and Hanieh Deilamsalehy and Puneet Mathur and Ryan A. Rossi and Trung Bui and Viet Dac Lai and Franck Dernoncourt and Thien Huu Nguyen},\nyear={2025},\nurl={https://openreview.net/forum?id=9DnKZbOr4r}\n}"},"title":{"value":"Taipan: Efficient and Expressive State Space Language Models with Selective Attention"},"pdf":{"value":"/pdf/4c569111617737e5532042f859ac730be0ac18ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"nguyen|taipan_efficient_and_expressive_state_space_language_models_with_selective_attention"},"authorids":{"value":["~Chien_Van_Nguyen1","~Huy_Huu_Nguyen1","~Thang_M._Pham1","~Ruiyi_Zhang3","~Hanieh_Deilamsalehy1","~Puneet_Mathur2","~Ryan_A._Rossi2","~Trung_Bui1","~Viet_Dac_Lai1","~Franck_Dernoncourt1","~Thien_Huu_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chien Van Nguyen","Huy Huu Nguyen","Thang M. Pham","Ruiyi Zhang","Hanieh Deilamsalehy","Puneet Mathur","Ryan A. Rossi","Trung Bui","Viet Dac Lai","Franck Dernoncourt","Thien Huu Nguyen"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.03810v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"ren|can_mamba_always_enjoy_the_free_lunch"},"authorids":{"value":["~Ruifeng_Ren3","https://dblp.org/search/pid/api?q=author:Zhicong_Li:","https://dblp.org/search/pid/api?q=author:Yong_Liu:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.03810"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-03810,\n  publtype={informal},\n  author={Ruifeng Ren and Zhicong Li and Yong Liu},\n  title={Can Mamba Always Enjoy the \"Free Lunch\"?},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.03810},\n  url={https://doi.org/10.48550/arXiv.2410.03810}\n}\n"},"abstract":{"value":"Transformers have been the cornerstone of current Large Language Models (LLMs); however, its linear growth in overhead during inference with respect to sequence length poses challenges for modeling long sequences. In this context, Mamba has gradually attracted attention due to its constant-level size during inference and existing empirical results have shown that it can perform comparably to Transformers in sequence modeling while offering significant savings. However, one may ask that, can Mamba always enjoy the ``free lunch\"? In this paper, we focus on analyzing the expressive ability of Mamba from a theoretical standpoint. First, inspired by the connection between Mamba and linear attention, we investigate potential shortcomings of the Mamba when performing the COPY operation. Our results indicate that Mamba with constant size may encounter bottlenecks when handling COPY, while it can achieve perfect performance when the size scales linearly with sequence length. Based on this observation, we analyze Mamba's ability to tackle DP problems when equipped with Chain of Thought (CoT). Our findings suggest that to solve arbitrary DP problems, the total cost of Mamba is comparable to standard and efficient Transformers. However, similar to efficient Transformers, when facing DP problems with favorable properties such as locality, Mamba can provide savings in overhead. Our results contribute to a deeper understanding of Mamba."},"title":{"value":"Can Mamba Always Enjoy the \"Free Lunch\"?"},"authors":{"value":["Ruifeng Ren","Zhicong Li","Yong Liu"]}},"tmdate":1747323477880,"pdate":1704067200000,"tcdate":1747323474819,"writers":["~"],"signatures":["~Ruifeng_Ren3"],"forum":"bevJa5D1ON","license":"CC BY-SA 4.0","number":492737,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747323477880,"domain":"DBLP.org","id":"bevJa5D1ON","version":2},{"content":{"TLDR":{"value":"This paper explores the expressive power of Mamba."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Expressive Power","Chain of Thought"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Transformers have been the cornerstone of current Large Language Models (LLMs); however, its linear growth in overhead during inference with respect to sequence length poses challenges for modeling long sequences. In this context, Mamba has gradually attracted attention due to its constant-level size during inference and existing empirical results have shown that it can perform comparably to Transformers in sequence modeling while offering significant savings. However, one may ask that, can Mamba always enjoy the ``free lunch\"? In this paper, we focus on analyzing the expressive ability of Mamba from a theoretical standpoint. First, inspired by the connection between Mamba and linear attention, we investigate potential shortcomings of the Mamba when performing the COPY operation. Our results indicate that Mamba with constant size may encounter bottlenecks when handling COPY, while it can achieve perfect performance when the size scales linearly with sequence length. Based on this observation, we analyze Mamba's ability to tackle DP problems when equipped with Chain of Thought (CoT). Our findings suggest that to solve arbitrary DP problems, the total cost of Mamba is comparable to standard and efficient Transformers. However, similar to efficient Transformers, when facing DP problems with favorable properties such as locality, Mamba can provide savings in overhead. Our results contribute to a deeper understanding of Mamba."},"_bibtex":{"value":"@misc{\nren2025can,\ntitle={Can Mamba Always Enjoy the ''Free Lunch''?},\nauthor={Ruifeng Ren and Zhicong Li and Yong Liu},\nyear={2025},\nurl={https://openreview.net/forum?id=UYarAv7rUx}\n}"},"title":{"value":"Can Mamba Always Enjoy the \"Free Lunch\"?"},"pdf":{"value":"/pdf/b4552a0f966e40cf480dfcacf744f021f4b8b521.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ren|can_mamba_always_enjoy_the_free_lunch"},"authorids":{"value":["~Ruifeng_Ren3","~Zhicong_Li1","~Yong_Liu7"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Ruifeng Ren","Zhicong Li","Yong Liu"]}},"tmdate":1738735740607,"tcdate":1727357319718,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6956/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission6956/Authors"],"forum":"UYarAv7rUx","license":"CC BY 4.0","number":6956,"cdate":1727357319718,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission6956/-/Full_Submission","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735740607,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"UYarAv7rUx","version":2},{"content":{"summary":{"value":"LION a bidirectional sequence-to-sequence framework that casts attention (Transformers) as RNNs. \nThe authors propose LION-S, a model which combines LION with Selective SSMs. \nLION-S achieves competitive performance with state-of-the-art vision and masked language transformer models while being more resource efficient."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"The paper focuses on LION-S as the main contributing method. However, to my understanding, LION-S would use the parallel scan algorithm. However, Figure (1) middle illustrates a matrix formulation during training instead. Could you clarify this? \n\nLong Range Arena's Text dataset is described with an input length of 2048 in Table 2, but in previous papers such as in \"S5\", the dataset is described with an input length of 4096. Could you clarify if it is indeed different?\n\nAre 27a and 27b coefficients supposed to be $\\exp(a_i)$ and $(1 - \\exp(a_i))$ instead of $\\exp(a_i)$ and $(\\exp(a_i) - 1)$? Since $\\exp(a_i) = \\exp(\\log(\\sigma(w^T_a x_i + b))) = \\sigma(w^T_a x_i + b)$, we have the sum of the two coefficients summing to $1$, ensuring an attention mask. If not, it's not clear to me why the second coefficient would be $(\\exp(a_i) - 1)$.\n\nConsidering the similarities between LION(-S) with SSMs, what are the conceptual and practical differences compared with existing SSM approaches? In addition, what is the practical benefit of using LION(-S) over these SSMs?\n\nConsidering the poor performances of S5 and Mamba, could you explain these discrepancies and provide details on their implementation or experimental setup?\n\nCould you include LION-LIT results or explain why they were omitted in Table 2?\n\nSince the paper labels LSTMs and GRUs as bidirectional when they are traditionally unidirectional, could you include the definition of bidirectionality used in the paper and how it applies consistently across the different models in the table?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The paper's illustrations and equations are written in a way that makes it easier to understand. The colour coding is a nice touch."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper proposes LION emphasizing that the framework casts bidirectional Transformers as bidirectional RNNs. \nThe paper then introduces LION-S (a model based on state-space models) and focuses on comparisons with Transformers showing efficiency gains and substantial performance improvements. However, this is misleading since the framework LION and LION-S can be viewed simply as SSMs.\n\nBy viewing LION as simply SSMs, the claimed benefits of their proposed transformers are not surprising and are common in prior SSMs. For example, (1) \"does not require any additional positional encoding, enabling it to extrapolate beyond the context length or resolution during inference.\". (2) competitive performance and significantly more efficient than transformers. (3) Solving the Long Range Arena (LRA) task.\n\nMore concretely, consider Theorem 3.3 $y_i$'s numerator and denominator:\n$$\n\ty_i^F + y_i^B = q_i^T (S_i^F + S_i^B - k_i v_i)\n$$\n\n$$\n\tc_i^F + c_i^B = q_i^T (z_i^F + z_i^B - k_i)\n$$\n\n$S^F, S^B, z^F, z^B$ are simply SSMs. Both $(S^F, z^F)$ and $(S^B, z^B)$ are computable with forward and backward parallel scans respectively. \nEssentially the numerator is two SSMs (1 forward and 1 backward) summed together ensuring bidirectionality. The denominator is also two SSMs summed together.\n\nAs a result, several experimental results are misleading. LION-S is grouped with \"Transformer as Linear Recurrent Model\" but it's essentially sums of SSMs with scaling. Viewing it as such, LION-S's similar performance to S4 and S5 is not surprising. Furthermore, its efficiency gains over Transformers are not surprising either. \nNotably, Figure 3 (right) efficiency gains are normal since it's essentially comparing SSMs with a Transformer. A more fair comparison in terms of performance efficiency would be to compare with Vision SSM models such as Vision Mamba. \n\n\n\nSome other concerns I have are as follows:\n -  In Table 2, S5 results are worse than S4, and Mamba results are very poor compared to S4 and S5. However, one would expect Mamba and S5 to outperform S4 as these methods improved upon S4. It is very surprising to see Mamba perform so poorly and S5's performance significantly worse than what is reported in the original paper. Justification for these results is needed. \n - The paper mentions \"Transformers can be represented as SSMs,\" citing the Mamba-2 paper => This statement is imprecise. The Mamba-2 paper mentions in its footnote: \"Technically speaking, these connections only relate to certain flavors of attention.\" The original Transformer (using Softmax) cannot be represented with a single SSM as it does not have a linear relationship. \n - LION-LIT results are missing from Table 2.\n - LSTM and GRU are described in Table 1 as having bidirectionality. These methods are not bidirectional by default unless stacked in different directions. With a similar logic, the various modern RNNs such as linear transformers and state-space models are also bidirectional by stacking them in different directions. This has already been done in several works such as the S5 paper for their experiments. \n - The paper mentions \"SSMs often fall short in tasks requiring dense information processing\". This is due to their bottleneck due to also being viewable as an RNN. The way the introduction is framed suggests to me that since LION is based on Transformers, it won't have this issue. However, LION should also have this issue since they are also viewable as RNNs (and are essentially being SSMs in different directions summed together)."}},"nonreaders":[],"tmdate":1731428746676,"tcdate":1730620554478,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13771/Reviewer_sped"],"signatures":["ICLR.cc/2025/Conference/Submission13771/Reviewer_sped"],"forum":"VxJjw52dZu","number":2,"license":"CC BY 4.0","cdate":1730620554478,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13771/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428746676,"domain":"ICLR.cc/2025/Conference","replyto":"VxJjw52dZu","id":"wsfYqaAFzJ","forumContent":{"TLDR":{"value":"We provide a bidirectional selective recurrent form of full kernelized attention with learnable mask allowing scalability in context length and superior inference efficiency"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Self-attention","transformer","bidirectionality","mamba","ssm","rnn","inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We introduce LION, a novel sequence-to-sequence framework that unifies the bidirectionality and parallelized training of Transformers with the fast inference of recurrent neural networks. LION is built upon a mathematical formulation where full kernelized attention with a learnable mask is efficiently computed using a bidirectional selective recurrent model, matching the effectiveness of softmax-based attention with constant-time inference. Our framework naturally accounts for spatial and temporal relationships within input sequences, reducing reliance on heuristic positional embeddings and facilitating straightforward scalability in context length and resolution. Using our framework and inspired by the recent state-space models, we propose three main running examples LIOn-LIT, LION-RETNET, and LION-S, a transformer with selective mask and recurrent inference. Numerical evaluations on tasks such as language modeling, the Long-Range Arena, and image classification show that LION framework achieves performance on par with state-of-the-art models while delivering fast training and inference efficiency."},"_bibtex":{"value":"@misc{\nafzal2025lion,\ntitle={{LION}: A bidirectional framework that trains like a Transformer and infers like an {RNN}},\nauthor={Arshia Afzal and Elias Abad Rocamora and Leyla Naz Candogan and Pol Puigdemont and Francesco Tonin and Yongtao Wu and Mahsa Shoaran and Volkan Cevher},\nyear={2025},\nurl={https://openreview.net/forum?id=VxJjw52dZu}\n}"},"title":{"value":"LION: A bidirectional framework that trains like a Transformer and infers like an RNN"},"pdf":{"value":"/pdf/6e8b8585752d6f27da24ffe876859459a5b56efe.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"afzal|lion_a_bidirectional_framework_that_trains_like_a_transformer_and_infers_like_an_rnn"},"authorids":{"value":["~Arshia_Afzal1","~Elias_Abad_Rocamora1","~Leyla_Naz_Candogan1","~Pol_Puigdemont1","~Francesco_Tonin1","~Yongtao_Wu1","~Mahsa_Shoaran1","~Volkan_Cevher1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Arshia Afzal","Elias Abad Rocamora","Leyla Naz Candogan","Pol Puigdemont","Francesco Tonin","Yongtao Wu","Mahsa Shoaran","Volkan Cevher"]}},"version":2},{"content":{"summary":{"value":"For ego-vehicle motion forecasting, the authors leverage recent advances in state space models (SSMs) and propose the Multi-Stage State Space Model (MS3M). In MS3M, the Temporal Mamba Model is employed to capture fine-grained temporal information, while the Spatial Mamba Model handles spatial interactions. Furthermore, a hierarchical query-based decoder is introduced.\n\nTo demonstrate the effect of the proposed approach, the authors conducted experiments on the large-scale relevant Argoverse2 dataset."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. Judging by the scheme of the proposed Multi-Stage State Space Model (MS3M), one of the input data is the trajectories of other agents, it would be interesting to see the dependence of the performance of the proposed method on the number of these agents.\n2. How would this dependence compare with the result of transformer models, for example QCNet?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The authors propose the Multi-Stage State Space Model (MS3M), a novel multi-stage architecture that integrates a selective scanning mechanism into motion forecasting task. It incorporates a Temporal Mamba Model to capture fine-grained temporal information and a Spatial Mamba Model to model spatial interactions.\n2. The authors also propose an original hierarchical query-based decoder.\n3. The authors conducted experiments with the relevant and sufficiently diverse Argoverse2 dataset."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. A significant drawback of the article is the small number of experiments that would confirm the hypotheses on the quality and performance of the proposed approach compared to state-of-the-art approaches. In addition to Argoverse2 for the same task, there are benchmarks for motion forecasting based on Waymo Open Dataset, Lyft, Nuscenes, on which the quality of the proposed approach was not tested. Thus, I recommend the authors to conduct additional experimental validation of their approach on the mentioned datasets. This would raise important questions about the performance of the method and its generalizability to different data distributions.\n2. The positive effect and applicability of the proposed approach is questionable. In Table 1, the authors themselves show that the developed MS3M approach is slightly inferior to the QCNet transformer approach, and in Table 2, the latency ranges of these methods significantly overlap taking into account the variance. Thus, the authors' thesis that the proposed method \"achieves superior performance while significantly reducing model size and latency\" is questionable. I suggest that the authors clarify or revise their claims about superior performance and reduced latency in light of the overlapping ranges.\n3. The comparison of methods does not include approaches that appeared in 2024, such as UniTraj, JointMotion, etc. I recommend the authors to add an updated comparison to the paper and discuss what fundamental advantages MS3M may have over these recent methods.\n4. The text of the paper requires the elimination of typos and inaccuracies, for example, some formulas lack punctuation marks at the end (commas or dots)."}},"nonreaders":[],"tmdate":1731427842258,"tcdate":1730531819162,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3447/Reviewer_g3Ds"],"signatures":["ICLR.cc/2025/Conference/Submission3447/Reviewer_g3Ds"],"forum":"MmOQY71YHw","number":3,"license":"CC BY 4.0","cdate":1730531819162,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3447/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427842258,"domain":"ICLR.cc/2025/Conference","replyto":"MmOQY71YHw","id":"Nx2uujpriW","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"This paper proposes a Mamba-based motion forecasting architecture that achieves superior performance with significantly lower latency."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Motion Forecasting","Autonomous Driving","State Space Model"]},"supplementary_material":{"value":"/attachment/101a980240bdc46f184713f07f4571c1b29e062b.pdf"},"primary_area":{"value":"applications to robotics, autonomy, planning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Motion forecasting is a fundamental component of autonomous driving systems, as it predicts an agent's future trajectories based on its surrounding environment. Transformer architectures have dominated this domain due to their strong ability to model both temporal and spatial information. However, transformers often suffer from quadratic complexity with respect to input sequence length, limiting their ability to efficiently process scenarios involving numerous agents. Additionally, transformers typically rely on positional encodings to represent temporal or spatial relationships, a strategy that may not be as effective or intuitive as the inductive biases naturally embedded in convolutional architectures. To address these challenges, we leverage recent advancements in state space models (SSMs) and propose the Multi-Stage State Space Model (MS$^3$M). In MS$^3$M, the Temporal Mamba Model (TMM) is employed to capture fine-grained temporal information, while the Spatial Mamba Model efficiently handles spatial interactions. By injecting temporal and spatial inductive biases through Mamba’s state-space model structure, the model's capacity is significantly improved. MS$^3$M also strikes an exceptional trade-off between accuracy and efficiency, which is achieved through convolutional computations and near-linear computational strategies in the Mamba architecture. Furthermore, a hierarchical query-based decoder is introduced, further enhancing model performance and efficiency. Extensive experimental results demonstrate that the proposed method achieves superior performance while maintaining low latency, which is crucial for practical real-time autonomous driving systems."},"_bibtex":{"value":"@misc{\nliu2025msm,\ntitle={{MS}\\${\\textasciicircum}3\\$M: Multi-Stage State Space Model for Motion Forecasting},\nauthor={Chunyu Liu and Shijie Li and Xulei Yang and Jianjun Yu},\nyear={2025},\nurl={https://openreview.net/forum?id=MmOQY71YHw}\n}"},"title":{"value":"MS$^3$M: Multi-Stage State Space Model for Motion Forecasting"},"pdf":{"value":"/pdf/b5472cb1e15055084d263ef47ccb9412ed68e871.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"liu|ms^3m_multistage_state_space_model_for_motion_forecasting"},"authorids":{"value":["~Chunyu_Liu2","~Shijie_Li1","~Xulei_Yang2","~Jianjun_Yu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chunyu Liu","Shijie Li","Xulei Yang","Jianjun Yu"]}},"version":2},{"content":{"TLDR":{"value":"We identify three critical gaps in modern sequence models through  triangular connections among human memory consolidation, RL experience  replay, and neural architecture design, proposing mechanisms and  testable predictions."},"venue":{"value":"HCAIR 2026"},"keywords":{"value":["memory consolidation","complementary learning systems","selective attention","sub-quadratic models","experience replay","cognitive architectures"]},"abstract":{"value":"Recent sub-quadratic sequence models-including Mamba, DeltaNet, and Titans-have independently converged on selective memory mechanisms allocating write intensity proportional to input surprise, mirroring surprise-gated encoding in human episodic memory. We argue this convergence reflects shared computational pressures studied under complementary learning systems (CLS) theory. However, current models capture only encoding; the critical post-encoding processes of consolidation, replay, and reconsolidation remain absent. Drawing triangular connections among (i) prioritized experience replay in reinforcement learning, (ii) surprise-gated writing in sequence models, and (iii) sleep-dependent consolidation in cognition, we identify three architectural gaps: lack of offline replay across memory systems, absence of importance-sampling correction for selective writes, and missing reconsolidation pathways. We formalize each gap, propose mechanisms, and derive testable predictions at both machine learning and cognitive modeling levels, charting a design space for cognitively grounded sequence architectures."},"_bibtex":{"value":"@inproceedings{\nkim2026a,\ntitle={A Cognitive Lens on Selective Memory in Neural Sequence Models: Surprise, Replay, and Consolidation},\nauthor={SoHyung Kim and Jea Kwon},\nbooktitle={ICLR 2026 Workshop - From Human Cognition to AI Reasoning: Models, Methods, and Applications},\nyear={2026},\nurl={https://openreview.net/forum?id=sxo1BcVUgp}\n}"},"title":{"value":"A Cognitive Lens on Selective Memory in Neural Sequence Models: Surprise, Replay, and Consolidation"},"paper_type":{"value":"Blue Sky Paper"},"pdf":{"value":"/pdf/baf4fa7bc6b15831bcdaf7113f86b0bb019a1585.pdf"},"venueid":{"value":"ICLR.cc/2026/Workshop/HCAIR"},"paperhash":{"value":"kim|a_cognitive_lens_on_selective_memory_in_neural_sequence_models_surprise_replay_and_consolidation"},"authorids":{"value":["~SoHyung_Kim1","~Jea_Kwon1"]},"authors":{"value":["SoHyung Kim","Jea Kwon"]}},"tmdate":1777327548772,"pdate":1772668728359,"tcdate":1770332303716,"writers":["ICLR.cc/2026/Workshop/HCAIR","ICLR.cc/2026/Workshop/HCAIR/Submission66/Authors"],"signatures":["ICLR.cc/2026/Workshop/HCAIR/Submission66/Authors"],"forum":"sxo1BcVUgp","license":"CC BY 4.0","number":66,"cdate":1770332303716,"readers":["everyone"],"invitations":["ICLR.cc/2026/Workshop/HCAIR/-/Submission","ICLR.cc/2026/Workshop/HCAIR/-/Post_Submission","ICLR.cc/2026/Workshop/HCAIR/-/Edit","ICLR.cc/2026/Workshop/HCAIR/Submission66/-/Revision"],"mdate":1777327548772,"odate":1772814546018,"domain":"ICLR.cc/2026/Workshop/HCAIR","id":"sxo1BcVUgp","version":2},{"content":{"comment":{"value":"## List references used in the rebuttal\n[1] Shen, Xuan, et al. \"Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024. \n\n[2] Xiao, Guangxuan, et al. \"Smoothquant: Accurate and efficient post-training quantization for large language models.\" International Conference on Machine Learning. PMLR, 2023. \n\n[3] Lin, Yang, et al. \"Fq-vit: Post-training quantization for fully quantized vision transformer.\" arXiv preprint arXiv:2111.13824 (2021). \n\n[4] Dong, Peiyan, et al. \"Packqvit: Faster sub-8-bit vision transformers via full and packed quantization on the mobile.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[5] Sakr, Charbel, et al. \"Optimal clipping and magnitude-aware differentiation for improved quantization-aware training.\" International Conference on Machine Learning. PMLR, 2022.\n\n[6] Li, Rundong, et al. \"Fully quantized network for object detection.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n[7] Lin, Yang, et al. \"Fq-vit: Post-training quantization for fully quantized vision transformer.\" arXiv preprint arXiv:2111.13824 (2021).\n\n[8] Tang, Hanlin, et al. \"Razorattention: Efficient kv cache compression through retrieval heads.\" arXiv preprint arXiv:2407.15891 (2024).\n\n[9] Lieber, Opher, et al. \"Jamba: A hybrid transformer-mamba language model.\" arXiv preprint arXiv:2403.19887 (2024).\n\n[10] https://huggingface.co/ai21labs/Jamba-v0.1\n\n[11] Zhao, Han, et al. \"Cobra: Extending mamba to multi-modal large language model for efficient inference.\" arXiv preprint arXiv:2403.14520 (2024).\n\n[12] Xiao, Guangxuan, et al. \"Smoothquant: Accurate and efficient post-training quantization for large language models.\" International Conference on Machine Learning. PMLR, 2023.\n\n[13] Ashkboos, Saleh, et al. \"Quarot: Outlier-free 4-bit inference in rotated llms.\" arXiv preprint arXiv:2404.00456 (2024).\n\n[14] Kumar, Tanishq, et al. \"Scaling Laws for Precision.\" arXiv preprint arXiv:2411.04330 (2024).\n\n[15] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n\n[16] Gu, Albert, et al. \"Hippo: Recurrent memory with optimal polynomial projections.\" Advances in neural information processing systems 33 (2020): 1474-1487."}},"tmdate":1732250047683,"tcdate":1732226523172,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11122/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission11122/Authors"],"forum":"mnna9LUg7P","number":2,"license":"CC BY 4.0","cdate":1732226523172,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11122/-/Official_Comment"],"mdate":1732250047683,"domain":"ICLR.cc/2025/Conference","replyto":"n0ZMPpWkvG","id":"QC8Z0hFc99","forumContent":{"TLDR":{"value":"We propose a 8-bit static quantization method for selective state space models."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["State Space Models","Model quantization"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSMs) have emerged as an appealing alternative to Transformers for large language models, achieving state-of-the-art accuracy with constant memory complexity which allows for holding longer context lengths than attention-based networks. The superior computational efficiency of SSMs in long sequence modeling positions them favorably over Transformers in many scenarios. However, improving the efficiency of SSMs on request-intensive cloud-serving and resource-limited edge applications is still a formidable task. SSM quantization is a possible solution to this problem, making SSMs more suitable for wide deployment, while still maintaining their accuracy. Quantization is a common technique to reduce the model size and to utilize the low bit-width acceleration features on modern computing units, yet existing quantization techniques are poorly suited for SSMs. Most notably, SSMs have highly sensitive feature maps within the selective scan mechanism (i.e., linear recurrence) and massive outliers in the output activations which are not present in the output of token-mixing in the self-attention modules. To address this issue, we propose a static 8-bit per-tensor SSM quantization method which suppresses the maximum values of the input activations to the selective SSM for finer quantization precision and quantizes the output activations in an outlier-free space with Hadamard transform. Our 8-bit weight-activation quantized Mamba 2.8B SSM benefits from hardware acceleration and achieves a 1.72 $\\times$ lower generation latency on an Nvidia Orin Nano 8G, with only a 0.9\\% drop in average accuracy on zero-shot tasks. When quantizing Jamba, a 52B parameter SSM-style language model, we observe only a $1\\%$  drop in accuracy, demonstrating that our SSM quantization method is both effective and scalable for large language models, which require appropriate compression techniques for deployment. The experiments demonstrate the effectiveness and practical applicability of our approach for deploying SSM-based models of all sizes on both cloud and edge platforms."},"_bibtex":{"value":"@inproceedings{\nchiang2025quamba,\ntitle={Quamba: A Post-Training Quantization Recipe for Selective State Space Models},\nauthor={Hung-Yueh Chiang and Chi-Chih Chang and Natalia Frumkin and Kai-Chiang Wu and Diana Marculescu},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=mnna9LUg7P}\n}"},"title":{"value":"Quamba: A Post-Training Quantization Recipe for Selective State Space Models"},"pdf":{"value":"/pdf/0c4ea9cb96dd9e16a33f3931fb1a1347d00b5817.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"chiang|quamba_a_posttraining_quantization_recipe_for_selective_state_space_models"},"authorids":{"value":["~Hung-Yueh_Chiang1","~Chi-Chih_Chang1","~Natalia_Frumkin1","~Kai-Chiang_Wu1","~Diana_Marculescu4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hung-Yueh Chiang","Chi-Chih Chang","Natalia Frumkin","Kai-Chiang Wu","Diana Marculescu"]}},"version":2},{"content":{"TLDR":{"value":"a heterogeneous graph mamba network (HGMN) as the first exploration in leveraging the powerful selective state space models (SSSMs) for heterogeneous graph learning."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["graph neural network","state space model"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We propose a heterogeneous graph mamba network (HGMN) as the first exploration in leveraging the selective state space models (SSSMs) for heterogeneous graph learning. Compared with the literature, our HGMN overcomes two major challenges: (i) capturing long-range dependencies among heterogeneous nodes and (ii) adapting SSSMs to heterogeneous graph data. Our key contribution is a general graph architecture that can solve heterogeneous nodes in real-world scenarios, followed an efficient flow. Methodologically, we introduce a two-level efficient tokenization approach that first captures long-range dependencies within identical node types, and subsequently across all node types. Empirically, we conduct comparisons between our framework and 19 state-of-the-art methods on the heterogeneous benchmarks. The extensive comparisons demonstrate that our framework outperforms other methods in both the accuracy and efficiency dimensions."},"_bibtex":{"value":"@misc{\npan2024hetegraphmamba,\ntitle={HeteGraph-Mamba: Heterogeneous Graph Learning via Selective State Space Model},\nauthor={Zhenyu Pan and Yoonsung Jeong and Xiaoda Liu and Han Liu},\nyear={2024},\nurl={https://openreview.net/forum?id=mn61GWpEiK}\n}"},"title":{"value":"HeteGraph-Mamba: Heterogeneous Graph Learning via Selective State Space Model"},"pdf":{"value":"/pdf/f3e375dedad64b3fb6a91ac3fc520e533a0243a1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"pan|hetegraphmamba_heterogeneous_graph_learning_via_selective_state_space_model"},"authorids":{"value":["~Zhenyu_Pan1","~Yoonsung_Jeong1","~Xiaoda_Liu1","~Han_Liu4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zhenyu Pan","Yoonsung Jeong","Xiaoda Liu","Han Liu"]}},"tmdate":1732092315822,"tcdate":1727316421283,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5526/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5526/Authors"],"forum":"mn61GWpEiK","license":"CC BY 4.0","number":5526,"cdate":1727316421283,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission5526/-/Full_Submission","ICLR.cc/2025/Conference/-/Withdrawn_Submission"],"mdate":1732092315822,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"mn61GWpEiK","version":2},{"content":{"summary":{"value":"The authors study the recent state-space models (SSM) family of efficient sequence architectures and address some of their challenges, related to the inability to perform content-based reasoning. The core contribution of the work is the addition of a selection method to the SSM architecture, which results in simple and scalable architecture, Mamba. Then they demonstrate the superiority of Mamba on standard language benchmarks, as well as DNA and audio modeling. The authors also contribute efficient implementation and benchmarking of Mamba on modern hardware."},"presentation":{"value":"4 excellent"},"contribution":{"value":"3 good"},"soundness":{"value":"4 excellent"},"strengths":{"value":"S1: The paper addresses very efficiently and effectively pressing problems in sequential modeling. \n\nS2: The authors have identified simple toy tasks, such as selective copying and associative recall, that enable them to make design choices which state-of-the-art impact on real-world data.\n\nS3: The connection to the role of gating mechanisms in RNNs is well-appreciated.\n\nS4: The empirical part of the paper is very thorough, and the results are strong."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I do not identify any major weaknesses of the paper."},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"I am curious if we could build a better understanding of the selection mechanism that you propose. In Theorem 1 you link that mechanism to gating in RNNs as a special case. Is it possible to understand better the generalization through some discussion / qualitative examples?"},"rating":{"value":"8: accept, good paper"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636387094,"tcdate":1698672946961,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission4202/Reviewer_sf96"],"signatures":["ICLR.cc/2024/Conference/Submission4202/Reviewer_sf96"],"forum":"AL1fq05o7H","number":1,"license":"CC BY 4.0","cdate":1698672946961,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission4202/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636387094,"domain":"ICLR.cc/2024/Conference","replyto":"AL1fq05o7H","id":"ZJCXkIMKQY","forumContent":{"venue":{"value":"Submitted to ICLR 2024"},"TLDR":{"value":"We introduce a selection mechanism into state space models, leading to state-of-the-art results on general sequence modeling including language."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Sequence model","language model","state space model","RNN","SSM","S4","Mamba"]},"primary_area":{"value":"representation learning for computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language.  We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to *selectively* propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (**Mamba**). Mamba enjoys fast inference (5$\\times$ higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-1.4B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation."},"_bibtex":{"value":"@misc{\ngu2024mamba,\ntitle={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},\nauthor={Albert Gu and Tri Dao},\nyear={2024},\nurl={https://openreview.net/forum?id=AL1fq05o7H}\n}"},"title":{"value":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces"},"pdf":{"value":"/pdf/27144ad57139872b668e5c522eeed631b23f5dce.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"gu|mamba_lineartime_sequence_modeling_with_selective_state_spaces"},"authorids":{"value":["~Albert_Gu1","~Tri_Dao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Albert Gu","Tri Dao"]}},"version":2},{"content":{"venue":{"value":"AAAI 2026"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/40082/44043"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"zhang|mambadriven_multiview_discriminative_clustering_via_globallocal_crossview_sequence_modeling"},"html":{"value":"https://doi.org/10.1609/aaai.v40i34.40082"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/ZhangWZXCWYL26,\n  author={Yuanyang Zhang and Xinhang Wan and Chao Zhang and Jie Xu and Cunjian Chen and Tien-Tsin Wong and Li Yao and Yijie Lin},\n  title={Mamba-Driven Multi-View Discriminative Clustering via Global-Local Cross-View Sequence Modeling},\n  year={2026},\n  cdate={1767225600000},\n  pages={28519-28527},\n  url={https://doi.org/10.1609/aaai.v40i34.40082},\n  booktitle={AAAI},\n  crossref={conf/aaai/2026}\n}\n"},"abstract":{"value":"Multi-view clustering (MVC) has recently garnered increasing attention for its ability to partition unlabeled samples into distinct clusters by leveraging complementary and consistent information from different views. Existing MVC methods primarily combine deep neural networks with contrastive learning for cross-view representation learning, yet often overlook the inherent global-local structural relationships among samples. While GNN-based methods capture local structures, they struggle to model global dependencies, leading to inferior inter-cluster separability. In contrast, Transformer-based methods excel at global aggregation but suffer from quadratic complexity, and their attention smoothing effect weakens fine-grained local structures, resulting in suboptimal intra-cluster compactness. To address these limitations, we propose a novel end-to-end MVC framework called Mamba-Driven Multi-View Discriminative Clustering via Global-Local Cross-View Sequence Modeling (MGLC). By flexibly constructing multi-view sequences, MGLC fully exploits the efficient sequence modeling capabilities of Mamba to jointly model cross-view dependencies and global-local structural relationships among samples. Furthermore, MGLC introduces a Cross-Mamba Fusion module to dynamically integrate cross-view and global-local structural representations. Additionally, MGLC incorporates a Dual Calibration Contrastive Learning module, guided by high-confidence pseudo-labels, that adaptively refines both feature and semantic representations while mitigating false negatives among semantically similar samples. Extensive comparative experiments and ablation studies demonstrate the effectiveness of MGLC."},"title":{"value":"Mamba-Driven Multi-View Discriminative Clustering via Global-Local Cross-View Sequence Modeling"},"authors":{"value":[{"fullname":"Yuanyang Zhang","username":""},{"fullname":"Xinhang Wan","username":""},{"fullname":"Chao Zhang","username":""},{"fullname":"Jie Xu","username":""},{"fullname":"Cunjian Chen","username":""},{"fullname":"Tien-Tsin Wong","username":""},{"fullname":"Li Yao","username":""},{"fullname":"Yijie Lin","username":"~Yijie_Lin1"}]}},"tmdate":1784984434361,"pdate":1798675200000,"externalIds":["dblp:conf/aaai/ZhangWZXCWYL26"],"tcdate":1784984427549,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Yijie_Lin1"],"forum":"cRYaCP30Pf","license":"CC BY-SA 4.0","number":104602,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784984434361,"domain":"OpenReview.net/Public_Article","id":"cRYaCP30Pf","version":2},{"content":{"venue":{"value":"AAAI 2026"},"pdf":{"value":"/pdf/35acd913c56ea5a0a9db52e11da6012ed7f75cfb.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"zhang|mambadriven_multiview_discriminative_clustering_via_globallocal_crossview_sequence_modeling"},"authorids":{"value":["~Yuanyang_Zhang1","~Xinhang_Wan1","~Chao_Zhang22","~Jie_Xu8","~Cunjian_Chen2","~Tien-Tsin_Wong2","~Li_Yao2","~Yijie_Lin1"]},"html":{"value":"https://doi.org/10.1609/aaai.v40i34.40082"},"abstract":{"value":"Multi-view clustering (MVC) has recently garnered increasing attention for its ability to partition unlabeled samples into distinct clusters by leveraging complementary and consistent information from different views. Existing MVC methods primarily combine deep neural networks with contrastive learning for cross-view representation learning, yet often overlook the inherent global-local structural relationships among samples. While GNN-based methods capture local structures, they struggle to model global dependencies, leading to inferior inter-cluster separability. In contrast, Transformer-based methods excel at global aggregation but suffer from quadratic complexity, and their attention smoothing effect weakens fine-grained local structures, resulting in suboptimal intra-cluster compactness. To address these limitations, we propose a novel end-to-end MVC framework called Mamba-Driven Multi-View Discriminative Clustering via Global-Local Cross-View Sequence Modeling (MGLC). By flexibly constructing multi-view sequences, MGLC fully exploits the efficient sequence modeling capabilities of Mamba to jointly model cross-view dependencies and global-local structural relationships among samples. Furthermore, MGLC introduces a Cross-Mamba Fusion module to dynamically integrate cross-view and globallocal structural representations. Additionally, MGLC incorporates a Dual Calibration Contrastive Learning module, guided by high-confidence pseudo-labels, that adaptively refines both feature and semantic representations while mitigating false negatives among semantically similar samples. Extensive comparative experiments and ablation studies demonstrate the effectiveness of MGLC."},"title":{"value":"Mamba-Driven Multi-View Discriminative Clustering via Global-Local Cross-View Sequence Modeling"},"authors":{"value":["Yuanyang Zhang","Xinhang Wan","Chao Zhang","Jie Xu","Cunjian Chen","Tien-Tsin Wong","Li Yao","Yijie Lin"]}},"tmdate":1784704774399,"pdate":1773417600000,"tcdate":1784704774399,"writers":["~Yuanyang_Zhang1","~Xinhang_Wan1","~Chao_Zhang22","~Jie_Xu8","~Cunjian_Chen2","~Tien-Tsin_Wong2","~Li_Yao2","~Yijie_Lin1"],"signatures":["~Yuanyang_Zhang1"],"forum":"RDe4Z7yZcP","license":"CC BY 4.0","number":52515,"cdate":1784704774399,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1784704774399,"domain":"OpenReview.net/Archive","id":"RDe4Z7yZcP","version":2},{"content":{"summary":{"value":"The paper introduces PhyxMamba, a Mamba-based state-space modeling with physics-informed principles for long-term chaotic system forecasting from short historical data. It reconstructs attractor manifolds via time-delay embeddings, employs generative multi-patch training, and preserves attractor geometry through distribution regularization. Experiments on simulated and real-world datasets (Lorenz, Rössler, ECG, EEG) demonstrate superior accuracy and stability over state-of-the-art models in both short- and long-horizon predictions, while maintaining key dynamical and statistical properties of chaotic systems."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. Given the setting in 3.3, the paper focus on researching methods for dissipative chaotic systems. The paper only shows the trajectory results on the invariant measure. Can the paper elaborate on how the method handles the transition from dissipation to  the invariant measure both theoratically and experimentally?\n2. The paper demonstrated PhyxMamba's performance on relatively low dimensional on chaos systems, and the simulations are mainly ode based. How PhyxMamba perform on more complexed PDE chaos systems?\n3. Regarding the computation complexity of MMD, can the paper elaborate on how this term would work and scale on training PhyxMamba for high dimensional chaotic systems, e.g 2D/3D turbulence?\n4. Regarding the short history input, the paper claims using a window of steps covering one Lyapunov time for forecasting. Is it a necessity that  at leaset 'one Lyapunov time' long window is used in every inference? What would the model behave if a shorter length / an incomplete Lyapunov time window is given in inference?\n5. Following on question 4, what would the model behave when the input window has a coarse time steps? (e.g. instead of having 30 points in a Lyapunov time window, only 3-5 points covering a Lyapunov time window are sampled to make the dataset)\n\nHappy to think about increasing my score if the paper is significantly improved to address the questions and weakness concerns."},"rating":{"value":4},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The paper has the following strengths worth credits:\n\n1. Both simulation and real benchmarks are evaluated to demonstrate the generalisation of PhyxMamba on chaos systems;\n2. Building physics constrained Mamba model for chaos systems learning is a novel attempt.\n3. Introducing the multi-patch prediction induced learning constraint is a novel attempt."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper should be improved considering the following facts:\n\n1. The paper evaluates PhyxMamba on simulated ODE chaos systems, not covering simulated PDE chaos systems and more high-dimensional cases, which is a significant overclaim of PhyxMamba as a framework for chaotic systems.\n2. Motivations are not clear. The necesscity of using short history input compared with 1-step input in the literature is not specified.  To which concern that predicting autoregressively with Mamba is important/advanced compared with other methods, such as Markov Neural Operators and constrained Koopman Operators, is not analysed or discussed.\n3. Limited theoretical analysis of convergence and stability is provided for the student-forcing patch decomposition method and its optimzation.\n4. Given the strength of mamba models in interpretability on physical systems analogy, the paper lacks essential exploration and demonstation of hidden states in PhyxMamba, how the components of hidden states are learnt and aligned with the analogy.\n5. Data in Table 4 is hard to read."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927557870,"tcdate":1761849279084,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17726/Reviewer_GCDL"],"signatures":["ICLR.cc/2026/Conference/Submission17726/Reviewer_GCDL"],"forum":"KWsrSSNVSB","number":3,"license":"CC BY 4.0","cdate":1761849279084,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17726/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927557870,"domain":"ICLR.cc/2026/Conference","replyto":"KWsrSSNVSB","id":"gokqdJQnWU","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We develop PhyxMamba, a framework that integrates a Mamba state-space model with physics-informed principles to forecast long-term chaotic systems using only short-term historical data."},"keywords":{"value":["Chaotic Systems","State-space Models"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Long-term forecasting of chaotic systems remains a fundamental challenge due to the intrinsic sensitivity to initial conditions and the complex geometry of strange attractors. Conventional approaches, such as reservoir computing, typically require training data that incorporates long-term continuous dynamical behavior to comprehensively capture system dynamics. While advanced deep sequence models can capture transient dynamics within the training data, they often struggle to maintain predictive stability and dynamical coherence over extended horizons. Here, we propose PhyxMamba, a framework that integrates a Mamba-based state-space model with physics-informed principles to forecast long-term behavior of chaotic systems given short-term historical observations on their state evolution. We first reconstruct the attractor manifold with time-delay embeddings to extract global dynamical features. After that, we introduce a generative training scheme that enables Mamba to replicate the physical process. It is further augmented by multi-patch prediction and attractor geometry regularization for physical constraints, enhancing predictive accuracy and preserving key statistical properties of systems. Extensive experiments on simulated and real-world chaotic systems demonstrate that PhyxMamba delivers superior forecasting accuracy and faithfully captures essential statistics from short-term historical observations."},"_bibtex":{"value":"@misc{\nliu2026mamba,\ntitle={Mamba Integrated with Physics Principles Masters Long-term Chaotic System Forecasting},\nauthor={Chang Liu and Bohao Zhao and Jingtao Ding and Huandong Wang and Yong Li},\nyear={2026},\nurl={https://openreview.net/forum?id=KWsrSSNVSB}\n}"},"title":{"value":"Mamba Integrated with Physics Principles Masters Long-term Chaotic System Forecasting"},"pdf":{"value":"/pdf/b8497e44ef182aaaf814d83c67b5a9b1c64d0e30.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"liu|mamba_integrated_with_physics_principles_masters_longterm_chaotic_system_forecasting"},"authorids":{"value":["~Chang_Liu29","~Bohao_Zhao1","~Jingtao_Ding1","~Huandong_Wang2","~Yong_Li7"]},"authors":{"value":["Chang Liu","Bohao Zhao","Jingtao Ding","Huandong Wang","Yong Li"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2605.05214v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"he|medmamba_recasting_mamba_for_medical_time_series_classification"},"html":{"value":"https://doi.org/10.48550/arXiv.2605.05214"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2605-05214,\n  publtype={informal},\n  author={ZhengXiao He and Huayu Li and Xiwen Chen and Janet M. Roveda and Jinghao Wen and Siyuan Tian and Ao Li},\n  title={MedMamba: Recasting Mamba for Medical Time Series Classification},\n  year={2026},\n  month={May},\n  cdate={1777593600000},\n  journal={CoRR},\n  volume={abs/2605.05214},\n  url={https://doi.org/10.48550/arXiv.2605.05214}\n}\n"},"abstract":{"value":"Medical time series, such as electrocardiograms (ECG) and electroencephalograms (EEG), exhibit complex temporal dynamics and structured cross-channel dependencies, posing fundamental challenges for automated analysis. Conventional convolutional and recurrent models struggle to capture long-range dependencies, while Transformer-based approaches incur quadratic complexity and often introduce redundant interactions that are misaligned with the intrinsic structure of physiological signals. To address these limitations, we propose MedMamba, a principle-driven multi-scale bidirectional state space architecture tailored for medical time series classification. Our design is guided by three key inductive biases of physiological signals: spatial centralization, multi-timescale temporal composition, and non-causal contextual dependency. These principles are instantiated through a lightweight channel-mixing module for cross-channel reparameterization, multi-scale convolutional tokenization for temporal decomposition, and bidirectional Mamba blocks for efficient global context modeling with linear complexity. Extensive experiments on six benchmark datasets spanning EEG, ECG, and human activity signals demonstrate that MedMamba consistently outperforms state-of-the-art methods across diverse modalities. Notably, it achieves 85.97% accuracy on PTB and establishes new state-of-the-art performance on the challenging ADFTD dataset (54.72% accuracy and 52.01% F1-score). Strong results on long-sequence benchmarks, such as SleepEDF, further validate its capability in modeling long-range dependencies. Moreover, MedMamba achieves a speedup of 4.6x in inference, highlighting its practicality for real-time clinical deployment. These results suggest that principle-guided state space modeling offers an effective and scalable alternative to Transformer-based approaches for medical time series analysis."},"title":{"value":"MedMamba: Recasting Mamba for Medical Time Series Classification"},"authors":{"value":[{"fullname":"ZhengXiao He","username":""},{"fullname":"Huayu Li","username":""},{"fullname":"Xiwen Chen","username":""},{"fullname":"Janet M. Roveda","username":""},{"fullname":"Jinghao Wen","username":"~Jinghao_Wen1"},{"fullname":"Siyuan Tian","username":""},{"fullname":"Ao Li","username":""}]}},"tmdate":1784923750301,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2605-05214"],"tcdate":1784923740742,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jinghao_Wen1"],"forum":"tS4BqXzAua","license":"CC BY-SA 4.0","number":102072,"cdate":1777593600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784923750301,"domain":"OpenReview.net/Public_Article","id":"tS4BqXzAua","version":2},{"content":{"venue":{"value":"LoViF Oral"},"keywords":{"value":["state space model","quantization"]},"supplementary_material":{"value":"/attachment/a101403d96fcce7d1eb7a3a2cf71a46f48069cc7.pdf"},"corresponding_author_email":{"value":"zwxiong@ustc.edu.cn"},"abstract":{"value":"State Space Models (SSMs), as key components of Mamba, have gained increasing attention for vision models recently. \nGiven the computational cost of deploying SSMs on resource-limited edge devices, Post-Training Quantization (PTQ) is a technique with the potential for efficient deployment of SSMs.\nHowever, quantizing the selective operation in vision SSMs remains a challenge due to the unique activation distributions.\nIn this work, we propose SELective Operation Quantization (SELOQ), a PTQ method designed for the selective operation in vision SSMs. Based on the analysis of the activation distributions, we reveal that the distribution of discrete parameters exhibits long-tailed skewness and the distribution of the hidden state sequence exhibits highly dynamic variations. Correspondingly, we design Long-tailed Skewness Quantization (LtSQ) to quantize discrete parameters and Temporal Scale Quantization (TSQ) to quantize hidden states, which reduces the quantization errors.\nExtensive experiments on multiple vision tasks demonstrate that SELOQ significantly outperforms existing methods on quantized vision SSMs across multiple model sizes and architectures."},"_bibtex":{"value":"@inproceedings{\nli2026selective,\ntitle={Selective Operation Quantization for Vision State Space Models},\nauthor={Yinglong Li and Xiaoyu Liu and Yinuo Liao and Jiacheng Li and Ruikang Xu and Yinda Chen and Panjun Liu and Zhiwei Xiong},\nbooktitle={The Second Workshop on Low-Level Vision Frontiers with Generative AI, Preference Optimization, Agentic Systems and World Models@ECCV2026},\nyear={2026},\nurl={https://openreview.net/forum?id=4nSjclOHqr}\n}"},"title":{"value":"Selective Operation Quantization for Vision State Space Models"},"corresponding_author_name":{"value":"Zhiwei Xiong"},"pdf":{"value":"/pdf/b6d5770d899eda0013cd4d251d14be02e351518d.pdf"},"venueid":{"value":"thecvf.com/ECCV/2026/Workshop/LoViF"},"paperhash":{"value":"li|selective_operation_quantization_for_vision_state_space_models"},"authorids":{"value":["~Yinglong_Li1","~Xiaoyu_Liu6","~Yinuo_Liao1","~Jiacheng_Li5","~Ruikang_Xu2","~Yinda_Chen1","~Panjun_Liu1","~Zhiwei_Xiong1"]},"authors":{"value":["Yinglong Li","Xiaoyu Liu","Yinuo Liao","Jiacheng Li","Ruikang Xu","Yinda Chen","Panjun Liu","Zhiwei Xiong"]}},"tmdate":1788834383686,"pdate":1788834383648,"tcdate":1784020002121,"writers":["thecvf.com/ECCV/2026/Workshop/LoViF","thecvf.com/ECCV/2026/Workshop/LoViF/Submission40/Authors"],"signatures":["thecvf.com/ECCV/2026/Workshop/LoViF/Submission40/Authors"],"forum":"4nSjclOHqr","license":"CC BY 4.0","number":40,"cdate":1784020002121,"readers":["everyone"],"invitations":["thecvf.com/ECCV/2026/Workshop/LoViF/-/Submission","thecvf.com/ECCV/2026/Workshop/LoViF/-/Post_Submission","thecvf.com/ECCV/2026/Workshop/LoViF/Submission40/-/Camera_Ready","thecvf.com/ECCV/2026/Workshop/LoViF/-/Edit"],"mdate":1788834383686,"odate":1788834383648,"domain":"thecvf.com/ECCV/2026/Workshop/LoViF","id":"4nSjclOHqr","version":2},{"content":{"comment":{"value":"(...cont)\n\n### \"Why isn't transfer-learning or few-shot included...It would be great to see these results in the few-shot setting. I imagine that the performance improvements as you increase sequence length would be even more extreme.\"\n\nThank you for raising this point, we agree that the paper would be significantly improved by including few-shot results. In response, we have added **Appendix Section G (“Few-shot learning on EHRSHOT”)** which includes few-shot results for all models and context lengths on all EHRSHOT tasks in **Appendix Tables 11, 12, and 13** as well as **Appendix Figure 10.** \n\nThe results confirm that for models which benefit from longer contexts (e.g. Mamba and Llama), we see consistent performance gains of longer context across all values of $k$ (i.e. # of shots). Please see the updated manuscript for full details on our few-shot experiments, as the section is too large to reproduce in full below.\n\n### \"This paper should include comparisons to linear attention models that practitioners are interested in.\"\n\nThank you for the suggestion! Yes, we definitely agree that adding linear attention models to our evaluations would be beneficial to readers. We had originally flagged this as future work in **Section 5 (“Discussion”):**\n\n> Sixth, other promising transformer alternatives, such as linear attention models \\cite{arora2024based}, hybrid architectures \\cite{stripedhyena, lieber2024jamba}, and recurrent models \\cite{rwkv}, should be explored in future work that builds upon the framework introduced here.\n\n…but agree that it would be nice to include preliminary results on linear attention models in this work. \n\nThus, per your suggestion **we are now training a linear attention model.** Given compute constraints on our shared cluster, however, it will take a few more days to complete. We will update our response once the results are available, so thank you for your patience. In the meantime, please let us know if we have resolved your other comments or if you have any additional follow-up questions on our responses!\n\n### \"This paper does not communicate compute budgets, such as wall-times and the hardware used for these sequence lengths. Could a plot communicating performance vs the compute-load be provided to help justify whether these improvements are worth the added compute time.\"\n\nThank you for the feedback. Below, we provide a table depicting EHRSHOT performance v. max throughput (i.e. tokens/sec) during inference on a single h100 GPU. Following the benchmarking procedure in the original Mamba paper [1], we have each model generate 128 tokens on a 2048 length prompt across batch sizes ranging from 1 to 128. We then take the max throughput achieved by each model, excluding batch sizes that led to out-of-memory errors. Our results are below:\n\n| Model   |   Context Length |  Max Throughput (tokens/sec)|  Overall AUROC |\n|:-------------|-----------------:|-------------:|----------:|\n| gpt2         |              512 |      744.519 |  0.792351 |\n| gpt2         |             1024 |      743.835 |  0.783288 |\n| gpt2         |             2048 |      741.761 |  0.799709 |\n| gpt2         |             4096 |      746.849 |  0.79148  |\n| llama        |              512 |     1156.82  |  0.78899  |\n| llama        |             1024 |     1157.05  |  0.789721 |\n| llama        |             2048 |     1791.95  |  0.800335 |\n| llama        |             4096 |     1795.43  |  0.796923 |\n| mamba        |             1024 |     2205.27  |  0.784284 |\n| mamba        |             4096 |     2295.82  |  0.80187  |\n| mamba        |             8192 |     2297.37  |  0.798959 |\n| mamba        |            16384 |     2289.14  |  0.807706 |\n\nWhile it is difficult to give precise wall clock estimates of training time given the nature of the shared compute cluster on which we trained our models, we believe these numbers would be useful to a practitioner hoping to deploy such a model in practice, as models’ life-cycles get increasingly spent on inference rather than training [2].\n\n[1] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n[2] Sardana, Nikhil, et al. \"Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.\" arXiv preprint arXiv:2401.00448 (2023).\n\n(cont...)"},"title":{"value":"Author Response (cont.)"}},"tmdate":1732177288786,"tcdate":1731928796678,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13179/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13179/Authors"],"forum":"zg3ec1TdAP","number":7,"license":"CC BY 4.0","cdate":1731928796678,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13179/-/Official_Comment"],"mdate":1732177288786,"domain":"ICLR.cc/2025/Conference","replyto":"0ZtAa1joXa","id":"QvNu4K9bCd","forumContent":{"TLDR":{"value":"An analysis of the impact of context length on foundation models trained on structured electronic health record (EHR) data for clinical prediction tasks."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["ehr","foundation model","long context","clinical prediction making","healthcare"]},"supplementary_material":{"value":"/attachment/b6ee9ac9232dd3da6857ced0d9ddb2aa470cc775.pdf"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Foundation Models (FMs) trained on Electronic Health Records (EHRs) have achieved state-of-the-art results on numerous clinical prediction tasks. However, prior EHR FMs typically have context windows of $<$1k tokens, which prevents them from modeling full patient EHRs which can exceed 10k's of events. For making clinical predictions, both model performance and robustness to the unique properties of EHR data are crucial. Recent advancements in subquadratic long-context architectures (e.g. Mamba) offer a promising solution. However, their application to EHR data has not been well-studied. We address this gap by presenting the first systematic evaluation of the effect of context length on modeling EHR data. We find that longer context models improve predictive performance -- our Mamba-based model surpasses the prior state-of-the-art on 9/14 tasks on the EHRSHOT prediction benchmark. Additionally, we measure robustness to three unique, previously underexplored properties of EHR data: (1) the prevalence of ``copy-forwarded\" diagnoses which create artificial token repetition in EHR sequences; (2) the irregular time intervals between EHR events which can lead to a wide range of timespans within a context window; and (3) the natural increase in disease complexity over time which makes later tokens in the EHR harder to predict than earlier ones. Stratifying our EHRSHOT results, we find that higher levels of each property correlate negatively with model performance (e.g., a 14% higher Brier loss between the least and most irregular patients), but that longer context models are more robust to more extreme levels of these properties. Our work highlights the potential for using long-context architectures to model EHR data, and offers a case study on how to identify and quantify new challenges in modeling sequential data motivated by domains outside of natural language. We release all of our model checkpoints and code."},"_bibtex":{"value":"@inproceedings{\nwornow2025context,\ntitle={Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on {EHR} Data},\nauthor={Michael Wornow and Suhana Bedi and Miguel Angel Fuentes Hernandez and Ethan Steinberg and Jason Alan Fries and Christopher Re and Sanmi Koyejo and Nigam Shah},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=zg3ec1TdAP}\n}"},"title":{"value":"Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data"},"pdf":{"value":"/pdf/b21eff6a613bf00fac9c545c0a6c7f2df1c9e5a0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wornow|context_clues_evaluating_long_context_models_for_clinical_prediction_tasks_on_ehr_data"},"authorids":{"value":["~Michael_Wornow1","~Suhana_Bedi1","~Miguel_Angel_Fuentes_Hernandez1","~Ethan_Steinberg1","~Jason_Alan_Fries1","~Christopher_Re1","~Sanmi_Koyejo1","~Nigam_Shah1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Michael Wornow","Suhana Bedi","Miguel Angel Fuentes Hernandez","Ethan Steinberg","Jason Alan Fries","Christopher Re","Sanmi Koyejo","Nigam Shah"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2407.05993v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"ji|selfprior_guided_mambaunet_networks_for_medical_image_superresolution"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zexin_Ji:","https://dblp.org/search/pid/api?q=author:Beiji_Zou_0001:","https://dblp.org/search/pid/api?q=author:Xiaoyan_Kui:","https://dblp.org/search/pid/api?q=author:Pierre_Vera:","~Su_Ruan1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2407.05993"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2407-05993,\n  publtype={informal},\n  author={Zexin Ji and Beiji Zou and Xiaoyan Kui and Pierre Vera and Su Ruan},\n  title={Self-Prior Guided Mamba-UNet Networks for Medical Image Super-Resolution},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2407.05993},\n  url={https://doi.org/10.48550/arXiv.2407.05993}\n}\n"},"abstract":{"value":"In this paper, we propose a self-prior guided Mamba-UNet network (SMamba-UNet) for medical image super-resolution. Existing methods are primarily based on convolutional neural networks (CNNs) or Transformers. CNNs-based methods fail to capture long-range dependencies, while Transformer-based approaches face heavy calculation challenges due to their quadratic computational complexity. Recently, State Space Models (SSMs) especially Mamba have emerged, capable of modeling long-range dependencies with linear computational complexity. Inspired by Mamba, our approach aims to learn the self-prior multi-scale contextual features under Mamba-UNet networks, which may help to super-resolve low-resolution medical images in an efficient way. Specifically, we obtain self-priors by perturbing the brightness inpainting of the input image during network training, which can learn detailed texture and brightness information that is beneficial for super-resolution. Furthermore, we combine Mamba with Unet network to mine global features at different levels. We also design an improved 2D-Selective-Scan (ISS2D) module to divide image features into different directional sequences to learn long-range dependencies in multiple directions, and adaptively fuse sequence information to enhance super-resolved feature representation. Both qualitative and quantitative experimental results demonstrate that our approach outperforms current state-of-the-art methods on two public medical datasets: the IXI and fastMRI."},"title":{"value":"Self-Prior Guided Mamba-UNet Networks for Medical Image Super-Resolution"},"authors":{"value":["Zexin Ji","Beiji Zou","Xiaoyan Kui","Pierre Vera","Su Ruan"]}},"tmdate":1731484853013,"pdate":1704067200000,"tcdate":1731481773237,"writers":["~"],"signatures":["~Su_Ruan1"],"forum":"pP0huxK3AY","license":"CC BY-SA 4.0","number":210729,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1731484853013,"domain":"DBLP.org","id":"pP0huxK3AY","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.12068v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"weng|simplified_mamba_with_disentangled_dependency_encoding_for_longterm_time_series_forecasting"},"authorids":{"value":["~Zixuan_Weng1","~Jindong_Han1","https://dblp.org/search/pid/api?q=author:Wenzhao_Jiang:","https://dblp.org/search/pid/api?q=author:Hao_Liu_0026:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.12068"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-12068,\n  publtype={informal},\n  author={Zixuan Weng and Jindong Han and Wenzhao Jiang and Hao Liu},\n  title={Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.12068},\n  url={https://doi.org/10.48550/arXiv.2408.12068}\n}\n"},"abstract":{"value":"Recently many deep learning models have been proposed for Long-term Time Series Forecasting (LTSF). Based on previous literature, we identify three critical patterns that can improve forecasting accuracy: the order and semantic dependencies in time dimension as well as cross-variate dependency. However, little effort has been made to simultaneously consider order and semantic dependencies when developing forecasting models. Moreover, existing approaches utilize cross-variate dependency by mixing information from different timestamps and variates, which may introduce irrelevant or harmful cross-variate information to the time dimension and largely hinder forecasting performance. To overcome these limitations, we investigate the potential of Mamba for LTSF and discover two key advantages benefiting forecasting: (i) the selection mechanism makes Mamba focus on or ignore specific inputs and learn semantic dependency easily, and (ii) Mamba preserves order dependency by processing sequences recursively. After that, we empirically find that the non-linear activation used in Mamba is unnecessary for semantically sparse time series data. Therefore, we further propose SAMBA, a Simplified Mamba with disentangled dependency encoding. Specifically, we first remove the non-linearities of Mamba to make it more suitable for LTSF. Furthermore, we propose a disentangled dependency encoding strategy to endow Mamba with cross-variate dependency modeling capabilities while reducing the interference between time and variate dimensions. Extensive experimental results on seven real-world datasets demonstrate the effectiveness of SAMBA over state-of-the-art forecasting models."},"title":{"value":"Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting"},"authors":{"value":["Zixuan Weng","Jindong Han","Wenzhao Jiang","Hao Liu"]}},"tmdate":1739183768160,"pdate":1704067200000,"tcdate":1727793387088,"writers":["~"],"signatures":["~Jindong_Han1"],"forum":"rsGa1Vq1Gm","license":"CC BY-SA 4.0","number":132489,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1739183768160,"domain":"DBLP.org","id":"rsGa1Vq1Gm","version":2},{"content":{"summary":{"value":"### **Post-rebuttal update**\nGiven the authors' additional experiments and changes-to-be-made to the manuscript, raising my original score from a 5 to a 6.\n\n### **Original review**\nThe authors tackle the problem of explainability in Mamba SSMs, which have been recently proposed and widely adapted.  Towards this end, they leverage layer-wise relevance propagation (LBP) and derive the necessary procedures/equations to enable LBP in Mamba models, similar to those previously proposed in Conservative Propagation paper for Transformers (Ali et al, 2022).  They benchmark their procedure on both Mamba language and vision models across a range of benchmarks and other widely used explainability methods to demonstrate the effectiveness of their approach."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"\"We use an instruction-finetuned Mamba-2.8B model\" <- In order to make the paper as self-contained as possible, could the authors summarize the fine-tuning recipe for this model within the supplementary (rather than forward referencing to the huggingface page)?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"**Originality** - Given the wide-spread adaptation of Mamba SSMs since their recent debut, explainability of these models is a very important topic.  While the specific detach approach to enable LRP in LLMs is not new, the adaptation to Mamba models is new and an excellent application of this methodology.\n\n**Quality** - The proposed approach is a good addition to the growing list of Mamba works, and will make a good tool for practitioners/researchers to explain the decisions of Mamba models going forward.  The extensive benchmarks and evaluation of both NLP and vision models and tasks are also solid contributions.  However, the evaluation of Mamba LLMs was somewhat limited (see weaknesses for further discussion).\n\n**Clarity** - For the most part, the paper is well written.  However, important details regarding evaluation are either missing or could be better highlighted (discussed in weaknesses).\n\n**Significance** - As previously mentioned, Mamba models continue to gain traction, particularly with the recent release of Mamba 2.  Thus, this work has significant opportunity for significant impact as the need to explain the decision making processes behind Mamba models grows."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"### Quality\nAs previously noted, there is significant room for improvement regarding the evaluation of Mamba LLMs.  In particular, only the smallest and second largest checkpoints are used for the majority of experiments (with the exception of the 2.8B checkpoint used in the needle in the haystack tests).  However, for a paper dedicated to enabling the capability of this line of LLMs, this seems insufficient; why are the 130M and 1.4B checkpoints used for the majority of tests?  In particular, the 130M model performs the worst among the suite of Mamba 1 models, but is used to demonstrate qualitative claims about the method (e.g., Figure 3, Figure 4, and reported runtimes).  Given the 2.8B is the most accurate and, thus, most desirable for practical use, this should ideally be evaluated along with the 130M (and 1.4B) models to demonstrate the efficacy of this approach at 2.8B Mamba-parameter size.\n\n### Clarity\nFor both the presented experiments and compared methods, several important details were not clear in the paper.  In particular, for Figure 6, how is position difference calculated?  E.g.:\n>We use the HotpotQA [70] subset from the LongBench dataset [11], designed to test long context understanding. After selecting all 127 instances, containing sequences up to 8192 tokens, we prompt the model to summarize the full paragraph by generating ten additional tokens  Fig. 6, shows the distribution of the positional difference between a relevant token and the currently generated token.\nHow are the relevant tokens determined?  Upon first pass, I wondered whether they were somehow derived from the\nsupporting fact labels in the HotpotQA dataset, but it seems to be calculated as the difference\tbetween the tokens generated position vs position in the input.  Can you clarify this, as well as how (exactly) the histogram in Figure 6 is calculated?  Also, the HotpotQA subset of LongBench contains 200 instances, yet the paper evaluates 127; what is the source of this discrepancy?\n\nMost importantly, it is not clear whether the author's are comparing to Attention Rollout, which the Ali et al 2024 Mamba explainability paper was based on, or Mamba-Attribution (i.e., the new mechanism derived for Mamba models in the Ali et al 2024 paper).  As this is the most relevant Mamba-specific method given the proposed method, this point requires clarification within the main text.\n\nOther portions which are unclear:\n- \"The results without fast CUDA kernels\" <- what are fast CUDA kernels in this context? The hardware-optimized selective SSM scan kernel included in Mamba (in contrast to the full PyTorch SSM kernel)?  Or are these kernels the authors are contributing to speed up their described method?\n- \"All models were trained for a maximum of 10 epochs, with an early stopping mechanism in place.\" <- What is the early stopping criterion?\n- Lines 192-193 should forward reference fine-tuning details are further described in \"C.1 Models and dataset.\"  Otherwise, it is easy to miss that small excerpt on line 193 which alludes to 130M and 1.4B being fine-tuned for the various tasks (which could lead to thinking pretrained Mamba models are being evaluated)."},"limitations":{"value":"Could the authors include foreseeable limitations of their work?  E.g., potentially large memory utilization to enable explainability in Mamba models?  Also, in the case that the compared method AttnRoll was not Mamba-Attribute (or not evaluated using the released code of the Ali et al 2024 paper), these would be important evaluation limitations to list."}},"nonreaders":[],"tmdate":1730879312874,"tcdate":1720726758285,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission9423/Reviewer_mtfa"],"signatures":["NeurIPS.cc/2024/Conference/Submission9423/Reviewer_mtfa"],"forum":"2n1Ysn1EDl","number":2,"license":"CC BY 4.0","cdate":1720726758285,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission9423/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879312874,"domain":"NeurIPS.cc/2024/Conference","replyto":"2n1Ysn1EDl","id":"VcbTTaFhjI","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Explainable AI","explainability","interpretability","state space models","Mamba","long-range dependency"]},"primary_area":{"value":"interpretability_and_explainability"},"abstract":{"value":"Recent sequence modeling approaches using selective state space sequence models, referred to as Mamba models, have seen a surge of interest. These models allow efficient processing of long sequences in linear time and are rapidly being adopted in a wide range of applications such as language modeling, demonstrating promising performance. To foster their reliable use in real-world scenarios, it is crucial to augment their transparency. Our work bridges this critical gap by bringing explainability, particularly Layer-wise Relevance Propagation (LRP), to the Mamba architecture. Guided by the axiom of relevance conservation, we identify specific components in the Mamba architecture, which cause unfaithful explanations. To remedy this issue, we propose MambaLRP, a novel algorithm within the LRP framework, which ensures a more stable and reliable relevance propagation through these components. Our proposed method is theoretically sound and excels in achieving state-of-the-art explanation performance across a diverse range of models and datasets. Moreover, MambaLRP facilitates a deeper inspection of Mamba architectures, uncovering various biases and evaluating their significance. It also enables the analysis of previous speculations regarding the long-range capabilities of Mamba models."},"_bibtex":{"value":"@inproceedings{\njafari2024mambalrp,\ntitle={Mamba{LRP}: Explaining Selective State Space Sequence Models},\nauthor={Farnoush Rezaei Jafari and Gr{\\'e}goire Montavon and Klaus Robert Muller and Oliver Eberle},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=2n1Ysn1EDl}\n}"},"title":{"value":"MambaLRP: Explaining Selective State Space Sequence Models"},"pdf":{"value":"/pdf/d253dfe5b75f15bb96b1033361699ed2984da98a.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"jafari|mambalrp_explaining_selective_state_space_sequence_models"},"authorids":{"value":["~Farnoush_Rezaei_Jafari1","~Grégoire_Montavon1","~Klaus_Robert_Muller1","~Oliver_Eberle1"]},"authors":{"value":["Farnoush Rezaei Jafari","Grégoire Montavon","Klaus Robert Muller","Oliver Eberle"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2505.14062v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"li|scaling_vision_mamba_across_resolutions_via_fractal_traversal"},"html":{"value":"https://doi.org/10.48550/arXiv.2505.14062"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2505-14062,\n  publtype={informal},\n  author={Bo Li and Haoke Xiao and Lv Tang},\n  title={Scaling Vision Mamba Across Resolutions via Fractal Traversal},\n  year={2025},\n  month={May},\n  cdate={1746057600000},\n  journal={CoRR},\n  volume={abs/2505.14062},\n  url={https://doi.org/10.48550/arXiv.2505.14062}\n}\n"},"abstract":{"value":"Vision Mamba has recently emerged as a promising alternative to Transformer-based architectures, offering linear complexity in sequence length while maintaining strong modeling capacity. However, its adaptation to visual inputs is hindered by challenges in 2D-to-1D patch serialization and weak scalability across input resolutions. Existing serialization strategies such as raster scanning disrupt local spatial continuity and limit the model's ability to generalize across scales. In this paper, we propose FractalMamba++, a robust vision backbone that leverages fractal-based patch serialization via Hilbert curves to preserve spatial locality and enable seamless resolution adaptability. To address long-range dependency fading in high-resolution inputs, we further introduce a Cross-State Routing (CSR) mechanism that enhances global context propagation through selective state reuse. Additionally, we propose a Positional-Relation Capture (PRC) module to recover local adjacency disrupted by curve inflection points. Extensive experiments across diverse downstream tasks, including image classification, semantic segmentation and object detection, demonstrate that FractalMamba++ consistently outperforms previous Mamba-based backbones, with particularly notable gains under high-resolution settings."},"title":{"value":"Scaling Vision Mamba Across Resolutions via Fractal Traversal"},"authors":{"value":[{"fullname":"Bo Li","username":""},{"fullname":"Haoke Xiao","username":""},{"fullname":"Lv Tang","username":"~Lv_Tang1"}]}},"tmdate":1777733499746,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2505-14062"],"tcdate":1777733485852,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Lv_Tang1"],"forum":"OkaXmsu6Cl","license":"CC BY-SA 4.0","number":2159,"cdate":1746057600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1777733499746,"domain":"OpenReview.net/Public_Article","id":"OkaXmsu6Cl","version":2},{"content":{"venue":{"value":"NeurIPS 2025 poster"},"TLDR":{"value":"We extend DeltaNet by using products of householders as state-transition matrices allowing us to trade-off expressivity and computational complexity."},"keywords":{"value":["linear rnn","linear attention","state-space models","state-tracking","negative eigenvalues","householder","formal languages","regular languages","stability","finite-state automata"]},"supplementary_material":{"value":"/attachment/55c21c531ca7877cd8019f50fda8b3c4e8ed6318.zip"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference. However, existing architectures face a fundamental trade-off between expressivity and efficiency, dictated by the structure of their state-transition matrices. Diagonal matrices, used in models such as Mamba, GLA, or mLSTM, yield fast runtime but have limited expressivity. To address this, recent architectures such as DeltaNet and RWKV-7 adopted a diagonal plus rank-1 structure, which allows simultaneous token and channel mixing, \nimproving associative recall and, as recently shown, state-tracking when\nallowing negative eigenvalues in the state-transition matrices.\nBuilding on the interpretation of DeltaNet's recurrence as performing one step of online gradient descent per token on an associative recall loss, we introduce DeltaProduct, which instead takes multiple ($n_h$) steps per token. This naturally leads to diagonal plus rank-$n_h$ state-transition matrices, formed as products of $n_h$ generalized Householder transformations, providing a tunable mechanism to balance expressivity and efficiency. \nWe provide a detailed theoretical characterization of the state-tracking capability of DeltaProduct in finite precision and how it improves by increasing $n_h$.\nOur extensive experiments demonstrate that DeltaProduct outperforms DeltaNet in both state-tracking and language modeling, while also showing significantly improved length extrapolation capabilities."},"_bibtex":{"value":"@inproceedings{\nsiems2025deltaproduct,\ntitle={DeltaProduct: Improving State-Tracking in Linear {RNN}s via Householder Products},\nauthor={Julien Siems and Timur Carstensen and Arber Zela and Frank Hutter and Massimiliano Pontil and Riccardo Grazzi},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=SoRiaijTGr}\n}"},"title":{"value":"DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products"},"pdf":{"value":"/pdf/4824138cbd313fd3b423880b5e08abe5b857516b.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"siems|deltaproduct_improving_statetracking_in_linear_rnns_via_householder_products"},"authorids":{"value":["~Julien_Siems1","~Timur_Carstensen1","~Arber_Zela1","~Frank_Hutter1","~Massimiliano_Pontil4","~Riccardo_Grazzi2"]},"authors":{"value":["Julien Siems","Timur Carstensen","Arber Zela","Frank Hutter","Massimiliano Pontil","Riccardo Grazzi"]}},"tmdate":1783627305362,"pdate":1758217348345,"tcdate":1746979095058,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission22323/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission22323/Authors"],"forum":"SoRiaijTGr","license":"CC BY 4.0","number":22323,"cdate":1746979095058,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission22323/-/Full_Submission","NeurIPS.cc/2025/Conference/Submission22323/-/Supplementary_Material","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission22323/-/Camera_Ready_Revision"],"mdate":1783627305362,"odate":1761705026315,"domain":"NeurIPS.cc/2025/Conference","id":"SoRiaijTGr","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2409.08513v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"wang|mambayoloworld_marrying_yoloworld_with_mamba_for_openvocabulary_detection"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Haoxuan_Wang:","~Qingdong_He1","https://dblp.org/search/pid/api?q=author:Jinlong_Peng:","https://dblp.org/search/pid/api?q=author:Hao_Yang:","~Mingmin_Chi2","https://dblp.org/search/pid/api?q=author:Yabiao_Wang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2409.08513"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2409-08513,\n  publtype={informal},\n  author={Haoxuan Wang and Qingdong He and Jinlong Peng and Hao Yang and Mingmin Chi and Yabiao Wang},\n  title={Mamba-YOLO-World: Marrying YOLO-World with Mamba for Open-Vocabulary Detection},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2409.08513},\n  url={https://doi.org/10.48550/arXiv.2409.08513}\n}\n"},"abstract":{"value":"Open-vocabulary detection (OVD) aims to detect objects beyond a predefined set of categories. As a pioneering model incorporating the YOLO series into OVD, YOLO-World is well-suited for scenarios prioritizing speed and efficiency. However, its performance is hindered by its neck feature fusion mechanism, which causes the quadratic complexity and the limited guided receptive fields. To address these limitations, we present Mamba-YOLO-World, a novel YOLO-based OVD model employing the proposed MambaFusion Path Aggregation Network (MambaFusion-PAN) as its neck architecture. Specifically, we introduce an innovative State Space Model-based feature fusion mechanism consisting of a Parallel-Guided Selective Scan algorithm and a Serial-Guided Selective Scan algorithm with linear complexity and globally guided receptive fields. It leverages multi-modal input sequences and mamba hidden states to guide the selective scanning process. Experiments demonstrate that our model outperforms the original YOLO-World on the COCO and LVIS benchmarks in both zero-shot and fine-tuning settings while maintaining comparable parameters and FLOPs. Additionally, it surpasses existing state-of-the-art OVD methods with fewer parameters and FLOPs."},"title":{"value":"Mamba-YOLO-World: Marrying YOLO-World with Mamba for Open-Vocabulary Detection"},"authors":{"value":["Haoxuan Wang","Qingdong He","Jinlong Peng","Hao Yang","Mingmin Chi","Yabiao Wang"]}},"tmdate":1733975610765,"pdate":1704067200000,"tcdate":1731464822668,"writers":["~"],"signatures":["~Mingmin_Chi2"],"forum":"mdrEnbTW00","license":"CC BY-SA 4.0","number":190133,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1733975610765,"domain":"DBLP.org","id":"mdrEnbTW00","version":2},{"content":{"summary":{"value":"This paper presents a hybrid architecture combining Transformer and Mamba modules to enhance inference speed and computational efficiency. The idea is novel and relevant, showing potential in bridging attention-based and state-space models. However, the paper lacks clarity on when and how each module is applied, as well as theoretical justification for their integration. The model design and experimental details are insufficiently explained, and the absence of thorough ablation and generalization studies limits the strength of the empirical claims. Overall, the contribution is promising but not yet fully convincing."},"soundness":{"value":3},"confidence":{"value":2},"questions":{"value":"1. How does the model determine when to use the Transformer module versus the Mamba module? Is there a data-driven or theoretical basis for this choice?\n\n2. Are the integration strategies between modules such as information flow, gating mechanisms, or parameter sharing fixed or adaptive during training?\n\n3. Could the authors provide ablation studies to quantify the contribution of each module and demonstrate the stability of the hybrid architecture across tasks?\n\n4. How does the proposed hybrid approach generalize to different sequence lengths and domains beyond the tested benchmarks?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The paper introduces a novel combination of Transformer and Mamba modules, aiming to exploit their complementary strengths in sequence modeling and efficiency.\n\nThe work addresses a relevant challenge in modern deep learning — improving inference speed and memory efficiency without significantly compromising accuracy.\n\nPreliminary results demonstrate that the proposed hybrid architecture can outperform standard Transformer baselines on certain tasks, suggesting its promise for long-sequence or resource-constrained applications."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper does not clearly explain the theoretical motivation for combining Transformer and Mamba modules, nor provide criteria for when each should be activated.\n\nKey implementation aspects such as module interaction, feature fusion, and parameter sharing are not well described, which limits reproducibility.\n\nThe evaluation is narrow, lacking ablation studies, robustness analysis, and tests on diverse datasets to support the claimed generalization and efficiency improvements."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926039983,"tcdate":1761134135482,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15809/Reviewer_Lhf2"],"signatures":["ICLR.cc/2026/Conference/Submission15809/Reviewer_Lhf2"],"forum":"1aAFc4OANr","number":1,"license":"CC BY 4.0","cdate":1761134135482,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15809/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926039983,"domain":"ICLR.cc/2026/Conference","replyto":"1aAFc4OANr","id":"Km0S4o4Znd","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Hybrid Architecture","Transformer","Mamba","Language Modeling","Efficiency"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Recent progress in large language models demonstrates that hybrid architectures--combining self-attention mechanisms with structured state space models like Mamba--can achieve a compelling balance between modeling quality and computational efficiency, particularly for long-context tasks. While these hybrid models show promising performance, systematic comparisons of hybridization strategies and analyses on the key factors behind their effectiveness have not been clearly shared to the community. In this work, we present a holistic evaluation of hybrid architectures based on inter-layer (sequential) or intra-layer (parallel) fusion. We evaluate these designs from a variety of perspectives: language modeling performance, long-context capabilities, scaling analysis, and training and inference efficiency. By investigating the core characteristics of their computational primitive, we identify the most critical elements for each hybridization strategy and further propose optimal design recipes for both hybrid models. Our comprehensive analysis provides practical guidance and valuable insights for developing hybrid language models, facilitating the optimization of architectural configurations."},"_bibtex":{"value":"@misc{\nbae2026hybrid,\ntitle={Hybrid Architectures for Language Models: Systematic Analysis and Design Insights},\nauthor={Sangmin Bae and Bilge Acun and Haroun Habeeb and Seungyeon Kim and Chien-Yu Lin and Liang Luo and Junjie Wang and Carole-Jean Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=1aAFc4OANr}\n}"},"title":{"value":"Hybrid Architectures for Language Models: Systematic Analysis and Design Insights"},"pdf":{"value":"/pdf/0e89b5aa27c2bea5f31c5069d50f1ff9ac97c543.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"bae|hybrid_architectures_for_language_models_systematic_analysis_and_design_insights"},"authorids":{"value":["~Sangmin_Bae1","~Bilge_Acun1","~Haroun_Habeeb2","~Seungyeon_Kim1","~Chien-Yu_Lin1","~Liang_Luo2","~Junjie_Wang17","~Carole-Jean_Wu2"]},"authors":{"value":["Sangmin Bae","Bilge Acun","Haroun Habeeb","Seungyeon Kim","Chien-Yu Lin","Liang Luo","Junjie Wang","Carole-Jean Wu"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2601.05527v1"},"venueid":{"value":"dblp.org/journals/CORR/2026"},"paperhash":{"value":"an|dema_dualpath_delayaware_mamba_for_efficient_multivariate_time_series_analysis"},"authorids":{"value":["~Rui_An2","~Haohao_Qu1","","",""]},"html":{"value":"https://doi.org/10.48550/arXiv.2601.05527"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2601-05527,\n  publtype={informal},\n  author={Rui An and Haohao Qu and Wenqi Fan and Xuequn Shang and Qing Li},\n  title={DeMa: Dual-Path Delay-Aware Mamba for Efficient Multivariate Time Series Analysis},\n  year={2026},\n  month={January},\n  cdate={1767225600000},\n  journal={CoRR},\n  volume={abs/2601.05527},\n  url={https://doi.org/10.48550/arXiv.2601.05527}\n}\n"},"abstract":{"value":"Accurate and efficient multivariate time series (MTS) analysis is increasingly critical for a wide range of intelligent applications. Within this realm, Transformers have emerged as the predominant architecture due to their strong ability to capture pairwise dependencies. However, Transformer-based models suffer from quadratic computational complexity and high memory overhead, limiting their scalability and practical deployment in long-term and large-scale MTS modeling. Recently, Mamba has emerged as a promising linear-time alternative with high expressiveness. Nevertheless, directly applying vanilla Mamba to MTS remains suboptimal due to three key limitations: (i) the lack of explicit cross-variate modeling, (ii) difficulty in disentangling the entangled intra-series temporal dynamics and inter-series interactions, and (iii) insufficient modeling of latent time-lag interaction effects. These issues constrain its effectiveness across diverse MTS tasks. To address these challenges, we propose DeMa, a dual-path delay-aware Mamba backbone. DeMa preserves Mamba's linear-complexity advantage while substantially improving its suitability for MTS settings. Specifically, DeMa introduces three key innovations: (i) it decomposes the MTS into intra-series temporal dynamics and inter-series interactions; (ii) it develops a temporal path with a Mamba-SSD module to capture long-range dynamics within each individual series, enabling series-independent, parallel computation; and (iii) it designs a variate path with a Mamba-DALA module that integrates delay-aware linear attention to model cross-variate dependencies. Extensive experiments on five representative tasks, long- and short-term forecasting, data imputation, anomaly detection, and series classification, demonstrate that DeMa achieves state-of-the-art performance while delivering remarkable computational efficiency."},"title":{"value":"DeMa: Dual-Path Delay-Aware Mamba for Efficient Multivariate Time Series Analysis"},"authors":{"value":["Rui An","Haohao Qu","Wenqi Fan","Xuequn Shang","Qing Li"]}},"tmdate":1784644326487,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2601-05527"],"tcdate":1772589721378,"writers":["~"],"signatures":["~Haohao_Qu1"],"forum":"vLU4nvb42G","license":"CC BY-SA 4.0","number":847277,"cdate":1767225600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784644326487,"domain":"DBLP.org","id":"vLU4nvb42G","version":2},{"content":{"TLDR":{"value":"This paper introduces a multi-resolution SSM framework that addresses these limitations by representing sequence dynamics across multiple resolutions."},"venue":{"value":"ICLR 2025 Workshop World Models"},"pdf":{"value":"/pdf/3ebb0658c6227cf5ad096bf47856778bb902d994.pdf"},"keywords":{"value":["State Space Models","Sequence Models"]},"venueid":{"value":"ICLR.cc/2025/Workshop/World_Models"},"paperhash":{"value":"karami|msssm_a_multiscale_state_space_model_for_enhanced_sequence_modeling"},"authorids":{"value":["~Mahdi_Karami2","~Ali_Behrouz1","~Peilin_Zhong1","~Razvan_Pascanu1","~Vahab_Mirrokni2"]},"abstract":{"value":"State Space Models (SSMs) have emerged as a promising alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurrences to integrate information over time, which enables for fast inference while still allowing the model to be parallelized during training and to control the stability of the recurrence. However, a consequence is that the effective memory of traditional SSMs is limited, requiring larger state sizes for improved recall. This paper introduces a multi-resolution SSM framework that addresses these limitations by representing sequence dynamics across multiple levels of detail. This approach captures both fine-grained, high-frequency patterns and coarse, low-frequency trends, hence effectively capturing historical patterns at multiple time scales. This decompositions allow the SSM to make better use of its memory. Our multi-resolution SSM demonstrates superior performance in various sequence modeling tasks, particularly in domains where multi-resolution patterns naturally occur, such as time series analysis and image processing."},"_bibtex":{"value":"@inproceedings{\nkarami2025msssm,\ntitle={{MS}-{SSM}: A Multi-Scale State Space Model for Enhanced Sequence Modeling},\nauthor={Mahdi Karami and Ali Behrouz and Peilin Zhong and Razvan Pascanu and Vahab Mirrokni},\nbooktitle={ICLR 2025 Workshop on World Models: Understanding, Modelling and Scaling},\nyear={2025},\nurl={https://openreview.net/forum?id=gfm4W1g7NH}\n}"},"title":{"value":"MS-SSM: A Multi-Scale State Space Model for Enhanced Sequence Modeling"},"authors":{"value":["Mahdi Karami","Ali Behrouz","Peilin Zhong","Razvan Pascanu","Vahab Mirrokni"]}},"tmdate":1744717513387,"pdate":1741261194836,"tcdate":1740632988570,"writers":["ICLR.cc/2025/Workshop/World_Models","ICLR.cc/2025/Workshop/World_Models/Submission84/Authors"],"signatures":["ICLR.cc/2025/Workshop/World_Models/Submission84/Authors"],"forum":"gfm4W1g7NH","license":"CC BY 4.0","number":84,"cdate":1740632988570,"readers":["everyone"],"invitations":["ICLR.cc/2025/Workshop/World_Models/-/Submission","ICLR.cc/2025/Workshop/World_Models/-/Post_Submission","ICLR.cc/2025/Workshop/World_Models/-/Edit"],"mdate":1744717513387,"odate":1744717513374,"domain":"ICLR.cc/2025/Workshop/World_Models","id":"gfm4W1g7NH","version":2},{"content":{"venue":{"value":"TIME 2021"},"pdf":{"value":"https://drops.dagstuhl.de/storage/00lipics/lipics-vol206-time2021/LIPIcs.TIME.2021.2/LIPIcs.TIME.2021.2.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2021"},"paperhash":{"value":"pedersen|extremescale_modelbased_time_series_management_with_modelardb_invited_talk"},"authorids":{"value":["~Torben_Bach_Pedersen1"]},"html":{"value":"https://doi.org/10.4230/LIPIcs.TIME.2021.2"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/Pedersen21,\n  author={Torben Bach Pedersen},\n  title={Extreme-Scale Model-Based Time Series Management with ModelarDB (Invited Talk)},\n  year={2021},\n  cdate={1609459200000},\n  pages={2:1-2:2},\n  url={https://doi.org/10.4230/LIPIcs.TIME.2021.2},\n  booktitle={TIME},\n  crossref={conf/time/2021}\n}\n"},"abstract":{"value":"To monitor critical industrial devices such as wind turbines, high quality sensors sampled at a high frequency are increasingly used. Current technology does not handle these extreme-scale time series well [Søren Kejser Jensen et al., 2017], so only simple aggregates are traditionally stored, removing outliers and fluctuations that could indicate problems. As a remedy, we present a model-based approach for managing extreme-scale time series that approximates the time series values using mathematical functions (models) and stores only model coefficients rather than data values. Compression is done both for individual time series and for correlated groups of time series. The keynote will present concepts, techniques, and algorithms from model-based time series management and our implementation of these in the open source Time Series Management System (TSMS) ModelarDB[Søren Kejser Jensen et al., 2018; Søren Kejser Jensen et al., 2019; Søren Kejser Jensen et al., 2021] . Furthermore, it will present our experimental evaluation of ModelarDB on extreme-scale real-world time series, which shows that that compared to widely used Big Data formats, ModelarDB provides up to 14× faster ingestion due to high compression, 113× better compression due to its adaptability, 573× faster aggregatation by using models, and close to linear scale-out scalability. ModelarDB is being commercialized by the spin-out company ModelarData."},"title":{"value":"Extreme-Scale Model-Based Time Series Management with ModelarDB (Invited Talk)"},"authors":{"value":["Torben Bach Pedersen"]}},"tmdate":1769586939292,"pdate":1640908800000,"externalIds":["dblp:conf/time/Pedersen21"],"tcdate":1769586899952,"writers":["~"],"signatures":["~Torben_Bach_Pedersen1"],"forum":"fSaT9Tz85v","license":"CC BY-SA 4.0","number":810141,"cdate":1609459200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769586939292,"domain":"DBLP.org","id":"fSaT9Tz85v","version":2},{"content":{"venue":{"value":"TIME 2022"},"pdf":{"value":"https://drops.dagstuhl.de/storage/00lipics/lipics-vol247-time2022/LIPIcs.TIME.2022.15/LIPIcs.TIME.2022.15.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2022"},"paperhash":{"value":"gutierrez|giving_instructions_in_linear_temporal_logic"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Julian_Gutierrez_0001:","~Sarit_Kraus1","~Giuseppe_Perelli2","~Michael_J._Wooldridge1"]},"html":{"value":"https://doi.org/10.4230/LIPIcs.TIME.2022.15"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/0001KPW22,\n  author={Julian Gutierrez and Sarit Kraus and Giuseppe Perelli and Michael J. Wooldridge},\n  title={Giving Instructions in Linear Temporal Logic},\n  year={2022},\n  cdate={1640995200000},\n  pages={15:1-15:14},\n  url={https://doi.org/10.4230/LIPIcs.TIME.2022.15},\n  booktitle={TIME},\n  crossref={conf/time/2022}\n}\n"},"abstract":{"value":"Our aim is to develop a formal semantics for giving instructions to taskable agents, to investigate the complexity of decision problems relating to these semantics, and to explore the issues that these semantics raise. In the setting we consider, agents are given instructions in the form of Linear Temporal Logic (LTL) formulae; the intuitive interpretation of such an instruction is that the agent should act in such a way as to ensure the formula is satisfied. At the same time, agents are assumed to have inviolable and immutable background safety requirements, also specified as LTL formulae. Finally, the actions performed by an agent are assumed to have costs, and agents must act within a limited budget. For this setting, we present a range of interpretations of an instruction to achieve an LTL task Υ, intuitively ranging from \"try to do this but only if you can do so with everything else remaining unchanged\" up to \"drop everything and get this done.\" For each case we present a formal pre-/post-condition semantics, and investigate the computational issues that they raise."},"title":{"value":"Giving Instructions in Linear Temporal Logic"},"authors":{"value":["Julian Gutierrez","Sarit Kraus","Giuseppe Perelli","Michael J. Wooldridge"]}},"tmdate":1747663025866,"pdate":1640995200000,"tcdate":1718688738801,"writers":["~"],"signatures":["~Giuseppe_Perelli2"],"forum":"vDSAjfqkBf","license":"CC BY-SA 4.0","number":33518,"cdate":1640995200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747663025866,"domain":"DBLP.org","id":"vDSAjfqkBf","version":2},{"content":{"summary":{"value":"Authors propose to adopt so-called reasoning module to the realtime sequential reasoning task, in which the agent is required to engage in reasoning and inference at the same time as in-taking the realtime observation sequence. Toward this goal, the authors incorporate the recurrent module into the coupled system of fast and slow processes, in which the module reasons at the fast time scale by sequentially integrating into its state the sensory-observation that occurs at the slow time scale."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1) In the section on reinforcement learning, the authors write that their model is trained end-to-end using Proximal Policy Optimization. Why was this choice of RL method made?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- Authors show their approach in reinforcement learning tasks and synthetic tasks, and show that it performs competitively against other well-known models for sequential processing, such as LSTM, Mamba, and TransformerXL.\n\n- Authors demonstrate that their strategy enables effective test-time scaling of the recurrent module to the time scale of observation, so that an agent makes better decisions by observing more at the test time; That is, the longer the agent is immersed in the given environment, the deeper the reasoning and, hence, the decision making.\n\n- With this test-time scaling, the proposed approach leads to improved decision making on more complex tasks, surpassing LSTM, Mamba, and TransformerXL on Maze and MiniGrid tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1) The authors use the term \"reasoning,\" but in the result tables and visualizations of the solved tasks, it appears that they simply generate a sequence of output actions.\n\n2) The authors examined the effectiveness of their approach only on relatively simple maze-solving and MiniGrid tasks; they did not conduct experiments with more complex robotic simulations. The practical usefulness of the developed approach is questionable.\n\n3) Figure 1 is not informative enough. What do the captions in the figure mean? How do the reasoning module and memory module interact?\n\n4) The comparison with basic action generation methods is incomplete; for example, the recent RATE method [1] or the classic Decision Transformer [2] are not mentioned.\n\n[1] Cherepanov, E., Staroverov, A., Yudin, D., Kovalev, A. K., & Panov, A. I. (2023). Recurrent action transformer with memory. arXiv preprint arXiv:2306.09459.\n\n[2] Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., ... & Mordatch, I. (2021). Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 15084-15097.\n\n5) The article does not provide information on the inference performance of the developed model, which is critical for fast simulation.\n\n6) The article's formatting should be improved; punctuation marks are missing at the end of the formulas."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922616368,"tcdate":1761996428663,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11517/Reviewer_YVJH"],"signatures":["ICLR.cc/2026/Conference/Submission11517/Reviewer_YVJH"],"forum":"BxWhmN3iSt","number":5,"license":"CC BY 4.0","cdate":1761996428663,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11517/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922616368,"domain":"ICLR.cc/2026/Conference","replyto":"BxWhmN3iSt","id":"fdbMHvGYEq","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Recurrent Model","Sequencial Reasoning","Test time scaling"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"We propose a recurrent architecture designed to extend test-time scaling capabilities to sequential input streams. By interleaving fast, iterative reasoning loops between slow observation updates, our method facilitates dynamic compression of latent representations, where internal states self-organize into stable clusters that persist and evolve alongside the input. This mechanism allows the model to maintain coherent representations over long horizons, significantly improving out-of-distribution generalization in reinforcement learning and algorithmic tasks compared to standard sequential baselines such as LSTM, state space models, and Transformer variants.\n\nCode: https://anonymous.4open.science/r/fastslow-81DB"},"_bibtex":{"value":"@misc{\ntakashiro2026recurrent,\ntitle={Recurrent model for Sequential reasoning},\nauthor={Shota Takashiro and Masanori Koyama and Kohei Hayashi and Takeru Miyato and Yusuke Iwasawa and Yutaka Matsuo},\nyear={2026},\nurl={https://openreview.net/forum?id=BxWhmN3iSt}\n}"},"title":{"value":"Recurrent model for Sequential reasoning"},"pdf":{"value":"/pdf/0f1f68af19133b11c727566e0ab23adfcfa5cbbc.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"takashiro|recurrent_model_for_sequential_reasoning"},"authorids":{"value":["~Shota_Takashiro1","~Masanori_Koyama1","~Kohei_Hayashi1","~Takeru_Miyato1","~Yusuke_Iwasawa1","~Yutaka_Matsuo1"]},"authors":{"value":["Shota Takashiro","Masanori Koyama","Kohei Hayashi","Takeru Miyato","Yusuke Iwasawa","Yutaka Matsuo"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.11278v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"wu|umambatsf_a_ushaped_multiscale_longterm_time_series_forecasting_method_using_mamba"},"authorids":{"value":["~Li_Wu1","https://dblp.org/search/pid/api?q=author:Wenbin_Pei:","https://dblp.org/search/pid/api?q=author:Jiulong_Jiao:","https://dblp.org/search/pid/api?q=author:Qiang_Zhang_0008:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.11278"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-11278,\n  publtype={informal},\n  author={Li Wu and Wenbin Pei and Jiulong Jiao and Qiang Zhang},\n  title={UmambaTSF: A U-shaped Multi-Scale Long-Term Time Series Forecasting Method Using Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.11278},\n  url={https://doi.org/10.48550/arXiv.2410.11278}\n}\n"},"abstract":{"value":"Multivariate Time series forecasting is crucial in domains such as transportation, meteorology, and finance, especially for predicting extreme weather events. State-of-the-art methods predominantly rely on Transformer architectures, which utilize attention mechanisms to capture temporal dependencies. However, these methods are hindered by quadratic time complexity, limiting the model's scalability with respect to input sequence length. This significantly restricts their practicality in the real world. Mamba, based on state space models (SSM), provides a solution with linear time complexity, increasing the potential for efficient forecasting of sequential data. In this study, we propose UmambaTSF, a novel long-term time series forecasting framework that integrates multi-scale feature extraction capabilities of U-shaped encoder-decoder multilayer perceptrons (MLP) with Mamba's long sequence representation. To improve performance and efficiency, the Mamba blocks introduced in the framework adopt a refined residual structure and adaptable design, enabling the capture of unique temporal signals and flexible channel processing. In the experiments, UmambaTSF achieves state-of-the-art performance and excellent generality on widely used benchmark datasets while maintaining linear time complexity and low memory consumption."},"title":{"value":"UmambaTSF: A U-shaped Multi-Scale Long-Term Time Series Forecasting Method Using Mamba"},"authors":{"value":["Li Wu","Wenbin Pei","Jiulong Jiao","Qiang Zhang"]}},"tmdate":1738895531725,"pdate":1704067200000,"tcdate":1738895527310,"writers":["~"],"signatures":["~Li_Wu1"],"forum":"6DOidnJwzl","license":"CC BY-SA 4.0","number":310512,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1738895531725,"domain":"DBLP.org","id":"6DOidnJwzl","version":2},{"content":{"summary":{"value":"This work proposes a new architecture composed of two novel primitives, namely a tokenization-free embedding method and a sequence-mixer, as an alternative to Transformer and softmax attention. Effectively, this sequence mixer models interactions between tokens based primarily on their relative distance in the sequence, as opposed to modeling token embeddings’ similarity in softmax attention. Surprisingly, this method works on par or better than Transformer in small-scale (around 110M parameters) bidirectional language model validation on GLUE benchmark.\n\nThe ideas in the paper are interesting but they require more rigorous and detailed research, presentation and validation (see weaknesses). If the problems, especially with insufficient presentation and unfounded claims are resolved, I'm willing to reevaluate the work."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. What is the sequence length you used during DIFT-BERT pre-training and what is the total number of tokens seen by the model? How does it compare to the other model in Table 3?\n\n2. What is the specific formula for $f_{\\phi}$ and how does it compare to the Transformer’s FFN (Feed Forward Network). You mention that $f_{\\phi}$ is an instance of MLP, but Linear layers are functions of a single vector variable and equation 9 on line 669 takes two inputs.\n\n3. More generally, could you rigorously and unambiguously specify the whole new architecture as a mathematical model, consisting of formulas with specified sizes and dimensions for parameters and inputs?\n\n4. Given model dimension d, the Transformer layer has $12d^2$ parameters, $4d^2$ for attention weights (Q, K, V, O), and $8d^2$ for FFN sub-network. It is stated in the paper that the number of parameters and layers of the DIFT model is the same as in the BERT-base model. Than what is the composition of each layer in terms of parameters and their sizes?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"* Both architectural elements introduced are novel and interesting concepts. From my understanding, the new sequence-mixing mechanism deviates far enough from standard attention and is more light-weight.\n\n* The new architecture is significantly faster than parameters-matched Transformer despite having the same asymptotic run-time complexity.\n\n* The interactions between tokens in the sequence-mixing mechanism of the DIFT are dependent on the distance between their positions and a representation of **just one of the tokens**. In the baseline BERT Transformer, interactions depend on the similarity of high-dimensional representations of **both** tokens, which in turn already have positional information incorporated. From my point of view, the Transformer interaction modeling paradigm is more powerful. Yet it is even more remarkable and even surprising that this model shows similar modeling performance to parameters-matched BERT on the established GLUE benchmark.\n\n* The proposed tokenizer-free method of creating token embeddings seems to benefit performance on several tasks, such as XNLI."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Insufficient descriptions**\n\nDIFT algorithm is not fully specified. For example, neither lines 196-199, nor 668-670 establish a precise mathematical formula for $f_{\\phi}$. Likewise, there are no formulas for $\\alpha_i$ and $\\sigma_i$ params. It doesn’t help that there’s no model code listing in the Appendix or accompanying code archive in the submission.  \n\nThe paper positions DIFT against Transformer-based models, conceptually and empirically. But there is no objective comparison of the method with self-attention Transformer architecture that would highlight their similarities and differences. Ideally, the two architectures could be described side-by-side by their respective sets of equations, using compatible notation.\n\n\nThe exposition of the novel embeddings formation method and its distinctions from other methods is also rudimentary. In particular, lines 160-167 don’t offer a rigorous description of how word boundaries are formed from byte sequences and what are the differences from established methods like BPE or WordPiece. Moreover, the distinctions and advantages of the method in relation to other tokenization-free models, such as recent ByteLatentTransformer [6] and H-Net [7], are not clearly stated.\n\nEquation 1 doesn’t explain why the embedding method is called “hierarchical aggregation” – it just calculates mean over representations of individual bytes in a word. \nExperimentation.\n\nMany hyper-parameters are not specified. Specifically, sequence length, learning rate schedule, optimizer and its parameters, numerical precision, software frameworks used (e.g., PyTorch or JAX, HuggingFace, Lightning, DeepSpeed, etc.) and optimizations (custom CUDA or Triton kernels, CUDA Graphs, compilation, etc.). The lack of hyper-params reporting, especially sequence length or total number of tokens for pre-training, might cast doubts about the fairness of comparisons in Table 3, because the other models were pre-trained on about 100B tokens each.\n\n**Experimentation**\n\n\nDIFT combines two distinct architectural innovations – initial embedding creation and sequence-mixing mechanism. However, it is not clear to what extent each of them contributes individually to performance and efficiency of the model. Ablations where only embeddings and only self-attention are replaced in Transformer model by novel modules would help determine their individual effectiveness.\n\nThe scope of empirical validation is limited and could be expanded. Currently, the main result is pre-training of an encoder BERT-like model and its fine-tuning on GLUE. However, GLUE benchmark tests performance on short sequences only ($\\leq 256$ with majority of samples having length $\\leq 128$, [8]). Since DIFT has natural locality bias (line 235), it remains unclear whether the performance translates to longer sequences. At least results on perplexity on longer contexts could be provided and compared to other models.\n\nI'd recommend to validate the model on some of these additional modalities: autoregressive language modeling, image classification, speech recognition, long-range dependencies modeling and synthetic benchmarks. The appeal of byte-level tokenizer-free models lies in their ability to process heterogenous data (e.g., images and text simultaneously), but now all but text remains unexplored. \n\nGood examples of comprehensive experimentations with a limited compute budget are presented in papers [9-10], and FNet which is cited in the paper.\n\n**The claims about self-attention’s lack of theoretical foundation and interpretability are unfounded.**\n\nThe recurring claim throughout the paper is that different self-attention mechanisms are “opaque”, uninterpretable, and not theory-grounded. I beg to differ in the matters of standard attention’s interpretability and groundedness. \n\nThere is a vast literature on the interpretability of self-attention. I would recommend to review that literature before making such claims. A good starting point is papers [1-2], and you can proceed with subsequent papers which cite [1-2].\n\nI’d like to add that attention, either softmax or linear one, has a very simple interpretation – for any token (or “word” in the terminology of this work), attention scores show how much each other token in the sequence should influence the next-layer representation of that token. In other words, attention scores show *importances* of tokens, given the similarity of their representations in a high-dimensional space (dot product of **q** and **k**).\n\nRegarding theoretical foundation, one could argue that attention draws from psychology or the science of how the human brain works – see [4, 5] for example. \n\n“Unlike self-attention’s learned attention patterns, our field interactions are explicitly spatial and interpretable through field visualization,” (lines 201-202) – there’s no visualizations or any other corroboration of this claim present in the paper. On the contrary, attention mechanisms are easily visualised as heat maps (see, e.g. Figure 1 in [2]) or as graphs [3].\n\n\nReferences:\n\n1. Clark, Kevin et al. “What Does BERT Look at? An Analysis of BERT’s Attention.” BlackboxNLP@ACL (2019).\n\n2. Rogers, Anna et al. “A Primer in BERTology: What We Know About How BERT Works.” Transactions of the Association for Computational Linguistics 8 (2020): 842-866.\n\n3. ​​Vig, Jesse. “A Multiscale Visualization of Attention in the Transformer Model.” ACL (2019)\n\n4. Zhao, Minglu et al. “From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures.” ArXiv abs/2407.01548 (2024)\n\n5. Eberle, Oliver et al. “Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?” ACL (2022)\n\n6. Pagnoni, Artidoro et al. “Byte Latent Transformer: Patches Scale Better Than Tokens.” ArXiv abs/2412.09871 (2024)\n\n7. Hwang, Sukjun et al. “Dynamic Chunking for End-to-End Hierarchical Sequence Modeling.” ArXiv abs/2507.07955 (2025)\n\n8. Portes, Jacob et al., \"MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining.\" NeurIPS 2023\n\n9. Qin, Zhen et al. \"The Devil in Linear Transformer.\" EMNLP 2022\n\n10. Qin, Zhen et al. \"Toeplitz Neural Network for Sequence Modeling.\" ICLR 2023"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762934055989,"tcdate":1761779399104,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20670/Reviewer_3TFN"],"signatures":["ICLR.cc/2026/Conference/Submission20670/Reviewer_3TFN"],"forum":"3flsDiwwUs","number":2,"license":"CC BY 4.0","cdate":1761779399104,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20670/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762934055989,"domain":"ICLR.cc/2026/Conference","replyto":"3flsDiwwUs","id":"K1j4uiB94q","forumContent":{"TLDR":{"value":"We introduce DIFT, a novel tokenizer-free language architecture that uses hierarchical byte aggregation and continuous interaction fields to achieve competitive performance with BERT on GLUE."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["language modeling","transformer architectures","tokenizer-free models","continuous representations","self-attention alternatives","hybrid models","byte-level processing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Standard language models are limited by their reliance on subword tokenization, which introduces cross-lingual disparities and struggles with morphologically rich languages and out-of-domain text. Additionally, the opaque, all-to-all self-attention mechanisms employed by these models hinder their interpretability. In this work, we propose a novel theoretical framework that models language as a \\textbf{hybrid discrete-continuous system}, addressing both of these challenges. We introduce the \\textbf{Dynamic Interaction Field Transformer (DIFT)}, the first tokenizer-free transformer architecture that achieves word-level computational efficiency and competitive performance on standard benchmarks. DIFT operates directly on raw bytes, utilizing \\textbf{hierarchical aggregation} to learn word representations from scratch. To model context, DIFT replaces traditional self-attention with an interpretable \\textbf{continuous interaction field}, where concepts influence each other through proximity in a shared semantic space. We demonstrate that DIFT, when trained from scratch, achieves competitive performance with BERT-base on GLUE. Its tokenizer-free design also enhances robustness and enables superior zero-shot multilingual performance, effectively overcoming the core limitations of subword-based models. DIFT offers a new direction for more robust, interpretable, and theoretically grounded language architectures."},"_bibtex":{"value":"@misc{\nkhalil2025the,\ntitle={The Dynamic Interaction Field Transformer: A Universal, Tokenizer-Free Language Architecture},\nauthor={Khurram Khalil and Ripan Kumar Kundu and Khaza Anuarul Hoque},\nyear={2025},\nurl={https://openreview.net/forum?id=3flsDiwwUs}\n}"},"title":{"value":"The Dynamic Interaction Field Transformer: A Universal, Tokenizer-Free Language Architecture"},"pdf":{"value":"/pdf/648a00895ada970253fbd8f8a2fb477f2f47ce25.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"khalil|the_dynamic_interaction_field_transformer_a_universal_tokenizerfree_language_architecture"},"authorids":{"value":["~Khurram_Khalil1","~Ripan_Kumar_Kundu1","~Khaza_Anuarul_Hoque1"]},"authors":{"value":["Khurram Khalil","Ripan Kumar Kundu","Khaza Anuarul Hoque"]}},"version":2},{"content":{"TLDR":{"value":"DeltaProduct extends DeltaNet by taking multiple gradient steps per token, yielding diagonal plus rank-$n_h$ transitions that enhance expressivity for language modeling, state-tracking, and length extrapolation, backed by stronger theory."},"venue":{"value":"ICLR 2025 FM-Wild Workshop"},"pdf":{"value":"/pdf/ce6100479b67b0e8396cb400671d8b5353d4fdc6.pdf"},"keywords":{"value":["State Tracking","Linear RNN","Linear Attention","GLA","DeltaNet","Formal Languages","Group Theory","Length Extrapolation","Householder","Permutations"]},"venueid":{"value":"ICLR.cc/2025/Workshop/FM-Wild"},"paperhash":{"value":"siems|deltaproduct_improving_statetracking_in_linear_rnns_via_householder_products"},"authorids":{"value":["~Julien_Siems1","~Timur_Carstensen1","~Arber_Zela1","~Frank_Hutter1","~Massimiliano_Pontil4","~Riccardo_Grazzi2"]},"abstract":{"value":"Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference. However, existing architectures face a fundamental trade-off between expressivity and efficiency, dictated by the structure of their state-transition matrices. While diagonal matrices used in architectures like Mamba, GLA, or mLSTM yield fast runtime, they suffer from severely limited expressivity. To address this, recent architectures such as (Gated) DeltaNet and RWKV-7 adopted a diagonal plus rank-1 structure, allowing simultaneous token-channel mixing, which overcomes some expressivity limitations with only a slight decrease in training efficiency. Building on the interpretation of DeltaNet's recurrence as performing one step of online gradient descent per token on an associative recall loss, we introduce DeltaProduct, which instead takes multiple ($n_h$) steps per token. This naturally leads to diagonal plus rank-$n_h$ state-transition matrices, formed as products of $n_h$ generalized Householder transformations, providing a tunable mechanism to balance expressivity and efficiency and a stable recurrence. Through extensive experiments, we demonstrate that DeltaProduct achieves superior state-tracking and language modeling capabilities while exhibiting significantly improved length extrapolation compared to DeltaNet. Additionally, we also strengthen the theoretical foundation of DeltaNet by proving that it can solve dihedral group word problems in just two layers."},"_bibtex":{"value":"@inproceedings{\nsiems2025deltaproduct,\ntitle={DeltaProduct: Increasing the Expressivity of DeltaNet Through Products of Householders},\nauthor={Julien Siems and Timur Carstensen and Arber Zela and Frank Hutter and Massimiliano Pontil and Riccardo Grazzi},\nbooktitle={ICLR 2025 Workshop on Foundation Models in the Wild},\nyear={2025},\nurl={https://openreview.net/forum?id=nvb60szj5C}\n}"},"title":{"value":"DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products"},"authors":{"value":["Julien Siems","Timur Carstensen","Arber Zela","Frank Hutter","Massimiliano Pontil","Riccardo Grazzi"]}},"tmdate":1744032788362,"pdate":1741257920846,"tcdate":1739268285787,"writers":["ICLR.cc/2025/Workshop/FM-Wild","ICLR.cc/2025/Workshop/FM-Wild/Submission123/Authors"],"signatures":["ICLR.cc/2025/Workshop/FM-Wild/Submission123/Authors"],"forum":"nvb60szj5C","license":"CC BY 4.0","number":123,"cdate":1739268285787,"readers":["everyone"],"invitations":["ICLR.cc/2025/Workshop/FM-Wild/-/Submission","ICLR.cc/2025/Workshop/FM-Wild/-/Post_Submission","ICLR.cc/2025/Workshop/FM-Wild/-/Edit","ICLR.cc/2025/Workshop/FM-Wild/Submission123/-/Camera-Ready_Version"],"mdate":1744032788362,"odate":1741257920846,"domain":"ICLR.cc/2025/Workshop/FM-Wild","id":"nvb60szj5C","version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Diffusion language models","State space models","Mamba"]},"_bibtex":{"value":"@inproceedings{\nsingh2026diffumamba,\ntitle={DiffuMamba: High-Throughput Diffusion {LM}s with Mamba Backbone},\nauthor={Vaibhav Singh and Oleksiy Ostapenko and Pierre-Andre Noel and Eugene Belilovsky and Torsten Scholak},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=hLizFuTBsA}\n}"},"title":{"value":"DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone"},"paperhash":{"value":"singh|diffumamba_highthroughput_diffusion_lms_with_mamba_backbone"},"originally_submitted_PDF":{"value":"/pdf/b1cc609bb7116035e8f92ea140a2dc75bbb8ad61.pdf"},"TLDR":{"value":"Diffusion language model with mamba backbone for faster inference at long sequence lengths."},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2× and 4.3× higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants, combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems."},"pdf":{"value":"/pdf/2a0726f8dba38b2e0a314e73cb5a94e4bfeaad5f.pdf"},"lay_summary":{"value":"Today's AI chatbots write one word at a time, like reading a book aloud,  fast for short notes but slow for long ones. Diffusion language models instead generate whole paragraphs in parallel by polishing gibberish into text, like a sculptor at work. But each polishing pass uses attention, which compares every word to every other word, making the cost explode at long lengths. We replaced attention with Mamba, an engine that reads text in a single sweep instead of cross-comparing every pair of words. Our two models: DiffuMamba and the hybrid DiffuMamba-H plug this engine into the same diffusion recipe, trained up to 1.3B parameters. On standard benchmarks, they match or beat the attention baseline, generate text up to 8.2× faster on long inputs, and stay stable at lengths where the attention baseline collapses."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Vaibhav_Singh1","~Oleksiy_Ostapenko1","~Pierre-Andre_Noel1","~Eugene_Belilovsky1","~Torsten_Scholak1"]},"authors":{"value":["Vaibhav Singh","Oleksiy Ostapenko","Pierre-Andre Noel","Eugene Belilovsky","Torsten Scholak"]}},"tmdate":1790067370649,"pdate":1777576518607,"tcdate":1769099802534,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission16782/Authors"],"signatures":["ICML.cc/2026/Conference/Submission16782/Authors"],"forum":"hLizFuTBsA","license":"CC BY 4.0","number":16782,"cdate":1769099802534,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/-/PC_Revision","ICML.cc/2026/Conference/Submission16782/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission16782/-/Camera_Ready_Revision"],"mdate":1790067370649,"odate":1782341963902,"domain":"ICML.cc/2026/Conference","id":"hLizFuTBsA","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"Diffusion language model with mamba backbone for faster inference at long sequence lengths."},"venue":{"value":"ICLR 2026 Workshop MM Intelligence Poster"},"keywords":{"value":["Diffusion language models","State space models","Mamba"]},"abstract":{"value":"Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2× and 4.3× higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants, combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems."},"title":{"value":"DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone"},"anonymization":{"value":"This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs."},"pdf":{"value":"/pdf/aa81eef8944d43b8edf40eef885b40c95b8903b9.pdf"},"venueid":{"value":"ICLR.cc/2026/Workshop/MM_Intelligence"},"paperhash":{"value":"singh|diffumamba_highthroughput_diffusion_lms_with_mamba_backbone"},"authorids":{"value":["~Vaibhav_Singh1","~Oleksiy_Ostapenko1","~Pierre-Andre_Noel1","~Eugene_Belilovsky1","~Torsten_Scholak1"]},"track":{"value":"long paper (up to 8 pages)"},"authors":{"value":["Vaibhav Singh","Oleksiy Ostapenko","Pierre-Andre Noel","Eugene Belilovsky","Torsten Scholak"]}},"tmdate":1773250848289,"pdate":1772483087567,"tcdate":1770053946468,"writers":["ICLR.cc/2026/Workshop/MM_Intelligence","ICLR.cc/2026/Workshop/MM_Intelligence/Submission25/Authors"],"signatures":["ICLR.cc/2026/Workshop/MM_Intelligence/Submission25/Authors"],"forum":"4Q6TiBFiXy","license":"CC BY 4.0","number":25,"cdate":1770053946468,"readers":["everyone"],"invitations":["ICLR.cc/2026/Workshop/MM_Intelligence/-/Submission","ICLR.cc/2026/Workshop/MM_Intelligence/-/Submission_Change_Before_Reviewing","ICLR.cc/2026/Workshop/MM_Intelligence/-/Submission_Release","ICLR.cc/2026/Workshop/MM_Intelligence/Submission25/-/Camera_Ready_Revision"],"mdate":1773250848289,"odate":1772483087567,"domain":"ICLR.cc/2026/Workshop/MM_Intelligence","id":"4Q6TiBFiXy","version":2},{"content":{"summary":{"value":"This paper offers a new weight-space learning technique that iterates weights of a feedforward neural network in a linear state space, enabling in-context learning. The authors evaluate WARP across diverse tasks including image completion, time series forecasting and classification, and dynamical system reconstruction, demonstrating competitive or superior performance compared to standard RNNs, state-space models (S4, Mamba), and Transformers. Notably, a physics-informed variant achieves order-of-magnitude improvements on physical system reconstruction benchmarks."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"“Rather than relying on direct inputs, we draw inspiration from the human brain and compute signal differences to drive such recurrences.” - from where is the inspiration drawn? Did you try ablating this, i.e. just using x_t as the input to B? Similarily, did you try ablating the random noise applied to observations in AR mode? i.e. p_forcing = 0 (or 1, whatever means noise is never added)\n\nWhat is the operational difference between gradient-free adaptation and in-context learning? The definitions provided seem to be nested on page 2 (i.e. in-context learning implies gradient-free adaptation).\n\n\"This strategic initialisation also imposes a critical constraint wherein the initial hidden state θ_0 must encode semantically rich information applicable to the entire sequence.\" Could you give some clarity here - how can the weights encode information about a sequence prior to observing it?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The core idea of parametrising RNN hidden states as weights of an auxiliary neural network is conceptually interesting and, to the best of my knowledge, novel. The authors test their method on a diverse set of domains and perform a large range of ablations to show the necessity of design choices. The writing is generally accessible, with good intuitive explanations, and goes far to place itself in the larger test-time adaptation literature."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Claims are sometimes overstated and/or imprecise. For example, phrases like \"transformative paradigm for adaptive machine intelligence\" (Abstract, Conclusion) and \"redefine sequence modeling\" (Abstract) are not well-supported. The empirical results show WARP is competitive but not uniformly superior. \"Brain-inspired formulation\" (Abstract, page 2) refers only to using input differences, with citation to synaptic plasticity [16], but the connection is somewhat superficial - there is a rich literature concerned with modelling realistic synaptic plasticity rules which this approach to weight-space trajectory modelling does not engage with. \"Infinite-dimensional RNN hidden states\" (page 9, footnote 6) is misleading—the hidden state is finite-dimensional, though it parametrizes a function."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917574578,"tcdate":1762032542494,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4784/Reviewer_rGWP"],"signatures":["ICLR.cc/2026/Conference/Submission4784/Reviewer_rGWP"],"forum":"zHdKaF3ZM7","number":3,"license":"CC BY 4.0","cdate":1762032542494,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4784/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917574578,"domain":"ICLR.cc/2026/Conference","replyto":"zHdKaF3ZM7","id":"GyOp2eDthB","forumContent":{"TLDR":{"value":"We introduce a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["physics-informed machine learning","weight-space learning","meta-learning","deep sequence modeling","linear recurrence","test-time training"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"We introduce WARP (**W**eight-space **A**daptive **R**ecurrent **P**rediction), a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into fixed-dimensional hidden states, WARP explicitly parametrizes its hidden state as the weights and biases of a distinct auxiliary neural network, and uses input differences to drive its recurrence. This brain-inspired formulation enables efficient gradient-free adaptation of the auxiliary network at test-time, in-context learning abilities, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state-of-the-art baselines on diverse classification tasks, featuring in the top three in 4 out of 6 real-world challenging datasets. Furthermore, extensive experiments across sequential image completion, multivariate time series forecasting, and dynamical system reconstruction demonstrate its expressiveness and generalization capabilities. Remarkably, a physics-informed variant of our model outperforms the next best model by more than 10x. Ablation studies confirm the architectural necessity of key components, solidifying weight-space linear RNNs as a transformative paradigm for adaptive machine intelligence."},"_bibtex":{"value":"@inproceedings{\nnzoyem2026weightspace,\ntitle={Weight-Space Linear Recurrent Neural Networks},\nauthor={Roussel Desmond Nzoyem and Nawid Keshtmand and Enrique Crespo-Fernandez and Idriss TSAYEM and Raul Santos-Rodriguez and David A.W. Barton and Tom Deakin},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zHdKaF3ZM7}\n}"},"title":{"value":"Weight-Space Linear Recurrent Neural Networks"},"pdf":{"value":"/pdf/c01f4c1dbdaf1c6e821cde510623f871a07c3404.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"nzoyem|weightspace_linear_recurrent_neural_networks"},"authorids":{"value":["~Roussel_Desmond_Nzoyem1","~Nawid_Keshtmand1","~Enrique_Crespo-Fernandez1","~Idriss_TSAYEM1","~Raul_Santos-Rodriguez1","~David_A.W._Barton1","~Tom_Deakin1"]},"authors":{"value":["Roussel Desmond Nzoyem","Nawid Keshtmand","Enrique Crespo-Fernandez","Idriss TSAYEM","Raul Santos-Rodriguez","David A.W. Barton","Tom Deakin"]}},"version":2},{"content":{"comment":{"value":"First, we thank the reviewer for acknowledging the soundness, presentation, and contribution of our work. The reviewer's feedback is invaluable for improving the quality of our manuscript, which we will revise accordingly. Next, we will highlight some details in the manuscript that may have been overlooked and provide additional experimental results to address the reviewer's concerns. We note considerable overlap among the points raised, and thus have structured our responses collectively.\n\nOn the inspiration from SSM for our VMI-SA:\n\nFrom design philosophy to specific functional form, our VMI-SA is inspired by SSM. The success of Vision Mamba inspired us to recognize that carefully designed causal models can also be effectively applied to 2D visual information modeling. The S4 model (a classic SSM), i.e., Equation 3 in the manuscript, compresses and maintains historical context information in a fixed-length vector (hidden state $h_i$​), which is similar both in form and function to the context vector of separable self-attention. Unlike earlier models employing separable self-attention, Vision Mamba generally integrates more seamlessly with advanced architectures and achieves superior performance across various scales. This led us to investigate whether introducing causality could enhance the performance of separable self-attention.\n\nWe approached this from the perspective of matrix rank to theoretically analyze the value of introducing causality into separable self-attention. In Section 3.3.2, we note: \"Next, we consider how to enhance the rank … by referring to Eq. (3).\"\n\nArchitecturally, while building upon SSM, we introduced optimizations resulting in the recurrent formulation of VMI-SA (Eq. 9). In Section 3.3.3, we state: \"Similar to Eq. (3), we restrict the receptive field.…\"\n\nOn the rationale behind Eq. 9 and Eq. 10 and their connection:\n\nThe logic underlying the recurrent formulation of VMI-SA (Eq. 9) is detailed in the preceding response. The matrix formulation of VMI-SA was introduced primarily to exploit the high parallelism of matrix operations, thereby further enhancing efficiency. Section 3.3.4 describes how we derive the matrix formulation (Eq. 10) from the recurrent form by altering the receptive field.\n\nAdditional experiments:\n\nIn response to the reviewer's concerns, we have included Flatten Transformer [1] in our comparisons and report the Top-5 accuracy for both VMINet and VMIFormer. As Top-5 accuracy was not reported in the original Flatten Transformer paper, we omit its comparison for this metric. The results demonstrate that, under comparable parameter and computation budgets, VMINet and VMIFormer outperform Flatten Transformer [1].\n\n\n**Table: Comparison of different models on ImageNet-1K.**\n\n| Method | Params (M) | FLOPs (G) | Top-1 (%) | Top-5 (%) |\n|--------|------------|-----------|-----------|-----------|\n| $\\textbf{VMINet-Ti (ours)}$ | $\\textbf{2}$ | $\\textbf{0.3}$ | $\\textbf{70.7}$ | $\\textbf{90.3}$ |\n| FLatten-DeiT-T (ICCV 2023) | 6 | 1.1 | 74.1 | - |\n| FLatten-PVT-T (ICCV 2023) | 12 | 2.0 | 77.8 | - |\n| $\\textbf{VMINet-XS (ours)}$ | $\\textbf{7}$ | $\\textbf{1.4}$ | $\\textbf{78.6}$ | $\\textbf{94.3}$ |\n| FLatten-PVTv2-B1 (ICCV 2023) | 13 | 2.2 | 79.5 | - |\n| $\\textbf{VMINet-S (ours)}$ | $\\textbf{13}$ | $\\textbf{2.3}$ | $\\textbf{80.5}$ | $\\textbf{95.2}$ |\n| FLatten-PVT-S (ICCV 2023) | 22 | 4.0 | 81.7 | - |\n| FLatten-Swin-T (ICCV 2023) | 29 | 4.5 | 82.1 | - |\n| $\\textbf{VMINet-B (ours)}$ | $\\textbf{28}$ | $\\textbf{4.8}$ | $\\textbf{82.4}$ | $\\textbf{96.1}$ |\n| FLatten-PVTv2-B2 (ICCV 2023) | 23 | 4.3 | 82.5 | - |\n| FLatten-CSwin-T (ICCV 2023) | 21 | 4.3 | 83.1 | - |\n| $\\textbf{VMIFormer-T (ours)}$ | $\\textbf{24}$ | $\\textbf{4.2}$ | $\\textbf{83.2}$ | $\\textbf{96.4}$ |\n\nOn model efficiency for long sequences:\n\nThe inference speed of Mamba relies on the hardware-aware parallel algorithm proposed in [2], an optimization designed for the pure Mamba architecture. Since Visual Mamba models typically adopt hybrid architectures, they cannot fully leverage this optimization, thereby negating the theoretical acceleration advantage in processing images. Ablation studies of VMamba [3] (Table 9) show that the linear-complexity Vanilla-VMamba [3] underperforms the quadratic-complexity DeiT [4] across all resolutions, making comparisons between VMINet and EfficientVMamba at typical resolutions unnecessary. Mamba [2] demonstrates superior efficiency over most linear models in long-sequence tasks, suggesting that visual Mamba models may reach an efficiency inflection point at extreme resolutions—though this goes beyond the scope of our discussion.\n\n[1]Han et al. FLatten Transformer: Vision Transformer using Focused Linear Attention. ICCV2023.\n\n[2]Gu and Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. COLM 2024.\n\n[3]Liu et al. VMamba: Visual State Space Model. NeurIPS 2024.\n\n[4]Touvron et al.  Training data-efficient image transformers & distillation through attention. ICML 2021."},"title":{"value":"We Thank the Reviewer for the Comments"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764407788771,"tcdate":1763296450906,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6579/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission6579/Authors"],"forum":"1KUMxnrhnH","number":3,"license":"CC BY 4.0","cdate":1763296450906,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6579/-/Official_Comment"],"mdate":1764407788771,"domain":"ICLR.cc/2026/Conference","replyto":"wIbUoc2Xp0","id":"E67CEejO3z","forumContent":{"TLDR":{"value":"Inspired by the state space model at the core of Vision Mamba, we propose a novel separable self-attention mechanism termed Vision Mamba-Inspired Separable Self-Attention."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Separable Self-attention","State Space Models","Vision Mamba"]},"supplementary_material":{"value":"/attachment/0cd46f2349f939d5a4b0d712c9497c03b36d205a.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Separable self-attention is an early attention mechanism with linear complexity. When parameters and FLOPs are comparable, lightweight networks built upon separable self-attention and its variants underperform the recent Vision Mamba (ViM). By analyzing the strengths and weaknesses of separable self-attention, we distill four design principles and, inspired by the State Space Model (SSM) serving as the core of ViM, propose a novel separable self-attention termed Vision Mamba Inspired Separable self-Attention (VMI-SA). Notably, VMI-SA does not incorporate any SSM blocks, and its attention computation process differs from all existing attention mechanisms to the best of our knowledge. We introduce proof-of-concept networks, VMINet and VMIFormer, enabling fair comparisons with ViMs through deliberate control of parameters, FLOPs, and encoder numbers. Compared to state-of-the-art Transformers, CNNs, and ViMs, VMINet and VMIFormer achieve competitive results in image classification and high-resolution dense prediction tasks."},"_bibtex":{"value":"@misc{\nzhang2026a,\ntitle={A Separable Self-attention Inspired by the State Space Model for Computer Vision},\nauthor={Juntao Zhang and Jun Zhou and Kun Bian and zhou you and Jianning Liu},\nyear={2026},\nurl={https://openreview.net/forum?id=1KUMxnrhnH}\n}"},"title":{"value":"A Separable Self-attention Inspired by the State Space Model for Computer Vision"},"pdf":{"value":"/pdf/763025fad779b7f900ba99ff6984267142f9e588.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|a_separable_selfattention_inspired_by_the_state_space_model_for_computer_vision"},"authorids":{"value":["~Juntao_Zhang1","~Jun_Zhou16","~Kun_Bian1","~zhou_you1","~Jianning_Liu1"]},"authors":{"value":["Juntao Zhang","Jun Zhou","Kun Bian","zhou you","Jianning Liu"]}},"version":2},{"content":{"venue":{"value":"IEEE Trans. Geosci. Remote. Sens. 2025"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/36/10807682/10960652.pdf"},"venueid":{"value":"dblp.org/journals/TGRS/2025"},"paperhash":{"value":"tang|spiralmamba_spatialspectral_complementary_mamba_with_spatial_spiral_scan_for_hyperspectral_image_classification"},"authorids":{"value":["~Xu_Tang2","https://dblp.org/search/pid/api?q=author:Yuexi_Yao:","https://dblp.org/search/pid/api?q=author:Jingjing_Ma_0001:","https://dblp.org/search/pid/api?q=author:Xiangrong_Zhang:","https://dblp.org/search/pid/api?q=author:Yuqun_Yang:","https://dblp.org/search/pid/api?q=author:Bo_Wang_0016:","https://dblp.org/search/pid/api?q=author:Licheng_Jiao:"]},"html":{"value":"https://doi.org/10.1109/TGRS.2025.3559137"},"_bibtex":{"value":"@article{DBLP:journals/tgrs/TangYMZYWJ25,\n  author={Xu Tang and Yuexi Yao and Jingjing Ma and Xiangrong Zhang and Yuqun Yang and Bo Wang and Licheng Jiao},\n  title={SpiralMamba: Spatial-Spectral Complementary Mamba With Spatial Spiral Scan for Hyperspectral Image Classification},\n  year={2025},\n  cdate={1735689600000},\n  journal={IEEE Trans. Geosci. Remote. Sens.},\n  volume={63},\n  pages={1-19},\n  url={https://doi.org/10.1109/TGRS.2025.3559137}\n}\n"},"abstract":{"value":"Hyperspectral image (HSI) classification is crucial in the remote sensing (RS) community. In recent years, Transformers have been popular in this field due to their global information modeling capabilities. However, the quadratic complexity limits their performance under limited computational resources. Fortunately, a selective structured state space model (SSM) named Mamba emerges. Like Transformer, it is good at modeling the long-distance relationships hidden in the pending data. Unlike Transformer, its complexity remains at a linear level. Therefore, a growing number of studies have been proposed to explore the usefulness of Mamba in HSI classification. Nevertheless, most of them only apply Mamba to HSIs directly but do not consider the inherent characteristics of HSIs properly. To exploit the potential of Mamba in HSI classification deeply, this article presents a new spatial-spectral complementary Mamba with a spatial spiral scan named SpiralMamba. It mainly encloses three main components: a spatial Mamba encoder (SpaME), a spectral Mamba encoder (SpeME), and a spatial-spectral complementary fusion module (SSCFM). SpaME focuses on understanding the spatial context within HSIs. To this end, instead of the common scanning, a spatial spiral scan strategy is introduced to address the sequence transformation of noncausal HSIs. SpeME aims to comprehensively extract valuable spectral features from HSIs. To achieve this goal, besides developing a spectral bidirectional scan strategy, a multilayer convolution (MLC) is also incorporated to capture local variations within spectral tokens. SSCFM concentrates on building the complex connections between spatial and spectral features and fusing them. For this purpose, a relationship learning block (RLB) and a threshold enhancement mechanism (TEM) are developed. Positive experimental results counted on three public HSI datasets demonstrate the effectiveness of SpiralMamba. Our source codes are available at https://github.com/TangXu-Group/Hyperspectral-Images-Classification/tree/main/SpiralMamba"},"title":{"value":"SpiralMamba: Spatial-Spectral Complementary Mamba With Spatial Spiral Scan for Hyperspectral Image Classification"},"authors":{"value":["Xu Tang","Yuexi Yao","Jingjing Ma","Xiangrong Zhang","Yuqun Yang","Bo Wang","Licheng Jiao"]}},"tmdate":1754096990104,"pdate":1735689600000,"externalIds":["dblp:journals/tgrs/TangYMZYWJ25"],"tcdate":1754096984497,"writers":["~"],"signatures":["~Xu_Tang2"],"forum":"3Y171n9AYG","license":"CC BY-SA 4.0","number":610280,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1754096990104,"domain":"DBLP.org","id":"3Y171n9AYG","version":2},{"content":{"TLDR":{"value":"LrcSSM is a nonlinear RNN with parallel prefix-scan updates, formal gradient stability, and compute-optimal scaling. It outperforms LRU, S5, and Mamba on long-range tasks - matching SSM speed without attention or FFT overhead."},"venue":{"value":"ES-FoMo III"},"pdf":{"value":"/pdf/138e5cbfe3d51dcd9f0ac57eb4d79e14385b293e.pdf"},"keywords":{"value":["structural state space model","non-linear RNN","liquid neural network","parallelization","sequence modeling"]},"venueid":{"value":"ICML.cc/2025/Workshop/ES-FoMo-III"},"paperhash":{"value":"farsang|scaling_up_liquidresistance_liquidcapacitance_networks_for_efficient_sequence_modeling"},"authorids":{"value":["~Mónika_Farsang1","~Ramin_Hasani1","~Radu_Grosu1"]},"abstract":{"value":"We present LrcSSM, a $\\textit{nonlinear}$ recurrent model that processes long sequences as fast as today’s linear state-space layers. By forcing the state-transition matrix to be diagonal and learned at every step, the full sequence can be solved in parallel with a single prefix-scan, giving $\\mathcal{O}(TD)$ time and memory and only $\\mathcal{O}(\\log T)$ sequential depth, for input-sequence length $T$ and a state dimension $D$. Moreover, LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 and Mamba do not provide. Lastly, for network depth $L$, as the forward and backward passes cost $\\Theta(T\\,D\\,L)$ FLOPs, with its low sequential depth and parameter count $\\Theta(D\\,L)$, the model follows the compute-optimal scaling law regime ($\\beta \\approx 0.42$) recently observed for Mamba, outperforming quadratic-attention Transformers at equal compute while avoiding the memory overhead of FFT-based long convolutions. We show that on a series of long-range forecasting tasks, LrcSSM outperforms LRU, S5 and Mamba."},"_bibtex":{"value":"@inproceedings{\nfarsang2025scaling,\ntitle={Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling},\nauthor={M{\\'o}nika Farsang and Ramin Hasani and Radu Grosu},\nbooktitle={ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models},\nyear={2025},\nurl={https://openreview.net/forum?id=3qRWlWtPTo}\n}"},"title":{"value":"Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling"},"authors":{"value":["Mónika Farsang","Ramin Hasani","Radu Grosu"]}},"tmdate":1752186361417,"pdate":1749663447060,"tcdate":1748370305848,"writers":["ICML.cc/2025/Workshop/ES-FoMo-III","ICML.cc/2025/Workshop/ES-FoMo-III/Submission153/Authors"],"signatures":["ICML.cc/2025/Workshop/ES-FoMo-III/Submission153/Authors"],"forum":"3qRWlWtPTo","license":"CC BY 4.0","number":153,"cdate":1748370305848,"readers":["everyone"],"invitations":["ICML.cc/2025/Workshop/ES-FoMo-III/-/Submission","ICML.cc/2025/Workshop/ES-FoMo-III/-/Post_Submission","ICML.cc/2025/Workshop/ES-FoMo-III/-/Edit","ICML.cc/2025/Workshop/ES-FoMo-III/Submission153/-/Camera_Ready"],"mdate":1752186361417,"odate":1752186093971,"domain":"ICML.cc/2025/Workshop/ES-FoMo-III","id":"3qRWlWtPTo","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2405.15413v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"qin|mambavc_learned_visual_compression_with_selective_state_spaces"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Shiyu_Qin:","~Jinpeng_Wang3","~Yimin_Zhou1","https://dblp.org/search/pid/api?q=author:Bin_Chen_0011:","~Tianci_Luo1","https://dblp.org/search/pid/api?q=author:Baoyi_An:","~Tao_Dai3","https://dblp.org/search/pid/api?q=author:Shutao_Xia:","https://dblp.org/search/pid/api?q=author:Yaowei_Wang_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2405.15413"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2405-15413,\n  publtype={informal},\n  author={Shiyu Qin and Jinpeng Wang and Yimin Zhou and Bin Chen and Tianci Luo and Baoyi An and Tao Dai and Shutao Xia and Yaowei Wang},\n  title={MambaVC: Learned Visual Compression with Selective State Spaces},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2405.15413},\n  url={https://doi.org/10.48550/arXiv.2405.15413}\n}\n"},"abstract":{"value":"Learned visual compression is an important and active task in multimedia. Existing approaches have explored various CNN- and Transformer-based designs to model content distribution and eliminate redundancy, where balancing efficacy (i.e., rate-distortion trade-off) and efficiency remains a challenge. Recently, state-space models (SSMs) have shown promise due to their long-range modeling capacity and efficiency. Inspired by this, we take the first step to explore SSMs for visual compression. We introduce MambaVC, a simple, strong and efficient compression network based on SSM. MambaVC develops a visual state space (VSS) block with a 2D selective scanning (2DSS) module as the nonlinear activation function after each downsampling, which helps to capture informative global contexts and enhances compression. On compression benchmark datasets, MambaVC achieves superior rate-distortion performance with lower computational and memory overheads. Specifically, it outperforms CNN and Transformer variants by 9.3% and 15.6% on Kodak, respectively, while reducing computation by 42% and 24%, and saving 12% and 71% of memory. MambaVC shows even greater improvements with high-resolution images, highlighting its potential and scalability in real-world applications. We also provide a comprehensive comparison of different network designs, underscoring MambaVC's advantages. Code is available at https://github.com/QinSY123/2024-MambaVC."},"title":{"value":"MambaVC: Learned Visual Compression with Selective State Spaces"},"authors":{"value":["Shiyu Qin","Jinpeng Wang","Yimin Zhou","Bin Chen","Tianci Luo","Baoyi An","Tao Dai","Shutao Xia","Yaowei Wang"]}},"tmdate":1769672031933,"pdate":1704067200000,"tcdate":1747274265017,"writers":["~"],"signatures":["~Tao_Dai3"],"forum":"VQMlBQx5Ss","license":"CC BY-SA 4.0","number":469660,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769672031933,"domain":"DBLP.org","id":"VQMlBQx5Ss","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2404.18861v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"xu|a_survey_on_vision_mamba_models_applications_and_challenges"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Rui_Xu:","https://dblp.org/search/pid/api?q=author:Shu_Yang_0004:","~Yihui_Wang1","https://dblp.org/search/pid/api?q=author:Bo_Du_0001:","https://dblp.org/search/pid/api?q=author:Hao_Chen_0011:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2404.18861"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2404-18861,\n  publtype={informal},\n  author={Rui Xu and Shu Yang and Yihui Wang and Bo Du and Hao Chen},\n  title={A Survey on Vision Mamba: Models, Applications and Challenges},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2404.18861},\n  url={https://doi.org/10.48550/arXiv.2404.18861}\n}\n"},"abstract":{"value":"Mamba, a recent selective structured state space model, excels in long sequence modeling, which is vital in the large model era. Long sequence modeling poses significant challenges, including capturing long-range dependencies within the data and handling the computational demands caused by their extensive length. Mamba addresses these challenges by overcoming the local perception limitations of convolutional neural networks and the quadratic computational complexity of Transformers. Given its advantages over these mainstream foundation architectures, Mamba exhibits great potential to be a visual foundation architecture. Since January 2024, Mamba has been actively applied to diverse computer vision tasks, yielding numerous contributions. To help keep pace with the rapid advancements, this paper reviews visual Mamba approaches, analyzing over 200 papers. This paper begins by delineating the formulation of the original Mamba model. Subsequently, it delves into representative backbone networks, and applications categorized using different modalities, including image, video, point cloud, and multi-modal data. Particularly, we identify scanning techniques as critical for adapting Mamba to vision tasks, and decouple these scanning techniques to clarify their functionality and enhance their flexibility across various applications. Finally, we discuss the challenges and future directions, providing insights into new outlooks in this fast evolving area. A comprehensive list of visual Mamba models reviewed in this work is available at https://github.com/Ruixxxx/Awesome-Vision-Mamba-Models."},"title":{"value":"A Survey on Vision Mamba: Models, Applications and Challenges"},"authors":{"value":["Rui Xu","Shu Yang","Yihui Wang","Bo Du","Hao Chen"]}},"tmdate":1737376033294,"pdate":1704067200000,"tcdate":1737376030425,"writers":["~"],"signatures":["~Yihui_Wang1"],"forum":"bPrxhBH1Sk","license":"CC BY-SA 4.0","number":268847,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1737376033294,"domain":"DBLP.org","id":"bPrxhBH1Sk","version":2},{"content":{"comment":{"value":"We thank the reviewer for their detailed and informative feedback.\n1. We acknowledge that pre-training models with various seeds and at scales more than 7B would strengthen our claims, but we have been limited by compute resources. Therefore, we provide scaling results across five compute budgets against training FLOPs, to provide evidence that our findings are likely to extend to larger model scales. Additionally, we state in the abstract and conclusion that the results mentioned are at the sub-1.5 B scale, as already mentioned in the response to and acknowledged by reviewer PM8N.\n2. We acknowledge certain numerical discrepancies that have crept in the draft due to typographical errors. The throughput gain claim should be more accurately mentioned as $1.27  - 2.7 \\times$ instead of the mentioned values. We also note that BoolQ accuracy for MobileLLM-125M in Table 2 should be 60.64% which has been mistakenly copied as the value in the previous table entry. We will include these updated artefacts, along with the updated downstream accuracy average in the final version of the paper.\n3. We follow the experimental protocols from Flash-Attention [1] and Mamba [2]. We attribute the non-monotonicity in our efficiency benchmarks majorly to experimental noise, as pointed out by the reviewer.\n4. We will refine our claims with respect to CLA, YOCO, MixAttention, and Xue et al, as suggested by the reviewer and mentioned below too: \nWe will replace the statement\n\"This work is, to the best of our knowledge, among the first to directly translate insights from interpretability research into an architectural design choice at pre-training time\" with:\n\n\"Taking inspiration from research in cross-layer redundancy in attention (Brandon et al., 2024; Sun et al., 2024b; Rajput et al., 2024) and on hybrid MLP-attention designs (Xue et al., 2025), along with interpretability findings on depth-wise computation, we demonstrate an efficient architectural design choice at pre-training time\"\n\n\nWe also note that we have qualified the model sizes targeted in this paper, in the abstract and conclusion and have limited our conclusions to the sub-1.5B parameter regime. \n\n[1] Tri Dao. Flash Attention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. \n[2] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1791298828588,"tcdate":1791298828588,"writers":["TMLR","TMLR/Paper11586/Authors"],"signatures":["TMLR/Paper11586/Authors"],"forum":"AFw5w8FuIu","number":3,"license":"CC BY 4.0","cdate":1791298828588,"readers":["everyone"],"invitations":["TMLR/Paper11586/-/Official_Comment"],"mdate":1791298828588,"domain":"TMLR","replyto":"n6ufIUxW4e","id":"MWSem7We3E","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Under review for TMLR"},"abstract":{"value":"While the transformer architecture has achieved state-of-the-art performance on natural language processing tasks, these models impose substantial memory and computational overhead. Recent research has identified significant architectural redundancies within these models, particularly in the attention sub-layers in the top layers, presenting opportunities for optimization without compromising performance. Taking insights from research on inference-time layer pruning and depth-dependent computation in language models, we introduce an efficient language model architecture referred to as ShishuLM. By replacing full decoder layers at the top of the model with MLP-only blocks, we achieve up to  $10-60$% improvement in generation latency and $1.3 -5\\times$ gain in throughput on sub-1.5B models from the Llama architecture family. Upon further sharing parameters across adjacent MLP-only layers of ShishuLM, we obtain up to 20% savings in memory with minimal degradation in performance. Our findings provide insights towards building more efficient language modeling architectures from a pre-training standpoint by leveraging how information flows in transformers."},"_bibtex":{"value":"@article{\nanonymous2026shishulm,\ntitle={Shishu{LM}: Achieving Optimal and Efficient Parameterization with Low Attention Transformer Models},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2026},\nurl={https://openreview.net/forum?id=AFw5w8FuIu},\nnote={Under review}\n}"},"title":{"value":"ShishuLM: Achieving Optimal and Efficient Parameterization with Low Attention Transformer Models"},"pdf":{"value":"/pdf/d661fc83dae1cf0bc43c927bd77eded9465c62b9.pdf"},"venueid":{"value":"TMLR/Under_Review"},"assigned_action_editor":{"value":"~Yu_Cheng1"}},"version":2},{"content":{"comment":{"value":"We thank Reviewer c9Zu for acknowledging the sound rationale behind our new module design, the superiority of our experimental results, and the detailed ablation studies.\nOur detailed responses to your comments are as follows.\nThe revised manuscript has been uploaded, with all changes clearly highlighted in blue for your convenience.\n\n---\n\n### [W1] & [Q1] Lack of explanation for using attention instead of Mamba or RWKV\n\nThank you for this feedback.\nFirst of all, we would like to emphasize that our architectural improvements focus on two aspects: spatial modeling and temporal modeling. For the temporal dimension, we introduce the Visibility-aware Long-Temporal Attention (VLTA) module to replace the previous RNN-like design.\nOur choice to use attention instead of more advanced RNN-like structures such as Mamba [1] or RWKV [2] is based on the following considerations.\n\nWe fully acknowledge that Mamba, RWKV, and other improved RNN-like architectures have demonstrated strong potential and efficiency.\nHowever, the core insight behind VLTA is that the **initial-frame information of the target point should be treated as the most reliable anchor and must be preserved throughout the tracking process**.\nAlthough Mamba and RWKV improve long-range information propagation through state compression and selective forgetting, they fundamentally **rely on a single evolving hidden state to carry information over time**. Regardless of how efficient the mechanism is, this hidden state represents a lossy compression of the entire history.\nIn long videos, a point may disappear for hundreds of frames, during which the fine-grained appearance details stored in the hidden state may **degrade or become contaminated**.\n\nIn contrast, our approach keeps the initial-frame anchor intact and uses attention to directly retrieve information from historical frames to accommodate appearance changes.\nVisibility prediction is further incorporated to ensure that more reliable historical information is aggregated.\n\nIn summary, we believe that for video point tracking, hidden-state evolution is less suitable for handling the complex challenges of long videos.\nWhile attention may incur higher computational cost, by appropriately controlling the size of the historical buffer, our model still **achieves a good balance between performance and efficiency**.\nAs described in Appendix Sec. A.2, TAPTRv3 can process arbitrarily long videos in a streaming manner (tracking 100 points simultaneously with less than 2GB GPU memory), making it **deployment cost-efficient**.\n\nOnce again, thank you for this insightful feedback. We have **added the corresponding explanation in the Appendix Sec. A.6** to address potential concerns.\n\n- [1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces, COLM 2024.\n- [2] RWKV: Reinventing RNNs for the Transformer Era, EMNLP 2023.\n\n### [W2] & [Q2] Further explanation of the effectiveness of visibility-aware attention\n\nThank you for raising this question.\nFirst, we apply the predicted visibility in a **soft weighting manner**, where frames predicted as more likely to be invisible are down-weighted according to confidence rather than discarded.\nThe operation is intuitively sound when the visibility prediction is correct. When the visibility estimation is erroneous, it can be further categorized into the following two specific cases:\n\n1. Visible point -> predicted as invisible\n\n    In this case, the attention weights for these frames are reduced, leading the model to extract less historical information from them.\n    As a result, the model relies more on the earlier frames where the point is visible, and these frames provide reliable information.\n\n2. Invisible point -> predicted as visible\n\n    This case degenerates to the situation where visibility weighting is not applied.\n    The model may aggregate unreliable historical information, similar to the baseline without visibility-aware weighting.\n\nIn summary, the visibility-aware attention mechanism **provides an overall positive effect**.\nThanks for your review, and we have **added the corresponding explanation in the Appendix Sec. A.6** to address potential concerns."},"title":{"value":"Response to Reviewer c9Zu (Part 1/3)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763667175162,"tcdate":1763667175162,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7206/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7206/Authors"],"forum":"N3WAcxTX6J","number":4,"license":"CC BY 4.0","cdate":1763667175162,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7206/-/Official_Comment"],"mdate":1763667175162,"domain":"ICLR.cc/2026/Conference","replyto":"YrnKXILqxF","id":"AfhLF7l1I0","forumContent":{"TLDR":{"value":"A novel state-of-the-art online method for the Track Any Point task, optimized for long videos."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Tracking Any Point"]},"supplementary_material":{"value":"/attachment/fd6d625a96dac0b5d927ec3a48ca518cc2b89645.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"In this paper, built upon TAPTRv2, we present TAPTRv3. TAPTRv3 improves TAPTRv2 by addressing its shortage in querying high quality features from long videos, where the target tracking points normally undergo increasing variation over time. In TAPTRv3, we propose to utilize both spatial and temporal context to bring better feature querying along the spatial and temporal dimensions for more robust tracking in long videos. For better spatial feature querying, we identify that off-the-shelf attention mechanisms struggle with point-level tasks and present Context-aware Cross-Attention (CCA). CCA introduces spatial context into the attention mechanism to enhance the quality of attention scores when querying image features. For better temporal feature querying, we introduce Visibility-aware Long-Temporal Attention (VLTA), which conducts temporal attention over all past frames while considering their corresponding visibilities. This effectively addresses the feature drifting problem in TAPTRv2 caused by its RNN-like long-term modeling. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the challenging datasets and obtains state-of-the-art performance. Even when compared with methods trained on large-scale extra internal data, TAPTRv3 still demonstrates superiority."},"_bibtex":{"value":"@inproceedings{\nqu2026taptrv,\ntitle={{TAPTR}v3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Video},\nauthor={Jinyuan Qu and Hongyang Li and Shilong Liu and Tianhe Ren and Zhaoyang Zeng and Lei Zhang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=N3WAcxTX6J}\n}"},"title":{"value":"TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Video"},"pdf":{"value":"/pdf/fd6826f06d67007841e0d56e7862e05827461e93.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"qu|taptrv3_spatial_and_temporal_context_foster_robust_tracking_of_any_point_in_long_video"},"authorids":{"value":["~Jinyuan_Qu1","~Hongyang_Li6","~Shilong_Liu1","~Tianhe_Ren1","~Zhaoyang_Zeng1","~Lei_Zhang23"]},"authors":{"value":["Jinyuan Qu","Hongyang Li","Shilong Liu","Tianhe Ren","Zhaoyang Zeng","Lei Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper comprehensively investigates the possibility of leveraging Mamba for trajectory learning. The authors take Decision Mamba as a playground and analyse the performance of this model over trajectory learning scenarios (gym/mujoco) from several aspects. A group of conclusions are attained through rigorous experiments, which is solid and potentially valuable for further researches realted with Mamba."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"see Weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. Novelty: given that Mamba is still at its exploratory stage, this paper positively probes Mamba's potential for tranjectory learning, with surprising results that with some specific pre-conditions, Mamba is more suited than Transformer."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Most discoveries in this paper have been implicitly discussed for several months within the community, while it is firstly presented officially in this paper. Besides, these dicoveries lean to be emparical  evidence, which is relatively shallow. This would make this paper's technical contribution weak. I would appreciate the authors if they could provide more in-depth explanation over these discoveries, in particular: 1) Transfomer-like model favors short sequence. 2) The significant role of the hidden attention. \n\n2. Although the experimental results are solid, I found that this paper is more suitable for Benchmark Track, since the technical novelty revolves around benchmarking Decision Mamba. \n\n3. Figure 1 (the title and the pic) should be improved. For now, it confuses me, especially the corresponding relationship between the text content (title) and the illustration.\n\n3. Minor: line 295: may more suitable -> may be more suitable"},"limitations":{"value":"The authors provide a brief limitation summary in Conclusion. I would appreciate the authors if they could refine this part since the \"limitations\" listed there do not seem like limitations."}},"nonreaders":[],"tmdate":1730879217165,"tcdate":1720015458134,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission8092/Reviewer_85kG"],"signatures":["NeurIPS.cc/2024/Conference/Submission8092/Reviewer_85kG"],"forum":"yWSxjlFsmX","number":1,"license":"CC BY 4.0","cdate":1720015458134,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission8092/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879217165,"domain":"NeurIPS.cc/2024/Conference","replyto":"yWSxjlFsmX","id":"m4uv2OLFQ0","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"TLDR":{"value":"An investigation of Mamba in offline RL."},"keywords":{"value":["Offline RL; Trajectory Optimization; Mamba"]},"primary_area":{"value":"reinforcement_learning"},"abstract":{"value":"Transformer-based trajectory optimization methods have demonstrated exceptional performance in offline Reinforcement Learning (offline RL). Yet, it poses challenges due to substantial parameter size and limited scalability, which is particularly critical in sequential decision-making scenarios where resources are constrained such as in robots and drones with limited computational power. Mamba, a promising new linear-time sequence model, offers performance on par with transformers while delivering substantially fewer parameters on long sequences. As it remains unclear whether Mamba is compatible with trajectory optimization, this work aims to conduct comprehensive experiments to explore the potential of Decision Mamba (dubbed DeMa) in offline RL from the aspect of data structures and essential components with the following insights: (1) Long sequences impose a significant computational burden without contributing to performance improvements since DeMa's focus on sequences diminishes approximately exponentially. Consequently, we introduce a Transformer-like DeMa as opposed to an RNN-like DeMa. (2) For the components of DeMa, we identify the hidden attention mechanism as a critical factor in its success, which can also work well with other residual structures and does not require position embedding. Extensive evaluations demonstrate that our specially designed DeMa is compatible with trajectory optimization and surpasses previous methods, outperforming Decision Transformer (DT) with higher performance while using 30\\% fewer parameters in Atari, and exceeding DT with only a quarter of the parameters in MuJoCo."},"_bibtex":{"value":"@inproceedings{\ndai2024is,\ntitle={Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?},\nauthor={Yang Dai and Oubo Ma and Longfei Zhang and Xingxing Liang and Shengchao Hu and Mengzhu Wang and Shouling Ji and Jincai Huang and Li Shen},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=yWSxjlFsmX}\n}"},"title":{"value":"Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?"},"pdf":{"value":"/pdf/e8f05bc8b78365623dc8f45e047f65b46390a923.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"dai|is_mamba_compatible_with_trajectory_optimization_in_offline_reinforcement_learning"},"authorids":{"value":["~Yang_Dai2","~Oubo_Ma1","~Longfei_Zhang3","~Xingxing_Liang1","~Shengchao_Hu1","~Mengzhu_Wang1","~Shouling_Ji1","~Jincai_Huang1","~Li_Shen1"]},"authors":{"value":["Yang Dai","Oubo Ma","Longfei Zhang","Xingxing Liang","Shengchao Hu","Mengzhu Wang","Shouling Ji","Jincai Huang","Li Shen"]}},"version":2},{"content":{"summary":{"value":"This paper presents Hydra, an innovative framework that builds upon the Mamba model by introducing bidirectional capabilities. Hydra's approach centers on a matrix mixer perspective, which allows it to consolidate various sequence models, including Transformers and structured state space models, into a unified framework. The primary strength of Hydra is its capacity to surpass other sequence models in non-causal tasks while retaining efficiency and expressiveness. The study demonstrates how Hydra's novel bidirectional methodology and matrix parameterizations effectively enhance the performance of sequence models."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"- The experiments are mainly around ~ a 100 M parameter scale. I am curious about whether the model can be scaled up to ~1.5B parameters and how the model will perform.\n- I am wondering, in such non-causal tasks, how is a comparison between the proposed hydra framework with stacked bidirectional Mamba frameworks (https://arxiv.org/pdf/2401.09417,https://arxiv.org/abs/2404.15772)?\n- Compared to the original Mamba framework, does Hydra require additional computational resources and training time, as the training objective is harder? I am also curious about whether the matrix mix method will harm performance on general tasks."},"rating":{"value":9},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":4},"strengths":{"value":"- The authors proposed a novel framework, Hydra, which extends mamba with bidirectional capabilities and presents an interesting perspective on improving sequence models.\n- The proposed method of matrix mix offers a cohesive understanding of various sequence model architectures, which also offers valuable insights into how matrix parameterizations and sequence alignment affect model flexibility and performance.\n- The paper is well-structured, making it easy for readers to follow the development of the Hydra framework and its contributions to sequence modeling.\n- Abundant experiments covering both language and vision tasks illustrate the efficacy of the proposed method."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I don't identify a specific weakness, but I have a few questions regarding efficiency. Please see the following section."},"limitations":{"value":"See Questions."}},"nonreaders":[],"tmdate":1730878827144,"tcdate":1720916718719,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission3122/Reviewer_5fYM"],"signatures":["NeurIPS.cc/2024/Conference/Submission3122/Reviewer_5fYM"],"forum":"preo49P1VY","number":2,"license":"CC BY 4.0","cdate":1720916718719,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission3122/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878827144,"domain":"NeurIPS.cc/2024/Conference","replyto":"preo49P1VY","id":"7MBRc5Mh8G","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Deep learning","sequence models","state space models","mamba"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"A wide array of sequence models are built on a framework modeled after Transformers, comprising alternating sequence mixer and channel mixer layers.  This paper studies a unifying *matrix mixer* view of sequence mixers that can be conceptualized as a linear map on the input sequence. This framework encompasses a broad range of well-known sequence models, including the self-attention of Transformers as well as recent strong alternatives such as structured state space models (SSMs), and allows understanding downstream characteristics such as efficiency and expressivity through properties of their structured matrix class. We identify a key axis of matrix parameterizations termed *sequence alignment*, which increases the flexibility and performance of matrix mixers, providing insights into the strong performance of Transformers and recent SSMs such as Mamba. Furthermore, the matrix mixer framework offers a systematic approach to developing sequence mixers with desired properties, allowing us to develop several new sub-quadratic sequence models. In particular, we propose a natural bidirectional extension of the Mamba model (**Hydra**), parameterized as a *quasiseparable matrix mixer*, which demonstrates superior performance over other sequence models including Transformers on non-causal tasks. As a drop-in replacement for attention layers, \\name outperforms BERT by 0.8 points on the GLUE benchmark and ViT by 2% Top-1 accuracy on ImageNet."},"_bibtex":{"value":"@inproceedings{\nhwang2024hydra,\ntitle={Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers},\nauthor={Sukjun Hwang and Aakash Lahoti and Ratish Puduppully and Tri Dao and Albert Gu},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=preo49P1VY}\n}"},"title":{"value":"Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers"},"pdf":{"value":"/pdf/975e174ed724f910ffb63bad51c2578e145b4c50.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"hwang|hydra_bidirectional_state_space_models_through_generalized_matrix_mixers"},"authorids":{"value":["~Sukjun_Hwang1","~Aakash_Lahoti1","~Ratish_Puduppully1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Sukjun Hwang","Aakash Lahoti","Ratish Puduppully","Tri Dao","Albert Gu"]}},"version":2},{"content":{"summary":{"value":"This paper introduces NeuroMamba (NeuMa), a biologically inspired state-space model designed as a circuit-level analog of the hippocampal DG–CA3–CA1 circuits. The authors argue that existing selective SSMs (e.g., Mamba) implicitly approximate this structure but lack key functional pathways (dentate input and CA3 direct output). NeuMa explicitly implements these modules to achieve both engineering efficiency and biological interpretability. The paper provides (1) synthetic benchmark results showing improved learning stability, (2) neuroscientific validation showing that NeuMa reproduces hippocampal decorrelation dynamics, and (3) a “real-world validation” stage involving fine-tuning a pretrained NeuMa model (NeuMa-Chem) for CO_2 reduction catalysis."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see the weaknesses."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- Conceptual novelty\\\nThe paper presents an interesting perspective by interpreting state-space models through the lens of hippocampal circuitry. The proposed DG–CA3–CA1 modular structure offers a biologically inspired inductive bias that could improve interpretability and potentially influence model dynamics.\n\n- Empirical evaluation on synthetic tasks\\\nOn algorithmic benchmarks such as Selective Copying and Induction Heads, NeuMa achieves comparable or somewhat better convergence behavior than Mamba. The results suggest that the proposed structure can enhance stability in some settings.\n\n- Neuroscientific analysis\\\nThe experiments simulating hippocampal learning dynamics show qualitative similarities between NeuMa’s internal states and observed CA1 decorrelation patterns.\n\n- Ablation study\\\nThe ablations are generally helpful for understanding the role of each sub-module. The comparisons support the claim that DG, CA3-Out, and CA1 gating each contribute to model stability and performance, although the evidence is mainly qualitative."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Ambiguous contribution of each biological module\\\nThe paper introduces DG, CA3, and CA1 as functionally distinct components but does not provide a systematic analysis or theoretical justification of their respective computational roles. The ablation study, while informative, does not clearly demonstrate how each module contributes differently to learning dynamics or model performance. As a result, the improvement over Mamba appears somewhat ad hoc and difficult to attribute to specific mechanisms.\n\n- Lack of quantitative rigor in the neuroscience comparison\\\nThe reported resemblance between NeuMa’s internal dynamics and hippocampal recordings is largely qualitative. The choice of thresholds or correlation metrics used to claim “orthogonalized dynamics” is not well justified and could appear tuned to fit the narrative. A more quantitative or statistically grounded comparison would be needed to establish genuine biological correspondence.\n\n- Unclear relevance of the “scientific discovery” example to NeuMa’s claimed capabilities\\\nThe real-world validation section describes a CO₂ catalysis result with significant yield improvement, but it is not explained how this outcome specifically demonstrates NeuMa’s architectural advantages. It remains uncertain whether other architectures such as Transformer or standard Mamba could achieve similar predictive results under comparable training conditions. As currently written, this section feels detached from the main contribution and does not convincingly support NeuMa’s unique capabilities.\n\n- Empirical validation remains speculative\\\nIt is not clearly stated whether the catalytic results came from the authors’ own laboratory experiments, from simulation, or from reanalysis of prior literature. Without transparent methodology and reproducible data, the “real-world validation” claim lacks credibility."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922047489,"tcdate":1761707588174,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10846/Reviewer_MqWf"],"signatures":["ICLR.cc/2026/Conference/Submission10846/Reviewer_MqWf"],"forum":"clCsSQ5rKg","number":1,"license":"CC BY 4.0","cdate":1761707588174,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10846/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922047489,"domain":"ICLR.cc/2026/Conference","replyto":"clCsSQ5rKg","id":"naQEIbTMZn","forumContent":{"TLDR":{"value":"We reframe Mamba as an incomplete model of the hippocampus and demonstrate that a more neuro-faithful architecture unlocks superior algorithmic reasoning, emergent robustness, and profound biological fidelity."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["State Space Models","Computational Neuroscience","Hippocampal Modeling","Interpretability","Architectural Priors"]},"supplementary_material":{"value":"/attachment/a2ecf359bc475d91326c5bba6dcd339f63cbdc98.zip"},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Grounded in the evolutionary principle that resource constraints favor structural solutions for complex computation, we propose a neuro-centric framework that reframes the success of State Space Models (SSMs), e.g. Mamba, as an unconscious convergence to an incomplete model of the hippocampus—the brain's canonical circuit for sophisticated computations such as pattern separation and completion. Motivated by this, we introduce NeuroMamba (NeuMa), a novel architecture that consciously and faithfully implements the canonical hippocampal circuit, including the dentate gyrus (DG), Cornu Ammonis 3 (CA3), and Cornu Ammonis 1 (CA1), using foundational SSM blocks. Enabled by custom kernels, our design bridges the gap between biological plausibility and practical efficiency. Experiments demonstrate that NeuMa achieves superior performance and learning efficiency on synthetic benchmarks. More critically, it exhibits profound biological fidelity by spontaneously replicating the \"orthogonalized state machine\" dynamics of the biological hippocampus. Finally, we validate its capacity for real-world scientific discovery by developing a generative agent for piezoelectric catalysis that achieves superior performance in this complex, low-resource domain, thereby showcasing a new path for AI architecture design rooted in neuroscience."},"_bibtex":{"value":"@misc{\nyang2026neuma,\ntitle={NeuMa, Born to Work},\nauthor={Yuanda Yang and HAORAN ZHANG and Yuehui Li},\nyear={2026},\nurl={https://openreview.net/forum?id=clCsSQ5rKg}\n}"},"title":{"value":"NeuMa, Born to Work"},"pdf":{"value":"/pdf/50462e129b9287a504163b7223ce661545a64d8f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yang|neuma_born_to_work"},"authorids":{"value":["~Yuanda_Yang1","~HAORAN_ZHANG45","~Yuehui_Li1"]},"authors":{"value":["Yuanda Yang","HAORAN ZHANG","Yuehui Li"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2605.27406v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"saadatmand|a_simple_state_space_model_excels_at_multivariate_time_series_classification"},"html":{"value":"https://doi.org/10.48550/arXiv.2605.27406"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2605-27406,\n  publtype={informal},\n  author={Hassan Saadatmand and Geoffrey I. Webb and Hamid Rezatofighi and Mahsa Salehi},\n  title={A Simple State Space Model Excels at Multivariate Time Series Classification},\n  year={2026},\n  month={May},\n  cdate={1777593600000},\n  journal={CoRR},\n  volume={abs/2605.27406},\n  url={https://doi.org/10.48550/arXiv.2605.27406}\n}\n"},"abstract":{"value":"Structured state space models (SSMs) have recently emerged as a promising foundation for sequence modeling, with Mamba-based architectures demonstrating strong performance through input-dependent state transitions, albeit at considerable complexity. However, their application to time-series classification (TSC) has been largely limited to Mamba-style architectures, leaving the broader SSM design space underexplored. We present the first systematic study spanning diagonal SSMs (S4D) and input-dependent SSMs (Mamba family) on large-scale TSC benchmarks, asking whether such complexity is necessary for top performance. Our results reveal a surprising finding: S4D consistently outperforms Mamba-based variants in both accuracy and efficiency, challenging the assumption that increased complexity translates to meaningful gains in TSC. Building on this, we introduce MS4, lightweight modifications to S4D via a linear input projection and channel-mixing mechanism, and MS4N, a normalized variant that stabilizes state dynamics with negligible overhead. Evaluated on 59 datasets across MONSTER (up to 60 million samples, 50K timesteps, 82 classes) and the UEA benchmark, against 15 baselines, MS4 and MS4N consistently outperform Mamba-based models while remaining more efficient, and MS4N matches or surpasses competing deep learning models that are roughly 2x and 10x larger in parameters. These results position lightweight structured SSMs as a compelling alternative to scaling complexity for TSC."},"title":{"value":"A Simple State Space Model Excels at Multivariate Time Series Classification"},"authors":{"value":[{"fullname":"Hassan Saadatmand","username":""},{"fullname":"Geoffrey I. Webb","username":""},{"fullname":"Hamid Rezatofighi","username":"~Hamid_Rezatofighi1"},{"fullname":"Mahsa Salehi","username":""}]}},"tmdate":1792096623500,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2605-27406"],"tcdate":1792096612178,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Hamid_Rezatofighi1"],"forum":"p9x3vK6Kaq","license":"CC BY-SA 4.0","number":190425,"cdate":1777593600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1792096623500,"domain":"OpenReview.net/Public_Article","id":"p9x3vK6Kaq","version":2},{"content":{"TLDR":{"value":"We describe the dynamical properties of tokens in a deep selective state-space model, and based on that, we suggest improvements to the model by excluding negatively impactful scenarios and reordering tokens using importance scores."},"venue":{"value":"ICLR 2025 DeLTa Workshop Poster"},"keywords":{"value":["Selective state-space model","continuous-time limit","dynamical system","asymptotic behavior","token reordering"]},"abstract":{"value":"Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity.  We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance.  For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training.  Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance.  Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications."},"_bibtex":{"value":"@inproceedings{\nvo2025demystifying,\ntitle={Demystifying the Token Dynamics of Deep Selective State Space Models},\nauthor={Thieu Vo and Duy-Tung Pham and Xin T. Tong and Tan Minh Nguyen},\nbooktitle={ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy},\nyear={2025},\nurl={https://openreview.net/forum?id=UbNC96K6yL}\n}"},"title":{"value":"Demystifying the Token Dynamics of Deep Selective State Space Models"},"pdf":{"value":"/pdf/c11a52dfad394fa278e8f99f73bb9f2950584f6f.pdf"},"venueid":{"value":"ICLR.cc/2025/Workshop/DeLTa"},"paperhash":{"value":"vo|demystifying_the_token_dynamics_of_deep_selective_state_space_models"},"authorids":{"value":["~Thieu_Vo1","~Duy-Tung_Pham1","~Xin_T._Tong1","~Tan_Minh_Nguyen1"]},"Track":{"value":"long paper (up to 8 pages)"},"authors":{"value":["Thieu Vo","Duy-Tung Pham","Xin T. Tong","Tan Minh Nguyen"]}},"tmdate":1744628014269,"pdate":1741244033818,"tcdate":1738832662194,"writers":["ICLR.cc/2025/Workshop/DeLTa","ICLR.cc/2025/Workshop/DeLTa/Submission47/Authors"],"signatures":["ICLR.cc/2025/Workshop/DeLTa/Submission47/Authors"],"forum":"UbNC96K6yL","license":"CC BY 4.0","number":47,"cdate":1738832662194,"readers":["everyone"],"invitations":["ICLR.cc/2025/Workshop/DeLTa/-/Submission","ICLR.cc/2025/Workshop/DeLTa/-/Post_Submission","ICLR.cc/2025/Workshop/DeLTa/-/Edit","ICLR.cc/2025/Workshop/DeLTa/Submission47/-/Camera_Ready"],"mdate":1744628014269,"odate":1741949086590,"domain":"ICLR.cc/2025/Workshop/DeLTa","id":"UbNC96K6yL","version":2},{"content":{"TLDR":{"value":"We describe the dynamical properties of tokens in a deep selective state-space model, and based on that, we suggest improvements to the model by excluding negatively impactful scenarios and reordering tokens using importance scores."},"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Selective state-space model","continuous-time limit","dynamical system","asymptotic behavior","token reordering"]},"supplementary_material":{"value":"/attachment/f01b2b77a31c531086f0fefd1143d89ad8415a39.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity.  We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance.  For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training.  Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance.  Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications."},"_bibtex":{"value":"@inproceedings{\nvo2025demystifying,\ntitle={Demystifying the Token Dynamics of Deep Selective State Space Models},\nauthor={Thieu Vo and Duy-Tung Pham and Xin T. Tong and Tan Minh Nguyen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=qtTIP5Gjc5}\n}"},"title":{"value":"Demystifying the Token Dynamics of Deep Selective State Space Models"},"pdf":{"value":"/pdf/abc1efe1f910ee48b743f7ba26000b1d5e622f85.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"vo|demystifying_the_token_dynamics_of_deep_selective_state_space_models"},"authorids":{"value":["~Thieu_Vo1","~Duy-Tung_Pham1","~Xin_T._Tong2","~Tan_Minh_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Thieu Vo","Duy-Tung Pham","Xin T. Tong","Tan Minh Nguyen"]}},"tmdate":1741966064962,"pdate":1737562415446,"tcdate":1727318399527,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5608/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5608/Authors"],"forum":"qtTIP5Gjc5","license":"CC BY 4.0","number":5608,"cdate":1727318399527,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission5608/-/Full_Submission","ICLR.cc/2025/Conference/Submission5608/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit","ICLR.cc/2025/Conference/Submission5608/-/Camera_Ready_Revision"],"mdate":1741966064962,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"qtTIP5Gjc5","version":2},{"content":{"comment":{"value":"We appreciate Reviewer 53Ds's for the positive initial assessment for our work. Regarding your concerns on novelty and experiment settings, please see our responses to your questions:\n\n**1. The \"exponential memory\" results are not novel or surprising.**\n\nWhile the concept of \"exponential memory\" may seem intuitive to many researchers in this field, we emphasize that our work is the first to theoretically demonstrate this property in the context of **input-dependent mechanisms** such as selection [1], contextualized gating [2], and input-dependent convolution [3], which are integral to modern SSMs. Existing works [3, 4, 5, 6] provide valuable insights into SSMs from various perspectives. However, the results in [4] are limited to **linear SSMs** (e.g., S4), considering nonlinearity only after applying the linear SSMs. The method proposed in [4] does not naturally extend to this scenario, as it assumes linearity in the sequence mixer (cf. Eq. 9 in [4]). Please refer to Appendix B of our revision for a detailed comparison. [3] and [5] primarily analyze the limitations of SSMs using specific synthetic tasks, such as copying and associative recall, which do not directly expose the long-range dependencies among tokens. [6] establishes a strong benchmark for modern SSMs. Our theoretical results further elucidate these limitations. Additionally, our long-context retrieval experiments extend these findings to scaled-up industrial-level models, demonstrating that even with scaling-up, the long-range limitations of SSMs persist. Furthermore, we identify a novel security vulnerability in SSMs, which can lead to easier jailbreaks in LLMs (see additional discussions in Sec. 3.3 of our revision). To cover these points, we have added a new related work section in our revision.\n\n\n[1] Gu & Dao, Mamba: Linear-time sequence modeling with selective state spaces\n\n[2] Yang et al. Gated linear attention transformers with hardware-efficient training\n\n\n[3] Arora et al. Zoology: Measuring and improving recall in efficient language models\n\n[4] Wang & Xue, State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory\n\n[5] Jelassi et al., Repeat After Me: Transformers are Better than State Space Models at Copying\n\n[6] Waleffe et al., An Empirical Study of Mamba-based Language Models\n\n**2. Exponential memory does not imply loss in perplexity and length generalization.**\n\nWe agree with Reviewer 53Ds that exponential memory does not necessarily lead to a loss in perplexity and can even aid in length generalization, as demonstrated by xLSTM [1] and Alibi [2]. As discussed in Sec. 3.1, recency serves as a near-optimal bias for language modeling, meaning its imposition does not negatively impact perplexity.\n\nHowever, we wish to emphasize that a key contribution of our work is showing that, despite not affecting perplexity, this decaying design has notable impacts on many downstream tasks. These issues include, but are not limited to, substantial loss of long-distance context and more severe security vulnerabilities (refer to our responses to (3)). This observation also underscores that validation perplexity may not fully reflect all aspects of a model's performance and is often an inadequate metric for evaluating long-range dependencies.\n\n[1] Pöppel et. al., xLSTM: Extended long short-term memory\n\n[2] Press et. al., Train short, test long: Attention with linear biases enables input length extrapolation"}},"tmdate":1732272409810,"tcdate":1732272409810,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1195/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission1195/Authors"],"forum":"pymXpl4qvi","number":2,"license":"CC BY 4.0","cdate":1732272409810,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1195/-/Official_Comment"],"mdate":1732272409810,"domain":"ICLR.cc/2025/Conference","replyto":"5Fa23p4lSv","id":"XsrnvdpHEl","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"We theoretically and empirically reveal and tackle locality and over-smoothing bottlenecks for state space models."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["State Space Models","Long-context LLMs","Over-smoothing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Structured State Space Models (SSMs) have emerged as alternatives to transformers.\nWhile SSMs are often regarded as effective in capturing long-sequence dependencies, we rigorously demonstrate that they are inherently limited by strong recency bias.\nOur empirical studies also reveal that this bias impairs the models' ability to recall distant information and introduces robustness issues. Our scaling experiments then discovered that deeper structures in SSMs can facilitate the learning of long contexts.\nHowever, subsequent theoretical analysis reveals that as SSMs increase in depth, they exhibit another inevitable tendency toward over-smoothing, e.g., token representations becoming increasingly indistinguishable.\nThis *fundamental dilemma* between recency and over-smoothing hinders the scalability of existing SSMs. \nInspired by our theoretical findings, we propose to *polarize* two channels of the state transition matrices in SSMs, setting them to zero and one, respectively, simultaneously addressing recency bias and over-smoothing.\nExperiments demonstrate that our polarization technique consistently enhances the associative recall accuracy of long-range tokens and unlocks SSMs to benefit further from deeper architectures.\nAll source codes are released at https://github.com/VITA-Group/SSM-Bottleneck."},"_bibtex":{"value":"@inproceedings{\nwang2025understanding,\ntitle={Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing},\nauthor={Peihao Wang and Ruisi Cai and Yuehao Wang and Jiajun Zhu and Pragya Srivastava and Zhangyang Wang and Pan Li},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=pymXpl4qvi}\n}"},"title":{"value":"Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing"},"pdf":{"value":"/pdf/982acd7979eff95835985d1c18fb057152350bf1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wang|understanding_and_mitigating_bottlenecks_of_state_space_models_through_the_lens_of_recency_and_oversmoothing"},"authorids":{"value":["~Peihao_Wang1","~Ruisi_Cai1","~Yuehao_Wang1","~Jiajun_Zhu1","~Pragya_Srivastava1","~Zhangyang_Wang1","~Pan_Li2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Peihao Wang","Ruisi Cai","Yuehao Wang","Jiajun Zhu","Pragya Srivastava","Zhangyang Wang","Pan Li"]}},"version":2},{"content":{"summary":{"value":"The paper introduces LongMamba, a method to extend the context length of Mamba models without additional training, focusing on long-context understanding for state-space models. Mamba models, though efficient in handling long contexts due to their linear complexity, struggle with context lengths far beyond training sequences due to exponential decay in the attention value, especially toward early tokens. LongMamba addresses this by categorizing channels into local and global based on receptive field lengths, then enlarging global channels' receptive fields via adaptive decay adjustments through token filtering. This modification allows LongMamba to maintain performance over long contexts without the computational cost of fine-tuning, making it a practical improvement over previous methods like DeciMamba."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. Could the authors please address the points mentioned in the weaknesses section?\n\n2. Is a similar exponential decay observed in the Mamba2[2] model? If so, does LongMamba improve the long-context understanding in this architecture as well?\n\n3. Would the proposed approach generalize to other SSMs beyond Mamba, including hybrid Transformer/SSM architectures, or is it specifically tailored to Mamba’s architectural nuances?\n\n4. LongBench is a comprehensive benchmark. How does LongMamba perform on other LongBench tasks, namely 2wikimqa, TREC, TriviaQA, LCC, and RepoBench-P?\n\n5. Have the authors considered alternative strategies for decay adjustment besides token filtering? If so, why was filtering chosen over other methods? \n\nReferences:\n\n[1]Ben-Kish, Assaf, et al. \"DeciMamba: Exploring the Length Extrapolation Potential of Mamba.\" arXiv preprint arXiv:2406.14528 (2024).\n\n[2]Dao, Tri, and Albert Gu. \"Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality.\" arXiv preprint arXiv:2405.21060 (2024)."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":2},"strengths":{"value":"1 - The paper addresses a critical challenge in long-context modeling, particularly for SSMs like Mamba, which suffer from limited long-sequence capabilities despite their efficient state-space mechanics. Enhancing this capability without retraining aligns well with real-world demands for scalable, efficient language models.\n\n\n2 - Devising a “training-free” context extension technique is an important aspect of the paper, as directly training the SSMs on longer context requires significant compute/resource and expansive datasets. \n\n\n3 - Extensive benchmarks show that LongMamba consistently outperforms vanilla Mamba on long-context tasks, including language modeling on the PG-19 dataset and passkey/document retrieval. LongMamba’s results in real-world tasks, such as LongBench, also showcase its adaptability"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1- A substantial limitation is that LongMamba relies heavily on the DeciMamba[1] framework, raising questions about its independent novelty. Although LongMamba innovatively extends global channels, the reliance on DeciMamba's token handling might limit its perceived originality. In particular, the idea of token filtering based on the averaged delta_t value was proposed by DeciMamba. LongMamba merely improves upon this idea by limiting the scope of this averaging to some specific channels (global channels).\n\n2 -The paper does not systematically address threshold selection for distinguishing global versus local channels. This selection is left to empirical values, which may lack generalizability across different datasets or models. A more formal approach to tuning this threshold would strengthen LongMamba's applicability. This is of importance, as global and local channels are integral parts of the LongMamba framework. \n\n\n3 - The method for handling global channels’ decay distribution adjustment is not fully elaborated. It remains ambiguous whether averaging across global channels fully mitigates decay issues over extended contexts, potentially affecting LongMamba’s consistency in varied contexts (Figure 2c, Decay Distribution Adjustment)."}},"nonreaders":[],"tmdate":1731428409691,"tcdate":1730690257875,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13423/Reviewer_xMy1"],"signatures":["ICLR.cc/2025/Conference/Submission13423/Reviewer_xMy1"],"forum":"fMbLszVO1H","number":3,"license":"CC BY 4.0","cdate":1730690257875,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13423/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428409691,"domain":"ICLR.cc/2025/Conference","replyto":"fMbLszVO1H","id":"rSfRrR36LI","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Models","State Space Models","Long Context Understanding"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State space models (SSMs) have emerged as an efficient alternative to Transformer models for language modeling, offering linear computational complexity and constant memory usage as context length increases. However, despite their efficiency in handling long contexts, recent studies have shown that SSMs, such as Mamba models, generally underperform compared to Transformers in long-context understanding tasks. To address this significant shortfall and achieve both efficient and accurate long-context understanding, we propose LongMamba, a training-free technique that significantly enhances the long-context capabilities of Mamba models. LongMamba builds on our discovery that the hidden channels in Mamba can be categorized into local and global channels based on their receptive field lengths, with global channels primarily responsible for long-context capability. These global channels can become the key bottleneck as the input context lengthens. Specifically, when input lengths largely exceed the training sequence length, global channels exhibit limitations in adaptively extend their receptive fields, leading to Mamba’s poor long-context performance. The key idea of LongMamba is to mitigate the hidden state memory decay in these global channels by preventing the accumulation of unimportant tokens in their memory. This is achieved by first identifying critical tokens in the global channels and then applying token filtering to accumulate only those critical tokens. Through extensive benchmarking across synthetic and real-world long-context scenarios, LongMamba sets a new standard for Mamba’s long-context performance, significantly extending its operational range without requiring additional training. Our code is available at https://github.com/GATECH-EIC/LongMamba."},"_bibtex":{"value":"@inproceedings{\nye2025longmamba,\ntitle={LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement},\nauthor={Zhifan Ye and Kejing Xia and Yonggan Fu and Xin Dong and Jihoon Hong and Xiangchi Yuan and Shizhe Diao and Jan Kautz and Pavlo Molchanov and Yingyan Celine Lin},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=fMbLszVO1H}\n}"},"title":{"value":"LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement"},"pdf":{"value":"/pdf/066062c0673cfc50aeafa9e3c96d0cf7a33fc3ca.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"ye|longmamba_enhancing_mambas_longcontext_capabilities_via_trainingfree_receptive_field_enlargement"},"authorids":{"value":["~Zhifan_Ye1","~Kejing_Xia1","~Yonggan_Fu1","~Xin_Dong1","~Jihoon_Hong1","~Xiangchi_Yuan1","~Shizhe_Diao2","~Jan_Kautz1","~Pavlo_Molchanov1","~Yingyan_Celine_Lin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zhifan Ye","Kejing Xia","Yonggan Fu","Xin Dong","Jihoon Hong","Xiangchi Yuan","Shizhe Diao","Jan Kautz","Pavlo Molchanov","Yingyan Celine Lin"]}},"version":2},{"content":{"venue":{"value":"CVPR 2025"},"pdf":{"value":"https://openaccess.thecvf.com/content/CVPR2025/papers/Xiao_Event-based_Video_Super-Resolution_via_State_Space_Models_CVPR_2025_paper.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"xiao|eventbased_video_superresolution_via_state_space_models"},"html":{"value":"https://openaccess.thecvf.com/content/CVPR2025/html/Xiao_Event-based_Video_Super-Resolution_via_State_Space_Models_CVPR_2025_paper.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/cvpr/XiaoW25,\n  author={Zeyu Xiao and Xinchao Wang},\n  title={Event-based Video Super-Resolution via State Space Models},\n  year={2025},\n  cdate={1735689600000},\n  pages={12564-12574},\n  url={https://openaccess.thecvf.com/content/CVPR2025/html/Xiao_Event-based_Video_Super-Resolution_via_State_Space_Models_CVPR_2025_paper.html},\n  booktitle={CVPR},\n  crossref={conf/cvpr/2025}\n}\n"},"abstract":{"value":"Exploiting temporal correlations is crucial for video super-resolution (VSR). Recent approaches enhance this by incorporating event cameras. In this paper, we introduce MamEVSR, a Mamba-based network for event-based VSR that leverages the selective state space model, Mamba. MamEVSR stands out by offering global receptive field coverage with linear computational complexity, thus addressing the limitations of convolutional neural networks and Transformers. The key components of MamEVSR include: (1) The interleaved Mamba (iMamba) block, which interleaves tokens from adjacent frames and applies multidirectional selective state space modeling, enabling efficient feature fusion and propagation across bi-directional frames while maintaining linear complexity. (2) The crossmodality Mamba (cMamba) block facilitates further interaction and aggregation between event information and the output from the iMamba block. The cMamba block can leverage complementary spatio-temporal information from both modalities and allows MamEVSR to capture finermotion details. Experimental results show that the proposed MamEVSR achieves superior performance on various datasets quantitatively and qualitatively."},"title":{"value":"Event-based Video Super-Resolution via State Space Models"},"authors":{"value":[{"fullname":"Zeyu Xiao","username":""},{"fullname":"Xinchao Wang","username":"~Xinchao_Wang1"}]}},"tmdate":1792053897041,"pdate":1767139200000,"externalIds":["dblp:conf/cvpr/XiaoW25"],"tcdate":1792053888609,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Xinchao_WANG3"],"forum":"mS3eS5oOLD","license":"CC BY-SA 4.0","number":188497,"cdate":1735689600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1792053897041,"domain":"OpenReview.net/Public_Article","id":"mS3eS5oOLD","version":2},{"content":{"summary":{"value":"The authors propose a linear attention model called RetNet for language modeling, which has a linear training complexity and constant inference complexity."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"Can you also add the results on MMLU and Qasper with the standard metrics besides perplexity?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. RetNet has both linear time complexity and constant inference memory complexity. \n2. RetNet has a chunk recurrent form which can be beneficial for speculative decoding."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The authors introduce a new term called \"Retention,\" but this is essentially the same as Linear Attention without the denominator, which has already been proposed in [1].\n2. Lack of comparison with the baselines on open source pretraining data. All the training experiments are conducted on in-house data mixtures, which harms the reproducibility.\n3. The paper doesn't compare RetNet with other linear attention model (such as GLA, RWKV, Mamba) on downstream tasks with standard metrics instead of perplexity. Table 2 only include RetNet and Transformer. The efficiency measurment of RetNet+ is absent.\n4. The evaluation on MMLU/Qasper is using perplexity but not the widely-used accuracy/F1 metric. The perplexity results don't necessarily mean that the model can make correct choices for the samples in MMLU, and has less guidance for the model's downstream performance.\n5. Missing citations: The authors should also cite [1] for the normalization after retention, and discuss the details of the triton implementation of RetNet and its difference from the implementation in the Flash Linear Attention [2] library.\n\n[1] Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7025–7041, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.\n\n[2] Yang, Songlin and Zhang, Yu. FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism. https://github.com/sustcsonglin/flash-linear-attention"},"limitations":{"value":"No, the authors should have a limitation section to point out the strong assumptions of their approximation of self-attention and relative position embedding."}},"nonreaders":[],"tmdate":1730878979491,"tcdate":1720865037296,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission5128/Reviewer_maEf"],"signatures":["NeurIPS.cc/2024/Conference/Submission5128/Reviewer_maEf"],"forum":"sxZlp9ZoHD","number":2,"license":"CC BY 4.0","cdate":1720865037296,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission5128/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878979491,"domain":"NeurIPS.cc/2024/Conference","replyto":"sxZlp9ZoHD","id":"7KujjLarMR","forumContent":{"TLDR":{"value":"RetNet is a foundation architecture, simultaneously achieving training parallelism, low-cost inference, and good performance."},"venue":{"value":"Submitted to NeurIPS 2024"},"keywords":{"value":["Retentive Network","Model Architecture"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost \n inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference."},"_bibtex":{"value":"@misc{\nanonymous2024retentive,\ntitle={Retentive Network},\nauthor={Anonymous},\nyear={2024},\nurl={https://openreview.net/forum?id=sxZlp9ZoHD}\n}"},"title":{"value":"Retentive Network"},"pdf":{"value":"/pdf/39f86c5d412e6438649afe72632de944694a9a32.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"sun|retentive_network"},"authorids":{"value":["~Yutao_Sun1","~Li_Dong1","~Shaohan_Huang1","~Shuming_Ma1","~Yuqing_Xia1","~Jilong_Xue2","~Jianyong_Wang2","~Furu_Wei1"]},"authors":{"value":["Yutao Sun","Li Dong","Shaohan Huang","Shuming Ma","Yuqing Xia","Jilong Xue","Jianyong Wang","Furu Wei"]}},"version":2},{"content":{"venue":{"value":"ACML 2024"},"pdf":{"value":"https://raw.githubusercontent.com/mlresearch/v260/main/assets/zhai25a/zhai25a.pdf"},"venueid":{"value":"dblp.org/conf/ACML/2024"},"paperhash":{"value":"zhai|colormamba_towards_highquality_nirtorgb_spectral_translation_with_mamba"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Huiyu_Zhai:","https://dblp.org/search/pid/api?q=author:Guang_Jin:","~Xingxing_Yang1","https://dblp.org/search/pid/api?q=author:Guosheng_Kang:"]},"html":{"value":"https://proceedings.mlr.press/v260/zhai25a.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/acml/ZhaiJ0K24,\n  author={Huiyu Zhai and Guang Jin and Xingxing Yang and Guosheng Kang},\n  title={ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mamba},\n  year={2024},\n  cdate={1704067200000},\n  pages={765-780},\n  url={https://proceedings.mlr.press/v260/zhai25a.html},\n  booktitle={ACML},\n  crossref={conf/acml/2024}\n}\n"},"abstract":{"value":"Translating NIR to the visible spectrum is challenging due to cross-domain complexities. Current models struggle to balance a broad receptive field with computational efficiency, limiting practical use. Although the Selective Structured State Space Model, especially the improved version, Mamba, excels in generative tasks by capturing long-range dependencies with linear complexity, its default approach of converting 2D images into 1D sequences neglects local context. In this work, we propose a simple but effective backbone, dubbed ColorMamba, which first introduces Mamba into spectral translation tasks. To explore global long-range dependencies and local context for efficient spectral translation, we introduce learnable padding tokens to enhance the distinction of image boundaries and prevent potential confusion within the sequence model. Furthermore, local convolutional enhancement and agent attention are designed to improve the vanilla Mamba. Moreover, we exploit the HSV color to provide multi-scale guidance in the reconstruction process for more accurate spectral translation. Extensive experiments show that our ColorMamba achieves a 1.02 improvement in terms of PSNR compared with the state-of-the-art method. Our code is available at https://github.com/AlexYangxx/ColorMamba/."},"title":{"value":"ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mamba"},"authors":{"value":["Huiyu Zhai","Guang Jin","Xingxing Yang","Guosheng Kang"]}},"tmdate":1750046242697,"pdate":1704067200000,"tcdate":1750046238008,"writers":["~"],"signatures":["~Xingxing_Yang1"],"forum":"Rx3XqkyxV7","license":"CC BY-SA 4.0","number":562882,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1750046242697,"domain":"DBLP.org","id":"Rx3XqkyxV7","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.08087v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"zhai|colormamba_towards_highquality_nirtorgb_spectral_translation_with_mamba"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Huiyu_Zhai:","https://dblp.org/search/pid/api?q=author:Guang_Jin:","~Xingxing_Yang1","https://dblp.org/search/pid/api?q=author:Guosheng_Kang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.08087"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-08087,\n  publtype={informal},\n  author={Huiyu Zhai and Guang Jin and Xingxing Yang and Guosheng Kang},\n  title={ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.08087},\n  url={https://doi.org/10.48550/arXiv.2408.08087}\n}\n"},"abstract":{"value":"Translating NIR to the visible spectrum is challenging due to cross-domain complexities. Current models struggle to balance a broad receptive field with computational efficiency, limiting practical use. Although the Selective Structured State Space Model, especially the improved version, Mamba, excels in generative tasks by capturing long-range dependencies with linear complexity, its default approach of converting 2D images into 1D sequences neglects local context. In this work, we propose a simple but effective backbone, dubbed ColorMamba, which first introduces Mamba into spectral translation tasks. To explore global long-range dependencies and local context for efficient spectral translation, we introduce learnable padding tokens to enhance the distinction of image boundaries and prevent potential confusion within the sequence model. Furthermore, local convolutional enhancement and agent attention are designed to improve the vanilla Mamba. Moreover, we exploit the HSV color to provide multi-scale guidance in the reconstruction process for more accurate spectral translation. Extensive experiments show that our ColorMamba achieves a 1.02 improvement in terms of PSNR compared with the state-of-the-art method. Our code is available at https://github.com/AlexYangxx/ColorMamba."},"title":{"value":"ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mamba"},"authors":{"value":["Huiyu Zhai","Guang Jin","Xingxing Yang","Guosheng Kang"]}},"tmdate":1733985224840,"pdate":1704067200000,"tcdate":1733985220640,"writers":["~"],"signatures":["~Xingxing_Yang1"],"forum":"jdVc7xP7b7","license":"CC BY-SA 4.0","number":255874,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1733985224840,"domain":"DBLP.org","id":"jdVc7xP7b7","version":2},{"content":{"verify_author_list":{"value":"I have double-checked the author list and understand that additions and removals will not be allowed after the submission deadline."},"venue":{"value":"ACML 2024 Conference Track"},"student_author":{"value":"Yes"},"a_signed_permission_to_publish_form_in_pdf":{"value":"/attachment/47ef4279b955cc3c0aa4c2d3cc3dc5f7ad846041.pdf"},"keywords":{"value":["NIR Image","Spectral Translation","Colorization","Mamba","State Space Model"]},"supplementary_material":{"value":"/attachment/1b1d1828454a18c3e601a757bd9590b5fda74586.zip"},"primary_area":{"value":"Applications (bioinformatics, biomedical informatics, climate science, collaborative filtering, computer vision, healthcare, human activity recognition, information retrieval, natural language processing, social networks, etc.)"},"abstract":{"value":"Translating NIR to the visible spectrum is challenging due to cross-domain complexities. Current models struggle to balance a broad receptive field with computational efficiency, limiting practical use. Although the Selective Structured State Space Model, especially the improved version, Mamba, excels in generative tasks by capturing long-range dependencies with linear complexity, its default approach of converting 2D images into 1D sequences neglects local context. In this work, we propose a simple but effective backbone, dubbed ColorMamba, which first introduces Mamba into spectral translation tasks. To explore global long-range dependencies and local context for efficient spectral translation, we introduce learnable padding tokens to enhance the distinction of image boundaries and prevent potential confusion within the sequence model. Furthermore, local convolutional enhancement and agent attention are designed to improve the vanilla Mamba. Moreover, we exploit the HSV color to provide multi-scale guidance in the reconstruction process for more accurate spectral translation. Extensive experiments show that our ColorMamba achieves a 1.02 improvement in terms of PSNR compared with the state-of-the-art method. Our code is available at https://github.com/AlexYangxx/ColorMamba/."},"_bibtex":{"value":"@inproceedings{\nzhai2024colormamba,\ntitle={ColorMamba: Towards High-quality {NIR}-to-{RGB} Spectral Translation with Mamba},\nauthor={Huiyu Zhai and Guang Jin and Xingxing Yang and Guosheng Kang},\nbooktitle={The 16th Asian Conference on Machine Learning (Conference Track)},\nyear={2024},\nurl={https://openreview.net/forum?id=VZkiOns3rE}\n}"},"title":{"value":"ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mamba"},"pdf":{"value":"/pdf/ebad9531b63628aac20a124a255fc2ba4b07099d.pdf"},"venueid":{"value":"ACML.org/2024/Conference"},"paperhash":{"value":"zhai|colormamba_towards_highquality_nirtorgb_spectral_translation_with_mamba"},"authorids":{"value":["~Huiyu_Zhai1","~Guang_Jin2","~Xingxing_Yang1","~Guosheng_Kang1"]},"paper_checklist_guidelines":{"value":"I certify that all co-authors of this work have read and commit to adhering to the guidelines in Call for Papers."},"authors":{"value":["Huiyu Zhai","Guang Jin","Xingxing Yang","Guosheng Kang"]}},"tmdate":1729061724765,"pdate":1725538669494,"tcdate":1719567727211,"writers":["ACML.org/2024/Conference","ACML.org/2024/Conference/Submission236/Authors"],"signatures":["ACML.org/2024/Conference/Submission236/Authors"],"forum":"VZkiOns3rE","license":"CC BY 4.0","number":236,"cdate":1719567727211,"readers":["everyone"],"invitations":["ACML.org/2024/Conference/-/Submission","ACML.org/2024/Conference/-/Post_Submission","ACML.org/2024/Conference/-/Edit","ACML.org/2024/Conference/Submission236/-/Revision"],"mdate":1729061724765,"odate":1729061724750,"domain":"ACML.org/2024/Conference","id":"VZkiOns3rE","version":2},{"content":{"summary":{"value":"This paper proposes LOTFormer, a novel attention mechanism that achieves both linear-time complexity and doubly-stochastic normalization by reformulating attention as a low-rank optimal transport (OT) problem. The key idea is to introduce a learnable pivot measure with a small support size , which acts as an intermediate latent distribution connecting queries and keys. Two entropic OT problems are solved and composed via a glued coupling, yielding a provably doubly-stochastic and low-rank attention matrix.\nEmpirically, LOTFormer achieves state-of-the-art results on the Long Range Arena (LRA) benchmark and competitive performance on ImageNet-1K and IWSLT’14 translation tasks, outperforming prior linear and transport-based attention methods."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"**Q1. Causal Mask Compatibility:**\nThe proposed method enforces doubly-stochastic normalization, assuming balanced bidirectional attention. How would this formulation behave under causal masking, where attention is asymmetric and upper-triangular? Do the theoretical guarantees still hold in that setting, or would the normalization need to be modified? Additionally, how is the decoder implemented in Section 4.4 under this formulation?\n\n**Q2. LOTFormer with Pola:**\nConsidering that the polarity decomposition in PolaFormer [1] was originally proposed to mitigate the loss of negative qk values in linear attention with ReLU feature maps, the motivation for adopting the Pola mechanism in LOTFormer appears unclear. Could the authors elaborate on why polarity decomposition is necessary in this framework, and why a simpler global Pola + Softmax formulation would not suffice?\n\n**Q3. Runtime and Memory Efficiency:**\nCould the authors provide runtime and GPU memory benchmarks to empirically validate the claimed linear-time complexity?\n\n**Q4. More Extensive Experiments:**\nIt would strengthen the paper to include evaluations on longer-sequence benchmarks and larger backbones (e.g., DeiT-B, ViT-B, or GPT-like architectures with ≥340 M parameters) to better demonstrate scalability.\n\n**Suggestion – Convergence and Stability of Sinkhorn Iterations:**\nThe paper relies on Sinkhorn iterations to compute entropic OT couplings, yet a more detailed analysis of their convergence behavior and numerical stability would be valuable. Since Sinkhorn iterations can be sensitive to the regularization parameter ε and may suffer from instability in low-precision or deep-training settings, including empirical convergence diagnostics or stability experiments would help clarify the robustness of the proposed approach.\n\nThis is an interesting work, and I would consider raising my score if the authors adequately address the concerns above.\n\n[1] Meng, Weikang, et al. \"PolaFormer: Polarity-aware Linear Attention for Vision Transformers.\" The Thirteenth International Conference on Learning Representations."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"**Theoretical Foundation**. The paper provides a principled derivation of attention as an entropic optimal transport plan. The proposed “glued coupling” formulation ensures provable double stochasticity and rank <=r. The mathematical connection between transport plans and normalized attention is clearly established.\n\n**Complexity Efficiency**. Achieves linear complexity, O(nr), without constructing the full n×n matrix.\nSinkhorn iterations are applied in factorized form, which improves scalability.\n\n**Empirical Effectiveness**. LOTFormer sets new state-of-the-art results on LRA with the best average accuracy. On ImageNet-1K , it surpasses the baseline with negligible overhead. Consistently outperforms the baseline in the machine translation task.\n\n**Compatibility and Extensibility**. The method can be combined wit existing improvements, such as polarity decomposition (Pol-[cls]). Conceptually unifies linear attention and transport-based normalization.\n\n**Clear Visualization and writing**. This paper plot sufficient visualizations and present clearly."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Missing Speed and Memory Benchmarks**.\nThe paper lacks quantitative comparisons of training/inference speed and memory usage against other baselines, which linear attention is proposed to solve. Although theoretical complexity is discussed, practical runtime and memory improvements are not empirically demonstrated. Without these results, claims of linear-time scalability remain partly unverified.\n\n**Inadequate Long-Sequence Evaluation**.\nEmpirically, the performance of linear attention will degrade as the sequence length increasing comparing with softmax attention, which is the main issue recently. Only LRA benchmark is not convincing because it tests the model under only two transformer layers.\n\n**Limited Model Size and Scope**.\nThe experiments are conducted on small-scale backbones and there is no evaluation on large-scale or high-capacity models (Base/Large in CV or 1B+ LLM), which limits the evidence of scalability. This makes it unclear whether the proposed method scales favorably in real-world high-throughput settings."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919862259,"tcdate":1761623222902,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7817/Reviewer_iMvN"],"signatures":["ICLR.cc/2026/Conference/Submission7817/Reviewer_iMvN"],"forum":"JFAIchgiZr","number":2,"license":"CC BY 4.0","cdate":1761623222902,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7817/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919862259,"domain":"ICLR.cc/2026/Conference","replyto":"JFAIchgiZr","id":"kPGEypfVL1","forumContent":{"TLDR":{"value":"LOTFormer is a linear-time, doubly-stochastic attention mechanism that uses a learnable low-rank pivot transport plan to achieve state-of-the-art accuracy and efficiency on long-sequence tasks."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Linear Attention","Doubly-Stochastic Attention","Transformers","Optimal Transport"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Transformers have proven highly effective across a wide range of modalities. However, the quadratic complexity of the standard softmax attention mechanism poses a fundamental barrier to scaling them to long context windows. A large body of work addresses this with linear attention, which reformulates attention as a kernel function and approximates it with finite feature maps to achieve linear-time computation. Orthogonal to computational scaling, most attention mechanisms—both quadratic and linear—produce row-normalized maps that can over-focus on a few tokens, degrading robustness and information flow. Enforcing doubly-stochastic attention alleviates this by balancing token participation across rows and columns, but existing doubly-stochastic attention mechanisms typically introduce substantial overhead, undermining scalability. We propose LOTFormer, a principled attention mechanism that is simultaneously linear-time and doubly-stochastic. Our approach exploits the connection between attention maps and transportation plans between query and key measures. The central idea is to constrain the transport plan to be low-rank by conditioning it on a learnable pivot measure with small support. Concretely, we solve two entropic optimal transport problems (queries$\\to$pivot and pivot$\\to$keys) and compose them into a conditional (glued) coupling. This yields an attention matrix that is provably doubly-stochastic, has rank at most $r \\ll n$, and applies to values in $O(nr)$ time without forming the full $n\\times n$ map. The pivot locations and masses are learned end-to-end. Empirically, LOTFormer achieves state-of-the-art results on the Long Range Arena benchmark, surpassing prior linear and transport-based attention methods in both accuracy and efficiency."},"_bibtex":{"value":"@misc{\nshahbazi2026lotformer,\ntitle={{LOTF}ormer: Doubly-Stochastic Linear Attention via Low-Rank Optimal Transport},\nauthor={Ashkan Shahbazi and Chayne Thrash and Yikun Bai and Keaton Hamm and Navid NaderiAlizadeh and Soheil Kolouri},\nyear={2026},\nurl={https://openreview.net/forum?id=JFAIchgiZr}\n}"},"title":{"value":"LOTFormer: Doubly-Stochastic Linear Attention via Low-Rank Optimal Transport"},"pdf":{"value":"/pdf/87223ea9635b417df3134c73708a84570ffb10fb.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"shahbazi|lotformer_doublystochastic_linear_attention_via_lowrank_optimal_transport"},"authorids":{"value":["~Ashkan_Shahbazi1","~Chayne_Thrash1","~Yikun_Bai2","~Keaton_Hamm1","~Navid_NaderiAlizadeh2","~Soheil_Kolouri1"]},"authors":{"value":["Ashkan Shahbazi","Chayne Thrash","Yikun Bai","Keaton Hamm","Navid NaderiAlizadeh","Soheil Kolouri"]}},"version":2},{"content":{"summary":{"value":"The authors study In-Context-Learning (ICL) abilities of a broad and general class of recent sequence-to-sequence architectures, such as Mamba, RWKV, and Gated Linear Attention, which they collectively name GLA. They show that the models of this algorithm family with L layers can perform L steps of Weighted Projected Gradient Descent (WPGD) for the task of linear regression during forward pass when presented with several examples. Moreover, the authors examine the setting when multi-task ICL examples are drawn from different distributions and correlated with the target sample. They find out, both empirically and theoretically, that for several weighting mechanisms,  it is possible to reach the optimal loss in this task."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"See weaknesses.\n\nAlso, there might be some typos or minor mistakes that I would consider clarifying or correcting:\n\nLine 158: $Z^\\top Z$ instead of $Z Z^\\top$ in formula for $\\hat{y}$.\n\nLine 161: It seems “predictor B” is associated with linear regression, not with linear attention. \n\nLine 162: Perhaps, the authors meant one step **of** gradient descent?\n\nLines 202, 771, 775 and other throughout the paper: the authors interchangeably use bold **0** both as vector and a scalar, occasionally in the same line, which is confusing."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. Authors encompass different and seemingly distinct model families such as Mamba and Linear Transformer into one analytical framework and study their ICL properties jointly which is a very convenient and refreshing paradigm.\n2. If the results stated in the paper are correct, it would be exciting and reassuring to know that a very broad class of models, beside Transformers, is provably capable of In-Context Learning despite some findings in the literature (e. g. [1])\n\n[1] Jelassi, Samy, et al. \"Repeat after me: Transformers are better than state space models at copying.\""},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Unfortunately, I was not able to fully verify the preconditions and mathematical derivations for the results in the paper. Below there are several examples of what I found confusing: \n\n1. Presentation is not self-contained, and it’s difficult to follow the arguments without prior reading of several referenced papers, notably Von Oswald et al. (2023), Ahn et al. (2024), Li et al. (2024), as well as  attempting to independently draw missing analogies and reductions between them and the reviewed paper.\n2. In the papers (Ahn et al. (2024), Li et al. (2024)) mentioned above, their authors research Preconditioned Gradient Descent. In the reviewed paper, the authors discuss (Weighted) Projected Gradient Descent, although it’s ostensibly implied in line 68 to be the same algorithm. It’s unclear whether it’s indeed the same algorithm, and what are the differences between them if it’s not. It’s worth noting that the terms “projected GD” and “preconditioned GD” refer to different algorithms, in case the authors use them interchangeably.\n3. If I understand correctly, the equation 7 states that parameter **$\\hat{\\beta}$** is the resulting predictor of data-dependent WPGD algorithm. **$\\hat{\\beta}$**  is constructed as a function of parameters $P_1, P_2$ and $\\Omega$. However, formal definition of the algorithm, its connection to the parameters, and derivation on how it optimizes linear regression task and arrives at solution **$\\hat{\\beta}$** conditioned on priors $P_1, P_2$ and $\\Omega$,  are not provided. \n4. Moreover, The theorem 1 states that  the output of single-layer GLA model with a specific construction of weights matrices parametrized by $P_1, P_2$ and $\\Omega$  matches the prediction of **$\\hat{\\beta}$** , also specifically constructed and parametrized by $P_1, P_2$ and $\\Omega$.  This joint construction does not shed light on whether the forward pass of 1-layer GLA performs one step of gradient descent or WGPD. It would be helpful to explicitly demonstrate and prove it as in e.g. Lemma 1 of Ahn et al. (2024).\n\nReferences:\n\nKwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.\n\nYingcong Li, Ankit Singh Rawat, and Samet Oymak. Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond. arXiv preprint arXiv:2407.10005, 2024.\n\nJohannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp. 35151–35174. PMLR, 2023."}},"nonreaders":[],"tmdate":1731429451068,"tcdate":1730685509243,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10591/Reviewer_N2zo"],"signatures":["ICLR.cc/2025/Conference/Submission10591/Reviewer_N2zo"],"forum":"AC9FsaVIpk","number":4,"license":"CC BY 4.0","cdate":1730685509243,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10591/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429451068,"domain":"ICLR.cc/2025/Conference","replyto":"AC9FsaVIpk","id":"shllNEjITs","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"This work offers theoretical insights into the ICL capabilities of gated linear attention models, demonstrates how gating is crucial for achieving stronger data adaptivity, and characterizes the loss landscape of weighted projected gradient descent."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["linear attention","gating","in-context learning","weighted gradient descent","optimization landscape"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Linear attention methods provide a strong alternative to softmax attention as they allow for efficient recurrent decoding. Recent research has focused on enhancing standard linear attention by incorporating gating while retaining its computational benefits. Such Gated Linear Attention (GLA) architectures include highly competitive models such as Mamba and RWKV. In this work, we examine the in-context learning capabilities of the GLA model and make the following contributions. We show that a multilayer GLA can implement a general class of Weighted Preconditioned Gradient Descent (WPGD) algorithms with data-dependent weights. These weights are induced by the gating and allows the model to control the contribution of individual tokens to prediction. To further understand the mechanics of weighting, we introduce a novel data model with multitask prompts and characterize the optimization landscape of the problem of learning a WPGD algorithm. We identify mild conditions under which there is a unique (global) minimum up to scaling invariance, and the associated WPGD algorithm is unique as well. Finally, we translate these findings to explore the optimization landscape of GLA and shed light on how gating facilitates context-aware learning and when it is provably better than vanilla linear attention."},"_bibtex":{"value":"@misc{\nli2025gating,\ntitle={Gating is Weighting: Understanding Gated Linear Attention through In-context Learning},\nauthor={Yingcong Li and Davoud Ataee Tarzanagh and Ankit Singh Rawat and Maryam Fazel and Samet Oymak},\nyear={2025},\nurl={https://openreview.net/forum?id=AC9FsaVIpk}\n}"},"title":{"value":"Gating is Weighting: Understanding Gated Linear Attention through In-context Learning"},"pdf":{"value":"/pdf/c48f63c30fac8116a961818257be2e4d313276f2.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"li|gating_is_weighting_understanding_gated_linear_attention_through_incontext_learning"},"authorids":{"value":["~Yingcong_Li1","~Davoud_Ataee_Tarzanagh1","~Ankit_Singh_Rawat1","~Maryam_Fazel1","~Samet_Oymak2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yingcong Li","Davoud Ataee Tarzanagh","Ankit Singh Rawat","Maryam Fazel","Samet Oymak"]}},"version":2},{"content":{"summary":{"value":"The paper proposes a DenseAttention Network (DANet), which addresses inefficiencies in the Transformer architecture, especially its high memory and computational cost - O(N^2), with respect to sequence length - N. DANet uses a new MaxNormActivation and Cosine Relative Positional Embeddings, capturing N x N interactions at O(N) space and time complexity. Experimental results demonstrate that DANet outperforms FlashAttention on long sequences on the Long Range Arena benchmark."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See weaknesses."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1) The paper proposes an interesting approach to eliminate projection matrices in attention, considering that the multiplication $W_QW_K^{\\top}$ can be replaced with a single parameter, which I don't think exists in previous literature.\n2) The paper also proposes using local attention in conjunction with the proposed attention function."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1) The paper lacks comparison against mamba in experiments. Mamba-I and Mamba-II are fast approaches for long range sequence modeling. \n2) This is not the first paper which captures NXN correlations with O(N) complexity. Linear attention [1] uses linear approximations of attention. A fair comparison with this paper would be great.\n3) The mathematical writing in this paper is inconsistent. Here are some instances:\n\n\nBetter Notation:\n1. Standard operators max, var should be mentioned in times new roman using \\DeclareMathOperator\n2. Defined operators such as MaxNormActivation can be put in \\text{MaxNormActivation}, as done in 200-204. \n3. Line 240: has a typo open bracket.\n4. Line 284: << should be \\ll.\n5. Line 246: why is fp16 and bf16 bolded?\n\nMajor readability issues:\n1. Inconsistent definition of $X_i$ in line 247, 300 and 311.\n\nIf the above issues are resolved I am willing to increase my score.\n\n[1] Katharopoulos, Angelos, et al. \"Transformers are rnns: Fast autoregressive transformers with linear attention.\" International conference on machine learning. PMLR, 2020."}},"nonreaders":[],"tmdate":1731429704416,"tcdate":1730700192667,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13608/Reviewer_HAtu"],"signatures":["ICLR.cc/2025/Conference/Submission13608/Reviewer_HAtu"],"forum":"2bIQBDSfRk","number":3,"license":"CC BY 4.0","cdate":1730700192667,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13608/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429704416,"domain":"ICLR.cc/2025/Conference","replyto":"2bIQBDSfRk","id":"Vo3IuaJ8w9","forumContent":{"TLDR":{"value":"We propose a novel DenseAttention architecture which achieves 1) favorable computational efficiency;  2) linear time & space complexity by simplification and reduction of standard Transformer architecture."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["self-attention","deep learning","transformer architecture","nlp","efficient transformers","DenseAttention","long context","Long Range Arena"]},"supplementary_material":{"value":"/attachment/78e4b43b1d574d7e27aaf0271c25bf639f64491b.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The ubiquitous Transformer architecture suffers from two main bottlenecks: 1) low computational and memory efficiency, leading to suboptimal hardware utilization, and 2) quadratic time complexity with respect to sequence length $N$, making it slow and costly for large data contexts. We propose a novel DenseAttention Network architecture, a straightforward simplification of the standard Transformer block that addresses these issues and serves as a drop-in replacement for language modeling tasks. We eliminate memory-bound components in DenseAttention, including Softmax, masking, one skip connection, and both LayerNorms, as well as key, value, and output projection matrices, as they become redundant. Despite these removals, it maintains exact $N \\times N$ pairwise interactions between tokens. By exploiting the associativity of matrix multiplications, DenseAttention can be computed with $O(N^2d)$ or $O(Nd^2)$ time and space complexity, depending on the context. To handle the absence of Softmax and prevent numerical instability, we introduce MaxNormActivation at both ends of the Transformer block. We also devise Cosine Relative Positional Embeddings as a computationally efficient replacement for RoPE, and simple LocalAttention variations of the block to help the model focus on details in extremely long contexts. \n\nDenseAttention competes with FlashAttention in speed on small sequences and outperforms it by orders of magnitude on large contexts. We pre-train encoder language models on sequences up to 16K in length, which perform similarly or better than baseline BERT-large, while significantly improving speed and efficiency.  Finally, we achieve state-of-the-art on the LRA benchmark among the Transformer-based architectures."},"_bibtex":{"value":"@misc{\nargatkiny2025denseattention,\ntitle={DenseAttention: No-Compromise Exact All \\$N {\\textbackslash}times N\\$  Interactions Algorithm with \\$O(N)\\$ Space and Time Complexity},\nauthor={Andrew Argatkiny},\nyear={2025},\nurl={https://openreview.net/forum?id=2bIQBDSfRk}\n}"},"title":{"value":"DenseAttention: No-Compromise Exact All $N \\times N$  Interactions Algorithm with $O(N)$ Space and Time Complexity"},"pdf":{"value":"/pdf/efeb6787f0c4f6394dce7d2e3d58e8e6dae7e3ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"argatkiny|denseattention_nocompromise_exact_all_n_\\times_n_interactions_algorithm_with_on_space_and_time_complexity"},"authorids":{"value":["~Andrew_Argatkiny1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Andrew Argatkiny"]}},"version":2},{"content":{"summary":{"value":"This paper addresses a well-known weakness of linear sequence models (such as linear RNNs and state-space models): their reliance on a single, fixed-size memory state, which leads to memory interference and poor performance on recall-intensive tasks. The authors propose Mixture-of-Memories (MoM), a new architecture that utilizes multiple independent memory states. A top-k routing network directs each input token to a specific subset of these memories for updating. This mechanism aims to increase the model's effective memory capacity and reduce interference by allowing different types of information to be stored in separate states. The final output is computed by querying a weighted mixture of the activated memory states. The paper demonstrates through extensive experiments that MoM significantly outperforms strong linear model baselines on a variety of downstream tasks, particularly those requiring information recall. Notably, the model achieves performance comparable to standard Transformer models while retaining the linear-time training and constant-time inference benefits of linear RNNs."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See my weakness part."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- The paper targets a critical and widely recognized weakness of linear-time sequence models: their poor performance on recall-intensive tasks due to the bottleneck of a single fixed-size memory state.\n- The experimental results are comprehensive and robust. \n\n- The core idea of using a top-k router to manage multiple, independent RNN memory states is a novel and clever application of sparse activation principles."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The ablation study in Table 6 indicates that the \"Shared Memory\" component is critical. Removing it causes a large performance drop (e.g., 2.1 points on Recall tasks). This makes it difficult to disentangle the gains from the \"Mixture\" mechanism (routing to $k$ of $N$ memories) from the gains of simply having a parallel, always-on \"Shared Memory\" state. The paper's narrative focuses heavily on the top-k mixture, but a large portion of the gains might be coming from this simpler shared component.\n- The main results in Table 2 appear to compare models of slightly different sizes. For example, Table 4 shows the \"Gated DeltaNet MoM\" model has 444M parameters, while its baseline \"Gated DeltaNet\" is 380M. This ~17% parameter difference complicates the main comparison. While a fairer study with equal activated parameters is commendably included in Appendix G, the primary results table in the main body of the paper should ideally be based on a more strictly controlled comparison.\n- The paper repeatedly claims MoM is a \"new paradigm\" and \"fundamentally differs\" from MoE (e.g., in Appendix B). This distinction feels overstated. The core mechanism, a top-k router that sparsely activates a subset of modules, which is the defining feature of MoE. Applying this concept to RNN states and key/value projections instead of FFN layers is a novel and useful application, but it does not seem to be a fundamental departure from the MoE paradigm.\n-  The analysis of memory specialization in Section 4.2.6 and Table 5 is weak. The paper provides no quantitative methodology for how token categories (e.g., \"Basic nouns/verbs\" vs. \"Proper nouns/scientific terms\") were defined or how the \"Potential Function\" (e.g., \"Simplify semantic information\") was determined. As presented, these conclusions appear entirely subjective and anecdotal, lacking the empirical rigor demonstrated in the rest of the paper."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762920365616,"tcdate":1761847133439,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8491/Reviewer_EVGR"],"signatures":["ICLR.cc/2026/Conference/Submission8491/Reviewer_EVGR"],"forum":"3PdOq8Rgue","number":3,"license":"CC BY 4.0","cdate":1761847133439,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8491/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762920365616,"domain":"ICLR.cc/2026/Conference","replyto":"3PdOq8Rgue","id":"hcp1dEEnB7","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Efficient/Low-Resource Methods for NLP","Linear Sequence Modeling","Machine Learning for NLP"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Linear sequence modeling methods, such as linear attention, state space modeling, and linear RNNs, offer significant efficiency improvements by reducing the complexity of training and inference. However, these methods typically compress the entire input sequence into a single fixed-size memory state, which leads to suboptimal performance on recall-intensive tasks. To address this limitation, we introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes multiple independent memory states, with a router network directing input tokens to specific memory states. This approach greatly enhances the overall memory capacity while minimizing memory interference. MoM serves as a general framework that can be seamlessly combined with diverse memory update mechanisms across linear models. As a result, MoM performs exceptionally well on recall-intensive tasks, surpassing existing linear sequence modeling techniques. Despite incorporating multiple memory states, the computation of each memory state remains linear in complexity, allowing MoM to retain the linear-complexity advantage during training, while constant-complexity during inference. Our experimental results show that MoM outperforms current linear sequence models on downstream language tasks, particularly recall-intensive tasks, and even achieves performance comparable to Transformer models."},"_bibtex":{"value":"@inproceedings{\ndu2026mom,\ntitle={MoM: Linear Sequence Modeling with Mixture-of-Memories},\nauthor={Jusen Du and Weigao Sun and Disen Lan and Jiaxi Hu and Tao Zhang and Yu Cheng},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=3PdOq8Rgue}\n}"},"title":{"value":"MoM: Linear Sequence Modeling with Mixture-of-Memories"},"pdf":{"value":"/pdf/6d8ebeff897bedb6faacfae3dae56e0c956cf7ca.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"du|mom_linear_sequence_modeling_with_mixtureofmemories"},"authorids":{"value":["~Jusen_Du1","~Weigao_Sun1","~Disen_Lan1","~Jiaxi_Hu1","~Tao_Zhang9","~Yu_Cheng1"]},"authors":{"value":["Jusen Du","Weigao Sun","Disen Lan","Jiaxi Hu","Tao Zhang","Yu Cheng"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a linear complexity attention mechanism for sequence modeling. The central concept involves processing the Q and K matrices with the softmax operation independently. Additionally, the paper explores mixed attention through sliding window attention, demonstrating enhanced performance in language modeling. Experimental results on both language modeling and LRA tasks indicate competitive performance. The paper also presents distillation results using pre-trained models."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"As above."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The paper is well-written."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The concept is quite similar to Efficient Attention: Attention with Linear Complexity. Although the author clarifies the differences, such as from vision tasks to language modeling and latent variable interpretation, I believe the novelty is still limited. First, vision tasks are a 2D sequence modeling problem, which is more complex than a 1D language modeling problem. Second, the latent variable interpretation treats the Q and K matrices as attention matrices, which seems a bit strange to me.\n\n2. There is a significant lack of linear models for comparison in this context. For instance, models such as HGRN (NeurIPS), HGRN2 (COLM), Lightning Attention (ICML), and GLA (ICML) are missing. Additionally, it is well-known that linear models may perform well on a small scale but often fail to scale effectively. The experiments conducted with 150 million parameters are insufficient to validate the actual scaling capabilities of the proposed method. Furthermore, the distillation results do not provide evidence of these scaling capabilities.\n\n3. Is the standard causal attention implemented with flash attention or not for the speed comparison? If not, the comparison results are not helpful. Also I would suggest include sota linear attention variants for comparison as well.\n\n4. It is well known that the limitation of linear models is their retrieval capability. The paper lacks experiments on \"Needle in a Haystack\" to demonstrate its performance on long sequence modeling."}},"nonreaders":[],"tmdate":1731429409060,"tcdate":1730536009020,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11402/Reviewer_sTmJ"],"signatures":["ICLR.cc/2025/Conference/Submission11402/Reviewer_sTmJ"],"forum":"Cj3B4SoWuT","number":3,"license":"CC BY 4.0","cdate":1730536009020,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11402/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429409060,"domain":"ICLR.cc/2025/Conference","replyto":"Cj3B4SoWuT","id":"gno9oVBXEP","forumContent":{"TLDR":{"value":"A latent variable model for linear time attention"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Sequence Modelling","Long Sequences","Linear Attention","Latent Variable Model"]},"supplementary_material":{"value":"/attachment/3cb6f199517bebe750bae9b35217afc14e0b3fcf.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The time complexity of the standard attention mechanism in transformers scales quadratically with sequence length. We propose a probabilistic framework for attention, enabling us to derive a novel low-rank linear re-parameterisation of both bidirectional and causal cases, based on defining a latent variable model. Our method can be seamlessly integrated as a drop-in replacement for the standard attention mechanism. Additionally, this framework provides a natural extension for combining local standard attention with our global linear attention. This approach allows us to extend the context length of existing large pre-trained models with only a few additional training steps. The resulting ``Latte Transformer'' achieves performance comparable to standard attention and other state-of-the-art models, while maintaining linear time and memory complexity, along with constant-time next-token prediction during inference."},"_bibtex":{"value":"@misc{\ndolga2025latte,\ntitle={Latte: Latent Attention for Linear Time Transformers},\nauthor={Rares Dolga and Lucas Maystre and Marius Cobzarenco and David Barber},\nyear={2025},\nurl={https://openreview.net/forum?id=Cj3B4SoWuT}\n}"},"title":{"value":"Latte: Latent Attention for Linear Time Transformers"},"pdf":{"value":"/pdf/fb6bc4a21b9df462a947bb33850431be6f06e97f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"dolga|latte_latent_attention_for_linear_time_transformers"},"authorids":{"value":["~Rares_Dolga1","~Lucas_Maystre1","~Marius_Cobzarenco1","~David_Barber1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Rares Dolga","Lucas Maystre","Marius Cobzarenco","David Barber"]}},"version":2},{"content":{"summary":{"value":"The main argument of this paper is that while current NL-to-TL systems excel at translating abstract formulas, they fail badly at the crucial step of connecting those formulas to real-world environments (so called \"grounding\"). To address this, the authors  introduce VLTL-Bench, a new benchmark designed specifically to address this gap and enable end-to-end evaluation of NL->TL translation including verification. This benchmark includes multiple distinct state spaces (environments), Thousands of diverse NL specifications with corresponding formal TL specs,  sample traces for verifying the grounded formulas within each state space and it supports the evaluation of the full process: \"Lifting\", \"Grounding\", \"Translation\", and \"Verification\"\nThe authors show that existing systems perform very well on \"lifting\" and \"translation\", confirming prior benchmark results, but they struggle significantly with the critical step of \"grounding\" atomic propositions into a provided state space."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"The benchmark includes only three distinct state spaces. How do you justify that this small number sufficiently captures the complexity and variability  required to evaluate grounding generalization in real-world scenarios? What steps were taken to ensure these environments are not biased or overlapping?\n\nHow were the sample traces for verification generated? Can you demonstrate statistically that they cover corner cases, violations, and  satisfactions of the LTL formulas? How do you address concerns about trace completeness impacting verification reliability?\n\nVLTL-Bench focuses exclusively on Linear Temporal Logic (LTL). Many practical applications require richer logics like CTL*, Signal Temporal Logic (STL), or Metric Temporal Logic (MTL). Does your approach fundamentally limit the benchmark's applicability? Is extending it to other TLs feasible?\n\nYou attribute poor grounding performance primarily to a 'lack of existing datasets'. However, could the failure also stem from fundamental architectural limitations in current NL-to-TL systems (e.g., LLMs lacking state-space reasoning)? How did you disentangle data scarcity from model capability?\n\nGiven the identified grounding bottleneck, what concrete architectural changes, training paradigms (e.g., using VLTL-Bench for fine-tuning), or hybrid approaches do you propose as most promising to address this?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"-- The paper addresses the major limitation in existing NL-to-TL research: the lack of evaluation for \"grounding\". Previous benchmarks ignored this essential step needed for real-world verification.\n\n-- The work enables end-to-end evaluation, providing not just formulas, but \"sample traces\" that allow researchers to verify if the final grounded TL formula behaves correctly in a specific state space/scenario. \n\n-- The paper considers NL-to-TL process as a pipeline of different stages (lifting, grounding, translation) and for each stage it provides the ground truth data.\n\n-- The paper provides a rich & diverse dataset, with \"thousands of diverse NL specifications\" providing corresponding formal TL specs for each NL spec.\n\n--The paper exposes the specific weakness of existing SOTA models: high accuracy on abstract translation but poor performance on grounding.\n\n-- Broad Evaluation Scope (`nl2spec`, `NL2TL`, `NL2LTL`, `Lang2LTL`) including sequence-to-sequence models and Various LLM prompting techniques."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"-- Scope Limited to LTL: The benchmark specifically targets Linear Temporal Logic (LTL), as indicated by \"VLTL-Bench\" and references to LTL. This does not evaluate translation  capabilities for other important temporal logics like Computation Tree Logic (CTL) or Signal Temporal Logic (STL).\n\n-- Limited State Space Diversity & Scale:While it includes three distinct state spaces, this is still a small number compared to the vast diversity of real-world systems NL-to-TL might  be applied to. Generalizability beyond these state spaces is not proven.\n\n-- Sample Trace Limitations: While including traces for verification is crucial, their representativeness are very important. The paper does not provide details how these traces were generated or if they comprehensively cover potential system behaviors (e.g., are corner cases included?)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762934010830,"tcdate":1761773392396,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20605/Reviewer_HCJe"],"signatures":["ICLR.cc/2026/Conference/Submission20605/Reviewer_HCJe"],"forum":"RUs4KC34yT","number":1,"license":"CC BY 4.0","cdate":1761773392396,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20605/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762934010830,"domain":"ICLR.cc/2026/Conference","replyto":"RUs4KC34yT","id":"Wra4TE2l8g","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We present VLTL-Bench, a benchmark unifying lifting, grounding, translation, and verification for NL-to-LTL. While prior models excel at lifting and translation, our evaluation reveals grounding into concrete state spaces remains a major challenge."},"keywords":{"value":["Benchmark","Temporal Logic","Linear Temporal Logic","Verification","Natural Language"]},"supplementary_material":{"value":"/attachment/c7037d92610176cde6a01262809a58fb7d50de65.zip"},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"Empirical evaluation of state-of-the-art natural language (NL) to temporal logic (TL) translation systems reveals near-perfect performance on existing benchmarks. However, current studies only measure the accuracy of the translation of NL logic into formal TL, ignoring a system’s capacity to ground atomic propositions into new scenarios or environments. This is a critical feature, necessary for the verification of resulting formulas in a concrete state space. In this paper, we introduce the Verifiable Linear Temporal Logic Benchmark (VLTL-Bench), a unifying benchmark for automated NL-to-LTL translation. The dataset consists of three unique state spaces and thousands of diverse natural language specifications and their corresponding formal temporal logic specifications. Moreover, the benchmark contains sample traces to verify the temporal logic expressions. While the benchmark directly supports end-to-end evaluation, we observe that many frameworks decompose the process into i) lifting,  ii) grounding, iii) translation, and iv) verification. The benchmark provides ground truths after each of these steps to enable researchers to improve and evaluate different substeps of the overall problem.  \n Using the benchmark, we evaluate several state‑of‑the‑art NL-to-TL translation models and frameworks, including nl2spec, NL2TL, NL2LTL, Lang2LTL, sequence-to-sequence translation, and various LLM prompting techniques. Our evaluation confirms that existing work is capable of reliably performing lifting and translation with high accuracy, while it exposes their struggles to ground the translation into a state space, which stems from the lack of existing datasets."},"_bibtex":{"value":"@misc{\nenglish2026verifiable,\ntitle={Verifiable Natural Language to Linear Temporal Logic Translation: A Benchmark Dataset and Evaluation Suite},\nauthor={William H English and Chase Walker and Dominic Simon and Sumit Kumar Jha and Rickard Ewetz},\nyear={2026},\nurl={https://openreview.net/forum?id=RUs4KC34yT}\n}"},"title":{"value":"Verifiable Natural Language to Linear Temporal Logic Translation: A Benchmark Dataset and Evaluation Suite"},"pdf":{"value":"/pdf/4a913f4403c564683ee70b528d9dcbc566738867.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"english|verifiable_natural_language_to_linear_temporal_logic_translation_a_benchmark_dataset_and_evaluation_suite"},"authorids":{"value":["~William_H_English1","~Chase_Walker1","~Dominic_Simon1","~Sumit_Kumar_Jha2","~Rickard_Ewetz1"]},"authors":{"value":["William H English","Chase Walker","Dominic Simon","Sumit Kumar Jha","Rickard Ewetz"]}},"version":2},{"content":{"summary":{"value":"The authors propose MVGamba, a general and lightweight Gaussian reconstruction model for unified 3D content generation. By replacing the previous LRM work’s transformer architecture with the recent model Mamba. MVGamba can generate long Gaussian sequences with linear complexity in a single forward process, eliminating the need for post hoc operations. Experiments demonstrate that MVGamba outperforms the current open-sourced LRM work in various 3D generation tasks with a smaller model size( 0.1× ). The authors also claim the proposed model has the potential to be applied to scene-level and 4D generation."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"- In lines 38-41, the authors state that previous Gaussian-based LRMs fail to achieve multi-view consistent results because they rely on local or mixed attention across limited multi-view image tokens or process each view separately before merging the predicted Gaussians. However, some prior works, like GS LRM, employ self-attention on all input image tokens. Despite merging the Gaussians afterward, they are predicted collectively. It is unclear why this approach leads to multi-view inconsistency.\n- In lines 281-286, the authors note that performance improves with increasing sequence length, and they model different sequence lengths by varying the patch size. If decreasing patch size enhances final performance without significantly impacting efficiency, as the authors claim, why didn't they choose a smaller patch size than 14, as indicated in the appendix? 14 is actually bigger than some transformer-based LRM’s patch size. (For example, GS LRM chose 8.)"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper is well-written and easy to understand, and the experiments are sufficient and well-analyzed.\n2. The motivation of this paper is sound. The current LRMs papers have achieved impressive results in 3D generation, but their architecture is highly computationally intensive, especially when handling long-sequence inputs. The recent Mama has the potential to solve it. \n3. The proposed pipeline achieves better results than the selected baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper’s experiments do not align with its claims regarding the inefficiencies of transformer-based LRMs in handling long-sequence inputs. The primary issue highlighted is that these transformer-based models are not efficient for long-sequence input. However, most experiments in the paper are for single or sparse image input 3D generation. In this context, the Mamba-based architecture does not demonstrate significant speed improvements over transformer-based architectures because the input tokens are relatively short. Table 1 shows no obvious speed enhancement compared to other feedforward methods, supporting this point. Therefore, I question the necessity of Mamba for most tasks discussed in the paper. The Mamba-based reconstruction model appears to be more suitable for dense view reconstruction tasks.\n\n2. The performance of paper doesn’t achieve the state of the art. Although in Table 1, the paper achieves better quantitative results over all baselines, some previous LRMs papers have better performance according to their paper. For example, GS-LRM attains 8 dB higher PSNR than LGM according to its table. 1, while MVGamba is just 2 dB higher than LGM. I can understand this paper didn’t compare with GS LRM because GS LRM doesn’t release its code. I am just not sure if Mamba is important for quality improvement as claimed by the authors, when some transformer-based LRMs can also achieve good (or even better) quality."},"limitations":{"value":"The limitations have been well discussed."}},"nonreaders":[],"tmdate":1730878754605,"tcdate":1720840598463,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission2178/Reviewer_XU7f"],"signatures":["NeurIPS.cc/2024/Conference/Submission2178/Reviewer_XU7f"],"forum":"AprsVxrwXT","number":3,"license":"CC BY 4.0","cdate":1720840598463,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission2178/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878754605,"domain":"NeurIPS.cc/2024/Conference","replyto":"AprsVxrwXT","id":"75w8DNyFC1","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["3D Generation and Reconstruction","State Space Model"]},"supplementary_material":{"value":"/attachment/836f2031251d5559938f7200420a18aa0a39f08c.zip"},"primary_area":{"value":"generative_models"},"abstract":{"value":"Recent 3D large reconstruction models (LRMs) can generate high-quality 3D content in sub-seconds by integrating multi-view diffusion models with scalable multi-view reconstructors. Current works further leverage 3D Gaussian Splatting as 3D representation for improved visual quality and rendering efficiency. However, we observe that existing Gaussian reconstruction models often suffer from multi-view inconsistency and blurred textures. We attribute this to the compromise of multi-view information propagation in favor of adopting powerful yet computationally intensive architectures (\\eg, Transformers). \nTo address this issue, we introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State Space Model (SSM). Our Gaussian reconstructor propagates causal context containing multi-view information for cross-view self-refinement while generating a long sequence of Gaussians for fine-detail modeling with linear complexity.\nWith off-the-shelf multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Extensive experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only $0.1\\times$ of the model size. The codes are available at \\url{https://github.com/SkyworkAI/MVGamba}."},"_bibtex":{"value":"@inproceedings{\nyi2024mvgamba,\ntitle={{MVG}amba: Unify 3D Content Generation as State Space Sequence Modeling},\nauthor={Xuanyu Yi and Zike Wu and Qiuhong Shen and Qingshan Xu and Pan Zhou and Joo Hwee Lim and Shuicheng YAN and Xinchao Wang and Hanwang Zhang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=AprsVxrwXT}\n}"},"title":{"value":"MVGamba: Unify 3D Content Generation as State Space Sequence Modeling"},"pdf":{"value":"/pdf/d1260f2186da4119a6e9c5f99613609eaa771e3e.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"yi|mvgamba_unify_3d_content_generation_as_state_space_sequence_modeling"},"authorids":{"value":["~Xuanyu_Yi1","~Zike_Wu1","~Qiuhong_Shen1","~Qingshan_Xu1","~Pan_Zhou3","~Joo_Hwee_Lim1","~Shuicheng_YAN3","~Xinchao_Wang1","~Hanwang_Zhang3"]},"authors":{"value":["Xuanyu Yi","Zike Wu","Qiuhong Shen","Qingshan Xu","Pan Zhou","Joo Hwee Lim","Shuicheng YAN","Xinchao Wang","Hanwang Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper focuses on the 3D point cloud segmentation task. It first compares the performance of three types of blocks: CNN, Transformer, and Mamba, and then improves the existing Mamba architecture. The authors provide extensive visualization results and ablation experiments."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"[Q1] Could the authors provide more detailed explanations of W1, such as whether there is theoretical evidence to support it or if there are sufficiently extensive experiments across multiple datasets?\n[Q2] Is the Bidirectional SSM mentioned by the authors consistent with VisionMamba (Zhu 2024)? I do not see a bidirectional structure in Fig(6b), only the Strided SSM.\n[Q2.5] If the answer of Q2 is yes, the result you reported in the second line of Tab(7d) shows that this structure can only bring 0.1% increasement, which seems inconsistent with the effects reported in VisionMamba (Zhu 2024) regarding this structure. From an efficiency perspective, is the introduction of the indirection necessary if it only brings a 0.1% improvement?\n[Q3] The casual-free conv block is not clearly explained. If this is something new you are proposing, it would be helpful to include a more detailed explanation or illustration.\n[Q4] Standardize the notation for SSM. In the abstract and introduction, it is referred to as \"Strided Bidirectional SSM,\" while later it is called \"Bidirectional Strided SSM.\"\n[Q5] There’s something wrong of your hyperlink in 3.3. It should be Fig 5 but yours is Fig 4."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"[S1] It is meaningful to compare the performance of the three types of blocks—CNN, Transformer, and Mamba—on the same task, point cloud segmentation.\n[S2] The paper is detailed and easy to read.\n[S3] The paper conducts detailed ablation experiments on the newly proposed module.\n[S4] The paper achieves state-of-the-art results on multiple datasets."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"[W1] The comparison of CNN, Transformer, and Mamba blocks lacks strong evidence. I believe that concluding \"CNN is more effective for local modeling and Transformer is better at handling contextual information\" based on just one example for each is insufficient.\n[W2] The proposed method is more like a small fix to Mamba-based method. \n[W3] The two solutions proposed in this paper: 1) the casual-free block is described in very little detail and lacks a clear explanation; 2) bidirectional SSM seems to have already been introduced in VisionMamba (Zhu 2024), and this paper's work only adds n-stride."}},"nonreaders":[],"tmdate":1731427394390,"tcdate":1730642447535,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1247/Reviewer_HJ9v"],"signatures":["ICLR.cc/2025/Conference/Submission1247/Reviewer_HJ9v"],"forum":"E1ML0nEReb","number":4,"license":"CC BY 4.0","cdate":1730642447535,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1247/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427394390,"domain":"ICLR.cc/2025/Conference","replyto":"E1ML0nEReb","id":"GOT72QPlCk","forumContent":{"TLDR":{"value":"This paper analyzes the key components that make a model architecture efficient and effective for point cloud segmentation and explores various methods to implement these components successfully."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["point cloud segmentation","efficient","contextual modeling"]},"supplementary_material":{"value":"/attachment/62537ba3107759f903cfd3e8f9ea61fd79348a56.zip"},"primary_area":{"value":"applications to robotics, autonomy, planning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Point cloud segmentation is an important topic in 3D understanding that has traditionally been tackled using either the CNN or Transformer. Recently, Mamba has emerged as a promising alternative, offering efficient long-range contextual modeling capabilities without the quadratic complexity associated with Transformer's attention mechanisms. However, despite Mamba's potential, early efforts have all failed to achieve better performance than the best CNN-based and Transformer-based methods. In this work, we address this challenge by identifying the key components of an effective and efficient point cloud segmentation architecture. Specifically, we show that: 1) Spatial locality and robust contextual understanding are critical for strong performance, and 2) Mamba features linear computational complexity, offering superior data and inference efficiency compared to Transformers, while still being capable of delivering strong contextual understanding. Additionally, we further enhance the standard Mamba specifically for point cloud segmentation by identifying its two key shortcomings. First, the enforced causality in the original Mamba is unsuitable for processing point clouds that have no such dependencies. Second, its unidirectional scanning strategy imposes a directional bias, hampering its ability to capture the full context of unordered point clouds in a single pass. To address these issues, we carefully remove the causal convolutions and introduce a novel Bidirectional Strided SSM to enhance the model's capability to capture spatial relationships. Our efforts culminate in a novel architecture named MEEPO that effectively integrates the strengths of CNN and Mamba. MEEPO surpasses the previous state-of-the-art method, PTv3, by up to +0.8 mIoU on multiple key benchmark datasets, while being 42.1\\% faster and 5.53$\\times$ more memory efficient. Our code will be released."},"_bibtex":{"value":"@misc{\nchng2025exploring,\ntitle={Exploring contextual modeling with linear complexity for point cloud segmentation},\nauthor={Yong Xien Chng and Xuchong QIU and Henry Zheng and Yizeng Han and Yifan Pu and Jiewei Cao and Gao Huang},\nyear={2025},\nurl={https://openreview.net/forum?id=E1ML0nEReb}\n}"},"title":{"value":"Exploring contextual modeling with linear complexity for point cloud segmentation"},"pdf":{"value":"/pdf/6670e7aa64624951a356099a0f6c5ec34dd15c15.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chng|exploring_contextual_modeling_with_linear_complexity_for_point_cloud_segmentation"},"authorids":{"value":["~Yong_Xien_Chng1","~Xuchong_QIU1","~Henry_Zheng1","~Yizeng_Han1","~Yifan_Pu1","~Jiewei_Cao2","~Gao_Huang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yong Xien Chng","Xuchong QIU","Henry Zheng","Yizeng Han","Yifan Pu","Jiewei Cao","Gao Huang"]}},"version":2},{"content":{"summary":{"value":"The authors propose a modification of the S5 sequence architecture with a resettable hidden state that leads to a drop-in replacement of RNNs and Transformers in partially observed / memory-intensive RL tasks. They show that S5 exceeds baseline performance while being more computationally efficient to train than LSTMs due to S5’s parallelization and faster at inference than Transformers’ N^2 attention."},"soundness":{"value":"3 good"},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"Please clarify if I am wrong in thinking that the DMControl tasks are zero-shot. The evaluation procedure is not as clearly discussed as it often is in meta-RL papers, and the appendix did not give more details.\n\nWere there any experiments to dramatically increase the context length towards the range seen with S4 in LRA? I’m curious if there are insights on limitations or changes that would be needed to make this work in an RL setting. According to Table 5 the DM Control experiments used 64 TPUv3s, so I assume compute was not the limiting factor here."},"rating":{"value":"7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":"3 good"},"contribution":{"value":"3 good"},"strengths":{"value":"State space models are a natural fit for in-context meta-RL and long-term memory because of their unique combination of train-time parallelism and test-time constant inference. They are also thought to be more compatible with the continuous inputs seen in RL than other long-sequence domains like NLP that use discrete tokens. In-context RL has a sequence length barrier that goes somewhat unnoticed because common benchmarks do not require adaptation over more than a few steps. Applying S5 and future versions here may have great long-term benefits. \n\nThe experiments do clearly demonstrate that resettable S5 can replace LSTMs and Transformers in partially observed tasks.\n\nThe paper also proposes a meta-learning version of the DM Control suite using randomly projected state and action spaces. While this is a somewhat arbitrary way to expand the meta-distribution of tasks from a small set of popular benchmarks, it is probably better than the common alternative of turning similar environments into goal-conditioned tasks and then hiding the goal (Ant Goal, Cheetah Fwd-Back, Humanoid Dir, ...). This benchmark will be useful for future work.\n\nThe method could be applied to most agents that support S5's computation, and should be easy to use and reproduce."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"S5 would be best suited for extending the sequence lengths used by current methods, and could unlock a new level of difficulty in long-context RL. However, the experiments here stop short of exploring those limits. Instead, sequence lengths are capped at ~1k where LSTMs/Transformers are feasible but slower and more expensive. Wall-clock efficiency is a nice advantage but could also be improved by other factors like using a more efficient base algorithm. If the goal was to show that S5 is a valid substitute at existing sequence lengths while leaving expansion for future work, it would have been simpler to use more standard benchmarks in the meta-RL portion and compare to external baselines. \n\nIn general, the narrative of the paper gets a bit lost in the gray area between more traditional long-term memory and “in-context” meta-learning. The need for long sequences is motivated by multi-episodic in-context learners like RL^2 (lines 36-42), but the method (Sec. 3.1 and Alg 1) does not distinguish between episode resets (which would not reset the sequence model’s hidden state) and task resets. The first two experiments are focused on standard long-term memory, while the third appears to only evaluate zero-shot generalization to a meta-distribution of tasks. It would have been more interesting to evaluate RL^2-style multi-episodic learning, which would naturally extend the sequence length to highlight S5’s core advantages.\n\nThe need for a resettable hidden state within S5 is presented as a widespread barrier to its application in RL. But this is primarily caused by the data collection and optimization implementations common to on-policy policy gradient algorithms with parallel actors. I think it would be helpful to explain this issue in more detail than is provided (lines 111-144, 43-48). For example, there would be no need for the modified architecture when substituting S5 for RNNs/Transformers in an off-policy in-context learner like in [Ni et al.](https://arxiv.org/abs/2110.05038) The issue may also be avoided by batching the policy updates into padded sequences that do not cross episode boundaries."},"limitations":{"value":"There is no Limitations section or a clear discussion on limitations. However, the paper is proposing an architectural change to an existing method that has known limitations, and some of those limitations are improved by the S5 model."}},"nonreaders":[],"tmdate":1702411395397,"tcdate":1688597473669,"writers":["NeurIPS.cc/2023/Conference","NeurIPS.cc/2023/Conference/Submission12708/Reviewer_R54Y"],"signatures":["NeurIPS.cc/2023/Conference/Submission12708/Reviewer_R54Y"],"forum":"4W9FVg1j6I","number":4,"license":"CC BY 4.0","cdate":1688597473669,"mdate":1702411395397,"readers":["everyone"],"invitations":["NeurIPS.cc/2023/Conference/Submission12708/-/Official_Review","NeurIPS.cc/2023/Conference/-/Edit"],"domain":"NeurIPS.cc/2023/Conference","replyto":"4W9FVg1j6I","id":"X36z0LD46m","forumContent":{"venue":{"value":"NeurIPS 2023 poster"},"keywords":{"value":["Reinforcement Learning","Meta-Learning","State Space Models"]},"supplementary_material":{"value":"/attachment/28dde0e402e34d8bf61f59cb22fbf7495a3c2aa8.zip"},"_bibtex":{"value":"@inproceedings{\nlu2023structured,\ntitle={Structured State Space Models for In-Context Reinforcement Learning},\nauthor={Chris Lu and Yannick Schroecker and Albert Gu and Emilio Parisotto and Jakob Nicolaus Foerster and Satinder Singh and Feryal Behbahani},\nbooktitle={Thirty-seventh Conference on Neural Information Processing Systems},\nyear={2023},\nurl={https://openreview.net/forum?id=4W9FVg1j6I}\n}"},"title":{"value":"Structured State Space Models for In-Context Reinforcement Learning"},"paperhash":{"value":"lu|structured_state_space_models_for_incontext_reinforcement_learning"},"TLDR":{"value":"We show that S4 models enable fast and efficient long-horizon memory in RL"},"abstract":{"value":"Structured state space sequence (S4) models have recently achieved state-of-the-art performance on long-range sequence modeling tasks. These models also have fast inference speeds and parallelisable training, making them potentially useful in many reinforcement learning settings. We propose a  modification to a variant of S4 that enables us to initialise and reset the hidden state in parallel, allowing us to tackle reinforcement learning tasks. We show that our modified architecture runs asymptotically faster than Transformers in sequence length and performs better than RNN's on a simple memory-based task. We evaluate our modified architecture on a set of partially-observable environments and find that, in practice, our model outperforms RNN's while also running over five times faster. Then, by leveraging the model’s ability to handle long-range sequences, we achieve strong performance on a challenging meta-learning task in which the agent is given a randomly-sampled continuous control environment, combined with a randomly-sampled linear projection of the environment's observations and actions. Furthermore, we show the resulting model can adapt to out-of-distribution held-out tasks. Overall, the results presented in this paper show that structured state space models are fast and performant for in-context reinforcement learning tasks. We provide code at https://github.com/luchris429/s5rl."},"pdf":{"value":"/pdf/3eb7fc8164913bb368cbc05a07288570734309cb.pdf"},"venueid":{"value":"NeurIPS.cc/2023/Conference"},"authorids":{"value":["~Chris_Lu1","~Yannick_Schroecker1","~Albert_Gu1","~Emilio_Parisotto1","~Jakob_Nicolaus_Foerster1","~Satinder_Singh2","~Feryal_Behbahani1"]},"authors":{"value":["Chris Lu","Yannick Schroecker","Albert Gu","Emilio Parisotto","Jakob Nicolaus Foerster","Satinder Singh","Feryal Behbahani"]}},"version":2},{"content":{"venue":{"value":"CVPR 2026"},"abstract":{"value":"Video object detection has gained notable progress with the advent of transformers. While transformers excel at modeling long-range contextual dependencies, the quadratic complexity limits their efficiency in long-sequence processing. In contrast, Mamba offers greater efficiency in modeling long sequences but tends to exhibit relatively limited contextual learning capability compared with transformers, and its application to video object detection remains unexplored. To harness the complementary strengths of transformers and Mamba, we propose a hybrid Transformer-Mamba network for video object detection (TMambaDet), a pioneering framework in this domain that combines the long-range modeling power of transformers with the efficient long-sequence processing capability of Mamba. Our TMambaDet is characterized by three core components: 1) a spatial adaptive deformable transformer encoder to effectively model the long-range dependencies within each frame, enabling intra-frame feature aggregation that substantially improves the spatial feature representations of objects; 2) a temporal cascaded bidirectional Mamba encoder to efficiently capture the long-range dependencies across frames in video sequences with linear complexity, enabling inter-frame feature aggregation that effectively enhances the temporal feature representations of objects; 3) a Mamba entangled transformer decoder to fully explore the interactions between object queries and spatial-temporal features, enabling fine-grained query-feature alignment that effectively enriches the instance-level representations of object queries. We conduct experiments on the ImageNet VID and EPIC-KITCHENS-55 datasets, showing that TMambaDet achieves state-of-the-art results."},"_bibtex":{"value":"@inproceedings{\nqi2026when,\ntitle={When Transformers Meet Mamba: A Hybrid Transformer-Mamba Network for Video Object Detection},\nauthor={Qiang Qi and Xiao Wang and Zongyuan Du and Yu Zhang},\nbooktitle={Conference on Computer Vision and Pattern Recognition 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=vDlDvYZhyh}\n}"},"title":{"value":"When Transformers Meet Mamba: A Hybrid Transformer-Mamba Network for Video Object Detection"},"pdf":{"value":"https://openaccess.thecvf.com/content/CVPR2026/papers/Qi_When_Transformers_Meet_Mamba_A_Hybrid_Transformer-Mamba_Network_for_Video_CVPR_2026_paper.pdf"},"venueid":{"value":"thecvf.com/CVPR/2026/Conference"},"paperhash":{"value":"qi|when_transformers_meet_mamba_a_hybrid_transformermamba_network_for_video_object_detection"},"authorids":{"value":["~Qiang_Qi1","~Xiao_Wang15","~Zongyuan_Du3","~Yu_Zhang149"]},"authors":{"value":["Qiang Qi","Xiao Wang","Zongyuan Du","Yu Zhang"]}},"tmdate":1789656576885,"pdate":1789656437831,"tcdate":1765222150807,"writers":["thecvf.com/CVPR/2026/Conference","thecvf.com/CVPR/2026/Conference/Submission39027/Authors"],"signatures":["thecvf.com/CVPR/2026/Conference/Submission39027/Authors"],"forum":"vDlDvYZhyh","license":"CC BY 4.0","number":39027,"cdate":1765222150807,"readers":["everyone"],"invitations":["thecvf.com/CVPR/2026/Conference/-/Submission","thecvf.com/CVPR/2026/Conference/Submission39027/-/Full_Submission","thecvf.com/CVPR/2026/Conference/-/Post_Submission","thecvf.com/CVPR/2026/Conference/Submission39027/-/Supplementary_Material","thecvf.com/CVPR/2026/Conference/-/Edit","thecvf.com/CVPR/2026/Conference/-/Compute_Flag"],"mdate":1789656576885,"odate":1789656437831,"domain":"thecvf.com/CVPR/2026/Conference","id":"vDlDvYZhyh","version":2},{"content":{"comment":{"value":"We thank the reviewer HAtu for thorough review, attention to details, and constructive feedback. We are encouraged by your recognition of the approach to eliminate the two low-rank matrices $\\mathbf{W}_Q$ and $\\mathbf{W}_K$ in favor of single high-rank matrix, which served as one of the reasons to name the method DenseAttention. Please let us address the concerns you have raised.\n\n\n\n> **W1. The paper lacks comparison against mamba in experiments. Mamba-I and Mamba-II are fast approaches for long range sequence modeling.**\n\nWe thank you for bringing up a discussion about Mamba. We believe comparing with Mamba on the benchmarks referred to in our paper would be a delicate issue due to the following reasons:\n\n**1. Absence of LRA tests by Mamba's authors**\n\nThe authors of the Mamba didn’t provide the results on the Long Range Arena (LRA) suite of benchmarks in the paper, and, it seems, they do not intend the model to be tested on the LRA. To quote the author of the paper (source: https://github.com/state-spaces/mamba/issues/282#issuecomment-2221135197), “We did not try LRA with Mamba. We don't believe that it's a good dataset, e.g. see: [1]“ \n\nDespite that, we found the Mamba’s results on the LRA in another paper [2]. We compare them with DenseAttention below:\n\n\n\n| Model             | ListOps | Text   | Retrieval | Image  | Pathfinder | Path-X | Avg   |\n|-------------------|---------|--------|-----------|--------|------------|--------|-------|\n| DenseAttention    | 50.50   | 81.19 | 87.51     | 72.55  | 87.40      | 88.82  | 75.83 |\n| Mamba             | 38.02   | 82.98 | 72.14     | 69.82  | 69.26      | 67.32  | 66.59 |\n\nThe metric for all tasks is accuracy, in %. Larger is better. DenseAttention significantly outperforms Mamba on average and in each benchmark individually. \n\nHowever, we don’t believe it might be a fair comparison because 1) the authors explicitly stated that this model is best suited for other causes, and 2) Mamba is an instance of a class of State Space Models (SSMs), whereas our model is Transformer-based.\n\nIt is worth noting that, generally, SSMs are in the league of their own in comparison with Transformer-based architectures and greatly outperform them on the LRA benchmarks due to their inherent inductive bias towards capturing hierarchical and long-range dependencies and lack of such bias in Transformers, as discussed in [1, 3-4]. Nevertheless, we found that DenseAttention also outperforms one of these much stronger SSM baselines in 4 of 6 benchmarks (S4-v1 [5], results reported in the Table 1 of the paper). To the best of our knowledge, this is the first case when a pure Transformer-based model compares favorably with an SSM on the LRA which is a valuable and interesting insight. \n\n**2. Mamba's incompatibility with bidirectional sequence processing**\n\nOur second group of experiments involves pre-training BERT-like architectures with DenseAttention. We are not able to draw comparisons with Mamba, because the task for the experiments – Masked Language Modeling – requires bidirectional architecture while Mamba is a solely unidirectional, left-to-right type of model. The authors of the paper have not provided an official bidirectional implementation as of now (https://github.com/state-spaces/mamba/issues/99). Devising a bidirectional Mamba-based architecture for general sequence processing would be an important piece of work on its own merits, worth a dedicated research paper, however it’s out of scope of our current work as it is focused on improving the Transformer-based architecture.\n\nAlso, we believe it’s natural to compare our models with BERT [6] specific architecture itself because we closely follow the implementation details and training process for the original model, except for replacing Transformer blocks with DANet blocks. Our key goal was to show that DenseAttention-BERT is at least on par with the original model in terms of LM quality, while being faster and more computationally efficient, and we successfully accomplished this goal.\n\nReferences:\n\n[1] Amos et al., \"Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors.\" ICLR 2024\n\n[2] Alonso et al., \"State Space Models as Foundation Models: A Control Theoretic Overview.\" arXiv preprint arXiv:2403.16899, 2024.\n\n[3] Ma et al., \"Mega: Moving Average Equipped Gated Attention.\" ICLR 2023\n\n[4] Tran et al., \"The Importance of Being Recurrent for Modeling Hierarchical Structure.\" EMNLP 2018\n\n[5] Gu et al., \"Efficiently Modeling Long Sequences with Structured State Spaces.\" ICLR 2022 \n\n[6] Devlin et al., \"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.\" NAACL 2019"},"title":{"value":"Author response. Part 1"}},"tmdate":1732516353260,"tcdate":1732515303963,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13608/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13608/Authors"],"forum":"2bIQBDSfRk","number":2,"license":"CC BY 4.0","cdate":1732515303963,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13608/-/Official_Comment"],"mdate":1732516353260,"domain":"ICLR.cc/2025/Conference","replyto":"Vo3IuaJ8w9","id":"IkavGOXXm7","forumContent":{"TLDR":{"value":"We propose a novel DenseAttention architecture which achieves 1) favorable computational efficiency;  2) linear time & space complexity by simplification and reduction of standard Transformer architecture."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["self-attention","deep learning","transformer architecture","nlp","efficient transformers","DenseAttention","long context","Long Range Arena"]},"supplementary_material":{"value":"/attachment/78e4b43b1d574d7e27aaf0271c25bf639f64491b.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The ubiquitous Transformer architecture suffers from two main bottlenecks: 1) low computational and memory efficiency, leading to suboptimal hardware utilization, and 2) quadratic time complexity with respect to sequence length $N$, making it slow and costly for large data contexts. We propose a novel DenseAttention Network architecture, a straightforward simplification of the standard Transformer block that addresses these issues and serves as a drop-in replacement for language modeling tasks. We eliminate memory-bound components in DenseAttention, including Softmax, masking, one skip connection, and both LayerNorms, as well as key, value, and output projection matrices, as they become redundant. Despite these removals, it maintains exact $N \\times N$ pairwise interactions between tokens. By exploiting the associativity of matrix multiplications, DenseAttention can be computed with $O(N^2d)$ or $O(Nd^2)$ time and space complexity, depending on the context. To handle the absence of Softmax and prevent numerical instability, we introduce MaxNormActivation at both ends of the Transformer block. We also devise Cosine Relative Positional Embeddings as a computationally efficient replacement for RoPE, and simple LocalAttention variations of the block to help the model focus on details in extremely long contexts. \n\nDenseAttention competes with FlashAttention in speed on small sequences and outperforms it by orders of magnitude on large contexts. We pre-train encoder language models on sequences up to 16K in length, which perform similarly or better than baseline BERT-large, while significantly improving speed and efficiency.  Finally, we achieve state-of-the-art on the LRA benchmark among the Transformer-based architectures."},"_bibtex":{"value":"@misc{\nargatkiny2025denseattention,\ntitle={DenseAttention: No-Compromise Exact All \\$N {\\textbackslash}times N\\$  Interactions Algorithm with \\$O(N)\\$ Space and Time Complexity},\nauthor={Andrew Argatkiny},\nyear={2025},\nurl={https://openreview.net/forum?id=2bIQBDSfRk}\n}"},"title":{"value":"DenseAttention: No-Compromise Exact All $N \\times N$  Interactions Algorithm with $O(N)$ Space and Time Complexity"},"pdf":{"value":"/pdf/efeb6787f0c4f6394dce7d2e3d58e8e6dae7e3ee.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"argatkiny|denseattention_nocompromise_exact_all_n_\\times_n_interactions_algorithm_with_on_space_and_time_complexity"},"authorids":{"value":["~Andrew_Argatkiny1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Andrew Argatkiny"]}},"version":2},{"content":{"comment":{"value":"We sincerely thank Reviewer veE3 for reviewing our work and providing insightful suggestions on connecting our findings with the existing literature and conducting additional experiments to further strengthen our claims. We address your concerns as follows.\n\n**1. Comparison with results in [Wang \\& Xue, 2023].**\n\nWe thank Reviewer veE3 for pointing out the recent work by [Wang \\& Xue, 2023]. Distinct from their findings, our primary contribution lies in analyzing **nonlinearity** and **input-dependent** mechanisms, which serves as the key to the success of modern SSMs [1,2,3]. The approach in [Wang \\& Xue, 2023] does not naturally extend to this case, as they assume the linearity of the sequence mixer (cf. Eq. 9 in [Wang \\& Xue, 2023]). To the best of our knowledge, our work presents the first counterargument showing that even with input-dependent SSMs, effective context filtering may not be achieved. Instead, these mechanisms impose a strict recency bias, inheriting the limitations of **linear** SSMs as previously highlighted by [Wang \\& Xue, 2023]. Our analysis considers both $A_t$ and $b_t$ as input-dependent, aligning with settings in Mamba and other follow-up works [2,3]. Furthermore, we conduct a finer-grained analysis, quantitatively relating the decay rate to the specific values within the input-dependent gating matrix $A_t$. We have summarized this discussion into our related work (Sec. 6) and include a full version in Appendix B.\n\n[1] Gu & Dao, Mamba: Linear-time sequence modeling with selective state spaces\n\n[2] Yang et al. Gated linear attention transformers with hardware-efficient training\n\n\n[3] Arora et al. Zoology: Measuring and improving recall in efficient language models\n\n**2. Experiments in Fig. 2 are limited to a few pre-trained models. Whether increasing hidden dimension and the number of heads can mitigate recency?**\n\nWe thank Reviewer veE3 for the suggestion to reproduce our empirical observations across a variety of models. However, we would like to point out that the model used in Fig. 2 is `Mamba-Codestral-7B`, which is currently the state-of-the-art SSM model known to us. At present, we are unable to find an open-source pretrained Mamba model that is larger and more advanced. Nevertheless, we still observe that such industry-level model aligns well with our theoretical findings. In contrast, transformer-based model `Mistral-7B` developed by the same team does not exhibit similar limitations. If the reviewer has any recommendations for models that are larger and more capable of long-context tasks than this baseline, we would greatly appreciate it if you could share them with us.\n\n**3. Training dynamics of $A_{max}$ and $A_{min}$.**\n\nWe agree with Reviewer veE3 that $A_{max} \\approx 1$ and $A_{min} \\approx 0$ can happen at the same time. However, our empirical findings show that these values are largely concentrated within a narrow range. To illustrate this, we visualize the distribution of $(A_{max} - A_{min})$ across different channels in Fig. 6 of our revised manuscript. Each bin represents the proportion of channels whose memory state satisfies $(A_{max} - A_{min})$ being smaller than the corresponding threshold on the x-axis. Notably, over 60% of channels exhibit $(A_{max} - A_{min})$ values smaller than 0.5, indicating that most channels cannot simultaneously achieve $A_{max} \\approx 1$ and $A_{max} \\approx 0$. Memory representations will inevitably undergo either exponential diminishing or over-smoothing."}},"tmdate":1732272892280,"tcdate":1732272892280,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1195/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission1195/Authors"],"forum":"pymXpl4qvi","number":5,"license":"CC BY 4.0","cdate":1732272892280,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1195/-/Official_Comment"],"mdate":1732272892280,"domain":"ICLR.cc/2025/Conference","replyto":"3WcVE8cZ0b","id":"TlP1iQ067F","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"We theoretically and empirically reveal and tackle locality and over-smoothing bottlenecks for state space models."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["State Space Models","Long-context LLMs","Over-smoothing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Structured State Space Models (SSMs) have emerged as alternatives to transformers.\nWhile SSMs are often regarded as effective in capturing long-sequence dependencies, we rigorously demonstrate that they are inherently limited by strong recency bias.\nOur empirical studies also reveal that this bias impairs the models' ability to recall distant information and introduces robustness issues. Our scaling experiments then discovered that deeper structures in SSMs can facilitate the learning of long contexts.\nHowever, subsequent theoretical analysis reveals that as SSMs increase in depth, they exhibit another inevitable tendency toward over-smoothing, e.g., token representations becoming increasingly indistinguishable.\nThis *fundamental dilemma* between recency and over-smoothing hinders the scalability of existing SSMs. \nInspired by our theoretical findings, we propose to *polarize* two channels of the state transition matrices in SSMs, setting them to zero and one, respectively, simultaneously addressing recency bias and over-smoothing.\nExperiments demonstrate that our polarization technique consistently enhances the associative recall accuracy of long-range tokens and unlocks SSMs to benefit further from deeper architectures.\nAll source codes are released at https://github.com/VITA-Group/SSM-Bottleneck."},"_bibtex":{"value":"@inproceedings{\nwang2025understanding,\ntitle={Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing},\nauthor={Peihao Wang and Ruisi Cai and Yuehao Wang and Jiajun Zhu and Pragya Srivastava and Zhangyang Wang and Pan Li},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=pymXpl4qvi}\n}"},"title":{"value":"Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing"},"pdf":{"value":"/pdf/982acd7979eff95835985d1c18fb057152350bf1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wang|understanding_and_mitigating_bottlenecks_of_state_space_models_through_the_lens_of_recency_and_oversmoothing"},"authorids":{"value":["~Peihao_Wang1","~Ruisi_Cai1","~Yuehao_Wang1","~Jiajun_Zhu1","~Pragya_Srivastava1","~Zhangyang_Wang1","~Pan_Li2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Peihao Wang","Ruisi Cai","Yuehao Wang","Jiajun Zhu","Pragya Srivastava","Zhangyang Wang","Pan Li"]}},"version":2},{"content":{"venue":{"value":"Inf. Fusion 2025"},"venueid":{"value":"dblp.org/journals/INFFUS/2025"},"paperhash":{"value":"he|panmamba_effective_pansharpening_with_state_space_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xuanhua_He:","~Ke_Cao2","https://dblp.org/search/pid/api?q=author:Jie_Zhang_0033:","https://dblp.org/search/pid/api?q=author:Keyu_Yan:","https://dblp.org/search/pid/api?q=author:Yingying_Wang_0005:","https://dblp.org/search/pid/api?q=author:Rui_Li_0027:","https://dblp.org/search/pid/api?q=author:Chengjun_Xie:","https://dblp.org/search/pid/api?q=author:Danfeng_Hong:","~Man_Zhou4"]},"html":{"value":"https://doi.org/10.1016/j.inffus.2024.102779"},"_bibtex":{"value":"@article{DBLP:journals/inffus/HeCZYWLXHZ25,\n  author={Xuanhua He and Ke Cao and Jie Zhang and Keyu Yan and Yingying Wang and Rui Li and Chengjun Xie and Danfeng Hong and Man Zhou},\n  title={Pan-Mamba: Effective pan-sharpening with state space model},\n  year={2025},\n  cdate={1735689600000},\n  journal={Inf. Fusion},\n  volume={115},\n  pages={102779},\n  url={https://doi.org/10.1016/j.inffus.2024.102779}\n}\n"},"abstract":{"value":"Highlights•Introduces Pan-Mamba: a novel pan-sharpening network for information modeling.•Custom-designed Mamba blocks for channel swapping and cross-modal info exchange.•Achieves state-of-the-art results across diverse datasets.•First exploration of Mamba model in pan-sharpening."},"title":{"value":"Pan-Mamba: Effective pan-sharpening with state space model"},"authors":{"value":["Xuanhua He","Ke Cao","Jie Zhang","Keyu Yan","Yingying Wang","Rui Li","Chengjun Xie","Danfeng Hong","Man Zhou"]}},"tmdate":1752151431922,"pdate":1735689600000,"tcdate":1747276352633,"writers":["~"],"signatures":["~Ke_Cao2"],"forum":"AarZaunEOd","license":"CC BY-SA 4.0","number":470898,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1752151431922,"domain":"DBLP.org","id":"AarZaunEOd","version":2},{"content":{"TLDR":{"value":"SMat-Attention uses the VC dimension of a structured causal mask as a tunable knob controlling long-range access complexity, yielding explicit tradeoffs between subquadratic computation and decoding memory."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Long Context","Efficient Attention","Structured Matrix","Causal masks","Decoding","ML Systems"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Long-context sequence models face a fundamental tradeoff: softmax attention uses flexible token-level interactions at quadratic cost, whereas linear attention obtains linear-time training and constant-time decoding by compressing history into a fixed-size state. In this work, we ask whether we can connect these regimes through a tunable notion of structure. To this end, we introduce Structured Matrix Attention (SMat-Attention) via a family of causal masks with structured long-range routing whose row supports have VC-dimension $d$. In our construction, $d=1$ recovers the standard causal mask, and increasing $d$ permits richer subset-routing patterns. We give chunkwise forward and backward algorithms to enable hardware-efficiency. For sequences of length $T$, the hard-routing construction takes $O(T^{2-3/d}+T)$ work, despite the mask being dense, for our prescribed family. In fixed-horizon streaming, decoding after the distant prefix takes constant time per token using $O(T^{1-1/d})$ cached states. SMat-Attention therefore makes VC-dimension an explicit knob governing access-pattern complexity, prefill cost, and decoding memory. Empirically, subset-routing and rule-assisted multi-key retrieval experiments illustrate the masks’ routing expressiveness. Extensions to Mamba-2 and Gated DeltaNet using learned routing with top-$k$ query reads retain subquadratic prefill, improve recall accuracy over the backbones in several settings, and achieve comparable small-scale language-modeling performance."},"_bibtex":{"value":"@inproceedings{\nanonymous2026smatattention,\ntitle={{SM}at-Attention: Structured Long-Context Sequence Modeling},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=J5Iz31xQlx},\nnote={under review}\n}"},"title":{"value":"SMat-Attention: Structured Long-Context Sequence Modeling"},"pdf":{"value":"/pdf/1c795ae4aa78e95802293edce6de31d5b8faef58.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230633491,"tcdate":1789584118718,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission26634/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission26634/Authors"],"forum":"J5Iz31xQlx","license":"CC BY 4.0","number":26634,"cdate":1789584118718,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission26634/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791230633491,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"J5Iz31xQlx","version":2},{"content":{"summary":{"value":"This paper introduces TRACE, an encoder-decoder transformer architecture aimed at temporal reconstruction of cellular events by modeling gene expression generation as a sequence-to-sequence task. In this setup, given a source single-cell gene expression sequence at a particular time point, the model generates a target sequence for a subsequent time point. Due to the non-autoregressive nature of single-cell data, TRACE employs an iterative unmasking generation style rather than the traditional autoregressive generation common in transformers. TRACE's efficacy is demonstrated across three biological datasets, highlighting its ability to generate data both in-distribution and out-of-distribution."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"- Could the authors expand on some of the methodological points mentioned in the “Areas for Improvement” section?\n- The iterative unmasking approach in TRACE bears a resemblance to discrete diffusion with an absorbing transition matrix. Did the authors explore this connection?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The authors apply sequence-to-sequence modeling techniques to modeling temporal dynamics of the single-cell data.\n2. The proposed method doesn’t suffer from high dimensionality which certain methods like OT-CFM suffers."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Some sections need better clarity, especially the Method section. Here are a few areas where additional explanation might be helpful:\n\n- The training and inference pipeline could be more thoroughly described, and Figure 1 could more closely align with the text description. A few specific questions came to mind:\n  - How is the embedding \\(Y_t\\) assigned?\n  - What is the process for generating gene tokens \\(X_t\\)?\n  - In Figure 1, should the target tokens be masked?\n\n- In addition, in the Introduction, the authors list several advantages of TRACE:\n   1. It handles high-dimensional single-cell data without dimensionality reduction.\n   2. It provides a learned embedding space for transformation between token and gene expression spaces.\n   3. It integrates well with foundation model pretraining and offers scalability.\n   4. It avoids challenges like posterior collapse often seen in VAE training.\n  5. Its learned embedding space may benefit diffusion and flow-matching models.\n\nHowever, only the first three points seem to be directly supported by experimental results. Providing further experimental validation for points (d) and (e) would strengthen the paper."}},"nonreaders":[],"tmdate":1731428975381,"tcdate":1730513041248,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11664/Reviewer_sYbB"],"signatures":["ICLR.cc/2025/Conference/Submission11664/Reviewer_sYbB"],"forum":"PHESUVacAw","number":1,"license":"CC BY 4.0","cdate":1730513041248,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11664/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428975381,"domain":"ICLR.cc/2025/Conference","replyto":"PHESUVacAw","id":"OY1UmhbrEq","forumContent":{"TLDR":{"value":"We propose a generative model for temporal single cell generation."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["self-supervised learning","single cell RNAseq","generative modeling","temporal sequence modeling"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Single-cell omics technologies capture molecular snapshots of cells, while most\nbiological processes unfold over time. Accurately predicting single-cell gene ex-\npression at unmeasured time points enhances our understanding of these processes,\nreducing costs and experimental effort by enabling the interpolation and extrap-\nolation of observed data. This helps study continuous development, response to\nperturbations, and disease progression. To address this problem, we propose an\nencoder-decoder transformer architecture for Temporal Reconstruction of Cellular\nEvents (TRACE). TRACE models gene expression generation as a sequence-to-\nsequence generation task by learning to transform a sequence of genes from a\nsource condition (e.g., previous time) into a sequence of genes in a target condition\n(e.g., next time point). TRACE decoder learns to generate gene tokens of the target\ncondition by iteratively unmaking tokens in the target sequence, overcoming the dis-\ncordance between autoregressive modeling and the non-sequential nature of gene\nexpression data. We evaluate TRACE both quantitatively and qualitatively on three\ndatasets, covering a range of tasks and biological scenarios. TRACE outperforms\nexisting models in generalizing across in-distribution and out-of-distribution tasks\nfor temporal prediction. Furthermore, we demonstrate the biological relevance of\nthe cell embeddings learned by TRACE by delineating activation-dependent cell\nstages in immune cells, measured across multiple time points. Our findings suggest\nthat TRACE can enhance in silico hypothesis generation, improving our under-\nstanding and prediction of cellular changes over time. This ultimately facilitates\ndisease understanding and supports the design of cost-effective experiments for\nbiological discovery."},"_bibtex":{"value":"@misc{\nvahidi2024sequencetosequence,\ntitle={Sequence-to-sequence modeling for Temporal Reconstruction of Cellular Events},\nauthor={Amirhossein Vahidi and Kevin Ly and Hesam Asadollahzadeh and Marie Moullet and Vijaya Baskar MS and Emily Stephenson and Muzlifah Haniffa and Gosia Trynka and Mohammad Lotfollahi},\nyear={2024},\nurl={https://openreview.net/forum?id=PHESUVacAw}\n}"},"title":{"value":"Sequence-to-sequence modeling for Temporal Reconstruction of Cellular Events"},"pdf":{"value":"/pdf/288d53ffab052342d26939cb6ecbe1dc9f512c80.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"vahidi|sequencetosequence_modeling_for_temporal_reconstruction_of_cellular_events"},"authorids":{"value":["~Amirhossein_Vahidi1","~Kevin_Ly1","~Hesam_Asadollahzadeh1","~Marie_Moullet1","~Vijaya_Baskar_MS1","~Emily_Stephenson1","~Muzlifah_Haniffa1","~Gosia_Trynka1","~Mohammad_Lotfollahi1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Amirhossein Vahidi","Kevin Ly","Hesam Asadollahzadeh","Marie Moullet","Vijaya Baskar MS","Emily Stephenson","Muzlifah Haniffa","Gosia Trynka","Mohammad Lotfollahi"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2024"},"TLDR":{"value":"GateLoop generalizes linear recurrent models (e.g. S4, S5, LRU, RetNet) using data-controlled state-transitions."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Data-controlled","Linear Recurrence","Sequence Modeling","GateLoop","Linear","RNN","State Space Model","SSM","S4","S5","LRU","RetNet","generalization"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Linear Recurrence has proven to be a powerful tool for modeling long sequences efficiently. In this work, we show that existing models fail to take full advantage of its potential. Motivated by this finding, we develop GateLoop, a foundational sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet, by employing data-controlled state transitions. \nUtilizing this theoretical advance, GateLoop empirically outperforms existing models for auto-regressive language modeling. Our method comes with a low-cost $O(l)$ recurrent mode and an efficient $O(l \\log_{2} l)$ parallel mode making use of highly optimized associative scan implementations. Furthermore, we derive an $O(l^2)$ surrogate-attention mode, revealing remarkable implications for Transformer and recently proposed architectures. \nSpecifically, we prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention. \nWhile many existing models solely rely on data-controlled cumulative sums for context aggregation, our findings suggest that incorporating data-controlled complex cumulative products may be a crucial step towards more powerful sequence models."},"_bibtex":{"value":"@misc{\nkatsch2024gateloop,\ntitle={GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling},\nauthor={Tobias Katsch},\nyear={2024},\nurl={https://openreview.net/forum?id=02Ug9N8DCI}\n}"},"title":{"value":"GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling"},"pdf":{"value":"/pdf/539cf74fff3ea51ab4ed275e6f34e07c55215962.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"katsch|gateloop_fully_datacontrolled_linear_recurrence_for_sequence_modeling"},"authorids":{"value":["~Tobias_Katsch1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tobias Katsch"]}},"tmdate":1707625743277,"tcdate":1695445280913,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6897/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission6897/Authors"],"forum":"02Ug9N8DCI","number":6897,"cdate":1695445280913,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/-/Submission","ICLR.cc/2024/Conference/-/Post_Submission","ICLR.cc/2024/Conference/Submission6897/-/Revision","ICLR.cc/2024/Conference/Submission6897/-/Rebuttal_Revision","ICLR.cc/2024/Conference/-/Edit"],"mdate":1707625743277,"odate":1697213872796,"domain":"ICLR.cc/2024/Conference","id":"02Ug9N8DCI","version":2},{"content":{"venue":{"value":"EMNLP (Findings) 2024"},"pdf":{"value":"https://aclanthology.org/2024.findings-emnlp.827.pdf"},"venueid":{"value":"dblp.org/conf/EMNLP/2024"},"paperhash":{"value":"eom|querybased_crossmodal_projector_bolstering_mamba_multimodal_llm"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:SooHwan_Eom:","https://dblp.org/search/pid/api?q=author:Jay_Shim:","https://dblp.org/search/pid/api?q=author:Gwanhyeong_Koo:","https://dblp.org/search/pid/api?q=author:Haebin_Na:","https://dblp.org/search/pid/api?q=author:Mark_Hasegawa-Johnson:","~Sungwoong_Kim1","https://dblp.org/search/pid/api?q=author:Chang_Dong_Yoo:"]},"html":{"value":"https://aclanthology.org/2024.findings-emnlp.827"},"_bibtex":{"value":"@inproceedings{DBLP:conf/emnlp/EomSKNHKY24,\n  author={SooHwan Eom and Jay Shim and Gwanhyeong Koo and Haebin Na and Mark Hasegawa-Johnson and Sungwoong Kim and Chang Dong Yoo},\n  title={Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM},\n  year={2024},\n  cdate={1704067200000},\n  pages={14158-14167},\n  url={https://aclanthology.org/2024.findings-emnlp.827},\n  booktitle={EMNLP (Findings)},\n  crossref={conf/emnlp/2024f}\n}\n"},"abstract":{"value":"The Transformer’s quadratic complexity with input length imposes an unsustainable computational load on large language models (LLMs). In contrast, the Selective Scan Structured State-Space Model, or Mamba, addresses this computational challenge effectively. This paper explores a query-based cross-modal projector designed to bolster Mamba’s efficiency for vision-language modeling by compressing visual tokens based on input through the cross-attention mechanism. This innovative projector also removes the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLM. Experimental results across various vision-language understanding benchmarks show that the proposed cross-modal projector enhances Mamba-based multimodal LLMs, boosting both performance and throughput."},"title":{"value":"Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM"},"authors":{"value":["SooHwan Eom","Jay Shim","Gwanhyeong Koo","Haebin Na","Mark Hasegawa-Johnson","Sungwoong Kim","Chang Dong Yoo"]}},"tmdate":1736611107911,"pdate":1704067200000,"tcdate":1736611103015,"writers":["~"],"signatures":["~Sungwoong_Kim2"],"forum":"1vUlsvEFaN","license":"CC BY-SA 4.0","number":264512,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1736611107911,"domain":"DBLP.org","id":"1vUlsvEFaN","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["speech enhancement","graph neural networks","state space models","cross-frame information reuse","non-uniform speech degradation"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Single-channel speech enhancement (SE) remains challenging when degradation exhibits temporal heterogeneity. Most existing methods process speech as temporally ordered sequences, without frame-level reliability estimation or direct reuse of information from reliable frames across time. To address this problem, we propose SFG-Mamba, a framework that incorporates a reliability-aware Speech Frame Graph (SFG) to model reliability-guided cross-temporal information transfer. A Speech Time-Frequency Graph Convolution (STGC) module further integrates non-local graph information with local time-frequency representations, followed by bidirectional Mamba for long-range temporal modeling. Experiments on VoiceBank+DEMAND and DNS demonstrate the effectiveness of SFG-Mamba, achieving PESQ scores of 3.57 and 3.52, respectively. Furthermore, additional experiments are designed to investigate the performance drop under increasing temporal heterogeneity. SFG-Mamba achieves the best performance among the compared methods."},"_bibtex":{"value":"@inproceedings{\nanonymous2026sfgmamba,\ntitle={{SFG}-Mamba: Reliability-Aware Speech Frame Graph Propagation with State-Space Modeling for Speech Enhancement},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=tfAcgxfUbN},\nnote={under review}\n}"},"title":{"value":"SFG-Mamba: Reliability-Aware Speech Frame Graph Propagation with State-Space Modeling for Speech Enhancement"},"pdf":{"value":"/pdf/eb105a9f6e20439d5f4adaee2fb19bbcd8458de5.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791234729606,"tcdate":1789787726160,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission52912/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission52912/Authors"],"forum":"tfAcgxfUbN","license":"CC BY 4.0","number":52912,"cdate":1789787726160,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission52912/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791234729606,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"tfAcgxfUbN","version":2},{"content":{"summary":{"value":"This paper proposes the Global Event Transformer (GET), a generative framework that fundamentally reframes dynamic link prediction as global event sequence generation. By processing the entire graph history as a single sequence and leveraging Transformer with ALiBi for continuous-time modeling, GET aims to overcome the limited receptive fields and inference scalability issues inherent in prior local/discriminative methods. The architecture integrates modular structural priors. Experiments show GET achieves competitive accuracy while delivering substantial speedup in inference, supporting its utility for large-scale dynamic graphs."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1.\tThe GNN enhancement is computed on a graph derived from *recent events* (L155). Given the goal of capturing global dependencies across the entire sequence $X_{1:n}$, why is the structural prior derived from a local temporal window rather than being a global invariant representation?\n\n2.\tThe next event is generated as $P(s_{n+1}) \\cdot P(d_{n+1}|s_{n+1}) \\cdot P(t_{n+1}|s_{n+1}, d_{n+1})$. In the dual-head architecture, $d_{n+1}$ and $t_{n+1}$ are predicted from the same context $\\mathbf{h}^{(L)}$. How do you ensure the predicted time $\\hat{t}_{n+1}$ is temporally plausible *given* the specific predicted destination $\\hat{d}_{n+1}$ without an explicit conditional link in the decoder?\n\n3.\tSince GET's final task is discriminative link prediction, but the model is optimized using contrastive classification over a sampled set $C$, what is the performance gap if you were to evaluate MRR over the *entire* node set $V$ (full ranking) instead of the contrastive candidate set?\n\n4.\tThe balance parameter $\\lambda$ controls the trade-off between $\\mathcal{L}_{\\text{node}}$ and $\\mathcal{L}_{\\text{time}}$. Given that $\\mathcal{L}_{\\text{time}}$ is crucial for temporal patterns (L417), what practical strategy or metric guided the selection of $\\lambda=0.4$ (Table 6) across diverse datasets like highly sparse `tgbl-review` and dense `tgbl-flight`?\n\n5.\tWhy was the 3-token subsequence $[\\text{SEP}, s_i, d_i]$ chosen over the 4-token sequence $[\\text{SEP}, s_i, d_i, t_i]$? Incorporating $t_i$ as a token might allow the Transformer to directly attend to temporal magnitude, potentially strengthening the role of the temporal dimension in the sequence."},"rating":{"value":4},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1.\tThis paper provides a fundamental paradigm shift for scalability. The shift from a multiplicative ($O(K \\cdot T_{\\text{propagate}})$) discriminative inference to an additive ($O(L^2 d + Kd)$) generative paradigm effectively resolves the major bottleneck for real-time applications with massive candidate sets.\n\n2.\tGET uniquely combines global sequence modeling with continuous time modeling, enabling the capture of long-range dependencies and non-local patterns previously inaccessible to local or memory-based methods.\n\n3.\tThe framework's modular design allows for flexible injection of structural priors via GNN or TGN-style memory modules, demonstrating complementary strengths tailored to graph density."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tThe GNN encoder (Eq. 2) relies on a computational graph from recent events (L155). Given GET's claim to model global, long-range dependencies, relying on a locally constructed graph for structural priors seems contradictory to the global modeling objective.\n\n2.\tThe dual-head decoder predicts $d_{n+1}$ (contrastive classification) and $t_{n+1}$ (regression) independently from $\\mathbf{H}^{(L)}$. There is no explicit mechanism ensuring that the predicted time $t_{n+1}$ is causally or consistently aligned with the predicted node $d_{n+1}$, which could introduce incoherence in the generated event.\n\n3.\tThe model is evaluated on discriminative link prediction but uses a generative mechanism. The cross-entropy loss $\\mathcal{L}_{\\text{node}}$ (Eq. 12) is inherently designed for classification over a *small, sampled* candidate set $C$, which might lead to inflated performance metrics (MRR) compared to full-set evaluation or rigorous ranking fidelity.\n\n4.\tAlthough the paper's core redefinition is \"generative,\" the evaluation primarily focuses on the discriminative task (link prediction). The brief generative analysis in Appendix E is insufficient to fully validate the fidelity and quality of the generated events, especially on metrics beyond edge overlap."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919729969,"tcdate":1761816989131,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7662/Reviewer_Zw8m"],"signatures":["ICLR.cc/2026/Conference/Submission7662/Reviewer_Zw8m"],"forum":"786oOfRVXO","number":2,"license":"CC BY 4.0","cdate":1761816989131,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7662/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919729969,"domain":"ICLR.cc/2026/Conference","replyto":"786oOfRVXO","id":"bsDqIgZAjQ","forumContent":{"TLDR":{"value":"A generative Transformer-based model for dynamic graph learning with structural-temporal context and efficient sequence prediction."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Graph Learning","Temporal Graph","Transformer","Link Prediction","Dynamic Graph"]},"supplementary_material":{"value":"/attachment/7801304ce9ec41fd5628c04e9a899fbba0d682b1.zip"},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"The prevailing paradigms in dynamic graph learning, which rely on local neighbor propagation or node-specific memory updates, are fundamentally limited in their ability to model global interaction patterns and to scale inference efficiently. Rethinking these challenges, we propose GET (Global Event Transformer), an event sequence modeling framework that reformulates dynamic graph learning as global event sequence modeling. By processing the entire interaction history as a unified stream of events, GET is designed to capture dependencies beyond the local receptive fields of prior methods, while its \"encode once, score many\" design removes the candidate-wise inference bottleneck of discriminative models. GET flexibly incorporates structural priors through lightweight GNN encoders and memory modules, which provide local structural and temporal cues that the global Transformer then reasons over. Our main focus in this paper is dynamic link prediction under standard Temporal Graph Benchmark (TGB) protocols, complemented by preliminary short-horizon generative evaluations on small/medium graphs. Extensive experiments on five large-scale TGB benchmarks and six additional datasets show that GET achieves strong competitive performance while delivering substantially faster inference throughput (up to 21.4 × on tgbl-wiki) compared to strong baselines."},"_bibtex":{"value":"@misc{\nchen2026get,\ntitle={{GET}: Rethinking Dynamic Graph Learning as Global Event Sequence Generation},\nauthor={Libin Chen and Xiyuan Wang and Xiaohui Zhang and Yanbo Wang and Luyao Wang and Hongfu Liu and Jing Chen and Muhan Zhang},\nyear={2026},\nurl={https://openreview.net/forum?id=786oOfRVXO}\n}"},"title":{"value":"GET: Rethinking Dynamic Graph Learning as Global Event Sequence Generation"},"pdf":{"value":"/pdf/25d8ba81cbc55477976c27c98b7feff8976f3264.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"chen|get_rethinking_dynamic_graph_learning_as_global_event_sequence_generation"},"authorids":{"value":["~Libin_Chen1","~Xiyuan_Wang1","~Xiaohui_Zhang9","~Yanbo_Wang2","~Luyao_Wang4","~Hongfu_Liu5","~Jing_Chen37","~Muhan_Zhang1"]},"authors":{"value":["Libin Chen","Xiyuan Wang","Xiaohui Zhang","Yanbo Wang","Luyao Wang","Hongfu Liu","Jing Chen","Muhan Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a robust and unified model that offers both flexibility and adaptability for designing sequence models intended for learning on graphs. The architecture includes three main modules: (1) Tokenization, (2) Local Encoding, and (3) Global Encoding, which facilitates effortless and efficient comparison between graph sequence models in terms of their performance on graph learning tasks, including RNN, SSM, etc. The authors also provide a theoretical analysis of the recurrent models and transformer-based graph models. An enhanced model, GSM++, is proposed based on empirical findings. GSM++ is specifically designed with three enhancement techniques: Hierarchical Affinity Clustering, a mixture of tokenization, and hybrid encoders."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. The term \"softmax non-causal Transformers\" requires a clearer definition. What is the relation between non-causality and permutation equivariance? How does this property specifically benefit graph-based tasks?\n2. The discrete router mechanism in the Model-of-Thought (MoT) framework requires further elaboration. Is it implemented as a learned component with trainable parameters, or does it follow predetermined heuristic rules?\n3. The HAC tree implementation raises several computational concerns.  What is the time complexity for constructing the HAC tree, especially for large graphs? How computationally intensive is the hierarchical positional encoding computation? Are there any optimizations or approximations that can be used to make these steps more efficient?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. The paper is well-written and easy to follow. Theorems are proved in appendix. \n2. The proposed framework (tokenizer, local encoding, and global encoding) unifies most existing graph sequence models, enabling effortless evaluation and creation of GSMs within a unified setting. \n3. Experimental results are promising. GSM++(MoT) achieves state-of-the-art performance in most cases."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Critical experimental details are missing. The paper omits essential hyper-parameter configurations, including the number of self-attention layers and hidden dimensions utilized in GSM++. Such omissions hinder reproducibility and meaningful comparison with other models.\n2. The architectural design choices in GSM++ lack comprehensive ablation studies. While the combination of Mamba and attention mechanisms demonstrate improved performance, alternative architectural variants (e.g., different stacking patterns of Mamba and attention layers) remain unexplored. The superior performance of the ensemble approach, though noteworthy, is an expected outcome rather than a novel contribution.\n3. Appendix H (Line1544) mentioned that baselines include state-of-the-art GTs and recurrent graph models (e.g., graph mamba and GRED). However, the empirical performance of these baselines are missing.\n4. MoT requires concatenation of sequence encodings generated by two different tokenizers, which significantly increases the computational complexity. GSM++ may suffer from scalability issues when applying larger datasets and more complex tasks.\n5. The efficiency and complexity of the proposed GSM++ is missing. The paper should provide theoretical complexity analysis compared to existing Graph Transformers and recurrent models; training and inference time, memory cost, etc."}},"nonreaders":[],"tmdate":1731428293129,"tcdate":1731210217937,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13617/Reviewer_brCn"],"signatures":["ICLR.cc/2025/Conference/Submission13617/Reviewer_brCn"],"forum":"iaHghgG8NR","number":4,"license":"CC BY 4.0","cdate":1731210217937,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13617/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428293129,"domain":"ICLR.cc/2025/Conference","replyto":"iaHghgG8NR","id":"OjXFuEb51K","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Graph Learning","Sequence Models","Graph Transformers","Hierarchical Clustering"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Modern sequence models (e.g., Transformers, linear RNNs, etc.) emerged as dominant backbones of recent deep learning frameworks, mainly due to their efficiency, representational power, and/or ability to capture long-range dependencies. Adopting these sequence models for graph-structured data has recently gained popularity as the alternative to Message Passing Neural Networks (MPNNs). There is, however, a lack of a common foundation about what constitutes a good graph sequence model, and a mathematical description of the benefits and deficiencies in adopting different sequence models for learning on graphs. To this end, we first present Graph Sequence Model (GSM), a unifying framework for adopting sequence models for graphs, consisting of three main steps: (1) Tokenization, which translates the graph into a set of sequences; (2) Local Encoding, which encodes local neighborhoods around each node; and (3) Global Encoding, which employs a scalable sequence model to capture long-range dependencies within the sequences. This framework allows us to understand, evaluate, and compare the power of different sequence model backbones in graph tasks. Our theoretical evaluations of the representation power of Transformers and modern recurrent models through the lens of global and local graph tasks show that there are both negative and positive sides for both types of models. Building on this observation, we present GSM++, a fast hybrid model that uses the Hierarchical Affinity Clustering (HAC) algorithm to tokenize the graph into hierarchical sequences, and then employs a hybrid architecture of Transformer to encode these sequences. Our theoretical and experimental results support the design of GSM++, showing that GSM++ outperforms baselines in most benchmark evaluations."},"_bibtex":{"value":"@misc{\nbehrouz2025towards,\ntitle={Towards an Understanding of Graph Sequence Models},\nauthor={Ali Behrouz and Ali Parviz and Mahdi Karami and Clayton Sanford and Bryan Perozzi and Vahab Mirrokni},\nyear={2025},\nurl={https://openreview.net/forum?id=iaHghgG8NR}\n}"},"title":{"value":"Towards an Understanding of Graph Sequence Models"},"pdf":{"value":"/pdf/bada696af3040c83282cf5eb2af8e982e43e9a1e.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"behrouz|towards_an_understanding_of_graph_sequence_models"},"authorids":{"value":["~Ali_Behrouz1","~Ali_Parviz1","~Mahdi_Karami2","~Clayton_Sanford1","~Bryan_Perozzi1","~Vahab_Mirrokni2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Ali Behrouz","Ali Parviz","Mahdi Karami","Clayton Sanford","Bryan Perozzi","Vahab Mirrokni"]}},"version":2},{"content":{"summary":{"value":"The paper presents DeMo, a novel framework for motion forecasting in autonomous driving systems. DeMo decouples the motion forecasting task into two distinct components: mode queries for capturing directional intentions and state queries for modeling dynamic states over time. This separation allows DeMo to separately optimize for multi-modality and dynamic state evolution, leading to a more comprehensive representation of future trajectories. The framework employs a combination of Attention and Mamba techniques for global information aggregation and state sequence modeling. Extensive experiments on the Argoverse 2 and nuScenes benchmarks demonstrate that DeMo achieves state-of-the-art performance in motion forecasting."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. In the first line of Eq. (2), what do the symbols {$\\{ t_1, t_2, ...t_{T_s} \\}$} represent, and how do $T_s$ and $T_f$ differ from or relate to each other? How are these time values obtained and utilized in the model?\n\n2. For the motion model $Q_m$, what specific features are considered for different motion models? The paper does not clearly specify the features used in these motion models.\n\n3. In Section 3.4, the loss function $L_{ts}$ is described as related to \"intermediate features of time state.\" What exactly are these intermediate features, and what is their role in the model? This concept is not clearly defined in the paper.\n\n4. Table 1 does not include baseline methods such as SEPT and SEPT++ that have better performance and earlier submission dates than DeMo. Similarly, Table 3 omits models like QCNet, which outperform DeMo. Why are these models not included in the comparisons?\n\n5. The paper lacks detailed information about the model’s parameters and settings. Can you provide more specific details on the parameters used for DeMo?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The overall idea behind DeMo is reasonable and technical soundness. Additionally, the experiments are comprehensive, demonstrating the results of DeMo across two different datasets, Argoverse 2 and nuScenes."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"There are two major weaknesses: 1. Technical Descriptions Lack Clarity: The technical explanations of the methods and algorithms used in DeMo are not sufficiently clear, making it challenging for readers to fully understand the proposed techniques and their implementations. 2. Unconvincing Contributions: The claimed contributions of the paper are not entirely convincing. For instance, the paper doesn’t compare DeMo to models that have superior performance on existing leaderboards, questioning the novelty of its improvements. Additionally, there are previous works (e.g. Motion Mamba by Zhang et.al.) that have already combined Mamba and Transformer techniques for time series motion data, which undermines the claim that DeMo introduces a novel approach."},"limitations":{"value":"The technical descriptions lack clarity and the overall contribution is limited."}},"nonreaders":[],"tmdate":1730878665765,"tcdate":1720837027260,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission896/Reviewer_pCB4"],"signatures":["NeurIPS.cc/2024/Conference/Submission896/Reviewer_pCB4"],"forum":"rbtnRsiXSN","number":4,"license":"CC BY 4.0","cdate":1720837027260,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission896/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878665765,"domain":"NeurIPS.cc/2024/Conference","replyto":"rbtnRsiXSN","id":"REM4cHslRL","forumContent":{"TLDR":{"value":"A framework that decouples multi-modal trajectory queries into mode queries for directional intentions and state queries for dynamic states, utilizing Attention and Mamba."},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Motion Forecasting","Autonomous Driving","Mamba","Attention"]},"primary_area":{"value":"robotics"},"abstract":{"value":"Accurate motion forecasting for traffic agents is crucial for ensuring the safety and efficiency of autonomous driving systems in dynamically changing environments. Mainstream methods adopt a one-query-one-trajectory paradigm, where each query corresponds to a unique trajectory for predicting multi-modal trajectories. While straightforward and effective, the absence of detailed representation of future trajectories may yield suboptimal outcomes, given that the agent states dynamically evolve over time. To address this problem, we introduce DeMo, a framework that decouples multi-modal trajectory queries into two types: mode queries capturing distinct directional intentions and state queries tracking the agent's dynamic states over time. By leveraging this format, we separately optimize the multi-modality and dynamic evolutionary properties of trajectories. Subsequently, the mode and state queries are integrated to obtain a comprehensive and detailed representation of the trajectories. To achieve these operations, we additionally introduce combined Attention and Mamba techniques for global information aggregation and state sequence modeling, leveraging their respective strengths. Extensive experiments on both the Argoverse 2 and nuScenes benchmarks demonstrate that our DeMo achieves state-of-the-art performance in motion forecasting. In addition, we will make our code and models publicly available."},"_bibtex":{"value":"@inproceedings{\nzhang2024demo,\ntitle={DeMo: Decoupling Motion Forecasting into  Directional Intentions and Dynamic States},\nauthor={Bozhou Zhang and Nan Song and Li Zhang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=rbtnRsiXSN}\n}"},"title":{"value":"DeMo: Decoupling Motion Forecasting into  Directional Intentions and Dynamic States"},"pdf":{"value":"/pdf/3d4994978d6ca281a3b560f62d73cecc4d2310f7.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"zhang|demo_decoupling_motion_forecasting_into_directional_intentions_and_dynamic_states"},"authorids":{"value":["~Bozhou_Zhang1","~Nan_Song4","~Li_Zhang5"]},"authors":{"value":["Bozhou Zhang","Nan Song","Li Zhang"]}},"version":2},{"content":{"comment":{"value":"**1. Limited discussion of causal sequence modeling approaches: Although the method addresses linear sequence encoding with foreground focus, it omits explicit comparison to other causal sequence modeling alternatives or other advanced state-space models.**\n\nWe appreciate the reviewer’s suggestion regarding the performance of diverse causal sequence modeling approaches. The results in Tables 1, 2, and 3 in the paper demonstrate the effectiveness of our approach compared with previous Mamba-based 3D detectors, such as LION and Voxel-Mamba. Moreover, we also compare the performance of the Mamba model with other alternatives, which is shown in the following Table. Specifically, we only replace the sequence encoding part with existing methods (e.g. RetNet (Ref.1), RWKV (Ref.2), and LSTM (Ref.3)) and maintain other components consistent with our original pipeline. We train all these alternatives from scratch in the nuScenes dataset with the same training configuration. The results prove that the Mamba model achieves the highest scores in both NDS and mAP metrics. Meanwhile, the computation of Mamba is also more efficient compared with the recent RetNet and RWKV approaches. Given the theoretically and empirically validated capability of Mamba in sequence modeling, we adopt it as the default linear encoder throughout our paper. We have also included this comparison in Table 12 of the supplementary material in the revised version (highlighted in 842-858 lines of our revision).\n\n| Model | NDS | mAP | FLOPs (G) $\\downarrow$ | FPS |\n| :--- | :---: | :---: | :---: | :---: |\n| RetNet | 72.1 | 67.7 | 31.12 | 59 |\n| RWKV | 71.9 | 67.1 | 36.15 | 47 |\n| LSTM | 70.8 | 65.9 | 23.92 | 78 |\n| Mamba (Ours) | 72.3 | 68.4 | 26.04 | 67 |\n\nRef.1: Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei, “Retentive network: A successor to transformer for large language models,” arXiv preprint arXiv:2307.08621, 2023.\n\nRef.2: B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella et al., “Rwkv: Reinventing rnns for the transformer era,” arXiv preprint arXiv:2305.13048, 2023.\n\nRef.3: S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997."},"title":{"value":"Response to Reviewer cXEa Q1"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763950253731,"tcdate":1763950253731,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10258/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission10258/Authors"],"forum":"e4t1775UJ1","number":1,"license":"CC BY 4.0","cdate":1763950253731,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10258/-/Official_Comment"],"mdate":1763950253731,"domain":"ICLR.cc/2026/Conference","replyto":"VcvZ9cIFvD","id":"fC6S9lqCqW","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["3D Object Detection","Mamba","Foreground Representation"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Linear modeling methods like Mamba have been merged as the effective backbone for the 3D object detection task. However, previous Mamba-based methods utilize the bidirectional encoding for the whole non-empty voxel sequence, which contains abundant useless background information in the scenes. Though directly encoding foreground voxels appears to be a plausible solution, it tends to degrade detection performance. We attribute this to the response attenuation and restricted context representation in the linear modeling for fore-only sequences. To address this problem, we propose a novel backbone, termed Fore-Mamba3D, to focus on the foreground enhancement by modifying Mamba-based encoder. The foreground voxels are first sampled according to the predicted scores. Considering the response attenuation existing in the interaction of foreground voxels across different instances, we design a regional-to-global slide window (RGSW) to propagate the information from regional split to the entire sequence. Furthermore, a semantic-assisted and state spatial fusion module (SASFMamba) is proposed to enrich contextual representation by enhancing semantic and geometric awareness within the Mamba model. Our method emphasizes foreground-only encoding and alleviates the distance-based and causal dependencies in the linear autoregression model. The superior performance across various benchmarks demonstrates the effectiveness of Fore-Mamba3D in the 3D object detection task."},"_bibtex":{"value":"@inproceedings{\nning2026foremambad,\ntitle={Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection},\nauthor={Zhiwei Ning and Xuanang Gao and Jiaxi Cao and Runze Yang and Huiying Xu and Xinzhong Zhu and JIE YANG and Wei Liu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=e4t1775UJ1}\n}"},"title":{"value":"Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection"},"pdf":{"value":"/pdf/c88d1a4b5a3d89c3710599dac8b4ca8c7c9c1d6e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"ning|foremamba3d_mambabased_foregroundenhanced_encoding_for_3d_object_detection"},"authorids":{"value":["~Zhiwei_Ning2","~Xuanang_Gao1","~Jiaxi_Cao1","~Runze_Yang2","~Huiying_Xu1","~Xinzhong_Zhu1","~JIE_YANG18","~Wei_Liu38"]},"authors":{"value":["Zhiwei Ning","Xuanang Gao","Jiaxi Cao","Runze Yang","Huiying Xu","Xinzhong Zhu","JIE YANG","Wei Liu"]}},"version":2},{"content":{"summary":{"value":"The paper “MBMamba: When Memory Buffer Meets Mamba for Structure-Aware Image Deblurring” aims to enhance the spatial structure modeling capability of Mamba-based image deblurring frameworks. The authors point out that the traditional Mamba’s “flatten-and-scan” strategy leads to local pixel forgetting and channel redundancy, thereby weakening spatial feature aggregation. To address this issue, they propose the MBMamba framework, whose core component is the Memory Buffering Mechanism (MemVSSM). This module preserves historical feature information through a memory buffer and fuses it via a Feature Cross-Attention Module (FCAM), strengthening local context modeling. In addition, an Ising-inspired regularization loss is designed to simulate pixel-level “mutual attraction” energy minimization, thereby maintaining structural consistency."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"See weakness"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The MemVSSM module is designed to enhance local context preservation via feature partitioning and temporal buffering. The physically inspired Ising Loss is incorporated into the deblurring loss function, enforcing spatial coherence through energy minimization. Notably, this module extends the spatial-awareness capability of the Mamba backbone without altering its main architecture, achieving both efficiency and accuracy.\n\nThe proposed method provides a new perspective on memory-based state-space modeling in visual tasks. It achieves superior results across four major benchmarks — GoPro, HIDE, RealBlur-R, and RealBlur-J — while maintaining significantly lower FLOPs and inference latency compared to Transformer-based methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The title and abstract emphasize “Structure-Aware”, yet the manuscript does not present any structural-awareness visualizations or heatmaps (No direct evidence found in the manuscript).\n\nThe paper does not discuss how the depth (K) of MemVSSM or the number of channel partitions (N) affects performance. Similarly, no comparison is provided against a non-memory Mamba baseline trained in a single step.\n\nSections 3.2 and 3.3 describe MemVSSM and the Ising Loss separately but do not explain their training interaction or coupling mechanism. The hierarchical integration of SFEM/FCAM is also unclear — Figure 3 shows only a schematic representation without specifying whether the connection is serial or parallel.\n\nFinally, the authors omit details about hardware configurations, such as GPU type or training time. The hyperparameters λ and δ are adopted from prior work without justification or tuning analysis, leaving their choice insufficiently explained."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919381960,"tcdate":1761479731204,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7244/Reviewer_jJFq"],"signatures":["ICLR.cc/2026/Conference/Submission7244/Reviewer_jJFq"],"forum":"uAmbHy4R8f","number":1,"license":"CC BY 4.0","cdate":1761479731204,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7244/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919381960,"domain":"ICLR.cc/2026/Conference","replyto":"uAmbHy4R8f","id":"isC13JB8tM","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Image restoration","Image deblurring","Mamba"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"The Mamba architecture has emerged as a promising alternative to CNNs and Transformers for image deblurring. However, its flatten-and-scan strategy often results in local pixel forgetting and channel redundancy, limiting its ability to effectively aggregate 2D spatial information. Although existing methods mitigate this by modifying the scan strategy or incorporating local feature modules, it increase computational complexity and hinder real-time performance. In this paper, we propose a structure-aware image deblurring network  without changing the original Mamba architecture. Specifically, we design a memory buffer mechanism to preserve historical information for later fusion, enabling reliable modeling of relevance between adjacent features. Additionally, we introduce an Ising-inspired regularization loss that simulates the energy minimization of the physical system's \"mutual attraction\" between pixels, helping to maintain image structure and coherence. Building on this, we develop MBMamba. Experimental results show that our method outperforms state-of-the-art approaches."},"_bibtex":{"value":"@misc{\ngao2026mbmamba,\ntitle={{MBM}amba: When Memory Buffer Meets Mamba for Structure-Aware Image Deblurring},\nauthor={Hu Gao and XiaoNing Lei and Xichen Xu and Depeng Dang and Lizhuang Ma},\nyear={2026},\nurl={https://openreview.net/forum?id=uAmbHy4R8f}\n}"},"title":{"value":"MBMamba: When Memory Buffer Meets Mamba for Structure-Aware Image Deblurring"},"pdf":{"value":"/pdf/aea2419687a8eab97916ee02e0d79b5e35b65c47.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"gao|mbmamba_when_memory_buffer_meets_mamba_for_structureaware_image_deblurring"},"authorids":{"value":["~Hu_Gao1","~XiaoNing_Lei1","~Xichen_Xu1","~Depeng_Dang1","~Lizhuang_Ma1"]},"authors":{"value":["Hu Gao","XiaoNing Lei","Xichen Xu","Depeng Dang","Lizhuang Ma"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["time delay system"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Existing sequence models share a critical limitation: they lack an explicit, interpretable mechanism for content-aware temporal recall. LSTMs compress everything into a single hidden state, thereby losing information about specific past events. Transformers attend to all tokens indiscriminately at quadratic cost, while state-space models like Mamba compress history into a fixed-dimensional state without explicit content-addressable memory. To address this gap, we introduce a novel framework grounded in nonlinear time-delay systems, in which the effective delay is dynamically estimated based on the semantic similarity between current and historical inputs. Our approach features three key innovations: (i) a differentiable semantic lag estimator that computes content-aware soft attention weights over history, producing a weighted historical context and similarity score; (ii) a diagonal state space with closed-form cumulative sum solution, achieving O(1) per-step complexity with no sequential loops; and (iii) an interpretable fuzzy semantic modulation mechanism that learns context-dependent rules for balancing instantaneous processing versus historical recall, where rule centers correspond to prototypical semantic contexts. Experimental results on text classification and generation outperform state-of-the-art methods."},"_bibtex":{"value":"@inproceedings{\nanonymous2026time,\ntitle={Time Delay System for Sequence Modeling},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=ykOlKfDke9},\nnote={under review}\n}"},"title":{"value":"Time Delay System for Sequence Modeling"},"pdf":{"value":"/pdf/4e289c1f6d66a85a7a7e91f95f544880ff7593d3.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791235486833,"tcdate":1789813436819,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission59821/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission59821/Authors"],"forum":"ykOlKfDke9","license":"CC BY 4.0","number":59821,"cdate":1789813436819,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission59821/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791235486833,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"ykOlKfDke9","version":2},{"content":{"summary":{"value":"This paper focuses on addressing the challenge of minimizing linear swap regret in extensive form games, which is considered a stronger notion compared to trigger regret in Extensive Form Correlated Equilibrium (EFCE). To achieve efficient implementation of the Phi-regret minimization problem (where Phi set contains all linear swap transformation), the authors provide a polynomial-sized description of all linear swap operators. This result shows that there exists a no linear swap regret algorithm with polynomial-time iteration complexity based on the size of the game tree. Additionally, the paper presents experimental results that illustrate how the equilibria achieved through the no linear swap regret algorithm exhibit greater strength compared to EFCE."},"soundness":{"value":"3 good"},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"1. Is there any intuition behind the linear swap transformation, apart from its linearity? For instance, can LCE be interpreted with the presence of a mediator? Why do you believe LCE is an equilibrium worth considering?\n\n2. Maximizing social welfare is known to be challenging. However, in your experiments, did you observe any significant differences in social welfare between No-linear-swap-regret dynamics and No-trigger-regret dynamics?\n\n3. Since No-linear-swap-regret dynamics are stronger than No-trigger-regret dynamics. The result in Figure 2 is expected. What about the running times? Is the running time of no-linear-swap-regret dynamics substantially larger?\n\n[1] Zhang, Brian, and Tuomas Sandholm. \"Polynomial-time optimal equilibria with a mediator in extensive-form games.\" Advances in Neural Information Processing Systems 35 (2022): 24851-24863."},"rating":{"value":"6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":"3 good"},"contribution":{"value":"3 good"},"strengths":{"value":"This paper introduces a novel and essential contribution by providing a polynomial characterization of the set of all linear transformations from a sequence-form strategy polytope to itself. This characterization holds significant value for future research on linear-swap-regret.\n\n\n"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The contribution of this paper may be perceived as limited. The primary and significant contribution lies in the polynomial description of the linear swap transformation set. Given the sequential structure of the sequence form strategy, this finding is not entirely unexpected.\n\nIt appears that the paper lacks some intuitive explanations regarding the concept of linear swap (further details can be found in the questions raised). There is a sense that LCE (Linear Swap Correlated Equilibrium) is merely one equilibrium between CE (Correlated Equilibrium) and EFCE (Extensive Form Correlated Equilibrium), which is relatively easy to compute. The experimental results may not comprehensively demonstrate the potential and strength of LCE\n\n"},"limitations":{"value":"The authors have partially addressed the limitations of their work, though there is space for improvement (see the section Weaknesses)."}},"nonreaders":[],"tmdate":1702411446727,"tcdate":1689331221804,"writers":["NeurIPS.cc/2023/Conference","NeurIPS.cc/2023/Conference/Submission13781/Reviewer_V8FX"],"signatures":["NeurIPS.cc/2023/Conference/Submission13781/Reviewer_V8FX"],"forum":"aCOKUvqHtD","number":4,"license":"CC BY 4.0","cdate":1689331221804,"mdate":1702411446727,"readers":["everyone"],"invitations":["NeurIPS.cc/2023/Conference/Submission13781/-/Official_Review","NeurIPS.cc/2023/Conference/-/Edit"],"domain":"NeurIPS.cc/2023/Conference","replyto":"aCOKUvqHtD","id":"HdST9ljL9S","forumContent":{"venue":{"value":"NeurIPS 2023 poster"},"keywords":{"value":["online learning","algorithmic game theory","extensive form games","correlated equilibrium","swap regret","linear swap regret"]},"_bibtex":{"value":"@inproceedings{\nfarina2023polynomialtime,\ntitle={Polynomial-Time Linear-Swap Regret Minimization in Imperfect-Information Sequential Games},\nauthor={Gabriele Farina and Charilaos Pipis},\nbooktitle={Thirty-seventh Conference on Neural Information Processing Systems},\nyear={2023},\nurl={https://openreview.net/forum?id=aCOKUvqHtD}\n}"},"title":{"value":"Polynomial-Time Linear-Swap Regret Minimization in Imperfect-Information Sequential Games"},"paperhash":{"value":"farina|polynomialtime_linearswap_regret_minimization_in_imperfectinformation_sequential_games"},"TLDR":{"value":"We show that it is possible to construct polynomial-time no-regret dynamics with respect to all linear strategy transformations in extensive-form games, thus giving an efficient algorithm converging to a stronger notion of sequential rationality."},"abstract":{"value":"No-regret learners seek to minimize the difference between the loss they cumulated through the actions they played, and the loss they would have cumulated in hindsight had they consistently modified their behavior according to some strategy transformation function. The size of the set of transformations considered by the learner determines a natural notion of rationality. As the set of transformations each learner considers grows, the strategies played by the learners recover more complex game-theoretic equilibria, including correlated \nequilibria in normal-form games and extensive-form correlated equilibria in extensive-form games. At the extreme, a no-swap-regret agent is one that minimizes regret against the set of all functions from the set of strategies to itself. While it is known that the no-swap-regret condition can be attained efficiently in nonsequential (normal-form) games, understanding what is the strongest notion of rationality that can be attained efficiently in the worst case in sequential (extensive-form) games is a longstanding open problem. In this paper we provide a positive result, by showing that it is possible, in any sequential game, to retain polynomial-time (in the game tree size) iterations while achieving sublinear regret with respect to all linear transformations of the mixed strategy space, a notion called no-linear-swap regret. This notion of hindsight rationality is as strong as no-swap-regret in nonsequential games, and stronger than no-trigger-regret in sequential games—thereby proving the existence of a subset of extensive-form correlated equilibria robust to linear deviations, which we call linear-deviation correlated equilibria, that can be approached efficiently."},"pdf":{"value":"/pdf/4fd86c56c6869c7fa4ba995b5385b6a3d39678b8.pdf"},"venueid":{"value":"NeurIPS.cc/2023/Conference"},"authorids":{"value":["~Gabriele_Farina1","~Charilaos_Pipis1"]},"authors":{"value":["Gabriele Farina","Charilaos Pipis"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"Learning Cross-Window Memory for Video Geometry"},"keywords":{"value":["3D Reconstruction","Feed forward Reconstruction","VGGT"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Recovering camera motion and scene geometry from long videos requires strong multi-view reasoning and effective reuse of past observations. Pretrained geometric transformers provide powerful multi-view priors, but their global attention becomes increasingly costly as the input grows. Windowed inference limits this cost while interrupting feature-level communication across windows, which output-level geometric stitching cannot restore. We introduce Mamba-VGGT, a recurrent adapter that extends a frozen Visual Geometry Grounded Transformer with fixed-capacity cross-window memory. A Mamba-based two-dimensional selective scan (SS2D) jointly processes the incoming state and current patch tokens, producing coupled residual feature updates and memory writes. This design retains pretrained within-window attention while allowing subsequent windows to use historical context. State decay and periodic segment resets control state accumulation during long-sequence inference. On KITTI-360 sequences with 2,000 sampled frames, Mamba-VGGT reduces mean absolute trajectory error by 42.3\\% and mean LiDAR-supported reconstruction distance by 56.3\\% relative to LoGeR with sliding-window attention under the evaluated inference configuration. It also achieves lower trajectory error on VBR inputs of 8,000 and 10,000 frames under matched window and reset schedules. Mixer-replacement experiments favor SS2D over a parameter-matched pointwise MLP under the tested training setup, while state-propagation ablations show that carried context further improves trajectory estimation. Together, these results support joint spatial adaptation and cross-window memory as an effective approach to extending frozen geometric models for long-sequence camera estimation and reconstruction."},"_bibtex":{"value":"@inproceedings{\nanonymous2026mambavggt,\ntitle={Mamba-{VGGT}: Explicit Spatial State for Cross-Window Video Geometry},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=boOHV1kROZ},\nnote={under review}\n}"},"title":{"value":"Mamba-VGGT: Explicit Spatial State for Cross-Window Video Geometry"},"pdf":{"value":"/pdf/e59df984aa5fb8c824b685d92f02397ef57105cc.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791220163670,"tcdate":1787646463193,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission4421/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission4421/Authors"],"forum":"boOHV1kROZ","license":"CC BY 4.0","number":4421,"cdate":1787646463193,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission4421/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791220163670,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"boOHV1kROZ","version":2},{"content":{"summary":{"value":"This paper presents a novel approach to improving temporal sequence prediction by introducing history tokens, dedicated tokens that accumulate and represent historical context. These tokens function as dynamic memory units, allowing the model to construct richer and more informative representations of past events during both pretraining and inference. Furthermore, the authors focus their investigation on two primary design dimensions:\n\n1. Placement: They evaluate two strategies for positioning history tokens: uniform distribution across the sequence and end-biased placement closer to the sequence's conclusion.
\n\n2. Attention: They examine how sequence tokens should attend to history tokens, comparing last-token attention with random attention mechanisms.
\n\nExtensive experiments on five datasets spanning finance, e-commerce, and healthcare demonstrate that the proposed HT-Transformer consistently outperforms strong baselines, including RNNs, standard Transformers, Longformer, and Recurrent Memory Transformer (RMT) in different settings. These results underscore the value of integrating history tokens into transformer-based architectures for temporal and event sequence modeling."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. How does the HT-Transformer relate to traditional time series forecasting methods and recent models like PatchTST? Could it be applied to classic forecasting tasks such as next-step prediction or multivariate forecasting, and how well does it capture inductive patterns like seasonality, trends, or locality? \n\n2. What factors influence the optimal number or ratio of history tokens relative to the input length? Are there cases where using too many tokens might introduce redundancy or reduce attention efficiency?\n\n3. Could the placement of history tokens be adapted to the structure of the data or task, for example, aligning with seasonal events, or paragraph or document boundaries in text? Would such task-aware placement improve efficiency or performance?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Novelty and Practicality: This paper introduces a simple yet effective idea called history tokens to flexibly incorporate historical context addressing a key limitation in temporal sequence modeling.
 The concept of history token is very interesting.\n\n2. Well Explored Design Choices: Authors 
provide a systematic study of token placement and attention strategies with solid ablation experiments that enhance both interpretability and adaptability.
\n\n3. Strong Empirical Results
: Experiments demonstrate consistent performance gains over RNNs Transformers Longformer and RMT across five datasets in finance healthcare and e-commerce and so on.
\n\n4. Reproducibility
: Public code are available supporting transparency and facilitating future research."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While this paper presents valuable contributions, addressing the following points could further strengthen its impact and clarity:\n\n1. Clarify the Relationship with CLS Tokens: 
It would be more comprehensive to include a detailed comparison between history tokens and the CLS token mechanism used in models like BERT, other than the history tokens appearing multiple times in a sequence. Elaborating on how they differ functionally and architecturally will help clarify history token's unique role.
\n\n2. Compare with Compressed Vector Methods in Long-Context Transformers:
 Providing a methodological comparison between history tokens and compressed or pooled vector approaches used in long-context models such as Unlimiformer, Linformer, Reformer, sparse/blockwise attention mechanisms, and so on, would contextualize the contribution and highlight advantages of history tokens.
\n\n3. Broaden Experimental Baselines: 
Incorporating additional baselines like BigBird, FlashAttention, ModernBERT, Mamba, and Structured State Space Models (SSMs) would strengthen the empirical evaluation and better position the work within the current state of the art.
\n\n4. Include Efficiency and Computational Cost Analysis:
 Adding an analysis of computational efficiency and resource usage compared to other long-sequence models is important for assessing the practical applicability of the approach. Including such details in the main text will provide readers with a clearer understanding of its scalability.
\n\nAddressing these points would provide greater clarity, strengthen the motivation, and improve the overall impact of the work."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919569183,"tcdate":1761148815379,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7456/Reviewer_xrtb"],"signatures":["ICLR.cc/2026/Conference/Submission7456/Reviewer_xrtb"],"forum":"RxUt170deG","number":2,"license":"CC BY 4.0","cdate":1761148815379,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7456/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919569183,"domain":"ICLR.cc/2026/Conference","replyto":"RxUt170deG","id":"w3JyJQ3H2n","forumContent":{"TLDR":{"value":"We propose the concept of history tokens for unsupervised embedding pretraining in Transformer models using a next-token prediction objective."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Event Sequences","Classification","Transformer","Embedding Extraction"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Deep learning has achieved strong results in modeling sequential data, including event sequences, temporal point processes, and irregular time series. Recently, transformers have largely replaced recurrent networks in these tasks. However, transformers often underperform RNNs in sequence classification tasks that aim to predict future targets. The reason behind this performance gap remains largely unexplored. In this paper, we identify a key limitation of transformers: the absence of a single state vector that provides a compact and effective representation of the entire sequence. Additionally, we show that contrastive pretraining of embedding vectors fails to capture local context, which is crucial for accurate prediction. To address these challenges, we introduce history tokens, a novel concept that facilitates accumulating historical information during next-token prediction pretraining. Our approach significantly improves transformer-based models, achieving impressive results in finance, e-commerce, and healthcare tasks."},"_bibtex":{"value":"@misc{\nkarpukhin2025httransformer,\ntitle={{HT}-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens},\nauthor={Ivan Karpukhin and Andrey Savchenko},\nyear={2025},\nurl={https://openreview.net/forum?id=RxUt170deG}\n}"},"title":{"value":"HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens"},"pdf":{"value":"/pdf/2525062587421376e83d386ae341539b500f063f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"karpukhin|httransformer_event_sequences_classification_by_accumulating_prefix_information_with_history_tokens"},"authorids":{"value":["~Ivan_Karpukhin1","~Andrey_Savchenko1"]},"authors":{"value":["Ivan Karpukhin","Andrey Savchenko"]}},"version":2},{"content":{"comment":{"value":"> Question 1. There is a known equivalence between convolutions and SSMs (this is the basis of S4!). Is it just that directly parameterizing the convolutions is more parameter efficient? Why can't you achieve this fully recurrent? I think at least qualitative discussion of this is absent. \n\nWe agree with the reviewer that this topic warrants a short discussion in the paper. Accordingly, we have added a section to the appendix (see Appendix A.2) that we summarize in the following. \n\nIt is true that temporal convolutions and SSMs are theoretically equivalent. However, during deployment on an embedded system, representing an arbitrary convolution with learnable spacings using a fully recurrent SSM is not as parameter efficient as using the convolution directly. To capture an arbitrary kernel of length $\\Gamma$, a naively parameterized SSM would require a state size of $\\Gamma$ and a corresponding $\\Gamma \\times \\Gamma$ transition matrix, even if only $K \\ll \\Gamma$ weights are actually learnable. Thus, while it is theoretically possible to reformulate our convolution with learnable spacings as a fully recurrent SSM, it would decrease the memory efficiency. Note that this also relates to your questions about Table A6. In addition to the concerns about memory efficiency, reformulating the convolution component as an SSM would complicate the use of DCLS for learning the spacings which is a critical component of mGRADE’s overall architecture.  \n\n### Missing Literature\n\nWe have added a short remark on MEGA to our “Related Works” in Section 5. \n\n### References: \n\n[1] A. Gu et al. Efficiently modeling long sequences with structured state spaces. ICLR 2022. \n\n[2] J. Sieber et al. Understanding the differences in foundation models: Attention, state space models, and recurrent neural networks. Neurips 2024. \n\n[3] S. Siegel et al. IMSSA: Deploying modern state-space models on memristive in-memory compute hardware. ISCAS, 2025 \n\n[4] S. Ward-Foxton. Applied Brain Research Demos first silicon for state-space models. In EE Times, 2025. https://www.eetimes.com/applied-brain-research-demos-first-silicon-for-state-space-models/ \n\n[5] T. Lewis et al. Introducing tenn: Revolutionizing computing with energy efficiency. BrainChip, 2025. https://brainchip.com/introducing-tenn-revolutionizing-computing-with-an-energy-efficient-transformer-replacement/ \n\n[6] J. S. P. Giraldo et al: A 65-nm speech-triggered wake-up soc for 10- µ w keyword spotting and speaker verification. In IEEE Journal of Solid-State Circuits 2020. \n\n[7] V. Jain et al: A tiny versatile system-on-chip with state-retentive mram for ml inference at the extreme edge. In IEEE Journal of Solid-State Circuits 2023. \n\n[8] F. Tan et al.  A 1.8% far, 2 ms decision latency, 1.73 nj/decision keywords-spotting (kws) chip incorporating transfer-computing speaker verification, hybrid-if-domain computing and scalable 5t-sram. In IEEE Journal of Solid-State Circuits 2025. \n\n[9] H. Yang et al. Fsl-hdnn: A 5.7 tops/w end-to-end few-shot learning classifier accelerator with feature extraction and hyperdimensional computing. ESSERC 2024 \n\n[10] D. den Blanken and C. Frenkel. Chameleon: A MatMul-Free Temporal Convolutional Network Accelerator for End-to-End Few-Shot and Continual Learning from Sequential Data. In arXiv, 2025.  \n\n[11] C. Frenkel and G. Indiveri. ReckOn: A 28nm Sub-mm2 Task-Agnostic Spiking Recurrent Neural Network Processor Enabling On-Chip Learning over Second-Long Timescales. ISSCC 2022 \n\n[12] J. Park et al. A 65-nm 236.5 nJ/Classification Neuromorphic Processor with 7.5% Energy Overhead On-Chip Learning Using Direct Spike-Only Feedback. ISSCC 2019 \n\n[13] S. Zhou et al. An 8.62-μW 75-dB DRSoC Fully Integrated SoC for Spoken Language Understanding. In IEEE Journal of Solid-State Circuits 2025 \n\n[14] P. Miralles-Gonzalez et al. On the locality bias and results in the long range arena, arxiv 2025 \n\n[15] L. Feng et al. Were RNNs all we needed?, 2025. URL https://openreview.net/forum?id=GrmFFxGnOR. \n\n[16] D. Balduzzi and M. Ghifary. Strongly-Typed recurrent neural networks. ICML 2016 \n\n[17] J. Bradbury et al. Quasi-recurrent neural networks. ICML 2017 \n\n[18] E. Martin and C. Cundy. Parallelizing linear recurrent neural nets over sequence length. ICLR 2018 \n\n[19] A. Shafiee et al. ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars, ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) 2016 \n\n[20] M. Horowitz. Computing’s energy problem (and what we can do about it). ISSCC 2014 \n\n[21] Albert Gu. Issue #282: Mamba in Long range arena (LRA). GitHub. 2024. https://github.com/state-spaces/mamba/issues/282 \n\n[22] M. Beck et al. xLSTM: Extended Long Short-Term Memory. Neurips 2024 \n\n[23] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling 2024"},"title":{"value":"Reply - Part 5 (+ references)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763664330868,"tcdate":1763664330868,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19489/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission19489/Authors"],"forum":"nqUwo7R2vv","number":11,"license":"CC BY 4.0","cdate":1763664330868,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19489/-/Official_Comment"],"mdate":1763664330868,"domain":"ICLR.cc/2026/Conference","replyto":"pYcU9PGeZZ","id":"fEkX1ykyvj","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["RNN","sequence processing","model architecture","embedded systems"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Processing temporal data directly at the sensor source demands models that capture both short- and long-range dynamics under tight memory constraints. While State-of-the-Art (SotA) sequence models such as Transformers excel at these tasks, their quadratic memory scaling with sequence length makes them impractical for edge settings. Recurrent Neural Networks (RNNs) offer constant memory scaling, but train sequentially and slowly, and Temporal Convolutional Networks (TCNs), though efficiently trainable, also scale memory with kernel length. For more memory-efficient sequence modeling, we propose mGRADE (minimally Gated Recurrent Architecture with Delay Embedding), a hybrid-memory system that integrates a temporal convolution with learnable spacings with a gated recurrent component equivalent to a minimal Gated Recurrent Unit (minGRU). The convolution with learnable spacings can express a flexible delay embedding that captures rapid temporal variations, while the recurrent component efficiently maintains global context with minimal memory overhead. We theoretically ground and empirically validate our approach on two types of synthetic tasks, demonstrating that mGRADE effectively separates and preserves temporal features across multiple timescales. Furthermore, on the challenging Long-Range Arena (LRA) benchmark, mGRADE reduces the memory footprint by up to a factor of 8, while maintaining competitive performance compared to SotA models."},"_bibtex":{"value":"@misc{\ntorchet2026mgrade,\ntitle={m{GRADE}: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling},\nauthor={Tristan Torchet and Christian Metzner and Karthik Charan Raghunathan and Jimmy Weber and Laura Kriener and Melika Payvand},\nyear={2026},\nurl={https://openreview.net/forum?id=nqUwo7R2vv}\n}"},"title":{"value":"mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling"},"pdf":{"value":"/pdf/5c41faee958abf75002def8814a54ee2bdafa10b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"torchet|mgrade_minimal_recurrent_gating_meets_delay_convolutions_for_lightweight_sequence_modeling"},"authorids":{"value":["~Tristan_Torchet1","~Christian_Metzner1","~Karthik_Charan_Raghunathan1","~Jimmy_Weber1","~Laura_Kriener1","~Melika_Payvand1"]},"authors":{"value":["Tristan Torchet","Christian Metzner","Karthik Charan Raghunathan","Jimmy Weber","Laura Kriener","Melika Payvand"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.09338v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"huang|localmamba_visual_state_space_model_with_windowed_selective_scan"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Tao_Huang_0020:","~Xiaohuan_Pei1","~Shan_You1","https://dblp.org/search/pid/api?q=author:Fei_Wang_0032:","https://dblp.org/search/pid/api?q=author:Chen_Qian_0006:","https://dblp.org/search/pid/api?q=author:Chang_Xu_0002:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.09338"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-09338,\n  publtype={informal},\n  author={Tao Huang and Xiaohuan Pei and Shan You and Fei Wang and Chen Qian and Chang Xu},\n  title={LocalMamba: Visual State Space Model with Windowed Selective Scan},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.09338},\n  url={https://doi.org/10.48550/arXiv.2403.09338}\n}\n"},"abstract":{"value":"Recent advancements in state space models, notably Mamba, have demonstrated significant progress in modeling long sequences for tasks like language understanding. Yet, their application in vision tasks has not markedly surpassed the performance of traditional Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). This paper posits that the key to enhancing Vision Mamba (ViM) lies in optimizing scan directions for sequence modeling. Traditional ViM approaches, which flatten spatial tokens, overlook the preservation of local 2D dependencies, thereby elongating the distance between adjacent tokens. We introduce a novel local scanning strategy that divides images into distinct windows, effectively capturing local dependencies while maintaining a global perspective. Additionally, acknowledging the varying preferences for scan patterns across different network layers, we propose a dynamic method to independently search for the optimal scan choices for each layer, substantially improving performance. Extensive experiments across both plain and hierarchical models underscore our approach's superiority in effectively capturing image representations. For example, our model significantly outperforms Vim-Ti by 3.1% on ImageNet with the same 1.5G FLOPs. Code is available at: https://github.com/hunto/LocalMamba."},"title":{"value":"LocalMamba: Visual State Space Model with Windowed Selective Scan"},"authors":{"value":["Tao Huang","Xiaohuan Pei","Shan You","Fei Wang","Chen Qian","Chang Xu"]}},"tmdate":1731473118577,"pdate":1704067200000,"tcdate":1731464183392,"writers":["~"],"signatures":["~Xiaohuan_Pei1"],"forum":"acCKdQtyXA","license":"CC BY-SA 4.0","number":189701,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1731473118577,"domain":"DBLP.org","id":"acCKdQtyXA","version":2},{"content":{"venue":{"value":"AAAI 2026"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/37277/41239"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"chen|hifimamba_dualstream_laplacian_enhanced_mamba_for_highfidelity_mri_reconstruction"},"html":{"value":"https://doi.org/10.1609/aaai.v40i4.37277"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/ChenFCRHTCSL26,\n  author={Hongli Chen and Pengcheng Fang and Yuxia Chen and Yingxuan Ren and Jing Hao and Fangfang Tang and Xiaohao Cai and Shanshan Shan and Feng Liu},\n  title={HiFi-Mamba: Dual-Stream ?-Laplacian Enhanced Mamba for High-Fidelity MRI Reconstruction},\n  year={2026},\n  cdate={1767225600000},\n  pages={2868-2876},\n  url={https://doi.org/10.1609/aaai.v40i4.37277},\n  booktitle={AAAI},\n  crossref={conf/aaai/2026}\n}\n"},"abstract":{"value":"Reconstructing high-fidelity MR images from undersampled k-space data remains a challenging problem in MRI. While Mamba variants for vision tasks offer promising long-range modeling capabilities with linear-time complexity, their direct application to MRI reconstruction inherits two key limitations: (1) insensitivity to high-frequency anatomical details; and (2) reliance on redundant multi-directional scanning. To address these limitations, we introduce High-Fidelity Mamba (HiFi-Mamba), a novel dual-stream Mamba-based architecture comprising stacked ?-Laplacian (WL) and HiFi-Mamba blocks. Specifically, the WL block performs fidelity-preserving spectral decoupling, producing complementary low- and high-frequency streams. This separation enables the HiFi-Mamba block to focus on low-frequency structures, enhancing global feature modeling. Concurrently, the HiFi-Mamba block selectively integrates high-frequency features through adaptive state-space modulation, preserving comprehensive spectral details. To eliminate the scanning redundancy, the HiFi-Mamba block adopts a streamlined unidirectional traversal strategy that preserves long-range modeling capability with improved computational efficiency. Extensive experiments on standard MRI reconstruction benchmarks demonstrate that HiFi-Mamba consistently outperforms state-of-the-art CNN-based, Transformer-based, and other Mamba-based models in reconstruction accuracy while maintaining a compact and efficient model design."},"title":{"value":"HiFi-Mamba: Dual-Stream ?-Laplacian Enhanced Mamba for High-Fidelity MRI Reconstruction"},"authors":{"value":[{"fullname":"Hongli Chen","username":""},{"fullname":"Pengcheng Fang","username":"~Pengcheng_Fang1"},{"fullname":"Yuxia Chen","username":""},{"fullname":"Yingxuan Ren","username":""},{"fullname":"Jing Hao","username":""},{"fullname":"Fangfang Tang","username":""},{"fullname":"Xiaohao Cai","username":""},{"fullname":"Shanshan Shan","username":""},{"fullname":"Feng Liu","username":""}]}},"tmdate":1783893359876,"pdate":1798675200000,"externalIds":["dblp:conf/aaai/ChenFCRHTCSL26"],"tcdate":1783893355041,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Pengcheng_Fang1"],"forum":"ZmFINcUgQj","license":"CC BY-SA 4.0","number":51659,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1783893359876,"domain":"OpenReview.net/Public_Article","id":"ZmFINcUgQj","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.13074v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"wei|hierarchical_spatiotemporal_statespace_modeling_for_fmri_analysis"},"authorids":{"value":["~Yuxiang_Wei4","https://dblp.org/search/pid/api?q=author:Anees_Abrol:","https://dblp.org/search/pid/api?q=author:Reihaneh_Hassanzadeh:","https://dblp.org/search/pid/api?q=author:Vince_D._Calhoun:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.13074"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-13074,\n  publtype={informal},\n  author={Yuxiang Wei and Anees Abrol and Reihaneh Hassanzadeh and Vince D. Calhoun},\n  title={Hierarchical Spatio-Temporal State-Space Modeling for fMRI Analysis},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.13074},\n  url={https://doi.org/10.48550/arXiv.2408.13074}\n}\n"},"abstract":{"value":"Recent advances in deep learning structured state space models, especially the Mamba architecture, have demonstrated remarkable performance improvements while maintaining linear complexity. In this study, we introduce functional spatiotemporal Mamba (FST-Mamba), a Mamba-based model designed for discovering neurological biomarkers using functional magnetic resonance imaging (fMRI). We focus on dynamic functional network connectivity (dFNC) derived from fMRI and propose a hierarchical spatiotemporal Mamba-based network that processes spatial and temporal information separately using Mamba-based encoders. Leveraging the topological uniqueness of the FNC matrix, we introduce a component-wise varied-scale aggregation (CVA) mechanism to aggregate connectivity across individual components within brain networks, enabling the model to capture component-level and network-level information. Additionally, we propose symmetric rotary position encoding (SymRope) to encode the relative positions of each functional connection while considering the symmetric nature of the FNC matrix. Experimental results demonstrate significant improvements in the proposed FST-Mamba model on various brain-based classification and regression tasks. We further show brain connectivities and dynamics that are crucial for the prediction. Our work reveals the substantial potential of attention-free sequence modeling in brain discovery. The codes are publicly available here: https://github.com/yuxiangwei0808/FunctionalMamba/tree/main."},"title":{"value":"Hierarchical Spatio-Temporal State-Space Modeling for fMRI Analysis"},"authors":{"value":["Yuxiang Wei","Anees Abrol","Reihaneh Hassanzadeh","Vince D. Calhoun"]}},"tmdate":1747082301995,"pdate":1704067200000,"tcdate":1747081392175,"writers":["~"],"signatures":["~YUXIANG_WEI3"],"forum":"oot53jaja8","license":"CC BY-SA 4.0","number":422769,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747082301995,"domain":"DBLP.org","id":"oot53jaja8","version":2},{"content":{"venue":{"value":"RECOMB 2025"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-031-90252-9_6.pdf"},"venueid":{"value":"dblp.org/conf/RECOMB/2025"},"paperhash":{"value":"wei|hierarchical_spatiotemporal_statespace_modeling_for_fmri_analysis"},"authorids":{"value":["~Yuxiang_Wei4","https://dblp.org/search/pid/api?q=author:Anees_Abrol:","https://dblp.org/search/pid/api?q=author:Vince_D._Calhoun:"]},"html":{"value":"https://doi.org/10.1007/978-3-031-90252-9_6"},"_bibtex":{"value":"@inproceedings{DBLP:conf/recomb/WeiAC25,\n  author={Yuxiang Wei and Anees Abrol and Vince D. Calhoun},\n  title={Hierarchical Spatio-Temporal State-Space Modeling for fMRI Analysis},\n  year={2025},\n  cdate={1735689600000},\n  pages={86-98},\n  url={https://doi.org/10.1007/978-3-031-90252-9_6},\n  booktitle={RECOMB},\n  crossref={conf/recomb/2025}\n}\n"},"abstract":{"value":"Recent advances in deep learning structured state space models, especially the Mamba architecture, have demonstrated remarkable performance improvements while maintaining linear complexity. In this study, we introduce functional spatiotemporal Mamba (FST-Mamba), a Mamba-based model designed for discovering neurological biomarkers using functional magnetic resonance imaging (fMRI). We focus on dynamic functional network connectivity (dFNC) derived from fMRI and propose a hierarchical spatiotemporal Mamba-based network that processes spatial and temporal information separately using Mamba-based encoders. Leveraging the topological uniqueness of the FNC matrix, we introduce a component-wise varied-scale aggregation (CVA) mechanism to aggregate connectivity across individual components within brain networks, enabling the model to capture component-level and network-level information. Additionally, we propose symmetric rotary position encoding (SymRope) to encode the relative positions of each functional connection while considering the symmetric nature of the FNC matrix. Experimental results demonstrate significant improvements in the proposed FST-Mamba model on various brain-based classification and regression tasks. We further show brain connectivities and dynamics that are crucial for the prediction. Our work reveals the substantial potential of attention-free sequence modeling in brain discovery. The codes are publicly available here: https://github.com/yuxiangwei0808/FunctionalMamba/tree/main."},"title":{"value":"Hierarchical Spatio-Temporal State-Space Modeling for fMRI Analysis"},"authors":{"value":["Yuxiang Wei","Anees Abrol","Vince D. Calhoun"]}},"tmdate":1747082285415,"pdate":1735689600000,"tcdate":1747081363314,"writers":["~"],"signatures":["~YUXIANG_WEI3"],"forum":"mQqPKGEV8e","license":"CC BY-SA 4.0","number":422750,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747082285415,"domain":"DBLP.org","id":"mQqPKGEV8e","version":2},{"content":{"summary":{"value":"The paper introduces a method for efficient world modeling that combines a VQ-VAE autoencoder with a sequence model to enable frame-level prediction of discrete latent representations, avoiding token-level autoregression. The approach models temporal dynamics by predicting the full grid of VQ code distributions for the next frame in a single forward pass and aligns these predictions with encoder posteriors through a KL-divergence-based dynamics loss derived from variational principles. Evaluated on the Atari 100K benchmark, STEP-VQ achieves performance comparable to categorical and autoregressive VQ-based baselines while providing 11× faster training, 4× fewer parameters, and improved scalability at higher resolutions, maintaining efficiency across both Transformer and Mamba-2 sequence model architectures."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See the weaknesses"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":1},"strengths":{"value":"1. The paper demonstrates a substantial training speedup compared to IRIS-like autoregressive VQ-VAE world models, effectively addressing a known computational bottleneck in VQ-based MBRL.\n\n2. The authors provide a well-structured categorization of existing approaches (categorical bottlenecks vs. VQ-VAE) and clearly identify the trade-offs motivating the proposed method.\n\n3. The inclusion of both Transformer and Mamba-2 sequence models in experiments enhances the robustness of the evaluation and shows the method’s compatibility with different sequence modeling paradigms."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Unclear methodological exposition:** The description of the proposed approach is mathematically dense and lacks intuitive implementation detail. The role of distributions and losses is not directly linked to concrete architecture design, and the core pipeline (as shown in Figure 6) should appear in the main text to aid understanding.\n\n2. **Limited performance gain over categorical bottlenecks:** While efficiency improves compared with IRIS-like methods, the method does not show clear performance advantages compared to strong CB-based baselines such as DreamerV3 or STORM. Additionally, computational cost comparisons with these CB models are missing, leaving STEP-VQ's relative benefit uncertain.\n\n3. **Insufficient detail in high-resolution comparisons:** The implementation setup of categorical baselines in higher-resolution experiments is under-specified, which weakens the claim that STEP-VQ consistently outperforms CB methods at larger resolutions.\n\n4. **Restricted scalability analysis:** Although the paper highlights resolution scalability, the experiments stop at 96x96 inputs, far below the resolutions addressed by recent models like Dreamer4 [1] (up to 640x360), limiting the evidence for true scalability across visual domains.\n\n[1] Hafner, Danijar, Wilson Yan, and Timothy Lillicrap. \"Training agents inside of scalable world models.\" arXiv preprint arXiv:2509.24527 (2025)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931420447,"tcdate":1761801815100,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19531/Reviewer_zxCC"],"signatures":["ICLR.cc/2026/Conference/Submission19531/Reviewer_zxCC"],"forum":"aBgX5jCTcd","number":1,"license":"CC BY 4.0","cdate":1761801815100,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19531/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931420447,"domain":"ICLR.cc/2026/Conference","replyto":"aBgX5jCTcd","id":"7h5GdfkCXL","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Model based reinforcement learning","VQ-VAE","Representation learning","Sequence Modeling"]},"primary_area":{"value":"reinforcement learning"},"abstract":{"value":"Model-based reinforcement learning (MBRL) from pixels often encodes frames into discrete latent variables that form tokens for sequence model backbones to learn world model dynamics. Previous work adopts two main approaches, each facing distinct limitations. Categorical bottlenecks enable fast frame-level prediction by flattening spatial features into categorical distributions, but suffer from explosive parameter growing with resolution and code dimension. Conversely, vector-quantised variational autoencoder (VQ-VAE) methods achieve parameter efficiency through codebook quantisation but require slow token-level autoregressive prediction within frames, shifting computational complexity to the dynamics model and producing longer sequences that slow training and inference.\n  \n  We propose STEP-VQ, a novel frame-level VQ-VAE-based world model that enables prediction of entire frames through single forward passes. STEP-VQ follows the latent-imagination paradigm with two components: a world model (VQ-VAE + sequence model) and a behaviour policy. The approach is sequence-model agnostic, working with both Mamba-2 and Transformer architectures without modifications. Our key insight is that fine-grained spatial structure preservation may be unnecessary for effective behaviour learning in latent space, as temporal dynamics can implicitly capture spatial patterns through frame-level prediction. We provide rigorous theoretical analysis grounded in variational inference, showing how our training objective emerges from evidence lower bound (ELBO) optimisation and why Kullback--Leibler (KL) divergence formulations enable superior performance through bidirectional optimisation.\n  \n  On Atari-100k, STEP-VQ achieves competitive performance whilst dramatically improving efficiency: 11.2× faster training than a strong VQ-VAE based baseline, 4× parameter reduction compared to categorical bottlenecks, and growing advantages at higher resolutions (+27.4\\% mean improvement at 96×96). STEP-VQ reaches superhuman performance on 9 games versus 8 for categorical methods, with KL divergence providing 24.5\\% improvement over cross-entropy baselines. These results demonstrate that frame-level discrete quantisation offers a practical path to efficient, scalable MBRL using modern sequence architectures."},"_bibtex":{"value":"@misc{\nwang2025stepvq,\ntitle={{STEP}-{VQ}: Sequence-model Agnostic Frame-level Inference with {VQ}-{VAE} for Model-based Reinforcement Learning},\nauthor={Wenlong Wang and Ivana Dusparic and Xiaowen Tao and Vinny Cahill},\nyear={2025},\nurl={https://openreview.net/forum?id=aBgX5jCTcd}\n}"},"title":{"value":"STEP-VQ: Sequence-model Agnostic Frame-level Inference with VQ-VAE for Model-based Reinforcement Learning"},"pdf":{"value":"/pdf/0b132deec96dca29af2e365f85e4b980d0a66808.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"wang|stepvq_sequencemodel_agnostic_framelevel_inference_with_vqvae_for_modelbased_reinforcement_learning"},"authorids":{"value":["~Wenlong_Wang4","~Ivana_Dusparic2","~Xiaowen_Tao1","~Vinny_Cahill1"]},"authors":{"value":["Wenlong Wang","Ivana Dusparic","Xiaowen Tao","Vinny Cahill"]}},"version":2},{"content":{"comment":{"value":"Thank you for your review! We appreciate the attention to detail and have fixed the writing errors and typos in our revision. We also used your comments to improve our manuscript, as described below.  \n\n> **W1: Lack of overall summary**   \n\nWe updated the paper to include algorithm boxes to summarize LoLCATs (Algorithm 1, Algorithm 2; L397 - 409). We also restructured the code in the appendix to be easier to follow as pseudocode for the entire linearizing process (Appendix C.1).   \n\n> **Q1: Why was the proposed method not validated on a smaller model, such as llama 1B?**   \n\nWe initially focused on larger LLMs with at least 7B parameters as this model size more strongly motivates LoLCATs. Among related works that propose new subquadratic architectures, several report results for pretraining 1 or 1.3B parameter models [1, 2, 3, 4, 5, 6]. However, fewer do so at the 7B scale, suggesting a need to develop more cost-effective ways to scale up new architectures for larger LLMs.\n\nHowever, we agree that linearizing 1B LLMs is also important to validate, especially if we can save training time by starting with available Transformers. We thus added experiments validating LoLCATs on two recent 1B models: Llama 3.2 1B, and Phi 1.5 1.3B, comparing against readily available alternatives (see Table 12, 13, 14 for full results). We find LoLCATs is able to achieve state-of-the-art linearized LLM quality in all evaluations.\n\n---\n**Llama 3.2 1B Comparison**    \n| Model | PiQA | ARC-e | ARC-c (acc. norm) | HellaSwag (acc. norm) | Winogrande | MMLU (5-shot) |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n| Llama 3.2 1B | 74.4 | 65.5 | 35.8 | 63.7 | 60.5 | 31.9 |\n| -> T2R | 69.2 | 58.2 | 29.9 | 42.6 | 54.1 | 23.3 |\n| -> Hedgehog | 70.1 | 55.8 | 29.8 | 47.7 | 50.7 | 23.0 |\n| -> LoLCATs (Ours) | **74.6** | **63.0** | **35.1** | **63.7** | **61.5** | **27.3** |  \n\n---\n---\n\n**Phi 1.5 1.3B Comparison**  \n| Model | PiQA | ARC-e | ARC-c (acc. norm) | HellaSwag (acc. norm) | Winogrande | MMLU (5-shot) |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n| **Transformer** |  |  |  |  |  |  |\n| Phi 1.5 1.3B (Our run) | 76.6 | 76.1 | 47.6 | 62.6 | 72.8 | 43.6 |\n| Phi 1.5 1.3B (from [7]) | 76.6 | 75.6 | 48.0 | 62.6 | 73.4 | - |\n| **Linearized** |  |  |  |  |  |  |\n| Phi-Mamba 1.5 [7] | 75.5 | 74.0 | 44.1 | 60.2 | 71.7 | - |\n| Hybrid Phi-Mamba 1.5 [7] | 76.5 | 75.3 | 45.8 | 60.6 | 72.0 | - |\n| Phi 1.5 1.3B T2R | 71.0 | 69.1 | 36.6 | 46.2 | 53.6 | 24.3 |\n| Phi 1.5 1.3B Hedgehog | 72.7 | 70.9 | 38.0 | 49.4 | 54.1 | 23.5 |\n| Phi 1.5 1B LoLCATs (Ours) | **76.9** | **77.0** | **46.9** | **62.3** | **72.7** | **39.2** |\n\n---\n\nSimilar to the 7B scale, we find LoLCATs gets state-of-the-art linearized LLM quality when compared against available linearizing alternatives (Table 12, 13), while also resulting in strong performance against 1.3B subquadratic LLMs pretrained from scratch (Table 14). This is all despite only using 40M training tokens, or 1.3% of the next best reported linearizing method for Phi 1.5 1.3B, and parameter efficient training (only updating feature maps in step 1, and using LoRA in step 2).      \n\n---\n\n**References**  \n\n[1] xLSTM: Extended Long Short-Term Memory, Beck et al., 2024  \n[2] Gated Linear Attention Transformers with Hardware-Efficient Training, Yang et al., 2023  \n[3] Parallelizing Linear Transformers with the Delta Rule over Sequence Length, Yang et al., 2024  \n[4] Simple linear attention language models balance the recall-throughput tradeoff, Arora et al., 2024  \n[5] Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Gu and Dao, 2024  \n[6] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, Dao and Gu, 2024  \n[7] Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models, Bick et al., 2024"},"title":{"value":"Response to Reviewer ​​Dj9p"}},"tmdate":1732211402880,"tcdate":1732210392676,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8323/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission8323/Authors"],"forum":"8VtGeyJyx9","number":5,"license":"CC BY 4.0","cdate":1732210392676,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8323/-/Official_Comment"],"mdate":1732211402880,"domain":"ICLR.cc/2025/Conference","replyto":"x0EG4lHDOC","id":"3TMLc1CVVL","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Linear Attention","Linearizing Transformers","Low-rank Adaptation","Large Language Models","Architecture Distillation"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent works show we can linearize large language models (LLMs)—swapping the quadratic attentions of popular Transformer-based LLMs with subquadratic analogs, such as linear attention—avoiding the expensive pretraining costs. However, linearizing LLMs often significantly degrades model quality, still requires training over billions of tokens, and remains limited to smaller 1.3B to 7B LLMs. We thus propose Low-rank Linear Conversion via Attention Transfer (LoLCATs), a simple two-step method that improves LLM linearizing quality with orders of magnitudes less memory and compute. We base these steps on two findings. First, we can replace an LLM's softmax attentions with closely-approximating linear attentions, simply by *training* the linear attentions to match their softmax counterparts with an output MSE loss (“attention transfer”). Then, this enables adjusting for approximation errors and recovering LLM quality simply with *low-rank* adaptation (LoRA). LoLCATs significantly improves linearizing quality, training efficiency, and scalability. We significantly reduce the linearizing quality gap and produce state-of-the-art subquadratic LLMs from Llama 3 8B and Mistral 7B v0.1, leading to 20+ points of improvement on 5-shot MMLU. Furthermore, LoLCATs does so with only 0.2% of past methods' model parameters and 0.04-0.2% of their training tokens. Finally, we apply LoLCATs to create the first linearized 70B and 405B LLMs (50$\\times$ that of prior work). When compared with prior approaches under the same compute budgets, LoLCATs significantly improves linearizing quality, closing the gap between linearized and original Llama 3.1 70B and 405B LLMs by 77.8\\% and 78.1\\% on 5-shot MMLU."},"_bibtex":{"value":"@inproceedings{\nzhang2025lolcats,\ntitle={Lo{LCAT}s: On Low-Rank Linearizing of Large Language Models},\nauthor={Michael Zhang and Simran Arora and Rahul Chalamala and Benjamin Frederick Spector and Alan Wu and Krithik Ramesh and Aaryan Singhal and Christopher Re},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=8VtGeyJyx9}\n}"},"title":{"value":"LoLCATs: On Low-Rank Linearizing of Large Language Models"},"pdf":{"value":"/pdf/6878f5a04bcbe6a65f04b4176e2b7aa63474099a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"zhang|lolcats_on_lowrank_linearizing_of_large_language_models"},"authorids":{"value":["~Michael_Zhang4","~Simran_Arora1","~Rahul_Chalamala1","~Benjamin_Frederick_Spector1","~Alan_Wu3","~Krithik_Ramesh1","~Aaryan_Singhal1","~Christopher_Re1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Michael Zhang","Simran Arora","Rahul Chalamala","Benjamin Frederick Spector","Alan Wu","Krithik Ramesh","Aaryan Singhal","Christopher Re"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Linear Attention Sequence Parallel (LASP), a method for handling long sequences in linear attention-based models by using sequence parallelism (SP) in a distributed setting. LASP's approach integrates kernel fusion and state caching to improve efficiency. It reduces communication overhead (by using a right-product kernel) and allows for compatibility with data-sequence hybrid parallelism methods. The paper reports that LASP achieves substantial speedup."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- Could the authors clarify the specific contributions of LASP beyond the use of the right-product kernel trick?\n\n- Given that much of the speedup is due to characteristics of linear attention, how does LASP differ from previous ring-attention methods?\n\n- Could additional benchmarks with varying sequence dimensions better isolate LASP's true efficiency?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- Hybrid Parallelism: LASP’s compatibility with data-sequence hybrid parallelism methods, including distributed data parallelism, is a notable strength.\n\n- System Engineering Optimizations: The paper addresses practical GPU execution challenges, such as kernel fusion and KV state caching, enhancing LASP's efficiency on hardware.\n\n- Scaling to large number sequences demonstrated (up to 8M tokens)"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Lack of Novelty in Speedup Mechanism: The reported speedup primarily derives from inherent features of linear attention (i.e., the ability to cache and reduce communication volume because the dimensions of V are lower than Q). The method itself does not introduce new mechanisms to achieve these benefits, it is just a straightforward implementation of ring-attention for linear attention.\n\n- Dependency on Right-Product Kernel Trick: LASP's reliance on the right-product kernel trick is not novel but rather a straightforward application of existing linear attention properties.\n\n- Limited Contribution to Parallelism Theory: The paper does not advance SP theory beyond standard ring-attention techniques, with its improvements largely attributed to linear attention-specific engineering rather than innovative parallelism methods."}},"nonreaders":[],"tmdate":1731428086381,"tcdate":1730353023965,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5885/Reviewer_nWhh"],"signatures":["ICLR.cc/2025/Conference/Submission5885/Reviewer_nWhh"],"forum":"oVnfVnwh6y","number":1,"license":"CC BY 4.0","cdate":1730353023965,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5885/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428086381,"domain":"ICLR.cc/2025/Conference","replyto":"oVnfVnwh6y","id":"CLTnuUysnS","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["sequence parallelism","distributed training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Sequence parallelism (SP) serves as a prevalent strategy to handle long sequences that exceed the memory limit of a single device. However, for linear sequence modeling methods like linear attention, existing SP approaches do not take advantage of their right-product-first feature, resulting in sub-optimal communication efficiency and usability. In this paper, we introduce Linear Attention Sequence Parallelism (LASP), an efficient SP approach designed for linear attention-based transformer models. Specifically, we design an efficient point-to-point ring-style communication mechanism to leverage the right-product kernel trick of linear attention, which sharply decreases the communication overhead, comparing with existing SP methods. We enhance the computation efficiency of LASP by performing kernel fusion and intermediate state caching, making the implementation of LASP hardware-friendly on GPUs. Furthermore, we meticulously ensure the compatibility of sequence-level LASP with all types of batch-level data parallel methods, which is vital for distributed training on large clusters with very-long sequences. We also discuss the generalization of LASP on other linear sequence modeling methods. Extensive experiments on linear attention-based models are conducted with varying sequence lengths from 2K to 4096K. LASP scales sequence length up to 4096K on 128 GPUs, which is 8$\\times$ longer than existing SP methods."},"_bibtex":{"value":"@misc{\nsun2025linear,\ntitle={Linear Attention Sequence Parallelism},\nauthor={Weigao Sun and Zhen Qin and Dong Li and Xuyang Shen and Yu Qiao and Yiran Zhong},\nyear={2025},\nurl={https://openreview.net/forum?id=oVnfVnwh6y}\n}"},"title":{"value":"Linear Attention Sequence Parallelism"},"pdf":{"value":"/pdf/91f0bb8ec553fe88b189a2550d301b39cef48faa.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"sun|linear_attention_sequence_parallelism"},"authorids":{"value":["~Weigao_Sun1","~Zhen_Qin6","~Dong_Li11","~Xuyang_Shen1","~Yu_Qiao1","~Yiran_Zhong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Weigao Sun","Zhen Qin","Dong Li","Xuyang Shen","Yu Qiao","Yiran Zhong"]}},"version":2},{"content":{"summary":{"value":"The authors propose a time series modeling framework for forecasting, generation and imputation built on the idea of switching OU latent process which maps to observations through a normalizing flow, thereby defining an invertible transformation between the latents and the observations. This modeling choice allows the system to model irregularly spaced points and thus generalize to more real-world scenarios. The experiments indicate that it leads to improved performance as well as diverse generations, and in particular the choice of linear parameterizations leads to closed form evaluation of a number of relevant conditional distributions."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- Maybe I did not understand this part correctly, but how do the authors parallelize training given the latents follow a linear dynamical system? What are the assumptions and simplifications made to allow for this parallelized training as opposed to ODE-RNN?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- The authors combine strong generative models (normalizing flows) with a latent OU process to model irregularly sampled time-series, pushing the frontier of real-world time-series modeling.\n- The design choices allow computing of important conditional distributions tractable in closed form, thereby allowing ease of both sampling as well as likelihood computation.\n- Empirically, they show that the proposed approach outperforms some of the baselines that they consider, built on top of state-space models as well as diffusion-style generative models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- A potential weakness of the proposed method is that the choice of linear evolution of the latents is governed at the trajectory level as opposed to dynamically updated throughout the time-series, which feels quite limiting and non compositional. \n- While the authors compare to some of the baselines, it would be nice to have a comparison against Time-Grad (Rasul et. al) which is one of the predominant methods for doing time series forecasting.\n- Most of the quantitative results highlight performance as opposed to training and inference time, for both generation and forecasting/imputation tasks. It would be good to get a comparative estimate of the time required to train the proposed model (in contrast to baselines) as well as the computational cost at inference.\n\nThe authors should compare and contrast with the following existing works that look at time series forecasting using continuous time samplers as well as switching linear systems:\n\n*Chen, Yu, et al. \"Recurrent interpolants for probabilistic time series prediction.\" arXiv preprint arXiv:2409.11684 (2024).*\n\n*Linderman, Scott W., et al. \"Recurrent switching linear dynamical systems.\" arXiv preprint arXiv:1610.08466 (2016).*\n\n*Halmos, Peter, Jonathan Pillow, and David A. Knowles. \"System Identification for Continuous-time Linear Dynamical Systems.\" arXiv preprint arXiv:2308.11933 (2023).*\n\n*Rasul, Kashif, et al. \"Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting.\" International conference on machine learning. PMLR, 2021.*"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918893415,"tcdate":1761951220221,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6547/Reviewer_KBdz"],"signatures":["ICLR.cc/2026/Conference/Submission6547/Reviewer_KBdz"],"forum":"I66uArUBJM","number":3,"license":"CC BY 4.0","cdate":1761951220221,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6547/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918893415,"domain":"ICLR.cc/2026/Conference","replyto":"I66uArUBJM","id":"m0hKQrxGkw","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Generative model","Time series analysis","Deep learning","Normalizing flow","State space model"]},"supplementary_material":{"value":"/attachment/b6fdbd4be78ec183bb844eed1f039d38ddca8278.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"We propose OUFlow, a general-purpose time-series generative model that robustly handles irregular sampling and generates sequences at arbitrary time points. OUFlow integrates latent dynamics governed by a mixture of Ornstein-Uhlenbeck processes with expressive target distributions via normalizing flows. Leveraging our analytically derived, efficiently computable likelihoods and posteriors for high-dimensional time series, OUFlow supports unconditional time-series generation, probabilistic forecasting, and imputation from partial observations within a unified model after a single training phase. It also enables explicit likelihood evaluation (e.g., for anomaly detection), clustering via modes of the latent OU process, and, in some cases, denoising under noisy supervision. By exploiting parallelization through the scan algorithm, OUFlow attains logarithmic runtime scaling in the number of generated points, while maintaining high accuracy in all three tasks. Comprehensive experiments on both synthetic and real-world datasets demonstrate that OUFlow consistently outperforms other models capable of all three tasks, in both generation quality and computational efficiency."},"_bibtex":{"value":"@misc{\nmorinaga2026fast,\ntitle={Fast Generation, Forecasting, and Imputation of Multivariate Irregular Time Series with {OUF}low},\nauthor={Taiki Morinaga},\nyear={2026},\nurl={https://openreview.net/forum?id=I66uArUBJM}\n}"},"title":{"value":"Fast Generation, Forecasting, and Imputation of Multivariate Irregular Time Series with OUFlow"},"pdf":{"value":"/pdf/20fbd240f3dda2a10554b65cfb52f545a736cd53.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"morinaga|fast_generation_forecasting_and_imputation_of_multivariate_irregular_time_series_with_ouflow"},"authorids":{"value":["~Taiki_Morinaga1"]},"authors":{"value":["Taiki Morinaga"]}},"version":2},{"content":{"summary":{"value":"The paper discusses a approach to spatio-temporal modelling for tasks with irregular observation spaces. Experiments are conducted on multi-agent scenario's like robotic target-chasing simulations and StarCraft II gameplay analysis.\n\n\nMotivation: The paper aruges that current architectures like Transformers and LSTMs works well in scenarios with fixed observation spaces but struggle with tasks where observation spaces vary over time, such as multi-agent interactions. The paper aims to combine the efficiency of recurrent models with the parallelism of attention mechanisms, focusing on varying-dimension observation spaces.\n\nProposed Method:  The paper introduce an algorithm using a 2D latent state, alternating between cross-attention on observations and a discounted cumulative sum over time, capturing long context.  \n\nExperiments: Two benchmarks are used for evaluation: (a) Chasing Targets: Agents pursuing dynamic targets, evaluating the model's ability to infer intended goals in real-time; (b) StarCraft II: A environment with complex, temporally varying observations, where models infer combat unit assignments and predict unit movement.\n\nResults: The proposed model achieves competitive accuracy with fewer parameters and faster inference compared to standard methods (e.g., transformers). The \"resampling cycle,\" improves sequence modeling accuracy by conditioning observations on accumulated sequence data.\n\n===================\n\nAfter rebuttal: I've read the review by other reviewers as well as the rebuttal by the authors.\n\nThis statement by the authors summarizes the contribution of the paper well.\n\"Our results primarily set out to explore the trade-offs between efficiency and performance in the irregular observation case. This means results should be interpreted given the compute budget available to a practitioner. If only performance matters, practitioners should select a temporal transformer with sequential cross attention (X-Attn Sequential) for observation encoding. For efficiency and model parsimony, an inclusive scan should be selected with X-Attn Sequential.\"\n\nIt will be very helpful to revise the introduction and abstract accordingly."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See Weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The paper investigates the use of various encoders to encode irregular observations into a fixed dimensional latent, and corresponding sequence modelling approaches."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The model's efficiency enables its use in real-time applications with irregular observation space, but studying the effect on longer sequence durations as well as diversifying the tasks (i.e., conduct experiments on more tasks) would further validate its utility in long-term planning tasks.\n- There's some work in the literature like Temporal Latent Bottleneck (https://arxiv.org/abs/2205.14794) and Block Recurrent Transformers (https://arxiv.org/abs/2203.07852); which combines strengths of attention and recurrence."}},"nonreaders":[],"tmdate":1733305410659,"tcdate":1731405134117,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission14157/Reviewer_aVTi"],"signatures":["ICLR.cc/2025/Conference/Submission14157/Reviewer_aVTi"],"forum":"TySMCLoGVl","number":4,"license":"CC BY 4.0","cdate":1731405134117,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission14157/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733305410659,"domain":"ICLR.cc/2025/Conference","replyto":"TySMCLoGVl","id":"lmrYurDC79","forumContent":{"TLDR":{"value":"We propose an efficient sequence modeling algorithm for irregular observation data."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["sequence modeling","efficient training","efficient inference","spatio-temporal","multi-agent task"]},"supplementary_material":{"value":"/attachment/54c980b7057c197c54605c16cf2ab560e59cd62b.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Various works have aimed at combining the inference efficiency of recurrent models and training parallelism of MHA for sequence modeling. However, most of these works focus on tasks with fixed-dimension observation spaces, such as individual tokens in language modeling or pixels in image completion. Variably sized, irregular observation spaces are relatively under-represented, yet they occur frequently in multi-agent domains such as autonomous driving and human-robot interaction. To handle an observation space of varying size, we propose a novel algorithm that alternates between cross-attention between a 2D latent state and observation, and a discounted cumulative sum over the sequence dimension to efficiently accumulate historical information. We find this resampling cycle is critical for performance. To evaluate efficient sequence modeling in this domain, we introduce two multi-agent intention tasks: simulated agents chasing bouncing particles and micromanagement analysis in professional StarCraft II games. Our algorithm achieves comparable accuracy with a lower parameter count, faster training and inference compared to existing methods."},"_bibtex":{"value":"@misc{\nferenczi2025efficiently,\ntitle={Efficiently Scanning and Resampling Spatio-Temporal Tasks with Irregular Observations},\nauthor={Bryce Ferenczi and Michael Burke and Tom Drummond},\nyear={2025},\nurl={https://openreview.net/forum?id=TySMCLoGVl}\n}"},"title":{"value":"Efficiently Scanning and Resampling Spatio-Temporal Tasks with Irregular Observations"},"pdf":{"value":"/pdf/60aa39230761dfa3c2b0a9e3c039c157e84bfc7d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ferenczi|efficiently_scanning_and_resampling_spatiotemporal_tasks_with_irregular_observations"},"authorids":{"value":["~Bryce_Ferenczi1","~Michael_Burke1","~Tom_Drummond1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Bryce Ferenczi","Michael Burke","Tom Drummond"]}},"version":2},{"content":{"summary":{"value":"This paper provides theoretical analysis to the question of whether Mamba can always enjoy the “free lunch”, that is whether it is able to achieve similar performance as Transformer while remaining significantly lower cost. The paper provides analysis for two tasks: COPY and dynamic programming  (DP), and demonstrates that for the COPY task, fixed-size Mamba may encounter bottlenecks, unless the size of the model grows linearly with sequence length, and for the DP task there is no particular advantage of Mamba unless the problem has a local structure."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- The theoretical result in Theorem 1 is honestly a bit difficult to understand intuitively: Is this a loose or tight bound? It appears to be a characterization of the difficulty of a COPY problem, but it is difficult to get a sense of what it indicates in practice as there are many variables introduced to control this bound. It would be great to justify the interpretability of this theoretical result and how this bound is an accurate characterization of the model behavior (e.g. plot curves comparing theoretical bounds with empirical results).\n\n- Similarly, the analysis of the paper depends on critical Assumptions 1 and 2. While I understand these are mathematical constructions needed to make the derivations go through, these assumptions appear to be a bit too convoluted and difficult be measured or justified empirically. In general, there are many ways to derive theoretical upper and lower bounds, depending on the choice of assumptions, however an “optimal” theory should be based on minimal assumptions while producing interpretable outcomes.\n\n- It would be great to also provide some experimental examples to support the claim of Sec. 5 (Mamba does not offer additional cost savings compared to Transformers for DP problems), similar to the COPY problem.\n\n- In Fig. 2, what is the Transformer baseline in the rightmost chart?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"This paper provides some in-depth theoretical study which elucidates the limits of the Mamba model architecture. These contributions help researchers better understand the trade-off between performance and computational cost for the two popular model architectures."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Existing works already demonstrated similar limitations of Mamba (and SSM in general) in handling tasks like COPY (especially [1,2], which the paper also referred to). The conclusions of this paper did not appear to have provided much additional insight, and the contributions are incremental. What is more, the theoretical derivation and results of the paper (Theorem 1-4, and the assumptions they build upon) seem to be pure complex mathematical constructions, which do not appear to be concise and interpretable enough to be fully convincing or empirically verifiable.\n\n[1] Repeat After Me: Transformers are Better than State Space Models at Copying\n[2] The Illusion of State in State-Space Models"}},"nonreaders":[],"tmdate":1731428784015,"tcdate":1730666301265,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6956/Reviewer_vUfm"],"signatures":["ICLR.cc/2025/Conference/Submission6956/Reviewer_vUfm"],"forum":"UYarAv7rUx","number":1,"license":"CC BY 4.0","cdate":1730666301265,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6956/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428784015,"domain":"ICLR.cc/2025/Conference","replyto":"UYarAv7rUx","id":"h4Npuffi1G","forumContent":{"TLDR":{"value":"This paper explores the expressive power of Mamba."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Expressive Power","Chain of Thought"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Transformers have been the cornerstone of current Large Language Models (LLMs); however, its linear growth in overhead during inference with respect to sequence length poses challenges for modeling long sequences. In this context, Mamba has gradually attracted attention due to its constant-level size during inference and existing empirical results have shown that it can perform comparably to Transformers in sequence modeling while offering significant savings. However, one may ask that, can Mamba always enjoy the ``free lunch\"? In this paper, we focus on analyzing the expressive ability of Mamba from a theoretical standpoint. First, inspired by the connection between Mamba and linear attention, we investigate potential shortcomings of the Mamba when performing the COPY operation. Our results indicate that Mamba with constant size may encounter bottlenecks when handling COPY, while it can achieve perfect performance when the size scales linearly with sequence length. Based on this observation, we analyze Mamba's ability to tackle DP problems when equipped with Chain of Thought (CoT). Our findings suggest that to solve arbitrary DP problems, the total cost of Mamba is comparable to standard and efficient Transformers. However, similar to efficient Transformers, when facing DP problems with favorable properties such as locality, Mamba can provide savings in overhead. Our results contribute to a deeper understanding of Mamba."},"_bibtex":{"value":"@misc{\nren2025can,\ntitle={Can Mamba Always Enjoy the ''Free Lunch''?},\nauthor={Ruifeng Ren and Zhicong Li and Yong Liu},\nyear={2025},\nurl={https://openreview.net/forum?id=UYarAv7rUx}\n}"},"title":{"value":"Can Mamba Always Enjoy the \"Free Lunch\"?"},"pdf":{"value":"/pdf/b4552a0f966e40cf480dfcacf744f021f4b8b521.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ren|can_mamba_always_enjoy_the_free_lunch"},"authorids":{"value":["~Ruifeng_Ren3","~Zhicong_Li1","~Yong_Liu7"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Ruifeng Ren","Zhicong Li","Yong Liu"]}},"version":2},{"content":{"comment":{"value":"We are grateful for the reviewer’s highly positive evaluation of our work. We are glad they viewed the three core components of Mamba-3 as strengths and liked the theoretical analysis provided for our complex SSM. We hope to answer the questions posed by the reviewer below.\n\n> More complex tasks such as reasoning can be explored to fully demonstrate the capability of Mamba-3.\n\nWe evaluated our 1.5B models and baselines on five additional benchmarks: BoolQ, CommonsenseQA, and MMLU (continuation variant), SciQ, and LogiQA2. We find that our **Mamba-3 SISO model continues to perform well on multiple benchmarks with the hybrid variant further improving performance**. Unfortunately, more complex tasks, such as mathematical reasoning, often require increased pre-training plus post-training. Thus, we leave such tasks to subsequent work. We hope the increased suite of evaluations helps further show the full capabilities of Mamba-3.\n\n| Model               | BoolQ | CommonSense QA | MMLU | SciQ | LogiQA2 (Acc_n) |\n|---------------------|-------|----------------|------|------|------------------|\n| Llama               | 59.9  | 19.7           | 34.0 | 92.1 | **29.8**             |\n| Gated DeltaNet      | 59.8  | 19.7           | **34.9** | 90.7 | 27.7             |\n| Mamba 2             | 62.0  | 20.7           | **34.9** | 91.0 | 27.5             |\n| Mamba 3 SISO        | 60.6  | 21.3           | **34.9** | 91.0 | 28.2             |\n| Mamba 3 Hybrid SISO | **64.1**  | **22.5**           | 34.5 | **92.5** | 28.5             |\n\n-------------------------\n\n> Can the proposed Mamba-3 be generalized to vision tasks? \n\nAs Mamba-3 is a general sequence mixer, it can be generalized to vision tasks and other modalities, much like how its predecessors Mamba-1 and Mamba-2 have been [1,2,3].\n\n------------------\n\n> Does Mamba-3 still has advantages when compared to sparse attention based transformers?\n\nWhile sparse attention is viewed as an efficient but worse-performing approximation to quadratic attention, we believe that linear models such as state-space models and linear attention have distinct performance trade-offs compared to full attention: they are not only more efficient but can have better modeling performance in many scenarios. This is exemplified by the performance on other modalities [4], as well as the fact that hybrid linear and attention models are known to have better performance than a full Transformer [5]. For this reason, hybrid SSM/Linear+Transformer models have become increasingly popular as frontier models [6,7,8,9]\n\n---------\n## References\n\n[1] [2401.09417] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model\n\n[2] [2403.06977] VideoMamba: State Space Model for Efficient Video Understanding\n\n[3] [2510.12111] Chimera: State Space Models Beyond Sequences\n\n[4] [2403.03234] Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling\n\n[5] [2406.07887] An Empirical Study of Mamba-based Language Models\n\n[6] [2505.00949] Llama-Nemotron: Efficient Reasoning Models\n\n[7] [2505.15431] Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought\n\n[8] https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd\n\n[9] [2510.26692] Kimi Linear: An Expressive, Efficient Attention Architecture"},"title":{"value":"Response to Reviewer 5Dn9 (1/1)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764477185282,"tcdate":1764475257387,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13549/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13549/Authors"],"forum":"HwCvaJOiCj","number":9,"license":"CC BY 4.0","cdate":1764475257387,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13549/-/Official_Comment"],"mdate":1764477185282,"domain":"ICLR.cc/2026/Conference","replyto":"FDn90uYjhx","id":"pDspw2CNh4","forumContent":{"venue":{"value":"ICLR 2026 Oral"},"TLDR":{"value":"Mamba-3, an inference-first SSM that pushes on core SSM principles: improved discretization for better quality, complex dynamics for new capabilities, and MIMO updates for efficient inference."},"keywords":{"value":["State Space Models","Mamba","LLMs","Subquadratic Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While current Transformer models deliver strong quality, their quadratic compute and linear memory requirements make inference expensive. This has spurred the development of sub-quadratic models with reduced compute and constant memory requirements.\nHowever, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice.\nGuided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule enabling richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation that improves model performance without increasing decode latency.\nTogether with architectural refinements, Mamba-3 achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with the MIMO variant further improving accuracy by an additional 1.2 points, for a total gain of 1.8 points.\nAcross state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half the state size. These results demonstrate that Mamba-3 advances the performance–efficiency frontier."},"_bibtex":{"value":"@inproceedings{\nlahoti2026mamba,\ntitle={Mamba-3: Improved Sequence Modeling using State Space Principles},\nauthor={Aakash Lahoti and Kevin Li and Berlin Chen and Caitlin Wang and Aviv Bick and J Zico Kolter and Tri Dao and Albert Gu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=HwCvaJOiCj}\n}"},"title":{"value":"Mamba-3: Improved Sequence Modeling using State Space Principles"},"pdf":{"value":"/pdf/09c67b76b7d1ad254ea30eeba839b8592de9bd78.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lahoti|mamba3_improved_sequence_modeling_using_state_space_principles"},"authorids":{"value":["~Aakash_Lahoti1","~Kevin_Li3","~Berlin_Chen1","~Caitlin_Wang1","~Aviv_Bick1","~J_Zico_Kolter1","~Tri_Dao1","~Albert_Gu1"]},"authors":{"value":["Aakash Lahoti","Kevin Li","Berlin Chen","Caitlin Wang","Aviv Bick","J Zico Kolter","Tri Dao","Albert Gu"]}},"version":2},{"content":{"summary":{"value":"This paper presents the limitations of Mamba in terms of length generalization and proposes an algorithm that combines token dropping with the Mamba architecture to effectively enhance its length generalization capabilities."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"Please refer to the Weaknesses."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The proposed method is straightforward and can be directly combined with Mamba to improve the length generalization abilities.\n2. The authors analyze the shortcomings of Mamba in terms of its length generalization abilities."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The experimental results are limited. The authors only combine the method with the Mamba model and verify the effectiveness of their method. It would be more convincing if the method can be validated on a broader range of SSM-based and linear-attention-based models.\n2. The experimental results are not satisfactory. As shown in Table 1, DeciMamba fails to achieve more than a 10% LongBench score on most datasets. In contrast, models with a context window size of only 4k, as reported in the original LongBench paper, often perform better (e.g., Llama-2-7B-chat has an average score of 31 on English tasks). Additionally, for simpler long-context tasks like Passkey, the authors mention that DeciMamba requires fine-tuning to handle them effectively.\n3. The presentation of the paper could be improved. For instance, proper citation formats (\\citep and \\citet) should be used; table captions should appear above the tables; and in Table 1, the DeciMamba entry is missing the LB score for the LCC dataset."}},"nonreaders":[],"tmdate":1732432542313,"tcdate":1730615988073,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5066/Reviewer_44XJ"],"signatures":["ICLR.cc/2025/Conference/Submission5066/Reviewer_44XJ"],"forum":"iWSl5Zyjjw","number":2,"license":"CC BY 4.0","cdate":1730615988073,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5066/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732432542313,"domain":"ICLR.cc/2025/Conference","replyto":"iWSl5Zyjjw","id":"SuM4bGRv95","forumContent":{"TLDR":{"value":"We explore Mamba's length extrapolation potential, identifying its limitation due to a restricted effective receptive field, and introduce DeciMamba, the first context-extension method specifically designed for Mamba."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Long context","Context extension","Long-range language modeling"]},"supplementary_material":{"value":"/attachment/7b340eb64654c7225cf43f5c6138ec972b453804.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrates high performance and achieves Transformer-level capabilities while requiring substantially fewer computational resources. In this paper we explore the length-generalization capabilities of Mamba, which we find to be relatively limited. Through a series of visualizations and analyses we identify that the limitations arise from a restricted effective receptive field, dictated by the sequence length used during training. To address this constraint, we introduce DeciMamba, a context-extension method specifically designed for Mamba. This mechanism, built on top of a hidden filtering mechanism embedded within the S6 layer, enables the trained model to extrapolate well even without additional training. Empirical experiments over real-world long-range NLP tasks show that DeciMamba can extrapolate to context lengths that are significantly longer than the ones seen during training, while enjoying faster inference. We will release our code and models."},"_bibtex":{"value":"@inproceedings{\nben-kish2025decimamba,\ntitle={DeciMamba: Exploring the Length Extrapolation Potential of Mamba},\nauthor={Assaf Ben-Kish and Itamar Zimerman and Shady Abu-Hussein and Nadav Cohen and Amir Globerson and Lior Wolf and Raja Giryes},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=iWSl5Zyjjw}\n}"},"title":{"value":"DeciMamba: Exploring the Length Extrapolation Potential of Mamba"},"pdf":{"value":"/pdf/b2e18164fc1679d810d80cd5fdb09a2cc1a6003f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"benkish|decimamba_exploring_the_length_extrapolation_potential_of_mamba"},"authorids":{"value":["~Assaf_Ben-Kish1","~Itamar_Zimerman1","~Shady_Abu-Hussein1","~Nadav_Cohen1","~Amir_Globerson1","~Lior_Wolf1","~Raja_Giryes1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Assaf Ben-Kish","Itamar Zimerman","Shady Abu-Hussein","Nadav Cohen","Amir Globerson","Lior Wolf","Raja Giryes"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.09626v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"chen|video_mamba_suite_state_space_model_as_a_versatile_alternative_for_video_understanding"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Guo_Chen:","https://dblp.org/search/pid/api?q=author:Yifei_Huang:","~Jilan_Xu1","~Baoqi_Pei1","https://dblp.org/search/pid/api?q=author:Zhe_Chen:","https://dblp.org/search/pid/api?q=author:Zhiqi_Li:","https://dblp.org/search/pid/api?q=author:Jiahao_Wang:","https://dblp.org/search/pid/api?q=author:Kunchang_Li:","https://dblp.org/search/pid/api?q=author:Tong_Lu:","https://dblp.org/search/pid/api?q=author:Limin_Wang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.09626"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-09626,\n  publtype={informal},\n  author={Guo Chen and Yifei Huang and Jilan Xu and Baoqi Pei and Zhe Chen and Zhiqi Li and Jiahao Wang and Kunchang Li and Tong Lu and Limin Wang},\n  title={Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.09626},\n  url={https://doi.org/10.48550/arXiv.2403.09626}\n}\n"},"abstract":{"value":"Understanding videos is one of the fundamental directions in computer vision research, with extensive efforts dedicated to exploring various architectures such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state space model, e.g., Mamba, shows promising traits to extend its success in long sequence modeling to video modeling. To assess whether Mamba can be a viable alternative to Transformers in the video understanding domain, in this work, we conduct a comprehensive set of studies, probing different roles Mamba can play in modeling videos, while investigating diverse tasks where Mamba could exhibit superiority. We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks. Our extensive experiments reveal the strong potential of Mamba on both video-only and video-language tasks while showing promising efficiency-performance trade-offs. We hope this work could provide valuable data points and insights for future research on video understanding. Code is public: https://github.com/OpenGVLab/video-mamba-suite."},"title":{"value":"Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding"},"authors":{"value":["Guo Chen","Yifei Huang","Jilan Xu","Baoqi Pei","Zhe Chen","Zhiqi Li","Jiahao Wang","Kunchang Li","Tong Lu","Limin Wang"]}},"tmdate":1727750656391,"pdate":1704067200000,"tcdate":1716534688611,"writers":["~"],"signatures":["~Jilan_Xu1"],"forum":"dixZTZO6Ql","license":"CC BY-SA 4.0","number":21230,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1727750656391,"domain":"DBLP.org","id":"dixZTZO6Ql","version":2},{"content":{"venue":{"value":"ME-FoMo 2024 Poster"},"pdf":{"value":"/pdf/c806708ab4a9f90ed3a9fa33cfea62a01934781d.pdf"},"keywords":{"value":["Mamba","MoE","Pre-training","Large Language Models"]},"venueid":{"value":"ICLR.cc/2024/Workshop/ME-FoMo"},"abstract":{"value":"State-space models (SSMs) have recently demonstrated competitive performance to transformers at large-scale language modeling benchmarks while achieving linear time and memory complexity as a function of sequence length. Mamba, a recently released SSM model, shows impressive performance in both language modeling and long sequence processing tasks. Simultaneously, mixture-of-expert (MoE) models have shown remarkable performance while significantly reducing the compute and latency costs of inference at the expense of a larger memory footprint. In this paper, we present BlackMamba, a novel architecture that combines the Mamba SSM with MoE to obtain the benefits of both. We demonstrate that BlackMamba performs competitively against both Mamba and transformer baselines, and outperforms in inference and training FLOPs. We fully train and open-source 340M/1.5B and 630M/2.8B BlackMamba models on 300B tokens of a custom dataset. We show that BlackMamba inherits and combines both of the benefits of SSM and MoE architectures, combining linear-complexity generation from SSM with cheap and fast inference from MoE. We release all weights, checkpoints, and inference code open-source. Inference code at: https://github.com/Zyphra/BlackMamba"},"_bibtex":{"value":"@inproceedings{\nanthony2024blackmamba,\ntitle={BlackMamba: Mixture of Experts for State-Space Models},\nauthor={Quentin Gregory Anthony and Yury Tokpanov and Paolo Glorioso and Beren Millidge},\nbooktitle={ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models},\nyear={2024},\nurl={https://openreview.net/forum?id=10dsmPgq9L}\n}"},"title":{"value":"BlackMamba: Mixture of Experts for State-Space Models"}},"tmdate":1713743155520,"pdate":1709534844707,"tcdate":1706915210504,"writers":["ICLR.cc/2024/Workshop/ME-FoMo","ICLR.cc/2024/Workshop/ME-FoMo/Submission29/Authors"],"signatures":["ICLR.cc/2024/Workshop/ME-FoMo/Submission29/Authors"],"forum":"10dsmPgq9L","license":"CC BY 4.0","number":29,"cdate":1706915210504,"readers":["everyone"],"invitations":["ICLR.cc/2024/Workshop/ME-FoMo/-/Submission","ICLR.cc/2024/Workshop/ME-FoMo/-/Post_Submission","ICLR.cc/2024/Workshop/ME-FoMo/-/Edit","ICLR.cc/2024/Workshop/ME-FoMo/Submission29/-/Revision"],"mdate":1713743155520,"odate":1712079580517,"domain":"ICLR.cc/2024/Workshop/ME-FoMo","id":"10dsmPgq9L","version":2},{"content":{"comment":{"value":"Due to word limitations, we answer the remaining questions in this comment. Please review this part of the response after the rebuttal part.\n\n\n### [II].Reply to the Questions\n>**[1/5]Q1. Did the authors consider techniques such as key-value caching for improving inference speed of Transformers for results reported in Table 4?**\n\nThe transformer model we counted in Table 4 has applied key-value caching. When calculating the action of a new time step, we do not recalculate the key and value of the previous time step. However, the transformer's computational complexity grows quadratic with the sequence length, which makes it much slower than the Mamba method in the context of long sequences.\n\n>**[2/5]Q2. Why is Mamba worse in effectiveness (Table 1)? What is a particular (theoretical) reason for this? Why does Mamba shorten the training time?**\n\nMamba models long sequence dependencies through an input-dependent selection mechanism. This mechanism acts independently on each token, so its computational complexity is less than the attention mechanism based on token pairs. Therefore, Mamba's computational complexity grows linearly with the sequence, making it naturally superior to transformers in terms of training speed and inference speed. However, even under the MDP assumption, the states in RL tasks are not independent, which causes Mamba to perform lower than the transformer in terms of effectiveness.\n\n>**[3/5]Q3. Do bigger models help on these tasks? How large are the considered models?**\n\nAs shown in the hyperparameters in Table 3 of the manuscript, our Transformer model follows the setting of the AT method [3], and the Mamba model follows the setting of the DM method [6]. According to the conclusions in the Mamba paper, larger models generally improve performance. Due to resource and time constraints, we will add tests of larger models in the revised manuscript. However, it is worth mentioning that HM, with the current model size, can handle these tasks well.\n\n>**[4/5]Q4. How well does the construction of valuable sub-goals generalize to other environments (e.g., with sparse rewards)?**\n\nThe construction of valuable sub-goals also generalizes well to environments with sparse rewards. Figures 2 and 3 in the manuscript report the tasks in a sparse rewards setting, showing sub-goals' effectiveness. In particular, we will try our best to test HM on the benchmarks suggested by the reviewers in the revised manuscript.\n\n>**[5/5]Q5. How do in-context improvement curves look like on D4RL?**\n\nAs we clarify in W5, the curves of D4RL are shown in Figure 1 in the global PDF file.\n\n[1]Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML 2018.  \n[2]Amago: Scalable in-context reinforcement learning for adaptive agents. ICLR 2024.  \n[3]Emergent Agentic Transformer from Chain of Hindsight Experience. ICML 2023.  \n[4]In-context Reinforcement Learning with Algorithm Distillation. ICLR 2023.  \n[5]Supervised Pretraining Can Learn In-Context Reinforcement Learning. NeurIPS 2023.  \n[6]Decision Mamba: Reinforcement learning via sequence modeling with selective state spaces. 2024."},"title":{"value":"Rebuttal of the remaining questions"}},"tmdate":1730889458940,"tcdate":1722938932450,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission5823/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission5823/Authors"],"forum":"wFzIMbTsY7","number":2,"license":"CC BY 4.0","cdate":1722938932450,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission5823/-/Official_Comment","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730889458940,"domain":"NeurIPS.cc/2024/Conference","replyto":"g4MnMSbKBh","id":"5qbePmnMM5","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["In-context Reinforcement Learning","Mamba"]},"primary_area":{"value":"reinforcement_learning"},"abstract":{"value":"Recent works have shown the remarkable superiority of transformer models in reinforcement learning (RL), where the decision-making problem is formulated as sequential generation. Transformer-based agents could emerge with self-improvement in online environments by providing task contexts, such as multiple trajectories, called in-context RL. However, due to the quadratic computation complexity of attention in transformers, current in-context RL methods suffer from huge computational costs as the task horizon increases. In contrast, the Mamba model is renowned for its efficient ability to process long-term dependencies, which provides an opportunity for in-context RL to solve tasks that require long-term memory. To this end, we first implement Decision Mamba (DM) by replacing the backbone of Decision Transformer (DT). Then, we propose a Decision Mamba-Hybrid (DM-H) with the merits of transformers and Mamba in high-quality prediction and long-term memory. Specifically, DM-H first generates high-value sub-goals from long-term memory through the Mamba model. Then, we use sub-goals to prompt the transformer, establishing high-quality predictions. Experimental results demonstrate that DM-H achieves state-of-the-art in long and short-term tasks, such as D4RL, Grid World, and Tmaze benchmarks. Regarding efficiency, the online testing of DM-H in the long-term task is 28$\\times$ times faster than the transformer-based baselines."},"_bibtex":{"value":"@inproceedings{\nhuang2024decision,\ntitle={Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling},\nauthor={Sili Huang and Jifeng Hu and Zhejian Yang and Liwei Yang and Tao Luo and Hechang Chen and Lichao Sun and Bo Yang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=wFzIMbTsY7}\n}"},"title":{"value":"Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling"},"pdf":{"value":"/pdf/f79cbc369ef6968176c7cc958c79839cb99e59b0.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"huang|decision_mamba_reinforcement_learning_via_hybrid_selective_sequence_modeling"},"authorids":{"value":["~Sili_Huang1","~Jifeng_Hu1","~Zhejian_Yang1","~Liwei_Yang2","~Tao_Luo2","~Hechang_Chen2","~Lichao_Sun1","~Bo_Yang6"]},"authors":{"value":["Sili Huang","Jifeng Hu","Zhejian Yang","Liwei Yang","Tao Luo","Hechang Chen","Lichao Sun","Bo Yang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2402.12192v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"he|panmamba_effective_pansharpening_with_state_space_model"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xuanhua_He:","","https://dblp.org/search/pid/api?q=author:Keyu_Yan:","https://dblp.org/search/pid/api?q=author:Rui_Li_0027:","https://dblp.org/search/pid/api?q=author:Chengjun_Xie:","https://dblp.org/search/pid/api?q=author:Jie_Zhang_0033:","https://dblp.org/search/pid/api?q=author:Man_Zhou:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2402.12192"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2402-12192,\n  publtype={informal},\n  author={Xuanhua He and Ke Cao and Keyu Yan and Rui Li and Chengjun Xie and Jie Zhang and Man Zhou},\n  title={Pan-Mamba: Effective pan-sharpening with State Space Model},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2402.12192},\n  url={https://doi.org/10.48550/arXiv.2402.12192}\n}\n"},"abstract":{"value":"Pan-sharpening involves integrating information from low-resolution multi-spectral and high-resolution panchromatic images to generate high-resolution multi-spectral counterparts. While recent advancements in the state space model, particularly the efficient long-range dependency modeling achieved by Mamba, have revolutionized computer vision community, its untapped potential in pan-sharpening motivates our exploration. Our contribution, Pan-Mamba, represents a novel pan-sharpening network that leverages the efficiency of the Mamba model in global information modeling. In Pan-Mamba, we customize two core components: channel swapping Mamba and cross-modal Mamba, strategically designed for efficient cross-modal information exchange and fusion. The former initiates a lightweight cross-modal interaction through the exchange of partial panchromatic and multi-spectral channels, while the latter facilities the information representation capability by exploiting inherent cross-modal relationships. Through extensive experiments across diverse datasets, our proposed approach surpasses state-of-the-art methods, showcasing superior fusion results in pan-sharpening. To the best of our knowledge, this work is the first attempt in exploring the potential of the Mamba model and establishes a new frontier in the pan-sharpening techniques. The source code is available at \\url{https://github.com/alexhe101/Pan-Mamba}."},"title":{"value":"Pan-Mamba: Effective pan-sharpening with State Space Model"},"authors":{"value":["Xuanhua He","Ke Cao","Keyu Yan","Rui Li","Chengjun Xie","Jie Zhang","Man Zhou"]}},"tmdate":1786344076651,"pdate":1704067200000,"tcdate":1747276352633,"writers":["~"],"signatures":["~Ke_Cao2"],"forum":"HzjXfq7PDt","license":"CC BY-SA 4.0","number":470897,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1786344076651,"domain":"DBLP.org","id":"HzjXfq7PDt","version":2},{"content":{"summary":{"value":"The paper proposes a revised auto-regressive self-supervised training method, MAP, that is specifically designed for the Hybrid Mamba-Transformer network. Unlike the language domain, auto-regressive training underperforms masked image modeling as a transformer training method in the vision domain. However, the paper finds that it is not generalized to Mamba architecture and argues that auto-regressive training can outperform MAE with proper design and hyper-parameter tuning. Based on a lot of pilot studies, the proposed auto-regressive training, MAP, exhibits impressive performance surpassing MAE with Hybrid Mamba-Transformer on ImageNet."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"- What is the major difference between [A] and MAP? All details presented in the paper are necessary to make high-performance auto-regressive training recipes?\n- I think the paper's findings lack novelty and impact. Mamba+AR performance has already been demonstrated in [A]. Matching scanning orders on Mamba and AR looks like a straightforward choice.  Basic random masking with MSE loss is also the best choice for AR. Is there any novel point in MAP that was never presented in existing papers and doesn't align with MAE practices?\n- Where is the definition of `HybridMH` and `HybridMT3D`? It looks like the paper's original model, but I can't find a proper description of it.\n- Why is MAP super effective only in Mamba architecture, and is Hybrid needed to maximize the improvement? Is there any theoretical explanation or insight for this?\n- 3D performance looks marginal. Why is MAP less effective in 3D rather than 2D? Is this improvement meaningful? Because `HybridMT3D` requires additional params and FLOPs, it looks simple computation-performance trade-off for me."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"- MAP training achieves reasonable improvement with Mamba-based architecture. It might be a valuable finding for researchers developing Mamba to make it better than Transformer.\n- The paper presents extensive experiment results."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The effectiveness of auto-regressive training has already been verified by a previous paper [A]. It significantly reduces the novelty and contribution of the paper's findings.\n- Although the paper shows various findings, the findings lack general applicability and theoretical evidence. More experiments are needed to verifies the findings can be applied to general cases, which can contribute to the community.\n- The proposed method, MAP, seems more like a mixture of heuristics and hyper-parameter tuning than a novel training method. It is incremental.\n- Performance improvement in the 3D domain looks marginal. Considering that HybridMT3D requires additional # params and FLOPs, it is not clear that it is better than other methods.\n- The writing quality is not good. It is hard to understand the main argument through the texts\n\n[A] Autoregressive pretraining with mamba in vision, 2024"}},"nonreaders":[],"tmdate":1731427595659,"tcdate":1730360206653,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2428/Reviewer_ij9g"],"signatures":["ICLR.cc/2025/Conference/Submission2428/Reviewer_ij9g"],"forum":"hBVywPrw6d","number":1,"license":"CC BY 4.0","cdate":1730360206653,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2428/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427595659,"domain":"ICLR.cc/2025/Conference","replyto":"hBVywPrw6d","id":"5aJhagVp3p","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"TLDR":{"value":"We propose a new pre-training strategy tailored for the Mamba-Transformer backbone."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Hybrid Vision Backbone","Masked Autoregressive","Self-supervised Learning"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba has achieved significant advantages in long-context modeling and autoregressive tasks, but its scalability with large parameters remains a major limitation in vision applications. pretraining is a widely used strategy to enhance backbone model performance. Although the success of Masked Autoencoder in Transformer pretraining is well recognized, it does not significantly improve Mamba's visual learning performance. We found that using the correct autoregressive pretraining can significantly boost the performance of the Mamba architecture. Based on this analysis, we propose Masked Autoregressive Pretraining(MAP) to pretrain a hybrid Mamba-Transformer vision backbone network. This strategy combines the strengths of both MAE and Autoregressive pretraining, improving the performance of Mamba and Transformer modules within a unified paradigm. Experimental results show that both the pure Mamba architecture and the hybrid Mamba-Transformer vision backbone network pretrained with MAP significantly outperform other pretraining strategies, achieving state-of-the-art performance. We validate the effectiveness of the method on both 2D and 3D datasets and provide detailed ablation studies to support the design choices for each component."},"_bibtex":{"value":"@misc{\nliu2024map,\ntitle={{MAP}: Unleashing Hybrid Mamba-Transformer Vision Backbone{\\textquoteright}s Potential with Masked Autoregressive Pretraining},\nauthor={Yunze Liu and Li Yi},\nyear={2024},\nurl={https://openreview.net/forum?id=hBVywPrw6d}\n}"},"title":{"value":"MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone’s Potential with Masked Autoregressive Pretraining"},"pdf":{"value":"/pdf/10243949177ada96d6426f1e3d852f7611be953b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"liu|map_unleashing_hybrid_mambatransformer_vision_backbones_potential_with_masked_autoregressive_pretraining"},"authorids":{"value":["~Yunze_Liu2","~Li_Yi2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yunze Liu","Li Yi"]}},"version":2},{"content":{"venue":{"value":"Data Min. Knowl. Discov. 2025"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/s10618-025-01149-9.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"lin|crossmamba_multivariate_time_series_forecasting_model_for_crosstemporal_and_crossdimensional_dependencies_with_mamba"},"html":{"value":"https://doi.org/10.1007/s10618-025-01149-9"},"_bibtex":{"value":"@article{DBLP:journals/datamine/LinXHZZZ25,\n  author={Yuhan Lin and Liping Xiong and Zhiyong Hong and Zhiqiang Zeng and Jian Zeng and Guoqiang Zeng},\n  title={Crossmamba: multivariate time series forecasting model for cross-temporal and cross-dimensional dependencies with Mamba},\n  year={2025},\n  month={September},\n  cdate={1756684800000},\n  journal={Data Min. Knowl. Discov.},\n  volume={39},\n  number={5},\n  pages={68},\n  url={https://doi.org/10.1007/s10618-025-01149-9}\n}\n"},"abstract":{"value":"Recent deep models for multivariate time series (MTS) forecasting highlight Transformer-based approaches for capturing long-term (cross-temporal) dependencies. However, most of these models impose uniform transformations across heterogeneous dimensions, failing to capture cross-dimensional dependencies that are crucial for MTS forecasting. Furthermore, the quadratic complexity \\(O(L^2)\\) of the attention mechanism creates an unbearable computational bottleneck when processing long historical sequences. To address these challenges, we propose Crossmamba, a novel Mamba-empowered architecture featuring three key innovations: (1) The MamMLP module exploits Mamba’s selective state space paradigm to achieve linear time complexity for MTS processing; (2) The Full Dimensional State Extraction (FDSE) layer combines linear time series modeling with dimension-aware dependency learning; (3) The Spectral Noising (SN) technique injects frequency-band-specific perturbations to simulate real-world non-stationarity and periodicity. The main structure of the architecture is the Spectral Noising Encoder–Decoder (SNED). It starts with Time Cycle Embedding, which vectorizes the original MTS data into a 2D vector array with time and dimension information. Subsequently, the encoder of the SNED, which is built by multiple FDSE layers and SN layers, processes these to generate multi-granular representations enhanced by simulating non-stationary and periodic changes. The decoder of the SNED transforms these multi-granular representations and accumulates them to obtain the final prediction result. Extensive experimental results demonstrate the effectiveness of Crossmamba over previous techniques. Code is available at https://github.com/IHAN-1212/Crossmamba."},"title":{"value":"Crossmamba: multivariate time series forecasting model for cross-temporal and cross-dimensional dependencies with Mamba"},"authors":{"value":[{"fullname":"Yuhan Lin","username":""},{"fullname":"Liping Xiong","username":"~Liping_Xiong1"},{"fullname":"Zhiyong Hong","username":""},{"fullname":"Zhiqiang Zeng","username":""},{"fullname":"Jian Zeng","username":""},{"fullname":"Guoqiang Zeng","username":""}]}},"tmdate":1782191103459,"pdate":1767139200000,"externalIds":["dblp:journals/datamine/LinXHZZZ25"],"tcdate":1782191099867,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Liping1"],"forum":"YuUex4upZd","license":"CC BY-SA 4.0","number":46049,"cdate":1756684800000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1782191103459,"domain":"OpenReview.net/Public_Article","id":"YuUex4upZd","version":2},{"content":{"comment":{"value":"We thank all reviewers for their careful reading of our work and their thoughtful feedback. All reviewers acknowledged that our work makes a significant architectural contribution, is clearly written, well-motivated, and demonstrates strong empirical results.\n\nOur paper provides a novel perspective for computation where all components, including parameters, are represented as tokens. Interactions between different token types are handled by the attention mechanism. Then, all the linear projections in the Transformer are replaced with this design. This innovation allows for greater architectural flexibility and supports incremental model scaling without retraining, substantially reducing training costs.\n\nReviewers posed insightful questions and constructive comments, which we have addressed in detail in our individual responses.\n\n***\n\n**Discussion of the two-layer MLPs and our pattention.**\n\n**A:** Our motivation is to provide a novel and unified perspective to model computation where all the components, including parameters, are represented as tokens. Interactions between different token types, such as data tokens and parameter tokens, are managed using the attention mechanism.\n\nSpecifically, we start with the formulation that a two-layer MLPs can be implemented as attention [1], and **use this module as the basic computational unit to design the neural network, rather than the previous linear projection**, thereby unifying token-token and token-parameter interactions within a single framework.\n\nBased on this all-attention property, TokenFormer naturally supports incremental scalability, enabling seamless expansion without retraining from scratch. We treat a pair of key-value parameter tokens as a learnable pattern and incrementally scale the model by augmenting this parameter set. This approach enables our model to progressively grow into a larger model based on a pre-trained smaller model, achieving performance comparable to training from scratch but at a significantly lower training cost.\n\nThis concept of tokenizing everything and using attention to establish relationships has the potential to impact topics beyond pretraining models. For example, it can be applied to the hidden states of linear models, enabling incremental memory expansion and reducing excessive information loss observed in linear models (e.g., Mamba [2] and RWKV [3]).\n\n\n[1] Geva, Mor, et al. \"Transformer feed-forward layers are key-value memories.\" EMNLP 2021.\n\n[2] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" COLM 2024.\n\n[3] Peng, Bo, et al. \"Rwkv: Reinventing rnns for the transformer era.\" EMNLP 2023."},"title":{"value":"Response to All Reviewers"}},"tmdate":1732273186941,"tcdate":1732273186941,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission795/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission795/Authors"],"forum":"oQ4igHyh3N","number":2,"license":"CC BY 4.0","cdate":1732273186941,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission795/-/Official_Comment"],"mdate":1732273186941,"domain":"ICLR.cc/2025/Conference","replyto":"oQ4igHyh3N","id":"n0XsIBnjRW","forumContent":{"venue":{"value":"ICLR 2025 Spotlight"},"TLDR":{"value":"Designing a fully attention-based neural network for efficient model scaling through treating model parameters as tokens."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Fully Attention-based Neural Network","Large Language Model","Model Scaling","Tokenized Model Parameters"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Transformers have become the predominant architecture in foundation models due to their excellent performance across various domains. However, the substantial cost of scaling these models remains a significant concern. This problem arises primarily from their dependence on a fixed number of parameters within linear projections. When architectural modifications (e.g., channel dimensions) are introduced, the entire model typically requires retraining from scratch. As model sizes continue growing, this strategy results in increasingly high computational costs and becomes unsustainable. To overcome this problem, we introduce Tokenformer, a natively scalable architecture that leverages the attention mechanism not only for computations among input tokens but also for interactions between tokens and model parameters, thereby enhancing architectural flexibility. By treating model parameters as tokens, we replace all the linear projections in Transformers with our token-parameter attention layer, where input tokens act as queries and model parameters as keys and values. This reformulation allows for progressive and efficient scaling without necessitating retraining from scratch. Our model scales from 124M to 1.4B parameters by incrementally adding new key-value parameter pairs, achieving performance comparable to Transformers trained from scratch while greatly reducing training costs. Code and models are available at {\\color{red}\\url{https://github.com/Haiyang-W/TokenFormer.git}}"},"_bibtex":{"value":"@inproceedings{\nwang2025tokenformer,\ntitle={TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters},\nauthor={Haiyang Wang and Yue Fan and Muhammad Ferjad Naeem and Yongqin Xian and Jan Eric Lenssen and Liwei Wang and Federico Tombari and Bernt Schiele},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=oQ4igHyh3N}\n}"},"title":{"value":"TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters"},"pdf":{"value":"/pdf/430572f27d6b814d52dfb4df61bbb5ccad24d5d0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wang|tokenformer_rethinking_transformer_scaling_with_tokenized_model_parameters"},"authorids":{"value":["~Haiyang_Wang2","~Yue_Fan1","~Muhammad_Ferjad_Naeem1","~Yongqin_Xian1","~Jan_Eric_Lenssen1","~Liwei_Wang1","~Federico_Tombari1","~Bernt_Schiele1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Haiyang Wang","Yue Fan","Muhammad Ferjad Naeem","Yongqin Xian","Jan Eric Lenssen","Liwei Wang","Federico Tombari","Bernt Schiele"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2504.08964v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"liu|bidirectional_linear_recurrent_models_for_sequencelevel_multisource_fusion"},"html":{"value":"https://doi.org/10.48550/arXiv.2504.08964"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2504-08964,\n  publtype={informal},\n  author={Qisai Liu and Zhanhong Jiang and Joshua R. Waite and Chao Liu and Aditya Balu and Soumik Sarkar},\n  title={Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion},\n  year={2025},\n  month={April},\n  cdate={1743465600000},\n  journal={CoRR},\n  volume={abs/2504.08964},\n  url={https://doi.org/10.48550/arXiv.2504.08964}\n}\n"},"abstract":{"value":"Sequence modeling is a critical yet challenging task with wide-ranging applications, especially in time series forecasting for domains like weather prediction, temperature monitoring, and energy load forecasting. Transformers, with their attention mechanism, have emerged as state-of-the-art due to their efficient parallel training, but they suffer from quadratic time complexity, limiting their scalability for long sequences. In contrast, recurrent neural networks (RNNs) offer linear time complexity, spurring renewed interest in linear RNNs for more computationally efficient sequence modeling. In this work, we introduce BLUR (Bidirectional Linear Unit for Recurrent network), which uses forward and backward linear recurrent units (LRUs) to capture both past and future dependencies with high computational efficiency. BLUR maintains the linear time complexity of traditional RNNs, while enabling fast parallel training through LRUs. Furthermore, it offers provably stable training and strong approximation capabilities, making it highly effective for modeling long-term dependencies. Extensive experiments on sequential image and time series datasets reveal that BLUR not only surpasses transformers and traditional RNNs in accuracy but also significantly reduces computational costs, making it particularly suitable for real-world forecasting tasks. Our code is available here."},"title":{"value":"Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion"},"authors":{"value":[{"fullname":"Qisai Liu","username":"~Qisai_Liu1"},{"fullname":"Zhanhong Jiang","username":""},{"fullname":"Joshua R. Waite","username":""},{"fullname":"Chao Liu","username":""},{"fullname":"Aditya Balu","username":""},{"fullname":"Soumik Sarkar","username":""}]}},"tmdate":1791992091530,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2504-08964"],"tcdate":1791992088270,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Qisai_Liu1"],"forum":"NWjyG58kmm","license":"CC BY-SA 4.0","number":184240,"cdate":1743465600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1791992091530,"domain":"OpenReview.net/Public_Article","id":"NWjyG58kmm","version":2},{"content":{"venue":{"value":"ICME 2025"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/11208895/11208897/11209217.pdf"},"venueid":{"value":"dblp.org/conf/ICMCS/2025"},"paperhash":{"value":"li|spmamba_leveraging_longsequence_modeling_with_state_space_models_for_speech_separation"},"authorids":{"value":["~Kai_Li14","https://dblp.org/search/pid/api?q=author:Guo_Chen:","https://dblp.org/search/pid/api?q=author:Runxuan_Yang:","https://dblp.org/search/pid/api?q=author:Xiaolin_Hu_0001:"]},"html":{"value":"https://doi.org/10.1109/ICME59968.2025.11209217"},"_bibtex":{"value":"@inproceedings{DBLP:conf/icmcs/LiCYH25,\n  author={Kai Li and Guo Chen and Runxuan Yang and Xiaolin Hu},\n  title={SPMamba: Leveraging Long-Sequence Modeling with State Space Models for Speech Separation},\n  year={2025},\n  cdate={1735689600000},\n  pages={1-6},\n  url={https://doi.org/10.1109/ICME59968.2025.11209217},\n  booktitle={ICME},\n  crossref={conf/icmcs/2025}\n}\n"},"abstract":{"value":"Existing CNN-based speech separation models face local receptive field limitations and cannot effectively capture long time dependencies. Although LSTM and Transformer-based speech separation models can avoid this problem, their high complexity causes them to face the challenge of computational resources and inference efficiency when dealing with long audio. To address this challenge, we introduce an innovative speech separation method called SPMamba. This model builds upon the robust TF-GridNet architecture, replacing its traditional BLSTM modules with bidirectional Mamba modules. These modules effectively model the spatiotemporal relationships between the time and frequency dimensions, allowing SPMamba to capture long-range dependencies with linear computational complexity. Specifically, the bidirectional processing within the Mamba modules enables the model to utilize both past and future contextual information, thereby enhancing separation performance. Extensive experiments were conducted on public datasets, including the WSJ0-2Mix and WHAM! and Libri2Mix, as well as the newly constructed Echo2Mix dataset, demonstrated that SPMamba achieved superior results to previous state-of-the-art (SOTA) models with reduced computational complexity. These findings highlight the effectiveness of SPMamba in addressing the intricate challenges of speech separation in complex environments. The source code for SPMamba is publicly accessible at https://anonymous.4open.science/r/SPMamba-ICME/."},"title":{"value":"SPMamba: Leveraging Long-Sequence Modeling with State Space Models for Speech Separation"},"authors":{"value":["Kai Li","Guo Chen","Runxuan Yang","Xiaolin Hu"]}},"tmdate":1767595521544,"pdate":1735689600000,"externalIds":["dblp:conf/icmcs/LiCYH25"],"tcdate":1767595519217,"writers":["~"],"signatures":["~Kai_Li15"],"forum":"fSzQKuzduu","license":"CC BY-SA 4.0","number":714775,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1767595521544,"domain":"DBLP.org","id":"fSzQKuzduu","version":2},{"content":{"summary":{"value":"The paper presents a formal theoretical study of the Mamba architecture. It analyzes a simplified single-layer Mamba followed by a two-layer MLP trained with gradient descent on two synthetic data regimes — a majority-voting model and a locality-structured model — where class labels depend on subsets of relevant features within the input sequence. It derives non-asymptotic convergence and generalization guarantees, proving that Mamba’s gating mechanism learns to amplify class-relevant features while suppressing irrelevant ones. The analysis links learning efficiency and generalization to interpretable data parameters such as signal-to-noise ratio and the fraction of informative tokens. Synthetic experiments confirm these trends qualitatively."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Have you observed any empirical evidence suggesting these theoretical results hold beyond your simplified setup? Or how do you think these results fair in practice?\n2. Can your analysis or conclusions be generalized to more complex or non-Gaussian data distributions (e.g., correlated, heavy-tailed, or structured inputs as in language or vision)? If not, what aspects of Mamba’s behavior might your theory fail to capture?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper is well-written, clearly structured, and theoretical results are supported by rigorous proofs with proof intuitions given in the main text.\n2. The paper is well-situated in the literature. I would only suggest to add [1,2], when discussing the connection of SSMs and Transformers in L147-154, for additional reading on how Mamba relates to attention, altough not critical.\n3. The theoretical analysis of the learning process of Mamba is novel and timely, as Mamba has become an alternative to Transformers.\n4. The results are convincing, however their generality is not assessed yet.\n5. The paper does a good job in linking the empirical findings to the theoretical results, supporting each statement with empirical evidence.\n\n[1] Dao & Gu (2024), \"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality\", https://arxiv.org/abs/2405.21060\n\n[2] Sieber et al. (2024), \"Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks\", https://arxiv.org/abs/2405.15731"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The main drawback of the paper is its limited scope, both in its model selection and the application setting. While chosing a single model (in this case Mamba) for such a study is reasonable, I would have expected a broader application setting, i.e., not limiting the study to binary classification.\n2. The investigated data models are purely theoretical and are not representative of more realistic data models, e.g. the Gaussian distribution assumption is not realistic for practical applications. Having a discussion on this point in the paper would be appreciated.\n3. The theoretical results are not novel in itself and are mainly extensions to the Mamba specifics. Altough I think this is not a major problem, but it reduces the originality of the contribution.\n4. The paper could do a better job at clarifying its limitations, e.g. with a paragraph in the conclusion.\n\n\nMinor points:\n- The short paragraphs before lemmas/theorems already discuss the results of the lemmas/theorems before they are stated. It would be better for the flow to reserve this discussion for after the lemma/theorem and only give a brief introduction to which results is shown next.\n- The remarks after the results are nice, but should not be wrapped in a proof environment, i.e., should not have a square at the end.\n- the empirical results provide no quantitative comparison of actual to theoretical convergence rates and are purely qualitative. Including, such a comparison/result would strengthen the paper."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931501018,"tcdate":1761924600568,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19648/Reviewer_T7xH"],"signatures":["ICLR.cc/2026/Conference/Submission19648/Reviewer_T7xH"],"forum":"hvpKqEYJjj","number":1,"license":"CC BY 4.0","cdate":1761924600568,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19648/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931501018,"domain":"ICLR.cc/2026/Conference","replyto":"hvpKqEYJjj","id":"0bRayXUiMQ","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Training Dynamics","Feature Learning","Generalization","Mamba","Learning Theory","Selective State Space Models"]},"supplementary_material":{"value":"/attachment/c9fcb2ad298f51c4efaa9f51d0c97c0844014e27.zip"},"primary_area":{"value":"learning theory"},"abstract":{"value":"The recent empirical success of Mamba and other selective state space models (SSMs) has renewed interest in non-attention architectures for sequence modeling, yet their theoretical foundations remain underexplored. We present a first-step analysis of generalization and learning dynamics for a simplified but representative Mamba block: a single-layer, single-head selective SSM with input-dependent gating, followed by a two-layer MLP trained via gradient descent (GD). Our study adopts a structured data model with tokens that include both class-relevant and class-irrelevant patterns under token-level noise and examines two canonical regimes: majority-voting and locality-structured data sequences. We prove that the model achieves guaranteed generalization by establishing non-asymptotic sample complexity and convergence rate bounds, which improve as the effective signal increases and the noise decreases.\nFurthermore, we show that the gating vector aligns with class-relevant features while ignoring irrelevant ones, thereby formalizing a feature-selection role similar to attention but realized through selective recurrence. Numerical experiments on synthetic data justify our theoretical results. Overall, our results provide principled insight into when and why Mamba-style selective SSMs learn efficiently, offering a theoretical counterpoint to Transformer-centric explanations."},"_bibtex":{"value":"@inproceedings{\nshandirasegaran2026a,\ntitle={A Theoretical Analysis of Mamba{\\textquoteright}s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models},\nauthor={Mugunthan Shandirasegaran and Hongkang Li and Songyang Zhang and Meng Wang and Shuai Zhang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=hvpKqEYJjj}\n}"},"title":{"value":"A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models"},"pdf":{"value":"/pdf/7ff125c33f63bfb0017d10d5e63ca5d19e737a02.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"shandirasegaran|a_theoretical_analysis_of_mambas_training_dynamics_filtering_relevant_features_for_generalization_in_state_space_models"},"authorids":{"value":["~Mugunthan_Shandirasegaran1","~Hongkang_Li1","~Songyang_Zhang2","~Meng_Wang4","~Shuai_Zhang6"]},"authors":{"value":["Mugunthan Shandirasegaran","Hongkang Li","Songyang Zhang","Meng Wang","Shuai Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Q-Mamba, a post-training quantization (PTQ) approach aimed at improving the efficiency of Mamba models by converting linear projections and state caches into low-bit integers. The authors present two techniques: Decoupled Scale Quantization (DSQ), which addresses the outliers in both state and channel dimensions, and Efficient Selectivity Reconstruction (ESR) to preserve the model’s selectivity. They demonstrate Q-Mamba’s efficacy across various quantization settings and tasks, reporting a 50% reduction in memory with minimal accuracy degradation in some configurations."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1- **Comparison to Related Work**: How does Q-MAMBA compare to other quantization approaches such as Mamba-PTQ, OmniQuant, OPTQ, and AWQ in terms of performance and memory efficiency? The papers are cited in the weaknesses section.\n\n2- **Impact on Latency**: What is Q-Mamba's effect on real-world latency, especially for larger batch sizes or latency-sensitive tasks? Additionally, what are the overheads of the outer products for the quantization and dequantizaiton of tensors?\n\n3- **Scalability of DSQ**: Can DSQ be effectively scaled or applied to other architectures, or does it require specific adjustments for compatibility?\n\n4- Clarification on DSQ Calculation: In Equation 7, should the mean of the absolute values of elements in $h$ be used instead of the mean? Further clarification on this point would be helpful."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. **Relevance**: The paper addresses a key issue by optimizing Mamba’s memory and computational requirements, making it relevant to the field of efficient large language models (LLMs).\n\n2. **Novel Quantization Scheme**: DSQ’s approach to outlier mitigation in both state and channel dimensions adds a unique element to quantization in SSMs.\n\n3. **Practical Impac**: Q-MAMBA's claimed memory reduction is substantial, especially for real-world applications where memory efficiency is essential.\n\n4. **Clarity in Methodology**: The authors provide a theoretical foundation for DSQ and clear motivation for ESR, contributing to understanding their method."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1- **Computation Overhead in Quantization**: The quantization and dequantization process requires computing an outer product of two vectors, which can be computationally expensive compared to other group quantization methods available in the field.\n\n2- **Lack of Comprehensive Evaluation**: The experimental results lack sufficient diversity across tasks and model sizes. Evaluating Q-Mamba’s performance on additional Mamba family models would strengthen its claims of versatility.\n\n3- **Limited Baseline Comparisons**: The paper could benefit from comparisons with related quantization methods like OmniQuant [1], OPTQ [2], AWQ [3], and Mamba-PTQ [4]. Assessing Q-Mamba against these methods would provide better insights into its performance, especially given that Mamba is composed largely of linear layers for which these quantization schemes have proven effective.\n\n4- **Insufficient Analysis of Latency**: The paper does not sufficiently explore the impact of Q-Mamba on inference latency, which could be critical for real-world deployment. More timing results can enrich the experiments section further.\n\n5- **Potential Misstep in DSQ Calculation**: In Equation 7, it appears that the mean of the absolute values of elements in $h$ should be used rather than the mean. If this is not an oversight, additional explanation would be valuable.\n\n6- **Justification of ESR Design Choices**: ESR’s design choices, particularly which parameters are made learnable, lack sufficient exploration, and further analysis on this aspect would clarify ESR's effectiveness.\n\n\n[1] Shao et al. \"OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models\", ICLR 2024\n\n[2] Frantar et al. \"OPTQ: Accurate Quantization for Generative Pre-trained Transformers\", ICLR 2023\n\n[3] Lin et al. \"AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration\", MLSys 2024\n\n[4] Pierro et al. \"Mamba-PTQ: Outlier Channels in Recurrent Large Language Models\", ICML 2024"}},"nonreaders":[],"tmdate":1731427532103,"tcdate":1730086447780,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2045/Reviewer_tXAm"],"signatures":["ICLR.cc/2025/Conference/Submission2045/Reviewer_tXAm"],"forum":"AY1S52vr0a","number":2,"license":"CC BY 4.0","cdate":1730086447780,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2045/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427532103,"domain":"ICLR.cc/2025/Conference","replyto":"AY1S52vr0a","id":"4NNhkQrRHr","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Quantization"]},"supplementary_material":{"value":"/attachment/0924a0296e7a1a9ad7158edce07c33777608192e.pdf"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSMs), such as Mamba, have recently demonstrated the potential to match or even surpass Transformers in language understanding tasks, making them a promising alternative for designing Large Language Models (LLMs). \nConcurrently, model quantization, especially Post-Training Quantization (PTQ), has been proven effective in reducing memory usage and inference latency in LLMs.\nIn this paper, we explore post-training quantization for Mamba (\\textbf{Q-Mamba}) by turning both linear projections and state caches into low-bit integers for efficient inference. \nAfter a theoretical analysis of the causes of outliers in states, we propose \\textbf{Decoupled Scale Quantization (DSQ)}, which mitigates outliers in both the state and channel dimensions by applying separate quantization scales.\nTo preserve the selective ability of quantized Mamba, we introduce \\textbf{Efficient Selectivity Reconstruction (ESR)}, a block-wise reconstruction method that involves a novel quantization simulation scheme, enabling fast parallel scan algorithms with the non-linear quantization function.\nWe demonstrate the effectiveness of Q-Mamba across various quantization settings, model sizes, and both generation and zero-shot tasks. \nIn particular, for Mamba2-2.7B with W8A8H4 quantization, Q-Mamba achieves a 50\\% reduction in memory consumption with only a 2.13\\% average accuracy degradation on zero-shot tasks."},"_bibtex":{"value":"@misc{\ntianqi2025qmamba,\ntitle={Q-Mamba: Towards more efficient Mamba models via Post-Training Quantization},\nauthor={Chen Tianqi and Yuanteng Chen and Weixiang Xu and Zeyu Zhu and Peisong Wang and Jian Cheng},\nyear={2025},\nurl={https://openreview.net/forum?id=AY1S52vr0a}\n}"},"title":{"value":"Q-Mamba: Towards more efficient Mamba models via Post-Training Quantization"},"pdf":{"value":"/pdf/1c7e10c8b02e5d50be6957f056d845dde2575456.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"tianqi|qmamba_towards_more_efficient_mamba_models_via_posttraining_quantization"},"authorids":{"value":["~Chen_Tianqi1","~Yuanteng_Chen1","~Weixiang_Xu2","~Zeyu_Zhu1","~Peisong_Wang1","~Jian_Cheng7"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chen Tianqi","Yuanteng Chen","Weixiang Xu","Zeyu Zhu","Peisong Wang","Jian Cheng"]}},"version":2},{"content":{"summary":{"value":"The paper analyzes why selective state space models like Mamba forget over distance, formalizes this with a horizontal–vertical memory fidelity framework for token-level and cross-layer information loss, and shows that Mamba’s long-range contributions decay exponentially. It then introduces an architecture that couples lightweight state summarization (“Note Block”) with cross-token and sparsely triggered cross-layer attention to retain salient signals while preserving linear time/space complexity."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. The Note Block and MemMamba block rely on importance scores and dual thresholds (τ₁ for “take note,” τ₂ for cross-token attention). Are τ₁/τ₂ learned, scheduled, or fixed? Are these threshoulds sensitive for the model's performance?\n2. What exact priority metric is used for Note block? Please compare FIFO vs priority on quality/latency."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. “Note Block” state summarization + cross-token and sparse cross-layer attention improve long-range recall while keeping O(n) time."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Passkey Retrieval is a relatively simple task on in-context retrieval, and it is better to try a more difficult mutli-key-value retrival task., such as Phonebook and RULER.\n2. Missing technical details and ablations. See Questions."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919696454,"tcdate":1762131419347,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7614/Reviewer_ToNT"],"signatures":["ICLR.cc/2026/Conference/Submission7614/Reviewer_ToNT"],"forum":"jPYJuvzof2","number":4,"license":"CC BY 4.0","cdate":1762131419347,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7614/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919696454,"domain":"ICLR.cc/2026/Conference","replyto":"jPYJuvzof2","id":"ionImkQtf8","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Long-sequence modeling","Mamba","Memory mechanisms"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the explosive growth of data, long-sequence modeling has become increasingly important in tasks such as natural language processing and bioinformatics. However, existing methods face inherent trade-offs between efficiency and memory. Recurrent neural networks suffer from gradient vanishing and explosion, making them hard to scale. Transformers can model global dependencies but are constrained by quadratic complexity. Recently, selective state-space models such as Mamba have demonstrated high efficiency with $O(n)$ time and $O(1)$ recurrent inference, yet their long-range memory decays exponentially. In this work, we conduct mathematical derivations and information-theoretic analysis to systematically uncover the memory decay mechanism of Mamba, answering a fundamental question: what is the nature of Mamba’s long-range memory and how does it retain information? To quantify key information loss, we further introduce horizontal–vertical memory fidelity metrics that capture degradation both within and across layers. Inspired by how humans distill and retain salient information when reading long documents, we propose MemMamba, a novel architectural framework that integrates state summarization mechanism together with cross-layer and cross-token attention, which alleviates long-range forgetting while preserving linear complexity. MemMamba achieves significant improvements over existing Mamba variants and Transformers on long-sequence benchmarks such as PG19-PPL and Passkey Retrieval, while delivering a 48\\% speedup in inference efficiency. Both theoretical analysis and empirical results demonstrate that MemMamba achieves a breakthrough in the complexity–memory trade-off, offering a new paradigm for ultra-long sequence modeling. The code and the pre-trained models will be released upon acceptance."},"_bibtex":{"value":"@misc{\nwang2026memmamba,\ntitle={MemMamba: Rethinking Memory Patterns in State Space Model},\nauthor={Youjin Wang and Yangjingyi Chen and Jiahao yan and Jiaxuan Lu and Xiao Sun},\nyear={2026},\nurl={https://openreview.net/forum?id=jPYJuvzof2}\n}"},"title":{"value":"MemMamba: Rethinking Memory Patterns in State Space Model"},"pdf":{"value":"/pdf/26376d032f685fb35659e4ab241730a5eb60a8ff.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wang|memmamba_rethinking_memory_patterns_in_state_space_model"},"authorids":{"value":["~Youjin_Wang1","~Yangjingyi_Chen1","~Jiahao_yan1","~Jiaxuan_Lu1","~Xiao_Sun8"]},"authors":{"value":["Youjin Wang","Yangjingyi Chen","Jiahao yan","Jiaxuan Lu","Xiao Sun"]}},"version":2},{"content":{"TLDR":{"value":"We propose FSMamba, a time series forecasting method that uses Mamba with regularization strategy to address the sequential order bias in capturing channel dependencies."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Time Series Forecasting","Mamba"]},"supplementary_material":{"value":"/attachment/d0aef3ef8995051541105dd0e448b12055805dee.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Mamba is a sequential model that has recently emerged as a promising alternative to Transformers, offering near-linear complexity.\nHowever, although channels in time series (TS) data generally lack a sequential order, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this, prior works have adopted bidirectional Mamba to scan channels in both forward and reverse orders. In this paper, we show that unidirectional Mamba can effectively replace the bidirectional Mamba with simple strategies. To this end, we propose FSMamba, a TS forecasting method employing a unidirectional Mamba that incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order. Furthermore, we introduce channel similarity modeling, a pretraining task to preserve similarities between channels from the data space to the latent space to enhance the ability to capture CD. Extensive experiments demonstrate the efficacy of our method, achieving state-of-the-art performance on diverse datasets."},"_bibtex":{"value":"@misc{\nlee2026is,\ntitle={Is Bidirectionality Necessary in Mamba for Time Series Forecasting?},\nauthor={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\nyear={2026},\nurl={https://openreview.net/forum?id=N6MIqE11eF}\n}"},"title":{"value":"Is Bidirectionality Necessary in Mamba for Time Series Forecasting?"},"pdf":{"value":"/pdf/bf428f2b7b338e6732b17e2ed317a213028a6cfe.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"lee|is_bidirectionality_necessary_in_mamba_for_time_series_forecasting"},"authorids":{"value":["~Seunghan_Lee1","~Juri_Hong1","~Kibok_Lee1","~Taeyoung_Park1"]},"authors":{"value":["Seunghan Lee","Juri Hong","Kibok Lee","Taeyoung Park"]}},"tmdate":1770805185291,"tcdate":1758351291899,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23977/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission23977/Authors"],"forum":"N6MIqE11eF","license":"CC BY 4.0","number":23977,"cdate":1758351291899,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission23977/-/Full_Submission","ICLR.cc/2026/Conference/Submission23977/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770805185291,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"N6MIqE11eF","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.16371v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"yang|uncovering_selective_state_space_models_capabilities_in_lifelong_sequential_recommendation"},"authorids":{"value":["","~Yuanzi_Li3","~Jingyu_Zhao5","~Hanbing_Wang1","https://dblp.org/search/pid/api?q=author:Muyang_Ma:","https://dblp.org/search/pid/api?q=author:Jun_Ma_0001:","~Zhaochun_Ren1","~Mengqi_Zhang1","~Xin_Xin2","~Zhumin_Chen1","~Pengjie_Ren1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.16371"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-16371,\n  publtype={informal},\n  author={Jiyuan Yang and Yuanzi Li and Jingyu Zhao and Hanbing Wang and Muyang Ma and Jun Ma and Zhaochun Ren and Mengqi Zhang and Xin Xin and Zhumin Chen and Pengjie Ren},\n  title={Uncovering Selective State Space Model's Capabilities in Lifelong Sequential Recommendation},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.16371},\n  url={https://doi.org/10.48550/arXiv.2403.16371}\n}\n"},"abstract":{"value":"Sequential Recommenders have been widely applied in various online services, aiming to model users' dynamic interests from their sequential interactions. With users increasingly engaging with online platforms, vast amounts of lifelong user behavioral sequences have been generated. However, existing sequential recommender models often struggle to handle such lifelong sequences. The primary challenges stem from computational complexity and the ability to capture long-range dependencies within the sequence. Recently, a state space model featuring a selective mechanism (i.e., Mamba) has emerged. In this work, we investigate the performance of Mamba for lifelong sequential recommendation (i.e., length>=2k). More specifically, we leverage the Mamba block to model lifelong user sequences selectively. We conduct extensive experiments to evaluate the performance of representative sequential recommendation models in the setting of lifelong sequences. Experiments on two real-world datasets demonstrate the superiority of Mamba. We found that RecMamba achieves performance comparable to the representative model while significantly reducing training duration by approximately 70% and memory costs by 80%. Codes and data are available at \\url{https://github.com/nancheng58/RecMamba}."},"title":{"value":"Uncovering Selective State Space Model's Capabilities in Lifelong Sequential Recommendation"},"authors":{"value":["Jiyuan Yang","Yuanzi Li","Jingyu Zhao","Hanbing Wang","Muyang Ma","Jun Ma","Zhaochun Ren","Mengqi Zhang","Xin Xin","Zhumin Chen","Pengjie Ren"]}},"tmdate":1768583088478,"pdate":1704067200000,"tcdate":1727701979322,"writers":["~"],"signatures":["~Mengqi_Zhang1"],"forum":"ASzx5gxx3S","license":"CC BY-SA 4.0","number":114213,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1768583088478,"domain":"DBLP.org","id":"ASzx5gxx3S","version":2},{"content":{"summary":{"value":"This paper proposes the Holistic Protean Block (HPB) for DNA sequence modeling, consisting of three components: (1) Locus Plasticity Layer (LPL) using dynamic multi-scale convolutions for local features, (2) Domain Plasticity Layer (DPL) combining global convolutions and wavelet transforms for time-frequency modeling, and (3) Saliency Plasticity Layer (SPL) for channel and position feature weighting. The authors claim HPB achieves \"multi-level plasticity\" to capture diverse DNA patterns including local motifs, long-range dependencies, and periodic signals. Evaluated on genomic benchmarks, the HPB-based model (HPB-DNA) reports state-of-the-art performance with quasi-linear complexity O(L log L)."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. Line 51 claims Transformers \"lack strong inductive biases for local patterns\" citing Enformer, but this requires stronger evidence or clarification of what Enformer actually states.\n\n2. Have the authors considered applying this architecture to sequence-to-function modeling tasks like Enformer (predicting gene expression from long genomic sequences)? The motivation emphasizes long-range dependencies and frequency-sensitive patterns, which are most relevant in such tasks requiring long-context predictions. The current evaluation on short-sequence benchmarks cannot adequately demonstrate whether the proposed complexity genuinely improves biological modeling or merely provides marginal gains on oversimplified tasks."},"rating":{"value":2},"details_of_ethics_concerns":{"value":"NA"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":1},"strengths":{"value":"1. Modeling DNA from a frequency perspective is novel and interesting, as DNA sequences indeed exhibit periodic patterns that current methods overlook. However, this novel point is obscured by excessive packaging of incremental architectural designs, diminishing the paper's clarity and impact.\n\n2. The experimental evaluation is comprehensive, covering diverse benchmarks including regulatory annotation, histone markers, and variant effect prediction.\n\n3. The ablation studies systematically validate the contribution of each component."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Most architectural components are incremental improvements: multi-scale convolutions, gated convolutions, and dual-axis feature gating are well-established techniques. While incremental engineering is acceptable, the authors unnecessarily complicate these straightforward designs with unclear terminology, making the paper difficult to read and understand.\n\n2. The paper emphasizes \"long-range DNA sequence modeling\" yet pre-trains on only 1K bp sequences (Appendix A.2), far shorter than Enformer (196K bp). This length cannot capture enhancer-promoter interactions mentioned in the motivation nor demonstrate the claimed dynamic modeling capabilities. Moreover, when both pre-training and evaluation use short sequences (<= 1K bp), the O(L log L) complexity advantage over linear methods becomes negligible. This undermines the \"long-range\" claim and makes comparisons with methods pre-trained on longer sequences potentially unfair.\n\n3. The abstract and introduction lack clarity from both biological and technical perspectives, obscuring rather than clarifying the contributions. I strongly recommend the authors use simpler, community-standard terminology rather than fancy-sounding phrases that obscure meaning. If the authors truly want to use these terms, they should provide clear explanations in context; otherwise, it will be very confusing for readers.  For examples: \n   - Lines 14-15: \"single/multi-domain\" operations are undefined, what is \"domain\" here?\n   - Line 17: \"plasticity\" is non-standard in both AI and genomics literature\n   - Line 18: \"Locus Plasticity Layer\" uses \"locus\" (a genomics term for gene position) when \"position\" suffices\n   - Line 85: \"construction coefficients\" is simply gating/weighting as in MoE but needlessly renamed\n   - Line 86: \"meta features\" lacks clear definition\n   - \"Holistic Protean Block\" uses uncommon terminology to convey only a general description of the model design  \n \n  4. The related work omits discussion of wavelet transforms and frequency-domain methods, despite frequency modeling being the paper's main novelty. Since most genomics researchers are unfamiliar with signal processing techniques, comprehensive background is essential.\n\n5. Line 82 claims the HPB is a \"scalable building block,\" but the largest model tested is under 2M parameters. Given the architectural complexity (three parallel modules with wavelets, FFT, and dynamic convolutions), it is unclear whether this design can scale up like foundation models in NLP or other genomics models.\n\n6. It appears that different pre-trained models were used for different downstream tasks, which seems to contradict the concept of a foundation model. Additionally, the hyperparameters across downstream tasks vary drastically, raising concerns that the downstream performance may be attributed to hyperparameter tuning rather than fundamental improvements."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360024911,"tcdate":1761818583106,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1587/Reviewer_R4oi"],"signatures":["ICLR.cc/2026/Conference/Submission1587/Reviewer_R4oi"],"forum":"Cxk7aaHt8o","number":2,"license":"CC BY 4.0","cdate":1761818583106,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1587/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360024911,"domain":"ICLR.cc/2026/Conference","replyto":"Cxk7aaHt8o","id":"EuFjsfpjAI","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["holistic protean block; DNA sequence modeling; foundation model; biological analysis"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Modeling DNA sequences, which are defined by a complex interplay of local motifs, long-range dependencies, and periodic patterns, is a fundamental challenge in computational biology. Existing foundation models based on CNNs, Transformers, and SSMs are constrained by static or time-domain-only signal processing operations, which limit the architectural flexibility and multi-domain perspective needed to fully capture the diverse features of DNA. Here, we introduce the **Holistic Protean Block (HPB)**, a novel scalable architecture that achieves multi-level plasticity through three synergistic layers. Its Locus Plasticity Layer (LPL) provides token-level plasticity by employing token-specific convolution operations, allowing it to precisely model fine-grained, local patterns. Its Domain Plasticity Layer (DPL) establishes perspective-level plasticity by concurrently modeling both sequential (time) and spectral (frequency) features, enabling it to form multi-domain, global representations. Its Saliency Plasticity Layer (SPL) realizes information flow plasticity by learning saliency scores along dual axes, thereby permitting it to focus information flow on the most critical features. These layers work in tandem, extracting a holistic representation of diverse genomic patterns by adaptively reshaping their computational strategy. The DNA model constructed with HPB (**HPB-DNA**) not only achieves state-of-the-art performance on various genomic benchmarks with a quasi-linear complexity, but is also validated by in-depth model analyses, which collectively establish the HPB as a more powerful and principled paradigm for DNA sequence modeling. Code will be available upon acceptance."},"_bibtex":{"value":"@misc{\nliu2026holistic,\ntitle={Holistic protean block for long-range {DNA} sequence modeling},\nauthor={Binghao Liu and Wenzheng Zhao and Zhijie Zheng and Fei Gu},\nyear={2026},\nurl={https://openreview.net/forum?id=Cxk7aaHt8o}\n}"},"title":{"value":"Holistic protean block for long-range DNA sequence modeling"},"pdf":{"value":"/pdf/ec99f4c0c492c878711d6ee97f1fd7a33d9b77a2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"liu|holistic_protean_block_for_longrange_dna_sequence_modeling"},"authorids":{"value":["~Binghao_Liu1","~Wenzheng_Zhao1","~Zhijie_Zheng3","~Fei_Gu1"]},"authors":{"value":["Binghao Liu","Wenzheng Zhao","Zhijie Zheng","Fei Gu"]}},"version":2},{"content":{"summary":{"value":"The paper addresses the Lookback Window Limitation (LWL) in long-term time series forecasting, where current models struggle with longer sequences due to redundancy and overfitting. To solve this, the authors propose the Information Bottleneck Filter (IBF) to reduce redundancy and the Hybrid-Transformer-Mamba (HTM) model to handle long sequences efficiently. These solutions enable improved performance with extended lookback windows."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Can a comparison of other baselines with different lookback window lengths?\n2. Why was the choice of 1024 made, and how does the performance with longer window lengths? Could more experimental analysis be provided to demonstrate that the model can indeed extract information more relevant to prediction?\n3. The paper mentions several key hyperparameters, such as K, β, and τ. Does the selection of these parameters have a significant impact on model performance? Why is there no discussion on the choice of parameters in the experimental results?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper introduces a novel approach to overcoming the Lookback Window Limitation (LWL) in long-term time series forecasting, which is a significant research contribution. The paper innovates both theoretically and practically by introducing the Information Bottleneck Filter (IBF) and the Hybrid-Transformer-Mamba (HTM) model."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper mentions the use of HYBRID-TRANSFORMER-MAMBA (HTM) to model long and short time series information separately, but it does not provide diagrams to detail how the Transformer and Mamba models are combined, reducing the comprehensibility of the method.\n2. Table 1 only shows results for L=1024, but some baseline methods may perform better with shorter lookback windows; therefore, it would be beneficial to provide the performance of each method under its original configuration.\n3. The authors select the best results for T=720 by choosing different lookback windows {24, 48, 96, 192, 336, 720, 1024}, which may seem unfair. The theory proposed by the authors should not only apply to long sequences; thus, the model's ability to extract information for any input length should be superior to the current best methods.\n4. The visualization results in Figure 3 (c) are intriguing. Could the authors please explain how this figure was obtained?"}},"nonreaders":[],"tmdate":1731428336038,"tcdate":1730445906668,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6771/Reviewer_RXSX"],"signatures":["ICLR.cc/2025/Conference/Submission6771/Reviewer_RXSX"],"forum":"hVpAjJPfgZ","number":3,"license":"CC BY 4.0","cdate":1730445906668,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6771/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428336038,"domain":"ICLR.cc/2025/Conference","replyto":"hVpAjJPfgZ","id":"Lx8SkyH2Sv","forumContent":{"TLDR":{"value":"We adopts a Transformer-Mamba hybrid architecture and information bottleneck for extreme long lookback windows in LSTF."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["long-term time series forcasting","Mamba","Information Bottleneck"]},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Long-term time series forecasting (LTSF) aims to predict future trends based on historical data. While longer lookback windows theoretically provide more comprehensive insights, current Transformer-based models face the Lookback Window Limitation (LWL). On one hand, longer windows introduce redundant information, which can hinder model learning. On the other hand, Transformers tend to overfit temporal noise rather than extract meaningful temporal information when dealing with longer sequences, compounded by their quadratic complexity. In this paper, we aim to overcome LWL, enabling models to leverage more historical information for improved performance. Specifically, to mitigate information redundancy, we introduce the Information Bottleneck Filter (IBF), which applies information bottleneck theory to extract essential subsequences from the input. Additionally, to address the limitations of the Transformer architecture in handling long sequences, we propose the Hybrid-Transformer-Mamba (HTM), which combines the linear complexity and long-range modeling capabilities of Mamba with the Transformer's strength in modeling short sequences. We integrate these two model-agnostic modules into various existing methods and conduct experiments on seven datasets. The results demonstrate that incorporating these modules effectively overcomes the lookback window limitations. Notably, by combining them with the Patch strategy, we design the PIH (\\textbf{P}atch-\\textbf{I}BF-\\textbf{H}TM), successfully extending the window length to 1024—a significantly larger window than previously achieved—and achieving state-of-the-art results, highlighting the potential of exploring even longer windows."},"_bibtex":{"value":"@misc{\nzhang2024overcoming,\ntitle={Overcoming Lookback Window Limitations: Exploring Longer Windows in Long-Term Time Series Forecasting},\nauthor={Jiahui Zhang and Wenjie Du and Zhengyang Zhou and Qihe Huang and Xuqiang Li and Yang Wang},\nyear={2024},\nurl={https://openreview.net/forum?id=hVpAjJPfgZ}\n}"},"title":{"value":"Overcoming Lookback Window Limitations: Exploring Longer Windows in Long-Term Time Series Forecasting"},"pdf":{"value":"/pdf/d81ea206a0e4871a9de5d22719240efde5117469.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhang|overcoming_lookback_window_limitations_exploring_longer_windows_in_longterm_time_series_forecasting"},"authorids":{"value":["~Jiahui_Zhang7","~Wenjie_Du2","~Zhengyang_Zhou1","~Qihe_Huang2","~Xuqiang_Li1","~Yang_Wang32"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jiahui Zhang","Wenjie Du","Zhengyang Zhou","Qihe Huang","Xuqiang Li","Yang Wang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.13229v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"chiang|quamba_a_posttraining_quantization_recipe_for_selective_state_space_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Hung-Yueh_Chiang:","https://dblp.org/search/pid/api?q=author:Chi-Chih_Chang:","https://dblp.org/search/pid/api?q=author:Natalia_Frumkin:","~Kai-Chiang_Wu1","~Diana_Marculescu2"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.13229"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-13229,\n  publtype={informal},\n  author={Hung-Yueh Chiang and Chi-Chih Chang and Natalia Frumkin and Kai-Chiang Wu and Diana Marculescu},\n  title={Quamba: A Post-Training Quantization Recipe for Selective State Space Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.13229},\n  url={https://doi.org/10.48550/arXiv.2410.13229}\n}\n"},"abstract":{"value":"State Space Models (SSMs) have emerged as an appealing alternative to Transformers for large language models, achieving state-of-the-art accuracy with constant memory complexity which allows for holding longer context lengths than attention-based networks. The superior computational efficiency of SSMs in long sequence modeling positions them favorably over Transformers in many scenarios. However, improving the efficiency of SSMs on request-intensive cloud-serving and resource-limited edge applications is still a formidable task. SSM quantization is a possible solution to this problem, making SSMs more suitable for wide deployment, while still maintaining their accuracy. Quantization is a common technique to reduce the model size and to utilize the low bit-width acceleration features on modern computing units, yet existing quantization techniques are poorly suited for SSMs. Most notably, SSMs have highly sensitive feature maps within the selective scan mechanism (i.e., linear recurrence) and massive outliers in the output activations which are not present in the output of token-mixing in the self-attention modules. To address this issue, we propose a static 8-bit per-tensor SSM quantization method which suppresses the maximum values of the input activations to the selective SSM for finer quantization precision and quantizes the output activations in an outlier-free space with Hadamard transform. Our 8-bit weight-activation quantized Mamba 2.8B SSM benefits from hardware acceleration and achieves a 1.72x lower generation latency on an Nvidia Orin Nano 8G, with only a 0.9% drop in average accuracy on zero-shot tasks. The experiments demonstrate the effectiveness and practical applicability of our approach for deploying SSM-based models of all sizes on both cloud and edge platforms."},"title":{"value":"Quamba: A Post-Training Quantization Recipe for Selective State Space Models"},"authors":{"value":["Hung-Yueh Chiang","Chi-Chih Chang","Natalia Frumkin","Kai-Chiang Wu","Diana Marculescu"]}},"tmdate":1747437919245,"pdate":1704067200000,"tcdate":1738007962698,"writers":["~"],"signatures":["~Kai-Chiang_Wu1"],"forum":"5paxY2DvHg","license":"CC BY-SA 4.0","number":284619,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747437919245,"domain":"DBLP.org","id":"5paxY2DvHg","version":2},{"content":{"venue":{"value":"IEEE Trans. Geosci. Remote. Sens. 2025"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/36/10807682/11150494.pdf"},"venueid":{"value":"dblp.org/journals/TGRS/2025"},"paperhash":{"value":"liang|dbmlla_doublebranch_mambalike_linear_attention_network_for_hyperspectral_image_classification"},"authorids":{"value":["","","","~Jiaxin_Li17","","",""]},"html":{"value":"https://doi.org/10.1109/TGRS.2025.3605798"},"_bibtex":{"value":"@article{DBLP:journals/tgrs/LiangXZLZLP25,\n  author={Lianhui Liang and Peiyi Xie and Ying Zhang and Jiaxin Li and Zhe Zhang and Jun Li and Antonio Plaza},\n  title={DBMLLA: Double-Branch Mamba-Like Linear Attention Network for Hyperspectral Image Classification},\n  year={2025},\n  cdate={1735689600000},\n  journal={IEEE Trans. Geosci. Remote. Sens.},\n  volume={63},\n  pages={1-15},\n  url={https://doi.org/10.1109/TGRS.2025.3605798}\n}\n"},"abstract":{"value":"Convolutional neural networks (CNNs) and transformers have made remarkable achievements in hyperspectral image classification (HSIC). Unfortunately, CNN-based methods struggle to capture the contextual dependencies between pixels in HSIs, while transformer-based methods suffer from quadratic computational complexity. Recently, the Mamba model has shown great potential as it can describe long-range dependencies between HSI pixels with linear computational complexity. Yet, Mamba still faces significant challenges in terms of global modeling. Inspired by the Mamba model framework and transformers, a novel double-branch Mamba-like linear attention (DBMLLA) network is proposed for HSIC, achieving efficient global dependency modeling. Specifically, the proposed DBMLLA combines an embedding module, a spatial–spectral Mamba-like linear attention (SS-MLLA) module, and a fusion module. In the embedding module, an absolute position embedding module is introduced for better extraction of global features. In the SS-MLLA module, we design the spatial MLLA (SpaMLLA) block and the spectral MLLA (SpeMLLA) block to exploit the spatial and spectral information of the HSI. In addition, SS-MLLA is improved using depthwise separable convolution (DSC) to enhance the model’s ability to extract deeper local feature information. Through experiments conducted on four public hyperspectral datasets, it is demonstrated that the proposed model consistently outperforms state-of-the-art approaches."},"title":{"value":"DBMLLA: Double-Branch Mamba-Like Linear Attention Network for Hyperspectral Image Classification"},"authors":{"value":["Lianhui Liang","Peiyi Xie","Ying Zhang","Jiaxin Li","Zhe Zhang","Jun Li","Antonio Plaza"]}},"tmdate":1775050747560,"pdate":1767139200000,"externalIds":["dblp:journals/tgrs/LiangXZLZLP25"],"tcdate":1775050744753,"writers":["~"],"signatures":["~Jiaxin_Li17"],"forum":"vXVP8qRflz","license":"CC BY-SA 4.0","number":862315,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1775050747560,"domain":"DBLP.org","id":"vXVP8qRflz","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2508.09179v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"chen|hifimamba_dualstream_wlaplacian_enhanced_mamba_for_highfidelity_mri_reconstruction"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Hongli_Chen:","~Pengcheng_Fang1","https://dblp.org/search/pid/api?q=author:Yuxia_Chen:","https://dblp.org/search/pid/api?q=author:Yingxuan_Ren:","https://dblp.org/search/pid/api?q=author:Jing_Hao:","https://dblp.org/search/pid/api?q=author:Fangfang_Tang:","~Xiaohao_Cai2","https://dblp.org/search/pid/api?q=author:Shanshan_Shan:","https://dblp.org/search/pid/api?q=author:Feng_Liu_0005:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2508.09179"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2508-09179,\n  publtype={informal},\n  author={Hongli Chen and Pengcheng Fang and Yuxia Chen and Yingxuan Ren and Jing Hao and Fangfang Tang and Xiaohao Cai and Shanshan Shan and Feng Liu},\n  title={HiFi-Mamba: Dual-Stream W-Laplacian Enhanced Mamba for High-Fidelity MRI Reconstruction},\n  year={2025},\n  month={August},\n  cdate={1754006400000},\n  journal={CoRR},\n  volume={abs/2508.09179},\n  url={https://doi.org/10.48550/arXiv.2508.09179}\n}\n"},"abstract":{"value":"Reconstructing high-fidelity MR images from undersampled k-space data remains a challenging problem in MRI. While Mamba variants for vision tasks offer promising long-range modeling capabilities with linear-time complexity, their direct application to MRI reconstruction inherits two key limitations: (1) insensitivity to high-frequency anatomical details; and (2) reliance on redundant multi-directional scanning. To address these limitations, we introduce High-Fidelity Mamba (HiFi-Mamba), a novel dual-stream Mamba-based architecture comprising stacked W-Laplacian (WL) and HiFi-Mamba blocks. Specifically, the WL block performs fidelity-preserving spectral decoupling, producing complementary low- and high-frequency streams. This separation enables the HiFi-Mamba block to focus on low-frequency structures, enhancing global feature modeling. Concurrently, the HiFi-Mamba block selectively integrates high-frequency features through adaptive state-space modulation, preserving comprehensive spectral details. To eliminate the scanning redundancy, the HiFi-Mamba block adopts a streamlined unidirectional traversal strategy that preserves long-range modeling capability with improved computational efficiency. Extensive experiments on standard MRI reconstruction benchmarks demonstrate that HiFi-Mamba consistently outperforms state-of-the-art CNN-based, Transformer-based, and other Mamba-based models in reconstruction accuracy while maintaining a compact and efficient model design."},"title":{"value":"HiFi-Mamba: Dual-Stream W-Laplacian Enhanced Mamba for High-Fidelity MRI Reconstruction"},"authors":{"value":["Hongli Chen","Pengcheng Fang","Yuxia Chen","Yingxuan Ren","Jing Hao","Fangfang Tang","Xiaohao Cai","Shanshan Shan","Feng Liu"]}},"tmdate":1783893355492,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2508-09179"],"tcdate":1768487602831,"writers":["~"],"signatures":["~Xiaohao_Cai2"],"forum":"63yINHD1hh","license":"CC BY-SA 4.0","number":765039,"cdate":1754006400000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1783893355492,"domain":"DBLP.org","id":"63yINHD1hh","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.03900v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"liu|mamba4rec_towards_efficient_sequential_recommendation_with_selective_state_space_models"},"authorids":{"value":["~Chengkai_Liu1","~Jianghao_Lin1","~Jianling_Wang1","~Hanzhou_Liu1","~James_Caverlee1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.03900"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-03900,\n  publtype={informal},\n  author={Chengkai Liu and Jianghao Lin and Jianling Wang and Hanzhou Liu and James Caverlee},\n  title={Mamba4Rec: Towards Efficient Sequential Recommendation with Selective State Space Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.03900},\n  url={https://doi.org/10.48550/arXiv.2403.03900}\n}\n"},"abstract":{"value":"Sequential recommendation aims to estimate the dynamic user preferences and sequential dependencies among historical user behaviors. Although Transformer-based models have proven to be effective for sequential recommendation, they suffer from the inference inefficiency problem stemming from the quadratic computational complexity of attention operators, especially for long behavior sequences. Inspired by the recent success of state space models (SSMs), we propose Mamba4Rec, which is the first work to explore the potential of selective SSMs for efficient sequential recommendation. Built upon the basic Mamba block which is a selective SSM with an efficient hardware-aware parallel algorithm, we design a series of sequential modeling techniques to further promote model performance while maintaining inference efficiency. Through experiments on public datasets, we demonstrate how Mamba4Rec effectively tackles the effectiveness-efficiency dilemma, outperforming both RNN- and attention-based baselines in terms of both effectiveness and efficiency. The code is available at https://github.com/chengkai-liu/Mamba4Rec."},"title":{"value":"Mamba4Rec: Towards Efficient Sequential Recommendation with Selective State Space Models"},"authors":{"value":["Chengkai Liu","Jianghao Lin","Jianling Wang","Hanzhou Liu","James Caverlee"]}},"tmdate":1769044844702,"pdate":1704067200000,"tcdate":1727454285725,"writers":["~"],"signatures":["~Chengkai_Liu1"],"forum":"5kPndaAnoY","license":"CC BY-SA 4.0","number":88341,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769044844702,"domain":"DBLP.org","id":"5kPndaAnoY","version":2},{"content":{"TLDR":{"value":"we present MindViT, a deployment-friendly hybrid vision backbone that synergizes the strengths of CNNs, Mamba, and Transformers and achieves strong performance."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Hybrid Architecture","Vision Backbone","Representation Learning","State Space Models"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"While Mamba-based models have shown strong potential for long sequence modeling, adapting them to vision is challenging due to the requirement of local neighborhood correlations and multi-directional spatial contexts for visual understanding. In this paper, we present MindViT, a novel hybrid vision backbone that synergizes the strengths of CNNs, Mamba, and Transformers. The core of our design is the Adaptive Bidirectional Mamba (ABM), which mitigates the directional bias of unidirectional SSMs through bidirectional selective scanning with learnable alpha blending, enabling content-adaptive directional fusion without the overhead of exhaustive multi-path routing. To further accelerate inference, we propose a deployment-friendly Reparameterized ConvSE (RepConvSE) module that leverages structural reparameterization to reduce latency and memory access overhead. \nExtensive experiments demonstrate that MindViT achieves strong performance on image classification, object detection, and semantic segmentation while enabling efficient inference through reparameterization."},"_bibtex":{"value":"@inproceedings{\nanonymous2026mindvit,\ntitle={MindViT: A Hybrid Vision Backbone with Adaptive Bidirectional Mamba},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=CfTqPziVKb},\nnote={under review}\n}"},"title":{"value":"MindViT: A Hybrid Vision Backbone with Adaptive Bidirectional Mamba"},"pdf":{"value":"/pdf/86ed2d17fdcc4fca8f1c5aa53ffd251258b0ada8.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230070525,"tcdate":1789529450706,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission23075/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission23075/Authors"],"forum":"CfTqPziVKb","license":"CC BY 4.0","number":23075,"cdate":1789529450706,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791230070525,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"CfTqPziVKb","version":2},{"content":{"venue":{"value":"CVPR 2025"},"pdf":{"value":"https://openaccess.thecvf.com/content/CVPR2025/papers/Zhang_2DMamba_Efficient_State_Space_Model_for_Image_Representation_with_Applications_CVPR_2025_paper.pdf"},"venueid":{"value":"dblp.org/conf/CVPR/2025"},"paperhash":{"value":"zhang|2dmamba_efficient_state_space_model_for_image_representation_with_applications_on_gigapixel_whole_slide_image_classification"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jingwei_Zhang:","https://dblp.org/search/pid/api?q=author:Anh_Tien_Nguyen:","https://dblp.org/search/pid/api?q=author:Xi_Han:","https://dblp.org/search/pid/api?q=author:Vincent_Quoc-Huy_Trinh:","~Hong_Qin1","https://dblp.org/search/pid/api?q=author:Dimitris_Samaras:","https://dblp.org/search/pid/api?q=author:Mahdi_S._Hosseini:"]},"html":{"value":"https://openaccess.thecvf.com/content/CVPR2025/html/Zhang_2DMamba_Efficient_State_Space_Model_for_Image_Representation_with_Applications_CVPR_2025_paper.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/cvpr/ZhangNHT0SH25,\n  author={Jingwei Zhang and Anh Tien Nguyen and Xi Han and Vincent Quoc-Huy Trinh and Hong Qin and Dimitris Samaras and Mahdi S. Hosseini},\n  title={2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification},\n  year={2025},\n  cdate={1735689600000},\n  pages={3583-3592},\n  url={https://openaccess.thecvf.com/content/CVPR2025/html/Zhang_2DMamba_Efficient_State_Space_Model_for_Image_Representation_with_Applications_CVPR_2025_paper.html},\n  booktitle={CVPR},\n  crossref={conf/cvpr/2025}\n}\n"},"abstract":{"value":"Efficiently modeling large 2D contexts is essential for various fields including Giga-Pixel Whole Slide Imaging (WSI) and remote sensing. Transformer-based models offer high parallelism but face challenges due to their quadratic complexity for handling long sequences. Recently, Mamba introduced a selective State Space Model (SSM) with linear complexity and high parallelism, enabling effective and efficient modeling of wide context in 1D sequences. However, extending Mamba to vision tasks, which inherently involve 2D structures, results in spatial discrepancies due to the limitations of 1D sequence processing. On the other hand, current 2D SSMs inherently model 2D structures but they suffer from prohibitively slow computation due to the lack of efficient parallel algorithms. In this work, we propose 2DMamba, a novel 2D selective SSM framework that incorporates the 2D spatial structure of images into Mamba, with a highly optimized hardware-aware operator, adopting both spatial continuity and computational efficiency. We validate the versatility of our approach on both WSIs and natural images. Extensive experiments on 10 public datasets for WSI classification and survival analysis show that 2DMamba improves up to 2.48% in AUC, 3.11% in F1 score, 2.47% in accuracy and 5.52% in C-index. Additionally, integrating our method with VMamba for natural imaging yields 0.5 to 0.7 improvements in mIoU on the ADE20k semantic segmentation dataset, and 0.2% accuracy improvement on ImageNet-1K classification dataset. Our code is available at https://github.com/AtlasAnalyticsLab/2DMamba."},"title":{"value":"2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification"},"authors":{"value":["Jingwei Zhang","Anh Tien Nguyen","Xi Han","Vincent Quoc-Huy Trinh","Hong Qin","Dimitris Samaras","Mahdi S. Hosseini"]}},"tmdate":1763038228244,"pdate":1735689600000,"externalIds":["dblp:conf/cvpr/ZhangNHT0SH25"],"tcdate":1763038225340,"writers":["~"],"signatures":["~Hong_Qin1"],"forum":"ZtRznjQp6i","license":"CC BY-SA 4.0","number":701287,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1763038228244,"domain":"DBLP.org","id":"ZtRznjQp6i","version":2},{"content":{"venue":{"value":"CVPR 2025"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"zhang|2dmamba_efficient_state_space_model_for_image_representation_with_applications_on_gigapixel_whole_slide_image_classification"},"authorids":{"value":["~Jingwei_Zhang7","~Anh_Tien_Nguyen1","~Xi_Han1","~Vincent_Quoc-Huy_Trinh1","~Hong_Qin1","~Dimitris_Samaras3","~Mahdi_S._Hosseini1"]},"html":{"value":"https://openaccess.thecvf.com/content/CVPR2025/papers/Zhang_2DMamba_Efficient_State_Space_Model_for_Image_Representation_with_Applications_CVPR_2025_paper.pdf"},"abstract":{"value":"Efficiently modeling large 2D contexts is essential for various fields including Giga-Pixel Whole Slide Imaging (WSI) and remote sensing. Transformer-based models offer high parallelism but face challenges due to their quadratic complexity for handling long sequences. Recently, Mamba introduced a selective State Space Model (SSM) with linear complexity and high parallelism, enabling effective and efficient modeling of wide context in 1D sequences. However, extending Mamba to vision tasks, which inherently involve 2D structures, results in spatial discrepancies due to the limitations of 1D sequence processing. On the other hand, current 2D SSMs inherently model 2D structures but they suffer from prohibitively slow computation due to the lack of efficient parallel algorithms. In this work, we propose 2DMamba, a novel 2D selective SSM framework that incorporates the 2D spatial structure of images into Mamba, with a highly optimized hardware-aware operator, adopting both spatial continuity and computational efficiency. We validate the versatility of our approach on both WSIs and natural images. Extensive experiments on 10 public datasets for WSI classification and survival analysis show that 2DMamba improves up to 2.48% in AUC, 3.11% in F1 score, 2.47% in accuracy and 5.52% in C-index. Additionally, integrating our method with VMamba for natural imaging yields 0.5 to 0.7 improvements in mIoU on the ADE20k semantic segmentation dataset, and 0.2% accuracy improvement on ImageNet-1K classification dataset. Our code is available at https://github. com/AtlasAnalyticsLab/2DMamba."},"title":{"value":"2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification"},"authors":{"value":["Jingwei Zhang","Anh Tien Nguyen","Xi Han","Vincent Quoc-Huy Trinh","Hong Qin","Dimitris Samaras","Mahdi S. Hosseini"]}},"tmdate":1762286670867,"pdate":1742014800000,"tcdate":1762285638548,"writers":["~Jingwei_Zhang7","~Anh_Tien_Nguyen1","~Xi_Han1","~Vincent_Quoc-Huy_Trinh1","~Hong_Qin1","~Dimitris_Samaras3","~Mahdi_S._Hosseini1"],"signatures":["~Anh_Tien_Nguyen1"],"forum":"OI9ciuKMN8","license":"CC BY 4.0","number":40052,"cdate":1762285638548,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1762286670867,"domain":"OpenReview.net/Archive","id":"OI9ciuKMN8","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2412.00678v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"zhang|2dmamba_efficient_state_space_model_for_image_representation_with_applications_on_gigapixel_whole_slide_image_classification"},"authorids":{"value":["~Jingwei_Zhang7","https://dblp.org/search/pid/api?q=author:Anh_Tien_Nguyen:","https://dblp.org/search/pid/api?q=author:Xi_Han:","~Vincent_Quoc-Huy_Trinh1","https://dblp.org/search/pid/api?q=author:Hong_Qin_0001:","~Dimitris_Samaras1","https://dblp.org/search/pid/api?q=author:Mahdi_S._Hosseini:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2412.00678"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-00678,\n  publtype={informal},\n  author={Jingwei Zhang and Anh Tien Nguyen and Xi Han and Vincent Quoc-Huy Trinh and Hong Qin and Dimitris Samaras and Mahdi S. Hosseini},\n  title={2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2412.00678},\n  url={https://doi.org/10.48550/arXiv.2412.00678}\n}\n"},"abstract":{"value":"Efficiently modeling large 2D contexts is essential for various fields including Giga-Pixel Whole Slide Imaging (WSI) and remote sensing. Transformer-based models offer high parallelism but face challenges due to their quadratic complexity for handling long sequences. Recently, Mamba introduced a selective State Space Model (SSM) with linear complexity and high parallelism, enabling effective and efficient modeling of wide context in 1D sequences. However, extending Mamba to vision tasks, which inherently involve 2D structures, results in spatial discrepancies due to the limitations of 1D sequence processing. On the other hand, current 2D SSMs inherently model 2D structures but they suffer from prohibitively slow computation due to the lack of efficient parallel algorithms. In this work, we propose 2DMamba, a novel 2D selective SSM framework that incorporates the 2D spatial structure of images into Mamba, with a highly optimized hardware-aware operator, adopting both spatial continuity and computational efficiency. We validate the versatility of our approach on both WSIs and natural images. Extensive experiments on 10 public datasets for WSI classification and survival analysis show that 2DMamba~improves up to $2.48\\%$ in AUC, $3.11\\%$ in F1 score, $2.47\\%$ in accuracy and $5.52\\%$ in C-index. Additionally, integrating our method with VMamba for natural imaging yields $0.5$ to $0.7$ improvements in mIoU on the ADE20k semantic segmentation dataset, and $0.2\\%$ accuracy improvement on ImageNet-1K classification dataset. Our code is available at https://github.com/AtlasAnalyticsLab/2DMamba."},"title":{"value":"2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification"},"authors":{"value":["Jingwei Zhang","Anh Tien Nguyen","Xi Han","Vincent Quoc-Huy Trinh","Hong Qin","Dimitris Samaras","Mahdi S. Hosseini"]}},"tmdate":1747083770394,"pdate":1704067200000,"tcdate":1741182224334,"writers":["~"],"signatures":["~Dimitris_Samaras3"],"forum":"RvSaoY3UvS","license":"CC BY-SA 4.0","number":354468,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747083770394,"domain":"DBLP.org","id":"RvSaoY3UvS","version":2},{"content":{"summary":{"value":"The paper performs a theoretical and mechanistic study of how Mamba solves associative recall, in particular the MQAR task. Focused on a simplified model (one-layer linear Mamba), the authors identity a solution for solving MQAR and verify such solutions present in the learned model weights. The authors then use JL lemma to theoretically show the required model size for solving associative recall for given task parameters. Finally, the theoretical claims are supported via empirical validations on both the simplified model and the original model."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. In most of the theoretical constructions, the hidden state is treated as a matrix with size (N, d) (e.g., eqn 3). However, in equation 2, the hidden state $h_t$ is treated as a vector of size N. Can the authors clarify the purpose of eqn 2 and discuss its relations to the rest of the paper?\n\n2. Fig.3 provides a very interesting inspection of the hidden state. I want to better understand the computation of $H''$: in lines 286-296, the authors comment $H''$ in eqn.13 does not exist as-is, but has to use $H'' = \\Pi_{v, out} H_t' \\Pi_{q, in}$. But it seems possible to compute eqn.13 from the learned embedding. Can the authors clarify?\n\n3. How do the mechanistic and theoretical findings generalize to Mamba2?\n\n4. The proof of Lemma 1: while the proof steps look correct to me, some wording and analysis look confusing. In line 1028-1030, the non-matching wrong fact pairs and non-matching empty pairs look the same from their mathematical definitions ($x_{\\tau-1} \\neq k_m, x_{\\tau} \\neq v_i$). Do the authors mean non-matching wrong fact as $x_{\\tau-1} = k \\neq k_m$ being a non-matching key, and non-matching empty pair as $x_{\\tau-1} = v$ being an arbitrary value?\n\n5. The proof of Thm 3, eqn (98): shouldn't it be $p_{\\text{success}} = p_w^{N_f - 1} p_n^{V_v - N_f}$? can the authors double check the rest of the equations are correct?\n\nMinor:\n\na. The notation $N_f$ in proof sketch of Lemma 1 and Theorem 3 means the same as $N_{\\text{facts}}$?\n\nb. What does the pixel color in Fig.6 mean (accuracy)? Also, the color bar is missing."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The mechanistic interpretability analysis on validating the simplified Mamba recall circuit is interesting and nicely done (taking in account suitable invariances).\n\n2. The quantitative analysis of how Mamba solves associative recall is thorough and empirically validated."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The theoretical construction of simplified one-layer linear Mamba largely follows from Huang et al. (2025), without providing enough remarks or discussions. Specifically, Theorem 1 uses the construction of Lemma 3 in Huang et al., and Theorem 2 largely reuses the idea of Theorem 2 in Huang et al. (i.e. using JL lemma to reduce dimensionality). Moreover, Theorem 4 in Huang et al. shows that using convolution and gating (together with a S4D-SSM) can solve MQAR, which takes a step towards understanding the role of gating, a direction that the authors remark for future work. The authors only highlight the difference with the study of Huang et al. in Sec 2, without properly attributing the similarities.\n\n2. The mechanistic and theoretical study focus on a particular task (MQAR) to identify a recall circuit. However, it is unclear how generalizable this circuit is applicable to other related recall tasks (e.g., MQAR with repeating keys, induction head where keys and values are from the same vocabulary)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762920003289,"tcdate":1761932033337,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8000/Reviewer_1qe9"],"signatures":["ICLR.cc/2026/Conference/Submission8000/Reviewer_1qe9"],"forum":"8cDoHzqDXP","number":2,"license":"CC BY 4.0","cdate":1761932033337,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8000/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762920003289,"domain":"ICLR.cc/2026/Conference","replyto":"8cDoHzqDXP","id":"QOWq1Q6cNy","forumContent":{"TLDR":{"value":"Using mechanistic interpretability, we show Mamba recalls via hash-like mechanisms and derive scaling laws that predict its recall capacity."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mamba","Memory","Associative Recall","Scaling Laws"]},"supplementary_material":{"value":"/attachment/889fd173211134007994b02141c1466e69caa77b.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Associative Recall (AR) is the cognitive ability to learn and retrieve links between items in memory. In NLP, AR is used as a benchmark for evaluating the in-context memory capacity of architectures such as Mamba, and has been found to strongly correlate with language modeling performance. This paper explores AR from the perspective of mechanistic interpretability, aiming to reverse-engineer the exact internal algorithm used by Mamba to perform recall. Our key insight is that Mamba performs recall by implicitly learning linear hash functions, and we identify the low-level circuit that enables this behavior. Building on these findings and inspired by theoretical tools in similarity-preserving hashing, such as the Johnson–Lindenstrauss (JL) lemma, we develop a theoretical framework for analyzing AR, which we term recall scaling laws. For example, given a model’s state, context length, and embedding dimensions, our theory predicts a lower bound on the largest vocabulary size that can be perfectly recalled in the AR task. Empirical results show that this bound is tight and predictive, offering insights into how AR capacity scales with vocabulary size, state size, embedding size, and model architecture."},"_bibtex":{"value":"@misc{\nkoren2026on,\ntitle={On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing},\nauthor={Yuval Koren and Itamar Zimerman and Assaf Ben-Kish and Raja Giryes and Lior Wolf},\nyear={2026},\nurl={https://openreview.net/forum?id=8cDoHzqDXP}\n}"},"title":{"value":"On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing"},"pdf":{"value":"/pdf/ef768fe2351b73310b384851405050a888cb6835.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"koren|on_the_recall_scaling_laws_in_mamba_a_theoretical_and_mechanistic_study_via_hashing"},"authorids":{"value":["~Yuval_Koren1","~Itamar_Zimerman1","~Assaf_Ben-Kish1","~Raja_Giryes1","~Lior_Wolf1"]},"authors":{"value":["Yuval Koren","Itamar Zimerman","Assaf Ben-Kish","Raja Giryes","Lior Wolf"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for acknowledging this paper as being well-written and would like to address the reviewer’s concern here.\n\n**W1.1: The authors compare RNNs and Transformers under different assumptions regarding their state sizes.**\n\nThe reason that we compare RNNs and Transformers with different state sizes is that RNNs indeed have smaller state sizes than Transformers with the same number of parameters.\n\n1. Having a smaller state size is a key feature of RNN, and it has motivated the recent empirical trend of developing modern RNNs (such as Mamba) to avoid the high memory cost of Transformers on long context inputs.  \n   1. For example, in \\[1\\], it is argued that *“RNNs require less memory, particularly for handling long sequences.”*  \n   2. Our work reveals a fundamental limitation of this approach, showing that such models with high memory efficiency are incapable of doing in-context retrieval.  \n   3. To overcome this limitation, the state size of RNN needs to scale linearly with respect to the context length for in-context retrieval.  \n   4. Given Transformer’s state size scales linearly with the context length, it already reaches the optimal asymptotic rate when the context length grows so it is impossible to improve the memory efficiency beyond a constant factor.  \n2. We compare RNNs and Transformers with the same order of number of **parameters** because it is typical in current empirical studies to test modern RNN architectures on in-context retrieval tasks with Transformers with the same amount of parameters \\[1,2,3,4\\]**.** More specifically, we assume the parameter size of both architectures to be constant and the precision is of order $\\\\log n$ when the context length is $n$. This is a standard assumption in previous works considering the representation power of Transformers \\[5, 6\\].\n\n\\[1\\] RWKV: Reinventing RNNs for the Transformer Era  \n\\[2\\] Mamba: Linear-Time Sequence Modeling with Selective State Spaces  \n\\[3\\] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality  \n\\[4\\] Parallelizing Linear Transformers with the Delta Rule over Sequence Length  \n\\[5\\] Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective  \n\\[6\\] Self-Attention Networks Can Process Bounded Hierarchical Languages\n\n**W1.2: Transformers can be viewed as RNNs with linear state size, and hence showing Transformers outperforming RNNs with state size $O(\\\\log n)$ is not surprising.** \n\n1. Models with larger state sizes do not automatically imply a stronger representation power. To prove our results, some key details in the Transformer architecture are crucial. For example, if the softmax mechanism is removed, the linear attention mechanism will not be able to perform in-context retrieval with context length $n$. This is because such Transformers are not effectively using their large state sizes. In fact, one can rewrite them as RNNs with $O(\\log n)$ state sizes.  \n2. Can we rewrite Transformers with softmax attention as small RNNs? This is unclear a priori, but our work is able to show that Transformers with softmax attention are more expressive than RNNs, especially on in-context retrieval tasks. So there is something *inherently specific to Transformers (with softmax attention) in this comparison.*  \n3. We not only show that RNNs are worse than Transformers on the simple in-context retrieval tasks but also show that even if we use CoT, RNNs are still worse than Transformers (without CoT)."}},"tmdate":1732342231924,"tcdate":1732342231924,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11974/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission11974/Authors"],"forum":"h3wbI8Uk1Z","number":4,"license":"CC BY 4.0","cdate":1732342231924,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11974/-/Official_Comment"],"mdate":1732342231924,"domain":"ICLR.cc/2025/Conference","replyto":"RyHCoJjLH1","id":"TYsP5w9FSi","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"We identify a representation gap between Transformers and RNNs (w/o CoT) due to the incapability to retrieve in context and show by adopting techniques to enhance this capability can bridge the gap."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["rnn","cot","representation theory"]},"primary_area":{"value":"learning theory"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"This paper investigates the gap in representation powers of Transformers and Recurrent Neural Networks (RNNs), which are more memory efficient than Transformers. We aim to understand whether RNNs can match the performance of Transformers, particularly when enhanced with Chain-of-Thought (CoT) prompting. Our theoretical analysis reveals that CoT improves RNNs but is insufficient to close the gap with Transformers. A key bottleneck lies in the inability of RNNs to perfectly retrieve information from the context, even with CoT: \nfor several tasks that explicitly or implicitly require this capability, such as associative recall and determining if a graph is a tree, we prove that RNNs are not expressive enough to solve the tasks while Transformers can solve them with ease.\nConversely, we prove that adopting techniques to enhance the in-context retrieval capability of RNNs, including Retrieval-Augmented Generation (RAG) and adding a single Transformer layer, can elevate RNNs to be capable of solving all polynomial-time solvable problems with CoT, hence closing the representation gap with Transformers. We validate our theory on synthetic and natural language experiments."},"_bibtex":{"value":"@inproceedings{\nwen2025rnns,\ntitle={{RNN}s are not Transformers (Yet):  The Key Bottleneck on In-Context Retrieval},\nauthor={Kaiyue Wen and Xingyu Dang and Kaifeng Lyu},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=h3wbI8Uk1Z}\n}"},"title":{"value":"RNNs are not Transformers (Yet):  The Key Bottleneck on In-Context Retrieval"},"pdf":{"value":"/pdf/9cd87a7fcca4b8336c6449ef6317a3319d4ab40f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wen|rnns_are_not_transformers_yet_the_key_bottleneck_on_incontext_retrieval"},"authorids":{"value":["~Kaiyue_Wen1","~Xingyu_Dang2","~Kaifeng_Lyu2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Kaiyue Wen","Xingyu Dang","Kaifeng Lyu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes InceptionMamba, an efficient segmentation framework that combines multi-stage CNN features with a lightweight global modeling block based on Selective State Space Models (Mamba). The core design includes a Feature Calibration Module (FCM) that enhances boundary details by separating high- and low-frequency cues, and an Inception Mamba Module (IMM) that mixes depth-wise multi-kernel convolutions with a Mamba branch for long-range context.  The method is evaluated on SegPC21, GlaS, ISIC2017, and ISIC2018, reporting Dice/IoU (and GFLOPs on SegPC21), and achieves competitive or state-of-the-art performance while claiming substantially lower computation than strong hybrid baselines such as GA2-Net.  Overall, the work argues that careful multi-scale enhancement plus efficient global context modeling can improve microscopic and skin lesion segmentation with favorable accuracy–efficiency trade-offs."},"justification_of_the_preliminary_rating":{"value":"I vote weakly reject because, while the paper presents a lightweight and well-performing segmentation model with broad experiments, several central claims (e.g., dropping stage-4 and dense skips) are not directly supported with clean, controlled ablations, and some comparisons/protocol details (missing table entries, ensemble settings) reduce confidence in the fairness and interpretability of the reported gains. These issues look fixable, but in the current form they make it harder to assess the true contribution and the robustness of the accuracy–efficiency conclusions."},"confidentiality_llm_acknowledgment":{"value":"Yes"},"strengths":{"value":"1. **Strong accuracy–efficiency trade-off with a lightweight design.** The method is built around efficiency-first choices (depthwise multi-branch processing, a simplified decoder, and a reduced ResNet feature hierarchy), and reports competitive Dice/IoU while using substantially fewer GFLOPs than strong hybrid baselines on SegPC21.\n\n2. **Well-motivated modules for microscopy-style segmentation failures.** The Feature Calibration Module (FCM) explicitly targets boundary ambiguity by separating “detail” and “region” cues, while the Inception-Mamba Module (IMM) combines multi-scale local context with a global modeling branch in a single block.  The overall design aligns with the long-standing multi-scale principle behind Inception-style architectures.\n\n3. **Broad empirical validation on widely used medical benchmarks.** The experiments span multiple problem types, cell/plasma segmentation (SegPC-2021), gland segmentation (GlaS), and skin lesion segmentation (ISIC), which increases confidence of cross dataset performance.\n\n4. **Extensive ablations that show the components matter.** The paper evaluates many variants (FCM vs none, different IMM configurations, replacing the global branch, kernel choices, and module placement) and generally demonstrates incremental gains when key components are added.  Even though the presentation could be more structured, the amount of evidence is valuable for readers who want to adopt or extend the model."},"weaknesses":{"value":"1. **Some reported comparisons are incomplete or unclear.** Tables 1 and 2 contain “–” entries (e.g., missing IoU for UDTransNet) without any explanation in the caption or main text.  This makes it harder to judge fairness and interpret the quantitative ranking.\n\n2. **Several key design claims are not backed by direct ablation evidence.** The paper states that (i) dense skip connections increase computation with negligible improvement, and (ii) ResNet stage-4 “high-level semantics” are not profitable for microscopic images, but does not provide explicit with/without experiments for these claims.  Since these choices affect both accuracy and compute, they should be supported with concrete numbers.\n\n3. **Ablation studies are extensive but not very systematic, which makes them hard to follow.** Table 4 mixes component ablations (FCM, IMM) with multiple design alternatives (Inception variants, self-attention replacement, decoder placement) in one table, and the baseline is a modified U-Net *without skip connections*, which can make “skip connections are unnecessary” harder to interpret.  A clearer step-by-step progression (baseline → +FCM → +IMM → decoder IMM → stem skip) would improve clarity.\n\n4. **The efficiency story is not consistently supported across datasets and settings.** The paper highlights GFLOPs/Params strongly on SegPC21, but compute numbers are not reported for the other main benchmark tables (GlaS/ISIC), even though efficiency is claimed as a core contribution.  In addition, all experiments are run at 224×224 resolution, which may not reflect practical microscopy/histology resolutions and can affect both accuracy and compute conclusions.\n\n5. **Protocol differences may affect fairness of the “state-of-the-art” claim.** For GlaS, the paper uses 3×5-fold cross-validation and a 5-model ensemble at inference.  It is not clear whether the compared methods in Table 2 use the same protocol, which can inflate performance relative to single-model baselines."},"confidence":{"value":5},"detailed_comments":{"value":"1. **Define what “–” means in the tables.**\n   Tables 1 and 2 contain “–” entries (e.g., missing IoU for UDTransNet), but the caption does not explain whether this means “not reported,” “not reproduced,” or “not applicable.” A one-line note would remove ambiguity.\n\n2. **Make the table captions more self-contained.**\n   Since compute is reported (Params / GFLOPs), it would help if the relevant captions explicitly stated that GFLOPs are computed at 224×224 input resolution (even though this is mentioned in the experimental details). \n\n3. **Clarify the evaluation protocol directly where results are shown.**\n   For GlaS, the paper uses 3×5-fold cross-validation and a 5-model ensemble at inference. This is important context for interpreting Table 2 and could be stated in the Table 2 caption (or immediately above the table). \n\n4. **Minor writing/typo cleanup.**\n   There are a few small typos (e.g., “kernal” → “kernel”) and repeated phrasing in the ablation discussion; quick cleanup would improve polish without changing content.\n\n5. **Improve readability/accessibility of best-result highlighting.**\nFigure 4. is hard to compare the prediction vs. GT. Consider add the false positive and false negative to better compare among different methods. Table 3. is color-coded in red/blue which is not friendly for color-sensitive readers."},"questions_to_address_in_the_rebuttal":{"value":"1. **What exactly does “–” mean in Tables 1 and 2, and can you complete those missing metrics?**\n   If the IoU for UDTransNet is missing because it was not reported in the source paper or not reproduced, please state that explicitly. Ideally, provide IoU under your evaluation protocol (or explain why it cannot be provided).\n\n2. **Can you provide direct ablation evidence for the two key claims: “no stage4” and “no dense skip”?**\n   A simple table showing (a) baseline+stage4 vs baseline (or your final model with/without stage4), and (b) decoder with dense skips vs your simplified decoder—reporting both accuracy and GFLOPs—would strongly strengthen the method justification.\n\n3. **Can you report efficiency (Params/GFLOPs) for GlaS and ISIC results, not only SegPC21?**\n   Since the paper highlights efficiency, adding compute numbers (at the same input size you use, 224×224) for Tables 2–3 would make the “efficient across tasks” claim much more solid.\n\n4. **Can you simplify the ablation narrative into a clear step-by-step progression?**\n   In the rebuttal, it would help to restate the ablation as one clean pipeline progression (baseline → +FCM → +IMM → +decoder IMM → +stem skip) and isolate “design choice” experiments (kernel size, IMM position) separately, so readers do not lose track."},"preliminary_rating":{"value":2}},"parentInvitations":"MIDL.io/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1771530040207,"tcdate":1768632549389,"writers":["MIDL.io/2026/Conference","MIDL.io/2026/Conference/Submission199/Reviewer_FWNc"],"signatures":["MIDL.io/2026/Conference/Submission199/Reviewer_FWNc"],"forum":"lJyX5YEdbu","number":3,"license":"CC BY 4.0","cdate":1768632549389,"readers":["everyone"],"invitations":["MIDL.io/2026/Conference/Submission199/-/Official_Review","MIDL.io/2026/Conference/-/Edit"],"mdate":1771530040207,"domain":"MIDL.io/2026/Conference","replyto":"lJyX5YEdbu","id":"D0Wh2O7Soa","forumContent":{},"version":2},{"content":{"summary":{"value":"To address the limitation that existing time series foundation models often overlook inter-channel modeling, this paper proposes a lightweight adapter, termed CoRA. CoRA innovatively decomposes the correlation matrix into time-varying and time-invariant components. It leverages learnable polynomials to capture dynamic patterns and introduces a novel dual contrastive learning mechanism to distinguish between positive and negative correlations. Optimized via a heterogeneous-local contrastive loss, CoRA incurs minimal computational overhead during the inference stage. Empirical results demonstrate that CoRA provides an effective solution for correlation modeling in multivariate time series forecasting."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"1. How does CoRA, as a plug-and-play module, accommodate the fundamental architectural differences between encoder-only and encoder-decoder models, particularly with respect to their distinct training objectives and inference processes?\n2. What is the underlying rationale for incorporating rule-based correlation relationships when the model is already designed to capture these dependencies through a learnable mechanism?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper is well-motivated, addressing the critical challenge of modeling complex inter-channel dependencies in multivariate time series.\n2. The manuscript is well-written, featuring clear and consistent notation throughout.\n3. The empirical evaluation demonstrates that CoRA consistently surpasses state-of-the-art baselines across a diverse set of real-world datasets."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The projection layers are not much different from the related modeling methods in existing methods (such as TSMixer [1]), a more detailed clarification is needed.\n2. The set of baselines for comparison is not comprehensive, as it omits recent state-of-the-art foundation models like TimerXL [2].\t\n3. Discuss and comparison with classical baselines (e.g., PatchTST  [3], Leddam [4], iTransformer [5], Autoformer [6], DLinear [7]) is suggested.\n\n*[1] TSMixer: An All-MLP Architecture for Time Series Forecasting.*\n\n*[2] Timer-XL: Long-Context Transformers for Unified Time Series Forecasting.*\n\n*[3] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers.*\n\n*[4] Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling.*\n\n*[5] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting.*\n\n*[6] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting.*\n\n*[7] Are Transformers Effective for Time Series Forecasting?.*"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762921841208,"tcdate":1761632592847,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10569/Reviewer_bdjv"],"signatures":["ICLR.cc/2026/Conference/Submission10569/Reviewer_bdjv"],"forum":"JRlNrcTllN","number":2,"license":"CC BY 4.0","cdate":1761632592847,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10569/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762921841208,"domain":"ICLR.cc/2026/Conference","replyto":"JRlNrcTllN","id":"pQmtvufkH0","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Multivariate Time Series Forecasting","Time Series Foundation Models","Correlation"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Most existing Time Series Foundation Models (TSFMs) use channel independent modeling and focus on capturing and generalizing temporal dependencies, while neglecting the correlations among channels or overlook the different aspects of correlations. However, these correlations play a vital role in Multivariate time series forecasting. To address this, we propose a Correlation-aware Adapter (**CoRA**), a lightweight plug-and-play method that requires only fine-tuning with TSFMs and is able to capture different types of correlations, so as to improve forecast performance. Specifically, to reduce complexity, we innovatively decompose the correlation matrix into low-rank Time-Varying and Time-Invariant components. For the Time-Varying component, we further design learnable polynomials to learn dynamic correlations by capturing trends or periodic patterns. To learn positive and negative correlations that appear only among some variables, we introduce a novel dual contrastive learning method that identifies correlations through projection layers, regulated by a Heterogeneous-Partial contrastive loss during training, without introducing additional complexity in the inference stage. Extensive experiments on 10 real-world datasets demonstrate that CoRA improves the state-of-the-art TSFMs in average forecast performance."},"_bibtex":{"value":"@inproceedings{\ncheng2026cora,\ntitle={Co{RA}: Boosting Time Series Foundation Models for Multivariate Forecasting through Correlation-aware Adapter},\nauthor={Hanyin Cheng and Xingjian Wu and Yang Shu and Zhongwen Rao and Lujia Pan and Bin Yang and Chenjuan Guo},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=JRlNrcTllN}\n}"},"title":{"value":"CoRA: Boosting Time Series Foundation Models for Multivariate Forecasting through Correlation-aware Adapter"},"pdf":{"value":"/pdf/fbe47a96cdc42d6b1c80e679c84d6fe196528601.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"cheng|cora_boosting_time_series_foundation_models_for_multivariate_forecasting_through_correlationaware_adapter"},"authorids":{"value":["~Hanyin_Cheng1","~Xingjian_Wu1","~Yang_Shu1","~Zhongwen_Rao1","~Lujia_Pan2","~Bin_Yang4","~Chenjuan_Guo1"]},"authors":{"value":["Hanyin Cheng","Xingjian Wu","Yang Shu","Zhongwen Rao","Lujia Pan","Bin Yang","Chenjuan Guo"]}},"version":2},{"content":{"summary":{"value":"SimTrack3D is a framework for 3D single object tracking in LiDAR point clouds. It treats point cloud or voxel features as dynamic sequences and integrates them with structured state space models (SSMs), such as Mamba, to achieve efficient and accurate motion modeling. The approach supports both point-based and voxel-based representations, and employs a spatiotemporal scanning strategy to encode spatial structural information into sequential representations. These are then processed by a Dynamic Sequential Feature Aggregation (DSFA-SSM) module for efficient inference."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"It needs to address the above comments in weakness."},"rating":{"value":2},"details_of_ethics_concerns":{"value":"None"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. High Efficiency and Real-Time Capability: By leveraging sequential modeling and the linear complexity of SSMs, the method achieves inference speeds significantly surpassing existing approaches while maintaining high accuracy, making it suitable for real-time applications such as autonomous driving.\n\n2. Strong Generality with Dual Representation Support: The framework is not constrained to a single point cloud representation. It is applicable to both point-based and voxel-based paradigms and demonstrates performance improvements across both, highlighting the versatility of its design."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Manual Design Dependency in Scanning Strategies: Although four scanning patterns are proposed, these strategies remain manually designed and may lack adaptability to varying scenes or target motion patterns, thereby limiting generalization capability.\n\n2. Sensitivity to Data Distribution: The method does not demonstrate strong performance on the NuScenes dataset (especially on scene bus and trailer), indicating sensitivity to data distribution and scene variations, which reveals certain limitations.\n\n3. Underutilization of Historical Information: The study acknowledges that \" However, more effective spatial scanning methods and modeling approaches that incorporate more historical motion data remain underexplored, which will be the focus of our future work \" suggesting limited exploitation of long-term temporal modeling potential.\n\n4. Lack of Extensibility to Multi-Object Tracking: The framework focuses solely on single-object tracking, with no discussion of extension to multi-object scenarios. The reviewers believe it restricts its applicability in complex interactive environments."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922717432,"tcdate":1761745988824,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11657/Reviewer_tWWS"],"signatures":["ICLR.cc/2026/Conference/Submission11657/Reviewer_tWWS"],"forum":"NOAuwgqr1J","number":3,"license":"CC BY 4.0","cdate":1761745988824,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11657/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922717432,"domain":"ICLR.cc/2026/Conference","replyto":"NOAuwgqr1J","id":"xjf2jceyFH","forumContent":{"TLDR":{"value":"Employing serialization processing techniques SSM to handle point cloud data and applying spatiotemporal scanning to construct efficient motion modeling."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["3D single object tracking","State Space Model","Point Cloud"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Accurate tracking of objects in 3D point clouds requires continuous and efficient motion modeling across spatial and temporal dimensions. Although voxel-based methods have recently achieved strong performance thanks to rich BEV representations, they inevitably introduce redundancy, thereby complicating motion extraction.Building on efficient point representations and their sequential modeling, we venture that voxel features can be reformulated as a state-sequence paradigm, serving as an intermediate representation for more effective motion modeling.To this end, we introduce a serialized motion modeling framework that sequentializes BEV features by embedding spatial positions within a structured voxel grid, naturally enabling more efficient processing.At its core is a simultaneous spatiotemporal scanning mechanism that enforces causal inference, preserves structural priors, and jointly disentangles motion features across adjacent domains.By leveraging geometric priors with a compact yet precise representation, our framework leads to significantly reduced computational overhead while enhancing tracking performance.  Interestingly, when integrated our approach with point-based methods, it further boosts performance through reinforced spatial modeling with minimal extra cost. Our method sets new SOTA records on the KITTI and NuScenes datasets, excelling in both accuracy and efficiency. Running at nearly 188 FPS on a single RTX 4090 GPU, it achieves a +19\\% speed improvement over the current best counterpart."},"_bibtex":{"value":"@misc{\nxu2025simtrackd,\ntitle={SimTrack3D: A Simple Sequential Motion Modeling for Efficient 3D Single Object Tracking},\nauthor={Yangzitai Xu and Fei Xie and Zongwei Wu and Xiaokang Yang and Chao Ma},\nyear={2025},\nurl={https://openreview.net/forum?id=NOAuwgqr1J}\n}"},"title":{"value":"SimTrack3D: A Simple Sequential Motion Modeling for Efficient 3D Single Object Tracking"},"pdf":{"value":"/pdf/3fdd468fc7b93b1b7601f733d572696fba9d43c5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xu|simtrack3d_a_simple_sequential_motion_modeling_for_efficient_3d_single_object_tracking"},"authorids":{"value":["~Yangzitai_Xu1","~Fei_Xie3","~Zongwei_Wu1","~Xiaokang_Yang1","~Chao_Ma3"]},"authors":{"value":["Yangzitai Xu","Fei Xie","Zongwei Wu","Xiaokang Yang","Chao Ma"]}},"version":2},{"content":{"summary":{"value":"This paper presents a version of RoPE with learned, input-dependent arbitrary rotations. Theoretical analysis provided indicates that softmax attention implicitly performs selective rotations, motiving the proposed architecture. Selective RoPE uses a learned linear projection and cumulative sum to produce input-dependent rotations. An analysis of diagonal SSMs is provided which shows distinct roles for the real and imaginary parts of the state matrix, motiving the incorporation of Selective RoPE with GLA to provide better memory. Experiments on language modeling and show improvements with Selective RoPE compared to RoPE and softmax attention in GLA models."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Did you compute RoPE setting for Table 3?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The paper gives a detailed theoretical justification for the architecture design. The analyses of softmax attention as implicit rotation and spectral leakage in SSMs may be useful to future work. Experimental evidence is provided to support claims."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The conclusion that softmax attention applies implicit selective rotation is based on the RFF approximation and additional normalization assumptions. The paper does not prove that the resultant normalized approximation converges in the limit to softmax attention, and so this analysis may be overstating the connection. \n\n\nThe real-data language modeling results in table 3 omit the RoPE condition."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762941771779,"tcdate":1761942578342,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21436/Reviewer_uVEP"],"signatures":["ICLR.cc/2026/Conference/Submission21436/Reviewer_uVEP"],"forum":"AQo1SEElNb","number":2,"license":"CC BY 4.0","cdate":1761942578342,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission21436/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762941771779,"domain":"ICLR.cc/2026/Conference","replyto":"AQo1SEElNb","id":"6H5GN7zbKm","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce Selective RoPE, an input-dependent rotary embedding that enhances gated linear transformers."},"keywords":{"value":["RoPE","Linear Transformer","Attention","State Space Models","Forget Gate"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Positional information is essential for language modeling. Softmax Transformers with Rotary Position Embeddings (RoPE) encode it with fixed-angle rotations, while linear Transformers rely on input-dependent gates that only decay past key-value norms. We provide a theoretical argument for the necessity of a rotation and decay component in well-performing sequence models, and observe that the missing ingredient in linear models is precisely the rotation that softmax attention performs implicitly. We introduce Selective Rotary Position Embedding (*Selective RoPE*), an input-dependent, learnable rotary embedding that generalizes RoPE to arbitrary angles and composes seamlessly with decay gates. Equipping gated linear attention with *Selective RoPE* yields a complex-valued recurrent layer that can be implemented efficiently with the “RoPE trick”. On synthetic benchmarks (MQAR, copying, state tracking) and 370M-parameter language-model pre-training, the method improves recall, downstream accuracy, and expressivity while adding minimal architectural overhead. We open-source our implementation [here](https://github.com/timurcarstensen/selective-rope)."},"_bibtex":{"value":"@inproceedings{\nmovahedi2026selective,\ntitle={Selective Rotary Position Embedding},\nauthor={Sajad Movahedi and Timur Carstensen and Arshia Afzal and Frank Hutter and Antonio Orvieto and Volkan Cevher},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=AQo1SEElNb}\n}"},"title":{"value":"Selective Rotary Position Embedding"},"pdf":{"value":"/pdf/943324e41b7e6a418a5aab0f6ad1261b8e48ad8e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"movahedi|selective_rotary_position_embedding"},"authorids":{"value":["~Sajad_Movahedi1","~Timur_Carstensen1","~Arshia_Afzal1","~Frank_Hutter1","~Antonio_Orvieto3","~Volkan_Cevher1"]},"authors":{"value":["Sajad Movahedi","Timur Carstensen","Arshia Afzal","Frank Hutter","Antonio Orvieto","Volkan Cevher"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"cong|uismamba_exploring_mamba_for_underwater_instance_segmentation_via_dynamic_tree_scan_and_hidden_state_weaken"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Runmin_Cong:","https://dblp.org/search/pid/api?q=author:Zongji_Yu:","~Hao_Fang5","https://dblp.org/search/pid/api?q=author:Haoyan_Sun:","https://dblp.org/search/pid/api?q=author:Sam_Kwong:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2508.00421"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2508-00421,\n  publtype={informal},\n  author={Runmin Cong and Zongji Yu and Hao Fang and Haoyan Sun and Sam Kwong},\n  title={UIS-Mamba: Exploring Mamba for Underwater Instance Segmentation via Dynamic Tree Scan and Hidden State Weaken},\n  year={2025},\n  month={August},\n  cdate={1754006400000},\n  journal={CoRR},\n  volume={abs/2508.00421},\n  url={https://doi.org/10.48550/arXiv.2508.00421}\n}\n"},"abstract":{"value":"Underwater Instance Segmentation (UIS) tasks are crucial for underwater complex scene detection. Mamba, as an emerging state space model with inherently linear complexity and global receptive fields, is highly suitable for processing image segmentation tasks with long sequence features. However, due to the particularity of underwater scenes, there are many challenges in applying Mamba to UIS. The existing fixed-patch scanning mechanism cannot maintain the internal continuity of scanned instances in the presence of severely underwater color distortion and blurred instance boundaries, and the hidden state of the complex underwater background can also inhibit the understanding of instance objects. In this work, we propose the first Mamba-based underwater instance segmentation model UIS-Mamba, and design two innovative modules, Dynamic Tree Scan (DTS) and Hidden State Weaken (HSW), to migrate Mamba to the underwater task. DTS module maintains the continuity of the internal features of the instance objects by allowing the patches to dynamically offset and scale, thereby guiding the minimum spanning tree and providing dynamic local receptive fields. HSW module suppresses the interference of complex backgrounds and effectively focuses the information flow of state propagation to the instances themselves through the Ncut-based hidden state weakening mechanism. Experimental results show that UIS-Mamba achieves state-of-the-art performance on both UIIS and USIS10K datasets, while maintaining a low number of parameters and computational complexity. Code is available at https://github.com/Maricalce/UIS-Mamba."},"title":{"value":"UIS-Mamba: Exploring Mamba for Underwater Instance Segmentation via Dynamic Tree Scan and Hidden State Weaken"},"authors":{"value":["Runmin Cong","Zongji Yu","Hao Fang","Haoyan Sun","Sam Kwong"]}},"tmdate":1762940628350,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2508-00421"],"tcdate":1762936187212,"writers":["~"],"signatures":["~Hao_Fang5"],"forum":"n4jvXStUMn","license":"CC BY-SA 4.0","number":691432,"cdate":1754006400000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1762940628350,"domain":"DBLP.org","id":"n4jvXStUMn","version":2},{"content":{"summary":{"value":"The paper proposes GF4FC, an approach to few-shot image classification that enhances CLIP by integrating dynamic cross-modal interaction and fine-grained visual feature supplementation. It introduces four key modules: (1) CLIMA, which uses a Cross-Modal Mamba (CMM) module to enable bidirectional text-image prompting with linear complexity; (2) FGC, which extracts multi-scale fine-grained features via a VSSM; (3) LFS, which supplements CLIP logits with local features through a residual structure; and (4) ALF, which adaptively fuses global and local logits. Experiments on seven benchmarks show improvements over SOTA CLIP-based FSL methods in higher-shot settings."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1) Could the authors provide qualitative examples (e.g., misclassified images on FGVC-Aircraft) showing how cross-modal interaction harms performance? Is the issue due to misaligned text prompts or Mamba’s sequence modeling bias?\n2) Why does the CLIP model perform so badly in Table 2?\n3) How about changing the fusion and alignment modules with existing mechanisms (such as CFA [1], XKanFuse[2], CoolNet[3], AlignMamba [4], or [5-6])?\n\n[1] Cross-Modal Fusion and Attention Mechanism for Weakly Supervised Video Anomaly Detection\n[2] XKanFuse: A novel cross-modal fusion method based on Kolmogorov-Arnold Network for multi-modal medical image fusion\n[3] Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis\n[4] AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment\n[5] Wan Y, Wang W, Zou G, et al. Cross-modal feature alignment and fusion for composed image retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 8384-8388.\n[6] Mingyong L, Yewen L, Mingyuan G, et al. CLIP-based fusion-modal reconstructing hashing for large-scale unsupervised cross-modal retrieval[J]. International Journal of Multimedia Information Retrieval, 2023, 12(1): 2."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1) The paper propose GF4FC, which combines Mamba-based State Space Models (SSMs) with cross-modal prompting in CLIP.\n2) The paper is clearly structured and the method is easy to conduct."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1) The motivation of this paper is not clear. In fact, the interaction among modalities is widely explored, such as CLIP-MSA: Incorporating Inter-Modal Dynamics and Common Knowledge to Multimodal Sentiment Analysis With Clip, CMFS: CLIP-Guided Modality Interaction for Mitigating Noise in Multi-Modal\nImage Fusion and Segmentation, and CLIP-BCA-Gated: A Dynamic Multimodal Framework for Real-Time Humanitarian Crisis Classification with Bi-Cross-Attention and Adaptive Gating\n\n2) In Table 1, GF4FC performs worse than Tip-Adapter-F in most cases (results are wrongly marked with bold), and the improvement over 16 shots is also limited.\n\n3) Fine-grained and global features are widely used for multimodel alignment. Also, logits-based fusion is a common strategy for multimodal fusion. All these show that the novelty of this paper is limited. The authors should compare these methods and clarify the differences. \n\n4) Some important references are missing, such as Li Y, Xing Y, Lan X, et al. AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24774-24784."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925326615,"tcdate":1760934231816,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14998/Reviewer_LGEW"],"signatures":["ICLR.cc/2026/Conference/Submission14998/Reviewer_LGEW"],"forum":"qz7g2MC32U","number":1,"license":"CC BY 4.0","cdate":1760934231816,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14998/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925326615,"domain":"ICLR.cc/2026/Conference","replyto":"qz7g2MC32U","id":"3lA864e1mg","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["CLIP","Multimodality","Few-Shot Learning","Mamba"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"CLIP is a highly efficient cross-modal text-image embedding model with remarkable generalization ability. However, the encoders in CLIP usually operate independently without dynamic cross-modal interaction, leading to suboptimal performance in few-shot classification. Therefore, we propose a Global and Fine-Grained Framework for CLIP with Cross-Modal Mamba in Few-Shot Image Classification (GF4FC). Specifically, the CLIP with Cross-Modal Mamba module (CLIMA) is conducted to leverage Transformer and Vision-Transformer to interdependently encode text and image. These cross-modal representations then serve as mutual prompts to refine the embedding space, while the proposed Cross-Modal Mamba module ensures efficient time complexity. Moreover, we design a Fine-Grained Capture module (FGC) to enhance CLIMA's image representations using a Vssm module to extract prior fine-grained information. Furthermore, the Local Feature Supplementation (LFS) module is conducted to supplement CLIP's logits with FGC-derived fine-grained representations through a residual structure. Finally, the Adaptive Logits Fusion module is constructed to dynamically fuses logits using learned adaptive weights. Experiments on seven datasets demonstrate that GF4FC achieves superior performance compared with state-of-the-art methods in few-show image classification."},"_bibtex":{"value":"@misc{\ncai2025global,\ntitle={Global and Fine-Grained Framework for {CLIP} with Cross-Modal Mamba in Few-Shot Image Classification},\nauthor={Junchen Cai and Zhi Chen and Qiaoqin Li and Rongyao Hu and Yongguo Liu},\nyear={2025},\nurl={https://openreview.net/forum?id=qz7g2MC32U}\n}"},"title":{"value":"Global and Fine-Grained Framework for CLIP with Cross-Modal Mamba in Few-Shot Image Classification"},"pdf":{"value":"/pdf/80ebe555fdd5e99bf78f296ed7496fee88100987.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"cai|global_and_finegrained_framework_for_clip_with_crossmodal_mamba_in_fewshot_image_classification"},"authorids":{"value":["~Junchen_Cai1","~Zhi_Chen7","~Qiaoqin_Li1","~Rongyao_Hu1","~Yongguo_Liu1"]},"authors":{"value":["Junchen Cai","Zhi Chen","Qiaoqin Li","Rongyao Hu","Yongguo Liu"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a improved approach to applying SSMs to 3D point cloud data. The study presents NIMBA, a Mamba-based model that uses a unique reordering strategy to convert 3D point clouds into 1D sequences while preserving spatial relationships, thus eliminating the need for positional embeddings and reducing data redundancy and computational overhead. Additionally, the NIMBA model demonstrates enhanced robustness to common data transformations."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"See Weakness. Besides, there are few other suggestions:\n\n1. Details in the writing require verification. For instance, the caption in Table 5 seems inaccurate; a full stop is missing at the end of lines 339 and 360.\n\n2. The reviewers understand that the authors trained from scratch to better highlight the difference between each setting. However, stronger data augmentation and pre-training fine-tuning could still be added to demonstrate the upper limit of NIMBA's performance."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The idea is simple and the paper is easy to follow.\n\n2. Introducing Mamba to point clouds is non-trival and further improve the efficiency and robustness is important.\n\n3. The analysis is soundness."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. PointMamba, the article's main comparator, has been accepted by NeurIPS, and its methodology has been updated. The authors should revise the relevant descriptions in the article accordingly.\n\n2. In the paper, the authors assert that NIMBA does not rely on positional embedding (PE). However, the ablation study in section 4.3.1 indicates that NIMBA performs better with PE, contradicting their claim (see line 97). Additionally, the results in Table 2 also seem to include PE, leading to confusion. Furthermore, since PE is easy to compute and doesn't significantly increase computational burden, the claim that NIMBA can contribute without it raises questions, especially given that omitting PE results in decreased performance\n\n3. The reviewer is also unsure how NIMBA validates global modeling with a sequence length N in the causal modeling Mamba. While sequence order can help preserve geometric relationships, the point patches still struggle to interact with each other. More discussion can be added."}},"nonreaders":[],"tmdate":1731428593787,"tcdate":1730822149869,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10086/Reviewer_Lgt4"],"signatures":["ICLR.cc/2025/Conference/Submission10086/Reviewer_Lgt4"],"forum":"aHSoKImhc2","number":4,"license":"CC BY 4.0","cdate":1730822149869,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10086/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428593787,"domain":"ICLR.cc/2025/Conference","replyto":"aHSoKImhc2","id":"K2dLKGfmV8","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["point clouds","SSMs","state-space models","Mamba","3D","transformers","pointcloud"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Transformers have become dominant in large-scale deep learning tasks across various domains, including text, 2D and 3D vision. However, the quadratic complexity of their attention mechanism limits their efficiency as the sequence length increases, particularly in high-resolution 3D data such as point clouds. Recently, state space models (SSMs) like Mamba have emerged as promising alternatives, offering linear complexity, scalability, and high performance in long-sequence tasks. The key challenge in the application of SSMs in this domain lies in reconciling the non-sequential structure of point clouds with the inherently directional (or bi-directional) order-dependent processing of recurrent models like Mamba. To achieve this, previous research proposed reorganizing point clouds along multiple directions or predetermined paths in 3D space, concatenating the results to produce a single 1D sequence capturing different views. In our work we introduce a method to convert point clouds into 1D sequences that maintains 3D spatial structure with no need for data replication, allowing Mamba’s sequential processing to be applied effectively in an almost permutation-invariant manner. In contrast to other works, we found that our method does not require positional embeddings, and allows for shorter sequence lengths while still achieving state-of-the-art results in ModelNet40 and ScanObjectNN datasets and surpassing Transformer-based models in both accuracy and efficiency."},"_bibtex":{"value":"@misc{\nkoprucu2025nimba,\ntitle={{NIMBA} : Towards Robust and Principled Processing of Point Clouds With {SSM}s},\nauthor={Nursena Koprucu and Destiny Okpekpe and Antonio Orvieto},\nyear={2025},\nurl={https://openreview.net/forum?id=aHSoKImhc2}\n}"},"title":{"value":"NIMBA : Towards Robust and Principled Processing of Point Clouds With SSMs"},"pdf":{"value":"/pdf/0e67e526c59087fbadbaa49be2898a7f6c87b665.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"koprucu|nimba_towards_robust_and_principled_processing_of_point_clouds_with_ssms"},"authorids":{"value":["~Nursena_Koprucu1","~Destiny_Okpekpe1","~Antonio_Orvieto3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Nursena Koprucu","Destiny Okpekpe","Antonio Orvieto"]}},"version":2},{"content":{"pdf":{"value":"/pdf/af92112e210032613fcc475f007f3e32366a0664.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"chen|hifimamba_dualstream_laplacian_enhanced_mamba_for_highfidelity_mri_reconstruction"},"authorids":{"value":["~Hongli_Chen1","~Pengcheng_Fang1","~Yuxia_Chen1","~Yingxuan_Ren1","~Jing_Hao2","~Fangfang_Tang1","~Xiaohao_Cai2","~Shanshan_Shan1"]},"abstract":{"value":"Reconstructing high-fidelity MR images from undersampled k-space data remains a challenging problem in accelerated MRI. While Mamba variants for vision tasks offer promising long-range modeling capabilities with linear-time complexity, their direct application to MRI reconstruction inherits two key limitations: (1) insensitivity to high-frequency anatomical details; and (2) reliance on redundant multi-directional scanning. To address these limitations, we introduce HiFi-Mamba, a novel dual-stream Mamba-based architecture comprising stacked 𝓌-Laplacian (WL) and HiFi-Mamba blocks. Specifically, the WL block performs fidelity-preserving spectral decoupling, producing complementary low- and high-frequency streams. This separation enables the HiFi-Mamba Block to focus on low-frequency structures, enhancing global feature modeling. Concurrently, the HiFi-Mamba block selectively integrates high-frequency features through adaptive state-space modulation, preserving comprehensive spectral details. To eliminate the scanning redundancy, the HiFi-Mamba Block adopts a streamlined unidirectional traversal strategy that preserves long-range modeling capability with improved computational efficiency. Extensive experiments on standard MRI reconstruction benchmarks demonstrate that HiFi-Mamba consistently outperforms state-of-the-art CNN-based, Transformer-based, and other Mamba-based models in reconstruction accuracy while maintaining a compact and efficient model design."},"title":{"value":"HiFi-Mamba: Dual-Stream  𝒲 -Laplacian Enhanced Mamba for High-Fidelity MRI Reconstruction"},"authors":{"value":["Hongli Chen","Pengcheng Fang","Yuxia Chen","Yingxuan Ren","Jing Hao","Fangfang Tang","Xiaohao Cai","Shanshan Shan"]}},"tmdate":1769619439230,"pdate":1762531200000,"tcdate":1769619439230,"writers":["~Hongli_Chen1","~Pengcheng_Fang1","~Yuxia_Chen1","~Yingxuan_Ren1","~Jing_Hao2","~Fangfang_Tang1","~Xiaohao_Cai2","~Shanshan_Shan1"],"signatures":["~Hongli_Chen1"],"forum":"LG1VEMgDzp","license":"CC BY 4.0","number":44090,"cdate":1769619439230,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1769619439230,"domain":"OpenReview.net/Archive","id":"LG1VEMgDzp","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2510.13046v3"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"jiang|one_dimensional_cnn_ecg_mamba_for_multilabel_abnormality_classification_in_12_lead_ecg"},"html":{"value":"https://doi.org/10.48550/arXiv.2510.13046"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2510-13046,\n  publtype={informal},\n  author={Huawei Jiang and Husna Mutahira and Gan Huang and Mannan Saeed Muhammad},\n  title={One Dimensional CNN ECG Mamba for Multilabel Abnormality Classification in 12 Lead ECG},\n  year={2025},\n  month={October},\n  cdate={1759276800000},\n  journal={CoRR},\n  volume={abs/2510.13046},\n  url={https://doi.org/10.48550/arXiv.2510.13046}\n}\n"},"abstract":{"value":"State space models offer linear-time sequence modeling and are a promising backbone for multi-label 12-lead ECG classification, but the design choices that drive their accuracy remain unclear. This letter presents ECG-Mamba-V2, a set of empirical refinements to a bidirectional Vision Mamba encoder: the class token is appended at the end of the token sequence instead of the midpoint, the forward and backward scan outputs are summed without the conventional 1/2 scaling, and dropout is applied at a uniform rate across blocks. On PhysioNet/CinC Challenge 2021, ECG-Mamba-V2 reaches 0.6494 macro AUPRC and 0.9716 macro AUROC, against 0.6100 and 0.9646 for its predecessor, while using 34\\% fewer parameters and delivering 38\\% higher throughput; it wins all 15 paired runs."},"title":{"value":"One Dimensional CNN ECG Mamba for Multilabel Abnormality Classification in 12 Lead ECG"},"authors":{"value":[{"fullname":"Huawei Jiang","username":""},{"fullname":"Husna Mutahira","username":""},{"fullname":"Gan Huang","username":"~Gan_Huang2"},{"fullname":"Mannan Saeed Muhammad","username":""}]}},"tmdate":1790970024788,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2510-13046"],"tcdate":1790970014312,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Gan_Huang2"],"forum":"0uTtesFGRu","license":"CC BY-SA 4.0","number":161768,"cdate":1759276800000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1790970024788,"domain":"OpenReview.net/Public_Article","id":"0uTtesFGRu","version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["linear attention","regression","preconditioning","gradient descent","efficiency"]},"_bibtex":{"value":"@inproceedings{\ntumma2026preconditioned,\ntitle={Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences},\nauthor={Neehal Tumma and Noel Loo and Daniela Rus},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=UC6YiTOeKb}\n}"},"title":{"value":"Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences"},"paperhash":{"value":"tumma|preconditioned_deltanet_curvatureaware_sequence_modeling_for_linear_recurrences"},"originally_submitted_PDF":{"value":"/pdf/108dfd09ecad7564d9b4dedc9e6a234312ca0184.pdf"},"primary_area":{"value":"deep_learning->attention_mechanisms"},"abstract":{"value":"To address the increasing long-context compute limitations of softmax attention, several subquadratic recurrent operators have been developed. This work includes models such as Mamba-2, DeltaNet, Gated DeltaNet (GDN), and Kimi Delta Attention (KDA). As the space of recurrences grows, a parallel line of work has arisen to taxonomize them. One compelling view is the test-time regression (TTR) framework, which interprets recurrences as performing online least squares updates that learn a linear map from the keys to values. Existing delta-rule recurrences can be seen as first-order approximations to this objective, but notably ignore the curvature of the least-squares loss during optimization. In this work, we address this by introducing preconditioning to these recurrences. Starting from the theory of online least squares, we derive equivalences between linear attention and the delta rule in the exactly preconditioned case. Next, we realize this theory in practice by proposing a diagonal approximation: this enables us to introduce preconditioned variants of DeltaNet, GDN, and KDA alongside efficient chunkwise parallel algorithms for computing them. Empirically, we find that our preconditioned delta-rule recurrences yield consistent performance improvements across synthetic recall benchmarks and language modeling at the 340M and 1B scale."},"link_to_code":{"value":"https://github.com/ntumm120/preconditioned-deltanet"},"pdf":{"value":"/pdf/4f0e5026212e683f0c11b105de1cf9b0f277c530.pdf"},"lay_summary":{"value":"Preconditioned DeltaNet is our approach for making efficient language models better at remembering useful information over long pieces of text. When a model reads new words, it has to decide how to update its internal memory, but existing methods often make these updates in a fairly blunt way. We give the model a more careful memory update rule, so it can better preserve, change, or emphasize information depending on what it has already seen. We show that this idea can be added to several existing efficient model families without making them much slower. In experiments, Preconditioned DeltaNet improves performance on tasks that require recalling earlier information, suggesting a practical path toward faster and more capable long-context language models."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Neehal_Tumma1","~Noel_Loo1","~Daniela_Rus1"]},"authors":{"value":["Neehal Tumma","Noel Loo","Daniela Rus"]}},"tmdate":1790067431810,"pdate":1777576863545,"tcdate":1769212326275,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission29325/Authors"],"signatures":["ICML.cc/2026/Conference/Submission29325/Authors"],"forum":"UC6YiTOeKb","license":"CC BY 4.0","number":29325,"cdate":1769212326275,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission29325/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission29325/-/Camera_Ready_Revision"],"mdate":1790067431810,"odate":1782342024837,"domain":"ICML.cc/2026/Conference","id":"UC6YiTOeKb","version":2},{"content":{"summary":{"value":"This paper presents a thorough analysis of the key factors contributing to the success of the S6 module in Mamba model, and introduces a new linear attention vision network, MLLA, inspired by the S6’s design. Extensive experimental results show outstanding performance, validating the effectiveness of the proposed model."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please see the main weaknesses."},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"The motivation and analysis underlying this work are fascinating and perceptive, offering a profound examination of the mathematical relationships between Mamba and Linear Attention. \n\nA thorough investigation of the performance contributions of each component from S6 is conducted, with corresponding refinements made to linear attention, ultimately leading to the development of MLLA. \n\nThe proposed MLLA serves as a generic vision backbone network, which outperforms recent Mamba-based models, demonstrating its superior capabilities"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I appreciate the paper's insightful analyses, but several key concerns regarding methodology and experiments need to be addressed. If the authors fully resolve these issues, I would be willing to reconsider my evaluation.\n\n**Possible inaccurate analysis of single-head attention**\n\nThe analysis of single-head attention appears to be inaccurate and unclear. Generally, multi-head attention in Transformer generates H dynamic matrices during attention computation, where H represents the number of heads. However, the paper seems to interpret S6 as a single-head module without providing sufficient evidence. The authors should explain this point from the perspective of the number of attention matrices generated. Moreover, according to Reference 1, S6 can actually produce multiple dynamic matrices instead of just one.\n\n**Unfair comparison on ImageNet-1K dataset**\n\nThe paper uses MESA as its optimizer, improving accuracy compared to AdamW. In contrast, the methods in Table 3 do not use MESA and may also be prone to overfitting. For fairness, authors should ensure that training strategies are consistent with competitors. Moreover, the authors should not hide this apparent difference in the appendix.\n\n\n**Questionable motivation for using Swin as the baseline**\n\nThe motivation for using Swin as a baseline is questionable. One important aspect of linear attention is its ability to efficiently establish global dependencies, so it is unclear why window-based linear attention is used. Additionally, the specific configuration of the baseline model, such as channels and depth, is not provided.\n\n**Unclear setting of d_state in SSM block design**\n\nIn the block design, it is unclear what value d_state of SSM is set to.\n\n\n**Insufficient experiments on downstream tasks**\n\n(1) Semantic segmentation should include tiny, small, and base models. (2) Mask R-CNN 3x should include the base model. (3) The performance comparison of MLLA-T and VMamba-T on object detection is lacking.\n\n**Unfair comparison in Table 6**\n\nThe comparison in Table 6 is unfair. The authors aim to compare the performance of different linear attention methods, but different block settings have a significant impact. The authors should use the same architecture, replacing only token mixer module.\n\n**Typo**\n\nIn line 294, 'SMM' should be 'SSM'."},"limitations":{"value":"Please see the main weaknesses."}},"nonreaders":[],"tmdate":1730878968326,"tcdate":1720455714503,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission4979/Reviewer_XBcf"],"signatures":["NeurIPS.cc/2024/Conference/Submission4979/Reviewer_XBcf"],"forum":"LvJ1R88KAk","number":1,"license":"CC BY 4.0","cdate":1720455714503,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission4979/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878968326,"domain":"NeurIPS.cc/2024/Conference","replyto":"LvJ1R88KAk","id":"ph6O0qYSba","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","Linear Attention Transformer"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba’s success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba’s success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-\nInspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https://github.com/LeapLabTHU/MLLA."},"_bibtex":{"value":"@inproceedings{\nhan2024demystify,\ntitle={Demystify Mamba in Vision: A Linear Attention Perspective},\nauthor={Dongchen Han and Ziyi Wang and Zhuofan Xia and Yizeng Han and Yifan Pu and Chunjiang Ge and Jun Song and Shiji Song and Bo Zheng and Gao Huang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=LvJ1R88KAk}\n}"},"title":{"value":"Demystify Mamba in Vision: A Linear Attention Perspective"},"pdf":{"value":"/pdf/419f5b34c2a7aefa14ac7e991c160ab502bd73f5.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"han|demystify_mamba_in_vision_a_linear_attention_perspective"},"authorids":{"value":["~Dongchen_Han1","~Ziyi_Wang21","~Zhuofan_Xia2","~Yizeng_Han1","~Yifan_Pu1","~Chunjiang_Ge1","~Jun_Song5","~Shiji_Song1","~Bo_Zheng5","~Gao_Huang1"]},"authors":{"value":["Dongchen Han","Ziyi Wang","Zhuofan Xia","Yizeng Han","Yifan Pu","Chunjiang Ge","Jun Song","Shiji Song","Bo Zheng","Gao Huang"]}},"version":2},{"content":{"summary":{"value":"This work proposes a new robust RL methods for MDP with uncertain transitions called IWOCS (Incremental Worst-Case Search). This method incrementally builds a discrete, non-sa-rectangular uncertainty set and a sequence of candidate robust policies. IWOCS can be utilized for classic RL tasks with discrete state/action spaces and Deep RL domains with continuous task settings. The experiments on windy gridworld and Mujoco tasks demonstrate its effectiveness."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. In this work, IWOCS only considers a unique starting $s_0$ (Line 067). However, in some tasks such as continuous control tasks, we need to tackle the distribution $p(s_0)$. For example, as shown in Algorithm 2, we need to compute $V_{T_{i+1}}^{\\pi_i}(s_0)$ and $Q_i(s_0,\\pi_i(s_0))$ during training. How to compute these values for uncertain $s_0$?  Discuss how IWOCS might be extended to handle a distribution of starting states, such as in IWOCS+SAC algorithm for Mujoco control tasks.  \n2. As shown in table 2&3, IWOCS* obtains higher performance than IWOCS, which is normal because IWOCS*  utilize grid-search rather than CMA-ES. Please provide a detailed comparison of computation times for IWOCS and IWOCS* across different uncertainty set dimensions. Discuss the trade-offs between performance and computation time when using grid-search vs CMA-ES, and provide guidelines for when to use IWOCS vs IWOCS* based on problem characteristics.\n3. Can the idea proposed in this work be utilized in other robust RL problems, such as robust RL under observation perturbations? Please discuss any modifications or potential challenges when applying IWOCS to observation perturbation problems."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. Improving robustness of RL against transition uncertainty is meaningful for downstream tasks, such as  RL applications on robotics.\n2. The extensive experiments on classic and deep RL settings demonstrate the effectiveness of the new method, especially in tasks with continuous state space and action spaces."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. In appendix. I, the author shows the training times for SAC and IWOCS in Mujoco control tasks. As shown in Table 6, IWOCS requires about twice time cost for training compared to SAC, leading to additional burdens in real applications. According to this phenomenon, please provide the following results to analyze and improve the new method:\n    1. Provide a more detailed breakdown of where the additional computation time is spent in IWOCS (e.g. worst-case search, policy optimization, etc.). This would help identify specific areas for potential optimization.\n    2. Provide potential methods or approaches to reduce the computational cost of IWOCS.\n    3. Include training times for other baseline robust RL methods to provide more context for evaluating IWOCS's computational efficiency relative to the state-of-the-art.\n2. This work focuses on the RL task with uncertain transition, such as robotics control with dynamic friction settings. This issue is also researched in Meta-RL domains, such as Meta RL methods with task inference. Such methods can adapt to new environment transitions with limited interactions. Please explain the relationship or difference of this work and such meta RL methods. A brief introduction of Meta-RL with uncertain transition should be included in the related work, with highlighting key similarities and differences between IWOCS and meta-RL approaches.\n3. Some typos in the manuscript, such as  line 085: denote -> denotes"}},"nonreaders":[],"tmdate":1731428614233,"tcdate":1729691021681,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10127/Reviewer_thpw"],"signatures":["ICLR.cc/2025/Conference/Submission10127/Reviewer_thpw"],"forum":"Zi1QNJKXAD","number":1,"license":"CC BY 4.0","cdate":1729691021681,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10127/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428614233,"domain":"ICLR.cc/2025/Conference","replyto":"Zi1QNJKXAD","id":"cStE7G2NTl","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Robust reinforcement learning"]},"supplementary_material":{"value":"/attachment/f25ba23d357ad8841194d3dbb1725bc85f1c2718.zip"},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"esigning control policies whose performance level is guaranteed to remain above a given threshold in a span of environments is a critical feature for the adoption of reinforcement learning (RL) in real-world applications. \nThe search for such robust policies is a notoriously difficult problem, related to the so-called dynamic model of transition function uncertainty, where the environment dynamics are allowed to change at each time step.\nBut in practical cases, one is rather interested in robustness to a span of static transition models throughout interaction episodes. \nThe static model is known to be harder to solve than the dynamic one, and seminal algorithms, such as robust value iteration, as well as most recent works on deep robust RL, build upon the dynamic model.\nIn this work, we propose to revisit the static model. \nWe suggest an analysis of why solving the static model under some mild hypotheses is a reasonable endeavor, based on an equivalence with the dynamic model, and formalize the general intuition that robust MDPs can be solved by tackling a series of static problems. \nWe introduce a generic meta-algorithm called IWOCS, which incrementally identifies worst-case transition models so as to guide the search for a robust policy. \nDiscussion on IWOCS sheds light on new ways to decouple policy optimization and adversarial transition functions and opens new perspectives for analysis.\nWe derive a deep RL version of IWOCS and demonstrate it is competitive with state-of-the-art algorithms on classical benchmarks."},"_bibtex":{"value":"@misc{\nzouitine2025solving,\ntitle={Solving robust {MDP}s as a sequence of static {RL} problems},\nauthor={Adil Zouitine and Matthieu Geist and Emmanuel Rachelson},\nyear={2025},\nurl={https://openreview.net/forum?id=Zi1QNJKXAD}\n}"},"title":{"value":"Solving robust MDPs as a sequence of static RL problems"},"pdf":{"value":"/pdf/f27e3d1c26f0716c8c28d5c801045d814a1025ed.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zouitine|solving_robust_mdps_as_a_sequence_of_static_rl_problems"},"authorids":{"value":["~Adil_Zouitine1","~Matthieu_Geist1","~Emmanuel_Rachelson1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Adil Zouitine","Matthieu Geist","Emmanuel Rachelson"]}},"version":2},{"content":{"comment":{"value":"Thank you very much for your valuable feedback. We sincerely appreciate the constructive feedback provided. However, we believe that the inclusion of additional test sets and baselines is not necessary for addressing the points presented in the paper. We address weakness points as below:\n\n> “I think this dataset cannot truly demonstrate the model's benefits for long text input and streaming inference. The authors should evaluate on more suitable test sets”\n\nIn our experiments, we demonstrated that our model can operate *beyond the maximum context length that it was trained on*. Our experiments use a filtered set of 10-20s samples (avg: 14.7s), while our model is trained on maximum 10s. We believe that effective shows that our model can handle infinitely long text stream by sliding a window, since our model *does not have access to the absolute position of text tokens* (in the same way as RoPE or a CNN/RNN network). We will, however, include a few longer samples in the attachment for qualitative listening.\n\n> “Additionally, the baselines chosen for comparison are relatively weak, as they do not include more advanced models like VoiceBox, NaturalSpeech 3, and CosyVoice.”\n\nOur strongest baseline is XTTS v2, which is currently the 4th model in the TTS Arena (https://huggingface.co/spaces/TTS-AGI/TTS-Arena), from which there are industrial candidates. We compare our model to only those that we can have access to a checkpoint since many factors can affect the results (e.g. enrollment speech can be chosen differently, not every paper uses the entire LibriTTS test sets, etc.)\n\nFurthermore, we only compare our model to language model-based approaches. Among suggested candidates, VoiceBox and Natural Speech 3 are diffusion-based and may not be suitable for streaming since the whole utterance is generated non-autoregressively, while more recent work like CosyVoice is not required for comparison under the ICLR guidelines (within 4 months). We believe that some of our baseline models (MetaVoice, XTTS v2) are among the best open-source candidates and others (SpeechX, LiveSpeech) are models trained with similar training data and model size. Our primary focus is also to demonstrate the streaming capabilities of our model, rather than achieving state-of-the-art performance.\n\n> “The authors did not demonstrate the necessity of the Mamba architecture, I think transformers could also achieve the same methods.”  \n\nMamba is not required functionally; however, this aims to reduce the inference time and allow the model to run on device (where streaming models usually run on), which is critical for streaming since the model needs to produce audio tokens at 75Hz. We identify the inference speed (different to latency) as one of challenges for on-device streaming (Introduction / paragraph 2 / last sentence) and propose Mamba to alleviate that. We refer to the original work [1,2] for an extensive comparison between the Mamba and the transformer architecture in terms of inference speed (as for your requested evidence). More importantly, we have showed that Mamba performs comparably to transformer counterparts overall. We want to highlight the fact that the Mamba architecture has not been used for large scale (>60k hours) training of zero-shot TTS models before, so validating the feasibility of the model is an important contribution for future streaming TTS research.\n\n[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces\n[2] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality\n\nWe hope these have also answered your questions. By responding early, we look forward to hearing from you to confirm if these points have been addressed satisfactorily!"},"title":{"value":"Thank you for your valuable feedback!"}},"tmdate":1731768869987,"tcdate":1731768869987,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1359/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission1359/Authors"],"forum":"RK3Gj9J5my","number":2,"license":"CC BY 4.0","cdate":1731768869987,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1359/-/Official_Comment"],"mdate":1731768869987,"domain":"ICLR.cc/2025/Conference","replyto":"IAD4Omxknx","id":"GyZS9eaR3x","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We introduce a zero-shot text-to-speech model that is capable of generating speech from a continuous text stream, with support for on-device streaming."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["zero-shot text-to-speech","streaming","state-space models"]},"supplementary_material":{"value":"/attachment/eb3ee0f6f9fc423a83e128f51cb138fde2d018cc.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Existing zero-shot text-to-speech (TTS) systems are typically designed to process complete sentences and are constrained by the maximum duration for which they have been trained. However, in many streaming applications, texts arrive continuously in short chunks, necessitating instant responses from the system. We identify the essential capabilities required for chunk-level streaming and introduce L3Speech, a stream-aware model that supports non-stop speech generation, text-audio stream synchronization, and seamless transitions between short speech chunks. To achieve these, we propose (1) adopting Mamba, a class of sequence modeling distinguished by linear-time decoding, which is augmented by cross-attention mechanisms for conditioning, (2) utilizing rotary positional embeddings in the computation of cross-attention, enabling the model to process a non-stop text stream by sliding a window, and (3) decoding with semantic guidance, a technique that aligns speech with the transcript during inference with minimal overhead. Experimental results demonstrate that our models are competitive with state-of-the-art language model-based zero-shot TTS models, while also providing flexibility to support a wide range of streaming scenarios."},"_bibtex":{"value":"@misc{\ndang2025zeroshot,\ntitle={Zero-Shot Text-to-Speech from Continuous Text Streams},\nauthor={Trung Dang and David Aponte and Dung Tran and Tianyi Chen and Kazuhito Koishida},\nyear={2025},\nurl={https://openreview.net/forum?id=RK3Gj9J5my}\n}"},"title":{"value":"Zero-Shot Text-to-Speech from Continuous Text Streams"},"pdf":{"value":"/pdf/ae6d24286703bb7fbfcd629543b4f9e8d221ff52.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"dang|zeroshot_texttospeech_from_continuous_text_streams"},"authorids":{"value":["~Trung_Dang1","~David_Aponte1","~Dung_Tran2","~Tianyi_Chen3","~Kazuhito_Koishida1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Trung Dang","David Aponte","Dung Tran","Tianyi Chen","Kazuhito Koishida"]}},"version":2},{"content":{"TLDR":{"value":"SW-Mamba selectively forgets transient weather-corrupted features while preserving reliable geometric states, enabling robust self-supervised monocular depth estimation under adverse conditions."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Monocular depth estimation","self-supervised learning","adverse-weather vision","state space model","selective forgetting"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Self-supervised monocular depth estimation relies on cross-frame photometric consistency, an assumption severely challenged by adverse weather where appearance changes deviate from scene geometry. To address these challenges, we propose Selective Weather-forgetting Mamba (SW-Mamba), which formulates robust adverse-weather depth estimation as selective latent-state correction. Rather than treating inter-frame variation as a weather mask, SW-Mamba interprets temporal cues jointly with multidirectionally propagated spatial context to estimate a spatiotemporal correction gate. The gate modulates reference-guided state correction, selectively updating unreliable responses while preserving geometric consistency. During paired training, aligned clean states provide supervision for gate calibration and serve as structural references. At inference, an internal structural proxy enables deployment without aligned clean observations. Extensive experiments on WeatherKITTI, nuScenes, and DrivingStereo show consistent gains under adverse weather and strong cross-domain generalization."},"_bibtex":{"value":"@inproceedings{\nanonymous2026swmamba,\ntitle={{SW}-Mamba: Selective Weather Forgetting for Self-Supervised Monocular Depth Estimation},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=kDMwZQK6mT},\nnote={under review}\n}"},"title":{"value":"SW-Mamba: Selective Weather Forgetting for Self-Supervised Monocular Depth Estimation"},"pdf":{"value":"/pdf/cc90653d853bfc6cb71ca92e7e095402153a70be.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791233031936,"tcdate":1789728863080,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission41452/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission41452/Authors"],"forum":"kDMwZQK6mT","license":"CC BY 4.0","number":41452,"cdate":1789728863080,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission41452/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791233031936,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"kDMwZQK6mT","version":2},{"content":{"summary":{"value":"This paper addresses the limitation of information loss in quantized protein structure representations for multimodal protein language models (pLMs) by proposing HD-Prot, a hybrid diffusion framework that integrates continuous structure tokens with discrete sequence tokens. HD-Prot uses a non-quantized autoencoder (salad) to generate high-fidelity continuous structure tokens, avoiding vector quantization (VQ)–induced information loss. It employs a unified masked diffusion process for inter-token dependency modeling and separate intra-token learning: categorical cross-entropy for discrete sequences and diffusion loss for continuous structures. Experiments on unconditional sequence-structure co-generation and motif-scaffolding show HD-Prot matches state-of-the-art models (e.g., DPLM-2) while using fewer computational resources. The work argues that continuous structure tokens offer a viable alternative to quantized representations for multimodal pLMs."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. The paper highlights the learning of inter-modal interaction where HD-Prot fuses sequence and structure tokens via element-wise summation (Equation 7), but the paper provides no mechanistic explanation of how this fusion captures cross-modal dependencies (e.g., how sequence embeddings guide structure diffusion or vice versa). There is no ablation of fusion methods (e.g., concatenation, cross-attention) to validate that element-wise summation is optimal, nor analysis of attention weights to confirm that the model actually leverages both modalities during generation.\n2. See weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper identifies a critical gap in existing multimodal pLMs caused by quantized structure tokens, and introduce the hybrid diffusion framework unifies discrete sequence and continuous structure modeling via shared pLM backbones, and the use of a pre-trained non-quantized autoencoder (salad) ensures high structural reconstruction fidelity. An idea validated in vision(-language) models but under-explored in protein modeling.\n2. Qualitative results further validate that HD-Prot generates proteins with amino acid and secondary structure distributions similar to natural proteins, supporting its biological plausibility."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Major: The work only evaluates two tasks (unconditional co-generation, motif-scaffolding) but ignores key protein modeling tasks that test practical utility, such as structure prediction (from sequence), inverse folding (from structure to sequence), or maybe structure-aware predictive downstream tasks. SOTA models like ESM, DPLM-2 or SaProt are validated across these tasks, so HD-Prot’s omission introduces concerns about its broader applicability.\n2. No comparison to pure continuous diffusion based multimodal pLMs, such as DiMA [1]. \n3. Minor: Some parts of the manuscript seems similar in wording and structure to DPLM-2 (Wang et al., 2024b). Better rephrase and reorganize a little bit. \n\n-- \n\n[1] Diffusion on language model encodings for protein sequence generation. In ICML 2025"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916719596,"tcdate":1762271372087,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission3427/Reviewer_NCdP"],"signatures":["ICLR.cc/2026/Conference/Submission3427/Reviewer_NCdP"],"forum":"lIIJDDxBrg","number":4,"license":"CC BY 4.0","cdate":1762271372087,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission3427/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916719596,"domain":"ICLR.cc/2026/Conference","replyto":"lIIJDDxBrg","id":"A9JjnJUGT4","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["ai for science","protein language model","protein sequence-structure co-generation"]},"supplementary_material":{"value":"/attachment/bbffeaa1873230f832fd5ab2e5572a8a12349a9f.zip"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"abstract":{"value":"Proteins inherently possess a consistent sequence-structure duality. The abundance of protein sequence data, which can be readily represented as discrete tokens, has enabled fruitful developments in protein language models (pLMs). A key remaining challenge, however, is how to effectively integrate continuous structural knowledge into pLMs. Current methods often discretize protein structure to accommodate the language modeling framework, which inevitably results in fine-grained information loss and limits the performance potential of multimodal pLMs. In this paper, we argue that such concerns can be circumvented: a sequence-based pLM can be extended to incorporate the structure modality through continuous tokens, i.e., high-fidelity protein structure latents that avoid vector quantization. Specifically, we propose a hybrid diffusion protein language model, HD-Prot, which embeds a continuous-valued diffusion generation head atop a discrete pLM, enabling seamless operation with both discrete and continuous tokens for sequence-structure joint-modeling in multimodal generative pLMs. The proposed model captures inter-token dependencies across modalities through a unified absorbing diffusion process, and estimates per-token distributions via categorical prediction for sequences and continuous diffusion for structures. Extensive empirical results show that our models achieve competitive performance in unconditional sequence–structure co-generation, motif-scaffolding, protein structure prediction, and inverse folding tasks, performing on par with state-of-the-art multimodal pLMs despite being developed under limited computational resources. It underscores the viability of jointly modeling discrete categorical and continuous arbitrary distributions using shared parameters within a pLM, pointing to an alternative and promising direction of progress for multimodal pLMs."},"_bibtex":{"value":"@misc{\nzhou2026empowering,\ntitle={Empowering Protein Language Model for Sequence-Structure Co-Generation with Continuous Structure Tokens},\nauthor={Yi Zhou and Haohao Qu and Yunqing LIU and Shanru Lin and Le Song and Wenqi Fan},\nyear={2026},\nurl={https://openreview.net/forum?id=lIIJDDxBrg}\n}"},"title":{"value":"Empowering Protein Language Model for Sequence-Structure Co-Generation with Continuous Structure Tokens"},"pdf":{"value":"/pdf/b26e444a9c1749d14ef6d2e663164c8e1ba9f7e1.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhou|empowering_protein_language_model_for_sequencestructure_cogeneration_with_continuous_structure_tokens"},"authorids":{"value":["~Yi_Zhou38","~Haohao_Qu1","~Yunqing_LIU1","~Shanru_Lin1","~Le_Song1","~Wenqi_Fan1"]},"authors":{"value":["Yi Zhou","Haohao Qu","Yunqing LIU","Shanru Lin","Le Song","Wenqi Fan"]}},"version":2},{"content":{"summary":{"value":"This paper propose a Mamba-based model for music information retrieval tasks. The proposed method is evaluated on a set of typical MIR tasks, following similar research."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"4.1: Why were the log-mel spectrogram parameters were matched to (Huang et al., 2022)?\n\n4.2: As a baseline, [6] was mentioned but [14] was cited and used. What is the reason?\n\n4.5: Meter2800: The task information is missing. \n\n4.6: Why was NSynth chosen for the ablation? \n\n5.4: \"Mamba effectively learns key transitions and harmonic relationships over extended sequences.\": What is the evidence for this?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper is well written, is based on extensive background research and similar previous works. The motivation is well-justified given the nature of music signal and the advantage of the Mamba architecture. The paper is written clearly and the overall experiment process is executed well."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I have a mixed feeling about the overall experiments. Table 2, the results without pretraining: What is the exact purpose of this experiment? Is the MoMamba designed and expected to be adopted when there is under 2000 samples? No, perhaps not, because in such a case, an SVM with musical features can outperform all of the baseline methods and the proposed method by a large margin. Thus, I put a lot more weight on Table 3 - the pretrained results, where MoMamba underperformed. Some of the results are probably not that bad, at least. But the result is not really supporting the motivation of efficient sequential modeling nature of the Mamba architecture. NSynth consists of 4 seconds signal, each of which corresponds to playing one note. In other words. NSynth is far from modeling sequence; but this is where the proposed architecture performed the best among all the tasks. \n\nAfter reading the paper, I rather thought it seems very important to pretrain MoMamba. That is also more relevant to the motivation of the original Mamba architecture; it was not proposed to require less amount of data (re: Table 2), but to *scale* better over long sequences (and it goes without saying that it assumes a large scale pretraining.)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933835755,"tcdate":1761949020600,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20385/Reviewer_w46Z"],"signatures":["ICLR.cc/2026/Conference/Submission20385/Reviewer_w46Z"],"forum":"TUbfsCpciR","number":2,"license":"CC BY 4.0","cdate":1761949020600,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20385/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933835755,"domain":"ICLR.cc/2026/Conference","replyto":"TUbfsCpciR","id":"K5lOhaa5up","forumContent":{"TLDR":{"value":"Mamba model adapted to music tasks"},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["MIR","Mamba","State Space Models","Music","Genre Classification","Global Key Classification","Emotion Regression","Instrument Classification","Pitch Classification"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Music Information Retrieval (MIR) tasks on raw audio have traditionally been tackled using convolutional neural networks (CNNs) and transformer-based models. While CNNs effectively capture local structures and transformers leverage attention for long-range dependencies, both architectures come with computational and scalability challenges. In this study, we introduce a novel extension of Mamba tailored to music. Our resulting method, MoMamba (Music Oriented Mamba), is a lightweight Mamba-based music classification model. We evaluate MoMamba's performance across several benchmark MIR tasks. Our results show that MoMamba consistently outperforms a number of baselines, including an existing Mamba-based method, on all of the benchmark datasets we considered. Importantly, all models were trained from scratch without any pretraining, making the performance gains especially notable since they cannot be attributed to transfer learning. Additionally, our model's performance rivals existing benchmarks from models pretrained on much larger datasets. Our work highlights the advantages of MoMamba in music analysis and retrieval such as accuracy and inference time, encouraging further research into its capabilities within the MIR domain."},"_bibtex":{"value":"@misc{\nliu2026momamba,\ntitle={MoMamba: A Lightweight Music Oriented Mamba-Based Model for Music Information Retrieval Tasks},\nauthor={David Liu and Mete Gumusayak and Dallin Gordon and Brian Kulis},\nyear={2026},\nurl={https://openreview.net/forum?id=TUbfsCpciR}\n}"},"title":{"value":"MoMamba: A Lightweight Music Oriented Mamba-Based Model for Music Information Retrieval Tasks"},"pdf":{"value":"/pdf/67017dc15e0d0c2385163394369fc33bb7b2ba08.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"liu|momamba_a_lightweight_music_oriented_mambabased_model_for_music_information_retrieval_tasks"},"authorids":{"value":["~David_Liu5","~Mete_Gumusayak1","~Dallin_Gordon1","~Brian_Kulis1"]},"authors":{"value":["David Liu","Mete Gumusayak","Dallin Gordon","Brian Kulis"]}},"version":2},{"content":{"venue":{"value":"NeurIPS 2024"},"pdf":{"value":"https://proceedings.neurips.cc/paper_files/paper/2024/file/e6231c5f46598cfd09ff1970524e0436-Paper-Conference.pdf"},"venueid":{"value":"dblp.org/conf/NIPS/2024"},"paperhash":{"value":"cirone|theoretical_foundations_of_deep_selective_statespace_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Nicola_Muca_Cirone:","~Antonio_Orvieto3","https://dblp.org/search/pid/api?q=author:Benjamin_Walker_0001:","https://dblp.org/search/pid/api?q=author:Cristopher_Salvi:","https://dblp.org/search/pid/api?q=author:Terry_J._Lyons:"]},"html":{"value":"http://papers.nips.cc/paper_files/paper/2024/hash/e6231c5f46598cfd09ff1970524e0436-Abstract-Conference.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/nips/CironeO0SL24,\n  author={Nicola Muca Cirone and Antonio Orvieto and Benjamin Walker and Cristopher Salvi and Terry J. Lyons},\n  title={Theoretical Foundations of Deep Selective State-Space Models},\n  year={2024},\n  cdate={1704067200000},\n  url={http://papers.nips.cc/paper_files/paper/2024/hash/e6231c5f46598cfd09ff1970524e0436-Abstract-Conference.html},\n  booktitle={NeurIPS},\n  crossref={conf/nips/2024}\n}\n"},"abstract":{"value":"Structured state-space models (SSMs) are gaining popularity as effective foundational architectures for sequential data, demonstrating outstanding performance across a diverse set of domains alongside desirable scalability properties. Recent developments show that if the linear recurrence powering SSMs allows for a selectivity mechanism leveraging multiplicative interactions between inputs and hidden states (e.g. Mamba, GLA, Hawk/Griffin, HGRN2), then the resulting architecture can surpass attention-powered foundation models trained on text in both accuracy and efficiency, at scales of billion parameters. In this paper, we give theoretical grounding to the selectivity mechanism, often linked to in-context learning, using tools from Rough Path Theory. We provide a framework for the theoretical analysis of generalized selective SSMs, fully characterizing their expressive power and identifying the gating mechanism as the crucial architectural choice. Our analysis provides a closed-form description of the expressive powers of modern SSMs, such as Mamba, quantifying theoretically the drastic improvement in performance from the previous generation of models, such as S4. Our theory not only motivates the success of modern selective state-space models, but also provides a solid framework to understand the expressive power of future SSM variants. In particular, it suggests cross-channel interactions could play a vital role in future improvements."},"title":{"value":"Theoretical Foundations of Deep Selective State-Space Models"},"authors":{"value":["Nicola Muca Cirone","Antonio Orvieto","Benjamin Walker","Cristopher Salvi","Terry J. Lyons"]}},"tmdate":1754034349266,"pdate":1704067200000,"externalIds":["dblp:conf/nips/CironeO0SL24"],"tcdate":1754034342292,"writers":["~"],"signatures":["~Antonio_Orvieto3"],"forum":"MsttjUmp5r","license":"CC BY-SA 4.0","number":608423,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1754034349266,"domain":"DBLP.org","id":"MsttjUmp5r","version":2},{"content":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["theory","deep learning theory","SSM","RNN","neural controlled differential equations","signatures","kernels"]},"supplementary_material":{"value":"/attachment/416fe722331399eafa4acc22b5fce96ef3d6cd13.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Structured state-space models (SSMs) are gaining popularity as effective foundational architectures for sequential data, demonstrating outstanding performance across a diverse set of domains alongside desirable scalability properties. Recent developments show that if the linear recurrence powering SSMs allows for a selectivity mechanism leveraging multiplicative interactions between inputs and hidden states (e.g. Mamba, GLA, Hawk/Griffin, HGRN2), then the resulting architecture can surpass attention-powered foundation models trained on text in both accuracy and efficiency, at scales of billion parameters. In this paper, we give theoretical grounding to the selectivity mechanism, often linked to in-context learning, using tools from Rough Path Theory. We provide a framework for the theoretical analysis of generalized selective SSMs, fully characterizing their expressive power and identifying the gating mechanism as the crucial architectural choice. Our analysis provides a closed-form description of the expressive powers of modern SSMs, such as Mamba, quantifying theoretically the drastic improvement in performance from the previous generation of models, such as S4. Our theory not only motivates the success of modern selective state-space models, but also provides a solid framework to understand the expressive power of future SSM variants. In particular, it suggests cross-channel interactions could play a vital role in future improvements."},"_bibtex":{"value":"@inproceedings{\ncirone2024theoretical,\ntitle={Theoretical Foundations of Deep Selective State-Space Models},\nauthor={Nicola Muca Cirone and Antonio Orvieto and Benjamin Walker and Cristopher Salvi and Terry Lyons},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=3SzrqwupUx}\n}"},"title":{"value":"Theoretical Foundations of Deep Selective State-Space Models"},"pdf":{"value":"/pdf/4e86fe9ae93de98a547f68ad2934a6a01ebc450e.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"cirone|theoretical_foundations_of_deep_selective_statespace_models"},"authorids":{"value":["~Nicola_Muca_Cirone1","~Antonio_Orvieto3","~Benjamin_Walker1","~Cristopher_Salvi1","~Terry_Lyons2"]},"authors":{"value":["Nicola Muca Cirone","Antonio Orvieto","Benjamin Walker","Cristopher Salvi","Terry Lyons"]}},"tmdate":1736079384479,"pdate":1727288007651,"tcdate":1715722775025,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission12481/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission12481/Authors"],"forum":"3SzrqwupUx","license":"CC BY 4.0","number":12481,"cdate":1715722775025,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/-/Submission","NeurIPS.cc/2024/Conference/-/Post_Submission","NeurIPS.cc/2024/Conference/Submission12481/-/Revision","NeurIPS.cc/2024/Conference/-/Edit","NeurIPS.cc/2024/Conference/Submission12481/-/Camera_Ready_Revision"],"mdate":1736079384479,"odate":1730873949421,"domain":"NeurIPS.cc/2024/Conference","id":"3SzrqwupUx","version":2},{"content":{"venue":{"value":"ISBRA (1) 2025"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-981-95-0698-9_20.pdf"},"venueid":{"value":"dblp.org/conf/ISBRA/2025"},"paperhash":{"value":"meng|mambast_hexagonal_state_space_modeling_for_spatial_domain_identification"},"authorids":{"value":["","~Kai_Hu6","",""]},"html":{"value":"https://doi.org/10.1007/978-981-95-0698-9_20"},"_bibtex":{"value":"@inproceedings{DBLP:conf/isbra/MengHCZ25,\n  author={Xianglong Meng and Kai Hu and Xuefeng Cui and Fa Zhang},\n  title={MambaST: Hexagonal State Space Modeling for Spatial Domain Identification},\n  year={2025},\n  cdate={1735689600000},\n  pages={236-248},\n  url={https://doi.org/10.1007/978-981-95-0698-9_20},\n  booktitle={ISBRA (1)},\n  crossref={conf/isbra/2025-1}\n}\n"},"abstract":{"value":"Spatial transcriptomics has transformed tissue analysis by preserving spatial context in gene expression data, enabling deeper insights into tissue microenvironments. However, current spatial domain identification methods largely focus on adjacent cellular similarities, limiting their ability to capture long-range spatial dependencies and identify identical cell types distributed across distant and non-contiguous areas. To address these challenges, we introduce MambaST, a hybrid deep-learning framework that integrates selective state space modeling (Mamba) and self-supervised learning for Spatial Transcriptomics data analysis. Specifically, MambaST incorporates a Six-Directional Selective Scan (SS6D) algorithm to convert graph-structured spatial data into topology-preserving pseudo-sequences, effectively bridging sequential modeling with spatial topology. Additionally, we propose HexMambaBlock (HMB), which applies Mamba to simultaneously denoise gene expression data and capture global spatial dependencies. Furthermore, contrastive learning enhanced with a Dynamic Context-aware Readout (DCR) module improves the biological specificity of local representations. Comprehensive evaluations across three spatial transcriptomic datasets demonstrate MambaST’s superior performance in spatial domain identification, achieving a 0.58 average Adjusted Rand Index (ARI) on the DLPFC dataset, which surpasses state-of-the-art methods by 2.7%."},"title":{"value":"MambaST: Hexagonal State Space Modeling for Spatial Domain Identification"},"authors":{"value":["Xianglong Meng","Kai Hu","Xuefeng Cui","Fa Zhang"]}},"tmdate":1772967338082,"pdate":1767139200000,"externalIds":["dblp:conf/isbra/MengHCZ25"],"tcdate":1772967334941,"writers":["~"],"signatures":["~Kai_Hu6"],"forum":"a9fWrtlxh9","license":"CC BY-SA 4.0","number":849807,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1772967338082,"domain":"DBLP.org","id":"a9fWrtlxh9","version":2},{"content":{"summary":{"value":"This paper introduces Decision Mamba, an offline RL backbone based on State Space Models. It enhances policy robustness by integrating a fine-grained SSM module alongside the original coarse-grained SSM in Mamba. Meanwhile, it adopts a progressive self-evolution regularization to prevent the policy from overfitting the noisy labels. Extensive experiments against a broad baseline demonstrate its effectiveness.\n\n-----------Post Rebuttal-----------------\n\nMain concerns resolved. Update rating from BA to WC."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- Table 3 indicates a significant impact of PSER on performance. Given its compatibility with other proposed baselines, it would be worth testing whether adding that into them would improve their performance results.\n- The context length in Mujoco environments is relatively short, minimizing the need for extensive historical temporal information. Can you provide experimental data on Atari environments to assess DM's capability in handling longer contextual sequences?\n- More experiments on Multi-Grained SSM modules are requested to bolster the supporting evidence for their effectiveness."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- Mamba have shown good performance across NLP and CV tasks and applying Mamba to offline reinforcement learning is an interesting direction.\n- The paper conducts a comprehensive comparison with SOTA offline RL methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The advantages of DM appear somewhat constrained when considering variance. It appears that performance enhancements stem more from the PSER rather than the Mamba architecture. Combining PSER with other architectures, such as transformers, could potentially yield superior performance outcomes.\n- In the experiments investigating various context lengths, the range examined is rather restricted, and there appears to be no notable difference in performance as the context length increases in either DT or DM. It may be beneficial to conduct experiments using longer context lengths to assess potential performance variations.\n- It is not appropriate to assume that readers are very familiar with Mamba related terminology. Enhancing the methodology section with additional details to explain such terms would improve readability. For instance, the term \"group branch\" used in the ablation experiments lacks an earlier clear definition within the text."},"limitations":{"value":"Unable to surpass the performance of the behavior policy, the model performs poorly on a suboptimal dataset."}},"nonreaders":[],"tmdate":1730879623198,"tcdate":1720680102131,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission13292/Reviewer_K8tX"],"signatures":["NeurIPS.cc/2024/Conference/Submission13292/Reviewer_K8tX"],"forum":"dc4xbVfdzy","number":4,"license":"CC BY 4.0","cdate":1720680102131,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission13292/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879623198,"domain":"NeurIPS.cc/2024/Conference","replyto":"dc4xbVfdzy","id":"g7WhDKTNFZ","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"TLDR":{"value":"We propose Decision Mamba tailored for the offline RL tasks, a novel multi-grained state space model with a self-evolving policy learning strategy."},"keywords":{"value":["Decision Transformer","Mamba","Offline Reinforcement Learning","Autoregressive Sequence Modeling","State Space Model"]},"primary_area":{"value":"reinforcement_learning"},"abstract":{"value":"While the conditional sequence modeling with the transformer architecture has demonstrated its effectiveness in dealing with offline reinforcement learning (RL) tasks, it is struggle to handle out-of-distribution states and actions.\nExisting work attempts to address this issue by data augmentation with the learned policy or adding extra constraints with the value-based RL algorithm. However, these studies still fail to overcome the following challenges: (1) insufficiently utilizing the historical temporal information among inter-steps, (2) overlooking the local intra-step relationships among return-to-gos (RTGs), states, and actions, (3) overfitting suboptimal trajectories with noisy labels. To address these challenges, we propose $\\textbf{D}$ecision $\\textbf{M}$amba ($\\textbf{DM}$), a novel multi-grained state space model (SSM) with a self-evolving policy learning strategy.\nDM explicitly models the historical hidden state to extract the temporal information by using the mamba architecture. To capture the relationship among RTG-state-action triplets, a fine-grained SSM module is designed and integrated into the original coarse-grained SSM in mamba, resulting in a novel mamba architecture tailored for offline RL. Finally, to mitigate the overfitting issue on noisy trajectories, a self-evolving policy is proposed by using progressive regularization. The policy evolves by using its own past knowledge to refine the suboptimal actions, thus enhancing its robustness on noisy demonstrations. Extensive experiments on various tasks show that DM outperforms other baselines substantially."},"_bibtex":{"value":"@inproceedings{\nlv2024decision,\ntitle={Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline {RL}},\nauthor={Qi Lv and Xiang Deng and Gongwei Chen and Michael Y Wang and Liqiang Nie},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=dc4xbVfdzy}\n}"},"title":{"value":"Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL"},"pdf":{"value":"/pdf/91aa3a5ea6196f252d1ae3ffa1ca9fe7e239e4ee.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"lv|decision_mamba_a_multigrained_state_space_model_with_selfevolution_regularization_for_offline_rl"},"authorids":{"value":["~Qi_Lv1","~Xiang_Deng1","~Gongwei_Chen1","~Michael_Y_Wang1","~Liqiang_Nie2"]},"authors":{"value":["Qi Lv","Xiang Deng","Gongwei Chen","Michael Y Wang","Liqiang Nie"]}},"version":2},{"content":{"summary":{"value":"The paper introduces Mixture-of-Mamba (MoM), a modality-aware extension of Mamba that applies structured sparsity inside the Mamba block rather than only in surrounding MLPs. Concretely, the core state-space projections are decoupled by modality so that only a subset is active per token/modality. Experiments span Transfusion, Chameleon, and a three-modality (text–image–speech) setup, with reports that MoM reaches comparable quality in ~2–2.5× fewer training steps at up to 1.5B parameters."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Regarding the modality-aware parameterization, are the three per-modality weight sets parallel (i.e., disjoint block partitions of the same projection), or is there a learned mask over a shared matrix that discovers the split?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"1. This method is effective for decoupling mamba blocks through different modality. \n\n2. Broad evaluation across multiple multi-modal pretraining regimes. Results suggest faster convergence in training steps versus dense Mamba and a flex-attention Transformer baseline."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Missing head-to-head comparisons with MoE-Mamba and BlackMamba under matched parameter and compute budgets. Without these, the claimed superiority or practical advantage is not fully substantiated.\n\n2. Limited theoretical/diagnostic analysis explaining why performance depends on input modalities. A testable hypothesis is that specific subsets of parameters specialize to each modality; the paper does not analyze this.\n\n3. The contribution is quite limited and does not meet the standard of publishing at ICLR, espcially considering the originality of the idea, which mainly extends Mixture-of-Transformers [1] to Mamba. \n\n[1] Liang, Weixin, et al. \"Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.\" TMLR (2025)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942792466,"tcdate":1761924286035,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23760/Reviewer_dos5"],"signatures":["ICLR.cc/2026/Conference/Submission23760/Reviewer_dos5"],"forum":"Valt8gMdfl","number":4,"license":"CC BY 4.0","cdate":1761924286035,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23760/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942792466,"domain":"ICLR.cc/2026/Conference","replyto":"Valt8gMdfl","id":"WQaC0IK6n1","forumContent":{"TLDR":{"value":"How can we reduce pretraining costs for multi-modal models without sacrificing quality? We introduce Mixture-of-Mamba, a sparse State Space Model architecture that achieves dense-level performance with up to 75% fewer FLOPs across multi-modal tasks."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Sparse architecture","Efficient deep architecture","Multi-modal foundation models","Mixture-of-Experts","State Space Model"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"State Space Models (SSMs) have emerged as efficient alternatives to Transformers for sequential modeling, but their inability to leverage modality-specific features limits their performance in multi-modal pretraining. Here, we propose Mixture-of-Mamba, a novel SSM architecture that introduces modality-aware sparsity through modality-specific parameterization of the Mamba block. We evaluate Mixture-of-Mamba across three multi-modal pretraining settings: Transfusion (interleaved text and continuous image tokens with diffusion loss), Chameleon (interleaved text and discrete image tokens), and an extended three-modality framework incorporating speech. Mixture-of-Mamba consistently reaches the same loss values at earlier training steps with significantly reduced computational costs. In the Transfusion setting, Mixture-of-Mamba achieves equivalent image loss using only 34.76% of the training FLOPs at the 1.4B scale. In the Chameleon setting, Mixture-of-Mamba reaches similar image loss with just 42.50% of the FLOPs at the 1.4B scale, and similar text loss with just 65.40% of the FLOPs. In the three-modality setting, MoM matches speech loss at 24.80% of the FLOPs at the 1.4B scale. Our ablation study highlights the synergistic effects of decoupling projection components, where joint decoupling yields greater gains than individual modifications. These results establish modality-aware sparsity as a versatile and effective design principle, extending its impact from Transformers to SSMs and setting new benchmarks in multi-modal pretraining."},"_bibtex":{"value":"@misc{\nliang2025mixtureofmamba,\ntitle={Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity},\nauthor={Weixin Liang and Junhong Shen and Genghan Zhang and Ning Dong and Mengjie Xie and Luke Zettlemoyer and Lili Yu},\nyear={2025},\nurl={https://openreview.net/forum?id=Valt8gMdfl}\n}"},"title":{"value":"Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity"},"pdf":{"value":"/pdf/e9862fc43f84d67dc434b0cbd4a96c699ae2f77a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"liang|mixtureofmamba_enhancing_multimodal_statespace_models_with_modalityaware_sparsity"},"authorids":{"value":["~Weixin_Liang1","~Junhong_Shen1","~Genghan_Zhang1","~Ning_Dong2","~Mengjie_Xie1","~Luke_Zettlemoyer1","~Lili_Yu2"]},"authors":{"value":["Weixin Liang","Junhong Shen","Genghan Zhang","Ning Dong","Mengjie Xie","Luke Zettlemoyer","Lili Yu"]}},"version":2},{"content":{"summary":{"value":"This paper propose Vector-Quantized Continual Diffuser (VQ-CD) for any offline continual reinforcement learning tasks, including the tasks with different observation and action space. Specifically, quantized spaces alignment (QSA) module aligns different state and action spaces (with even different dimensions) to perform continual learning within the same space. Selective weights activation diffuser (SWA) module preserves the previous knowledge by separating task-related parameters with task-related masking. VQ-CD achieves SOTA performance on 15 continual tasks comparing with 16 baselines."},"soundness":{"value":1},"confidence":{"value":3},"questions":{"value":"1. What is the \"diffusion-based lifelong learning systems\" in Abstract?\n2. Figure 1 appears somewhat cluttered and hard to understand. Labeling the sequence within the figure would enhance its clarity."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"1. Traditional continual reinforcement learning approaches are significantly limited as they only consider tasks with consistent state and action spaces. This paper successfully addresses the challenge of tasks with diverse state and action spaces through the innovative use of a quantized spaces alignment module. \n2. The baselines compared in the paper are thorough and comprehensive, and the experimental environments considered are sufficiently diverse. These aspects robustly demonstrate the superiority of VQ-CD."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper appears to lack a significant amount of crucial details, which hinders my understanding of the algorithms presented. \n1. The paper does not clarify how the 'return' is utilized. Based on Section 3.2, I guess that the algorithm employs the 'return' as a condition. So, is it the 'return' or the Q-function that is being used? Additionally, how is the target return or target function determined during the inference phase? \n2. Within the QSA module, which parameters are shared, and which need to be retrained for different tasks?"}},"nonreaders":[],"tmdate":1731428441727,"tcdate":1730691626788,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5556/Reviewer_ch1d"],"signatures":["ICLR.cc/2025/Conference/Submission5556/Reviewer_ch1d"],"forum":"Hr3TBaZl4S","number":3,"license":"CC BY 4.0","cdate":1730691626788,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5556/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428441727,"domain":"ICLR.cc/2025/Conference","replyto":"Hr3TBaZl4S","id":"kWET8z5dLa","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We propose to solve continual offline RL with various state and action spaces by adopting selective weights activation on the quantized alignment spaces."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["continual offline reinforcement learning"]},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based continual learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space. Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 16 baselines, our method reaches the SOTA performance."},"_bibtex":{"value":"@misc{\nhu2025solving,\ntitle={Solving Continual Offline {RL} through Selective Weights Activation on Aligned Spaces},\nauthor={Jifeng Hu and Sili Huang and Li Shen and Zhejian Yang and Shengchao Hu and Shisong Tang and Hechang Chen and Yi Chang and Dacheng Tao and Lichao Sun},\nyear={2025},\nurl={https://openreview.net/forum?id=Hr3TBaZl4S}\n}"},"title":{"value":"Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces"},"pdf":{"value":"/pdf/19ade0bc8e9c98b3410f06b22f7f2a308a2ed7fe.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"hu|solving_continual_offline_rl_through_selective_weights_activation_on_aligned_spaces"},"authorids":{"value":["~Jifeng_Hu1","~Sili_Huang1","~Li_Shen1","~Zhejian_Yang1","~Shengchao_Hu1","~Shisong_Tang1","~Hechang_Chen2","~Yi_Chang4","~Dacheng_Tao1","~Lichao_Sun1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jifeng Hu","Sili Huang","Li Shen","Zhejian Yang","Shengchao Hu","Shisong Tang","Hechang Chen","Yi Chang","Dacheng Tao","Lichao Sun"]}},"version":2},{"content":{"comment":{"value":"### Thank you for acknowledging the contributions of our work. We are thrilled and grateful for your insightful feedback, which has significantly contributed to enhancing the quality of our manuscript. In the following responses, **W** and **Q** represent Weaknesses and Questions, and **A** represents the corresponding answer.\n\n\n**W1**.  On the ablation study and the contribution of individual components.\n\n**A1**:  We appreciate the reviewer’s comments and have expanded the ablation study accordingly. The revised version now includes explicit ablations of the global descriptor $G$, the “distillation” path, and the self-attention layer, in addition to previously reported components (**please refer to lines 471 to 474 on Page 9**). These new results confirm that each module contributes to performance, and we have added standard deviations for clarity. We also renamed “distillation path” to \"*embedding path*\"  to avoid misunderstanding.\n\n**W2**. On the concern about spatial ordering and 3D cortical topology.\n\n**A2**:  Our model operates on preprocessed BOLD time series of shape (ROI × time), following standard neuroimaging pipelines [https://fmriprep.org/en/stable/]. Because the FC matrices are computed from region-wise time series rather than surface meshes or voxel grids, no 3D topology is available to the model or to baseline methods. Thus, the 1D sliding-window over ROIs is consistent with existing FC-based learning settings. We have clarified this point in the revision (**please refer to lines 333 to 335 on Page 7**).\n\n**W3**. On fairness of baseline adaptation and the input format.\n\n**A3**: Thanks for your valuable comment. For the Euclidean sequence baselines (Transformer, Mamba), the input was the **vectorized FC matrices** derived from Pearson-correlation of BOLD time series, following the standard practice in sequence modeling for FC-based representations. These methods were included because they represent state-of-the-art architectures for modeling temporal dependencies in complex sequences, and their performance provides an important reference for evaluating the effectiveness of our approach. Beyond these Euclidean models, we also compared against **graph-based brain network analysis methods** (e.g., BrainGNN, NeuroGraph, Contrast-Pool, and STAGIN), which explicitly model topological structure, as well as **manifold-based approaches** (e.g., SPDNet) that operate natively on non-Euclidean spaces. This comprehensive baseline selection spans methods from conventional Euclidean modeling to graph-structured and Riemannian manifold learning, ensuring that the comparison is both fair and informative.\n\nWe have now added detailed descriptions of the input pipelines and data formatting strategies for all baselines (**please refer to lines 340 to 349 on Page 7**). This clarification ensures that performance differences can be interpreted in terms of both input design and geometric modeling, allowing a more accurate assessment of the contribution of manifold-aware learning in Geo-Mamba.\n\n**W4**. On SSM layers and hierarchical spatial modeling.\n\n**A4**:  We thank the reviewer for pointing this out and in our Stacked Path, the architecture comprises one SSM$_1$ layer, one self-attention module, and dSSM$_1$ layer(2-layers), resulting in a total of four layers (**please refer to line 298 on Page 6**). For the Embedding Path, the number of compression layers is determined by the hyperparameter down_dim. In the current implementation, we adopted a four-layer configuration (**please refer to lines 311 to 312 on Page 6**). This design ensures sufficient capacity to capture multi-scale spatial dependencies while maintaining model efficiency. The revision also includes clarification on how stacking improves the representation capacity. Specifically, this is supported by the ablation studies, which demonstrate the individual contributions of SSM$_1$, the self-attention module, and dSSM$_1$ layers to the overall performance (**please refer to lines 471 to 474 on Page 9**).\n\n**W5**. On Bt, Ct, λt and their relation to Mamba’s selective parameterization.\n\n**A5**:  We thank the reviewer for the comment.  At each step $i$, the SPD input is first mapped to the Euclidean tangent space via the logarithmic map, and the resulting vectorized representation is processed by a lightweight GRU.  The GRU output is then passed through three linear layers to produce the gating signal $\\Delta_i$ and the selective operators $B_i$ and $C_i$, selective operators are projected back to the SPD manifold via Stiefel retraction to ensure orthonormality.  Unlike Mamba, which operates purely in Euclidean space, our GRU evolves in the tangent space and generates geometry-compatible operators, explicitly linking gating with SPD-preserving state updates (**please refer to lines 222 to 233 on Page 5**)."},"title":{"value":"Respone to Reviewer aJYZ"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763715787992,"tcdate":1763715787992,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11691/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission11691/Authors"],"forum":"7XonwHRkTS","number":4,"license":"CC BY 4.0","cdate":1763715787992,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11691/-/Official_Comment"],"mdate":1763715787992,"domain":"ICLR.cc/2026/Conference","replyto":"PNKlaxX9zU","id":"Xf4ZCn0l6d","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Brain functional connectivity","Riemannian geometry","State space model","Geometric deep learning"]},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Functional magnetic resonance imaging (fMRI)–derived functional connectivity (FC) is represented as graphs and as correlation/covariance matrices that live on non-Euclidean spaces—cortical graphs and the Riemannian manifold of symmetric positive-definite (SPD) matrices—so conventional Euclidean sequence models are misspecified. To this end, we introduce *Geo-Mamba*, a geometric variant of Mamba formulated on Riemannian manifolds. *Geo-Mamba* employs a dual-path selective state-space design: *a stacked path* performs hierarchical spatial modeling by aggregating pyramid multi-scale features to capture local and global dependencies, while *a embedding path* combats redundancy in high-dimensional SPD inputs via progressive, geometry-aware dimensionality reduction (operating in the appropriate manifold spaces) to produce compact states without violating Riemannian constraints. Their complementary outputs are fused through the tailored *GeoMix* operator to yield a compact, discriminative SPD representation. *Geo-Mamba* is evaluated on six public fMRI datasets—ADNI, OASIS, PPMI, Taowu, Neurocon, and Mātai—spanning Alzheimer’s and Parkinson’s cohorts as well as multi-site normative populations with diverse acquisition protocols. Across these benchmarks, it delivers consistently competitive accuracy and robustness, supporting the value of dual-path manifold modeling for neuroimaging and its potential for clinical translation."},"_bibtex":{"value":"@misc{\ncao2026geomamba,\ntitle={Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics},\nauthor={Yuwei Cao and Tingting Dan and Yang Yang and Guorong Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=7XonwHRkTS}\n}"},"title":{"value":"Geo-Mamba: Dual-Path Riemannian State-Space Models for Functional Dynamics"},"pdf":{"value":"/pdf/1ce491d063b410272cb740d9493deafc822b3586.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cao|geomamba_dualpath_riemannian_statespace_models_for_functional_dynamics"},"authorids":{"value":["~Yuwei_Cao3","~Tingting_Dan1","~Yang_Yang12","~Guorong_Wu1"]},"authors":{"value":["Yuwei Cao","Tingting Dan","Yang Yang","Guorong Wu"]}},"version":2},{"content":{"summary":{"value":"Linear models have proven effective in time series forecasting due to their capacity to capture and leverage the linear relationships inherent in many time series datasets. The challenge is to develop a powerful prediction model that retains the high efficiency of linear models. The authors develop an LLM-enhanced Mixture-of-Linear-Experts (MoLE) for time series forecasting. The authors mention that this is the first work on improving linear time series models based on mixture-of-expert learning and multimodal learning. Compared to several recent state-of-the-art prediction networks on long-term forecasting and few-shot tasks, this method demonstrates the effectiveness of this proposed LeMoLE in terms of accuracy and efficiency."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. LeMoLE can process the global and local text data, can you please show some examples of how well LeMoLE can integrate the prompt text with time series data?\n2. What is $H$ in Eq.1?\n3. What is the architecture of CNN in Figure 2?\n4. What is the limitation of this work?\n5. What is the difference between LeMoLE-F and LeMoLE-T in processing the input features?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. By integrating LLM with linear experts, especially multiple linear experts on time series forecasting tasks, LeMoLE achieve a new sota compared to existing LLM-based time series forecasting models;\n2. This study is the first work that uses linear time series models based on mixture-of-expert learning and multimodal learning."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"This article looks incomplete.\n\n1. The novelty is not strong enough. This architecture roughly assembles LLMs into a linear layer, and there is no clear presentation of the used CNN structure. It would be better to theoretically discuss how and why CNN can overperform other architecture and to provide evidence on how a lightweight CNN can fuse the linear features.\n2. Some typos and errors should be revised. For example, in Eq. 3, $b_{i}$ should be $b_{m}$. \"Based on Equation (??)\" should be revised.  \n3. The presentation sounds ambiguous in some sessions. For example, LeMoLE mentioned the use of varying window lengths, but in this article, there is no detail on the CNN when setting the range of $T$. Especially, when $T=96$ and the number of expert is $5$, the convolutional block should be $3 \\times 3$ or $7 \\times 7$? Or other settings?\n4. A lack of visualization results. For example, comparisons between time-based and frequency-based LeMoLE outputs, or visualizations of how different window lengths affect the results.\n5. Finally, the results are not consistent to the experiment settings. For example, the length of $T$ is not consistent to the $H$ of Table 2 and Table 3. Please double check if this is a typo."}},"nonreaders":[],"tmdate":1731429139800,"tcdate":1730600924911,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8663/Reviewer_hK8o"],"signatures":["ICLR.cc/2025/Conference/Submission8663/Reviewer_hK8o"],"forum":"ERBm5WK8nq","number":3,"license":"CC BY 4.0","cdate":1730600924911,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8663/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429139800,"domain":"ICLR.cc/2025/Conference","replyto":"ERBm5WK8nq","id":"DyrOCbUFVa","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["time series"]},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent research has shown that large language models (LLMs) can be effectively used for real-world time series forecasting due to their strong natural language understanding capabilities. However, aligning time series into semantic spaces of LLMs comes with high computational costs and inference complexity, particularly for long-range time series generation. Building on recent advancements in using linear models for time series, this paper introduces an LLM-enhanced mixture of linear experts for precise and efficient time series forecasting. This approach involves developing a mixture of linear experts with multiple lookback lengths and a new multimodal fusion mechanism. The use of a mixture of linear experts is efficient due to its simplicity, while the multimodal fusion mechanism adaptively combines multiple linear experts based on the learned features of the text modality from pre-trained large language models. In experiments, we rethink the need to align time series to LLMs by existing time-series large language models and further discuss their efficiency and effectiveness in time series forecasting. Our experimental results show that the proposed LeMoLE model presents lower prediction errors and higher computational efficiency than existing LLM models."},"_bibtex":{"value":"@misc{\nzhang2024lemole,\ntitle={LeMo{LE}: {LLM}-enhanced Mixture of Linear Experts for Time Series Forecasting},\nauthor={Lingzheng Zhang and Yimin Zheng and Lifeng Shen and Shiyuan Piao and Ziyue Li and Fugee Tsung},\nyear={2024},\nurl={https://openreview.net/forum?id=ERBm5WK8nq}\n}"},"title":{"value":"LeMoLE: LLM-enhanced Mixture of Linear Experts for Time Series Forecasting"},"pdf":{"value":"/pdf/733b799cf5e0d14449075a1946e6f24b38973ec6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhang|lemole_llmenhanced_mixture_of_linear_experts_for_time_series_forecasting"},"authorids":{"value":["~Lingzheng_Zhang1","~Yimin_Zheng1","~Lifeng_Shen1","~Shiyuan_Piao1","~Ziyue_Li2","~Fugee_Tsung1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Lingzheng Zhang","Yimin Zheng","Lifeng Shen","Shiyuan Piao","Ziyue Li","Fugee Tsung"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2510.18684v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"ali|mlma_towards_multilingual_asr_with_mambabased_architectures"},"html":{"value":"https://doi.org/10.48550/arXiv.2510.18684"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2510-18684,\n  publtype={informal},\n  author={Mohamed Nabih Ali and Daniele Falavigna and Alessio Brutti},\n  title={MLMA: Towards Multilingual ASR With Mamba-based Architectures},\n  year={2025},\n  month={October},\n  cdate={1759276800000},\n  journal={CoRR},\n  volume={abs/2510.18684},\n  url={https://doi.org/10.48550/arXiv.2510.18684}\n}\n"},"abstract":{"value":"Multilingual automatic speech recognition (ASR) remains a challenging task, especially when balancing performance across high- and low-resource languages. Recent advances in sequence modeling suggest that architectures beyond Transformers may offer better scalability and efficiency. In this work, we introduce MLMA (Multilingual Language Modeling with Mamba for ASR), a new approach that leverages the Mamba architecture -- an efficient state-space model optimized for long-context sequence processing -- for multilingual ASR. Using Mamba, MLMA implicitly incorporates language-aware conditioning and shared representations to support robust recognition across diverse languages. Experiments on standard multilingual benchmarks show that MLMA achieves competitive performance compared to Transformer-based architectures. These results highlight Mamba's potential as a strong backbone for scalable, efficient, and accurate multilingual speech recognition."},"title":{"value":"MLMA: Towards Multilingual ASR With Mamba-based Architectures"},"authors":{"value":[{"fullname":"Mohamed Nabih Ali","username":"~Mohamed_Nabih1"},{"fullname":"Daniele Falavigna","username":""},{"fullname":"Alessio Brutti","username":""}]}},"tmdate":1779733552830,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2510-18684"],"tcdate":1779733549051,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Mohamed_Nabih1"],"forum":"gUkhctxbaE","license":"CC BY-SA 4.0","number":24564,"cdate":1759276800000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1779733552830,"domain":"OpenReview.net/Public_Article","id":"gUkhctxbaE","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["test-time training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state. Inspired by prior work, we present a practical framework to instantiate sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Both TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer, they can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. With preliminary systems optimization, TTT-Linear is already faster than Transformer at 8k context and matches Mamba in wall-clock time. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research."},"_bibtex":{"value":"@misc{\nsun2025learning,\ntitle={Learning to (Learn at Test Time): {RNN}s with Expressive Hidden States},\nauthor={Yu Sun and Xinhao Li and Karan Dalal and Jiarui Xu and Arjun Vikram and Genghan Zhang and Yann Dubois and Xinlei Chen and Xiaolong Wang and Sanmi Koyejo and Tatsunori Hashimoto and Carlos Guestrin},\nyear={2025},\nurl={https://openreview.net/forum?id=eifW0W0xgt}\n}"},"title":{"value":"Learning to (Learn at Test Time): RNNs with Expressive Hidden States"},"pdf":{"value":"/pdf/be1b3f1cf37ec5812fb041e8bbdd1569d92a9992.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sun|learning_to_learn_at_test_time_rnns_with_expressive_hidden_states"},"authorids":{"value":["~Yu_Sun1","~Xinhao_Li3","~Karan_Dalal1","~Jiarui_Xu1","~Arjun_Vikram1","~Genghan_Zhang1","~Yann_Dubois1","~Xinlei_Chen1","~Xiaolong_Wang3","~Sanmi_Koyejo1","~Tatsunori_Hashimoto1","~Carlos_Guestrin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yu Sun","Xinhao Li","Karan Dalal","Jiarui Xu","Arjun Vikram","Genghan Zhang","Yann Dubois","Xinlei Chen","Xiaolong Wang","Sanmi Koyejo","Tatsunori Hashimoto","Carlos Guestrin"]}},"tmdate":1738735709013,"tcdate":1727309236042,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5310/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5310/Authors"],"forum":"eifW0W0xgt","license":"CC BY 4.0","number":5310,"cdate":1727309236042,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission5310/-/Full_Submission","ICLR.cc/2025/Conference/Submission5310/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735709013,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"eifW0W0xgt","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.12671v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"anand|denseperceptncssd_vision_mamba_towards_realtime_dense_visual_perception_with_noncausal_state_space_duality"},"html":{"value":"https://doi.org/10.48550/arXiv.2511.12671"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-12671,\n  publtype={informal},\n  author={Tushar Anand and Advik Sinha and Abhijit Das},\n  title={DensePercept-NCSSD: Vision Mamba towards Real-time Dense Visual Perception with Non-Causal State Space Duality},\n  year={2025},\n  month={November},\n  cdate={1761955200000},\n  journal={CoRR},\n  volume={abs/2511.12671},\n  url={https://doi.org/10.48550/arXiv.2511.12671}\n}\n"},"abstract":{"value":"In this work, we propose an accurate and real-time optical flow and disparity estimation model by fusing pairwise input images in the proposed non-causal selective state space for dense perception tasks. We propose a non-causal Mamba block-based model that is fast and efficient and aptly manages the constraints present in a real-time applications. Our proposed model reduces inference times while maintaining high accuracy and low GPU usage for optical flow and disparity map generation. The results and analysis, and validation in real-life scenario justify that our proposed model can be used for unified real-time and accurate 3D dense perception estimation tasks. The code, along with the models, can be found at https://github.com/vimstereo/DensePerceptNCSSD"},"title":{"value":"DensePercept-NCSSD: Vision Mamba towards Real-time Dense Visual Perception with Non-Causal State Space Duality"},"authors":{"value":[{"fullname":"Tushar Anand","username":""},{"fullname":"Advik Sinha","username":"~Advik_Sinha1"},{"fullname":"Abhijit Das","username":""}]}},"tmdate":1781847019424,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2511-12671"],"tcdate":1781847010975,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Advik_Sinha1"],"forum":"I3IN6aSDKL","license":"CC BY-SA 4.0","number":43746,"cdate":1761955200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1781847019424,"domain":"OpenReview.net/Public_Article","id":"I3IN6aSDKL","version":2},{"content":{"research_area_keywords":{"value":"generative models, word embeddings, representation learning"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2025 May Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025ts,\ntitle={{TS}4: Tensorized Structured State Space Sequence Models},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - May 2025},\nyear={2025},\nurl={https://openreview.net/forum?id=YrhAMWrH19},\nnote={under review}\n}"},"title":{"value":"TS4: Tensorized Structured State Space Sequence Models"},"abstract":{"value":"Recently, structured state space sequence (S4) models have generated considerable interest due to their simplicity and favorable performance compared to the transformer architecture in certain sequence modeling tasks. A very important property that distinguishes these models from traditional gated RNNs is the linear dependence of the model output on the latent space vector at each time step, even when an input dependent selection mechanism is incorporated. This means that the computation underlying inference and sequence mapping in these models involves linear time evolution of the latent space vector. Inspired by long standing studies of time evolution of matrix product states in quantum mechanics, we study the problem of compressing the latent space of sequence models using tensorization methods. Such tensorized sequence models, we call TS4. Various novel structures on the parameters of S4 models within the tensorization setting are imposed to propose new classes of structured sequence models."},"paper_type":{"value":"Short"},"pdf":{"value":"/pdf/9b101f604409bda2143015e683a842b15cc7bd33.pdf"},"research_area":{"value":"Machine Learning for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/May/Submission"}},"tmdate":1782962221647,"tcdate":1747733937433,"writers":["aclweb.org/ACL/ARR/2025/May","aclweb.org/ACL/ARR/2025/May/Submission6441/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/May/Submission6441/Authors"],"forum":"YrhAMWrH19","license":"CC BY 4.0","number":6441,"cdate":1747733937433,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/May/-/Submission","aclweb.org/ACL/ARR/2025/May/-/Edit","aclweb.org/ACL/ARR/2025/May/-/Post_Submission","aclweb.org/ACL/ARR/2025/May/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2025/May/-/Preprint_Post_Submission"],"mdate":1782962221647,"odate":1753766020949,"domain":"aclweb.org/ACL/ARR/2025/May","id":"YrhAMWrH19","version":2},{"content":{"summary":{"value":"The paper studies whether bidirectionality is necessary for Mamba in multivariate time series forecasting. Since channels lack natural order, applying bidirectional Mamba for channel dependencies (CD) adds complexity. The authors propose FSMamba, a unidirectional model with a regularization term that minimizes the distance between embeddings of reversed channel orders and a channel similarity modeling (CSM) pretraining step. Experiments on multiple datasets show FSMamba achieves comparable or better performance than bidirectional models with fewer parameters, proving that bidirectionality is unnecessary for effective CD modeling."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. The paper proposes a regularization method that minimizes the distance between two embedding vectors generated with reversed channel orders. However, this seems to prevent Mamba, as a sequence modeling method, from capturing sequential information. I do not understand the rationale behind this design. The final architecture appears equivalent to using Selective SSM to capture CD. Please explain why this design is necessary and why it works.\n\n2. Since the existing benchmarks contain too few dimensions, to further validate FSMamba’s capability in modeling CD, it would be helpful to include experiments on some datasets from Time-HD benchmark [1] and Wike2000 from TFB [2].\n\n[1] Ni, J., Wang, S., Liu, Z., Shi, X., Zhong, X., Ye, Z., & Jin, W. (2025). U-Cast: Learning Hierarchical Structures for High-Dimensional Time Series Forecasting. arXiv preprint arXiv:2507.15119.\n[2] Qiu, X., Hu, J., Zhou, L., Wu, X., Du, J., Zhang, B., ... & Yang, B. (2024). Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods. arXiv preprint arXiv:2403.20150"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":2},"strengths":{"value":"1. Clear and easy-to-understand writing with clean and visually appealing figures.\n2. Detailed experimental analysis"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Since Mamba itself is designed for TD, the motivation for applying it to CD is not well justified.\n2. The evaluation of Mamba’s modeling capability on CD is insufficient."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942879968,"tcdate":1761842511874,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23977/Reviewer_P4ub"],"signatures":["ICLR.cc/2026/Conference/Submission23977/Reviewer_P4ub"],"forum":"N6MIqE11eF","number":1,"license":"CC BY 4.0","cdate":1761842511874,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23977/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942879968,"domain":"ICLR.cc/2026/Conference","replyto":"N6MIqE11eF","id":"8ht4NKW9aq","forumContent":{"TLDR":{"value":"We propose FSMamba, a time series forecasting method that uses Mamba with regularization strategy to address the sequential order bias in capturing channel dependencies."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Time Series Forecasting","Mamba"]},"supplementary_material":{"value":"/attachment/d0aef3ef8995051541105dd0e448b12055805dee.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Mamba is a sequential model that has recently emerged as a promising alternative to Transformers, offering near-linear complexity.\nHowever, although channels in time series (TS) data generally lack a sequential order, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this, prior works have adopted bidirectional Mamba to scan channels in both forward and reverse orders. In this paper, we show that unidirectional Mamba can effectively replace the bidirectional Mamba with simple strategies. To this end, we propose FSMamba, a TS forecasting method employing a unidirectional Mamba that incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order. Furthermore, we introduce channel similarity modeling, a pretraining task to preserve similarities between channels from the data space to the latent space to enhance the ability to capture CD. Extensive experiments demonstrate the efficacy of our method, achieving state-of-the-art performance on diverse datasets."},"_bibtex":{"value":"@misc{\nlee2026is,\ntitle={Is Bidirectionality Necessary in Mamba for Time Series Forecasting?},\nauthor={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\nyear={2026},\nurl={https://openreview.net/forum?id=N6MIqE11eF}\n}"},"title":{"value":"Is Bidirectionality Necessary in Mamba for Time Series Forecasting?"},"pdf":{"value":"/pdf/bf428f2b7b338e6732b17e2ed317a213028a6cfe.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"lee|is_bidirectionality_necessary_in_mamba_for_time_series_forecasting"},"authorids":{"value":["~Seunghan_Lee1","~Juri_Hong1","~Kibok_Lee1","~Taeyoung_Park1"]},"authors":{"value":["Seunghan Lee","Juri Hong","Kibok Lee","Taeyoung Park"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2607.21000v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"lim|naju_a_native_discrete_statespace_model_with_independent_retention_and_writing_for_longsequence_memory"},"html":{"value":"https://doi.org/10.48550/arXiv.2607.21000"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2607-21000,\n  publtype={informal},\n  author={Hyuk Lim and Seunghyun Yoon},\n  title={Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory},\n  year={2026},\n  month={July},\n  cdate={1782864000000},\n  journal={CoRR},\n  volume={abs/2607.21000},\n  url={https://doi.org/10.48550/arXiv.2607.21000}\n}\n"},"abstract":{"value":"Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones. In our diagnostic suite, the strongest efficient baselines tend to solve only one side well. Continuous-time-parameterized state-space models (SSMs) such as Mamba obtain their discrete recurrence by zero-order-hold discretization of a continuous-time system; we argue that this detour is unnecessary for memory tracking and parameterize the discrete transition directly. Naju (Native Adaptive Junction Unit) factorizes the recurrent update, schematically $x_n = f_n\\odot x_{n-1} + i_n\\odot(B_n u_n)$, into an explicit discrete pole (a learned forget gate $f_n$), an independent write gain $i_n$, and input-dependent write/read maps. Since the sigmoid pole satisfies $0<f_n<1$, each frozen local coordinate is Schur-stable by construction, and the full time-varying recurrence satisfies a fading-memory/BIBO bound under uniform boundedness assumptions, with no stability regularizer. We formalize the key structural limitation of coupled designs: any non-expansive complementary single-gate recurrence ties the effective retention $r$ and write gain $w$ through $|r|+w\\le 1$, so near-complete retention forces weak writing; decoupling $f_n$ from $i_n$ removes this constraint. Empirically, Naju is the only evaluated model that remains strong on both retention and overwriting at 4x the training length. Beyond the diagnostic suite, we evaluate Naju on WikiText-103 language modeling, Long Range Arena, and multi-query associative recall. Across these settings, Naju consistently combines strong long-range memory with competitive or superior performance, outperforming the Mamba baselines in the principal comparisons while remaining competitive with the Transformer and preserving linear-time, linear-memory scaling."},"title":{"value":"Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory"},"authors":{"value":[{"fullname":"Hyuk Lim","username":""},{"fullname":"Seunghyun Yoon","username":"~Seunghyun_Yoon2"}]}},"tmdate":1789879319808,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2607-21000"],"tcdate":1789879308378,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Seunghyun_Yoon2"],"forum":"qlOAlrX4Hv","license":"CC BY-SA 4.0","number":151766,"cdate":1782864000000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1789879319808,"domain":"OpenReview.net/Public_Article","id":"qlOAlrX4Hv","version":2},{"content":{"research_area_keywords":{"value":"Quantization,Mamba,LLMs"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2025 February Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025qmamba,\ntitle={Q-Mamba: Towards more efficient Mamba models via post-training quantization},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - February 2025},\nyear={2025},\nurl={https://openreview.net/forum?id=DxPTm7Bxmh},\nnote={under review}\n}"},"title":{"value":"Q-Mamba: Towards more efficient Mamba models via post-training quantization"},"contribution_types":{"value":["Approaches to low-resource settings","Approaches low compute settings-efficiency"]},"abstract":{"value":"State Space Models (SSMs), such as Mamba, have recently demonstrated potential in language understanding tasks, positioning them as competitors to transformer architectures. However, our investigations reveal that the Mamba architecture still has room for further optimization—not only in linear projections but also in state caches, which contribute significantly to memory consumption, particularly after quantizing the former into low bits.\nAfter a theoretical analysis of the causes of outliers in states, we propose Decoupled Scale Quantization (DSQ), which mitigates outliers in both the state and channel dimensions by applying separate quantization scales.\nTo preserve the selective ability of quantized Mamba, we introduce Efficient Selectivity Reconstruction (ESR), a novel quantization simulation scheme in block-wise reconstruction that enables fast parallel scan algorithms with the non-linear quantization function.\nWe demonstrate the effectiveness of Q-Mamba across various quantization settings, model sizes, and both generation and zero-shot tasks. \nIn particular, for Mamba2-2.7B with W8A8H4 (8-bit weights and activations, 4-bit state caches) quantization, Q-Mamba achieves a 50\\% reduction in memory consumption with only a 2.13\\% average accuracy degradation on zero-shot tasks."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/edcc510e39f54a7d72310a62152b2793eb10fab0.pdf"},"research_area":{"value":"Efficient/Low-Resource Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/February/Submission"}},"tmdate":1782961958346,"tcdate":1739525232755,"writers":["aclweb.org/ACL/ARR/2025/February","aclweb.org/ACL/ARR/2025/February/Submission1837/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/February/Submission1837/Authors"],"forum":"DxPTm7Bxmh","license":"CC BY 4.0","number":1837,"cdate":1739525232755,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/February/-/Submission","aclweb.org/ACL/ARR/2025/February/-/Edit","aclweb.org/ACL/ARR/2025/February/-/Post_Submission","aclweb.org/ACL/ARR/2025/February/Submission1837/-/Change_Reviewer_Nomination","aclweb.org/ACL/ARR/2025/February/Submission1837/-/Blind_Submission_License_Agreement","aclweb.org/ACL/ARR/2025/February/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2025/February/-/Preprint_Post_Submission"],"mdate":1782961958346,"odate":1746821355472,"domain":"aclweb.org/ACL/ARR/2025/February","id":"DxPTm7Bxmh","version":2},{"content":{"venue":{"value":"ACL (Findings) 2025"},"pdf":{"value":"https://aclanthology.org/2025.findings-acl.551.pdf"},"venueid":{"value":"dblp.org/conf/ACL/2025"},"paperhash":{"value":"chen|qmamba_towards_more_efficient_mamba_models_via_posttraining_quantization"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Tianqi_Chen:","~Yuanteng_Chen1","https://dblp.org/search/pid/api?q=author:Peisong_Wang:","~Weixiang_Xu1","https://dblp.org/search/pid/api?q=author:Zeyu_Zhu:","https://dblp.org/search/pid/api?q=author:Jian_Cheng_0001:"]},"html":{"value":"https://aclanthology.org/2025.findings-acl.551/"},"_bibtex":{"value":"@inproceedings{DBLP:conf/acl/ChenCWXZ025,\n  author={Tianqi Chen and Yuanteng Chen and Peisong Wang and Weixiang Xu and Zeyu Zhu and Jian Cheng},\n  title={Q-Mamba: Towards more efficient Mamba models via post-training quantization},\n  year={2025},\n  cdate={1735689600000},\n  pages={10594-10610},\n  url={https://aclanthology.org/2025.findings-acl.551/},\n  booktitle={ACL (Findings)},\n  crossref={conf/acl/2025f}\n}\n"},"abstract":{"value":"State Space Models (SSMs), such as Mamba, have recently demonstrated potential in language understanding tasks, positioning them as competitors to transformer architectures. However, our investigations reveal that the Mamba architecture still has room for further optimization—not only in linear projections but also in state caches, which contribute significantly to memory consumption, particularly after quantizing the former into low bits. After a theoretical analysis of the causes of outliers in states, we propose Decoupled Scale Quantization (DSQ), which mitigates outliers in both the state and channel dimensions by applying separate quantization scales. To preserve the selective ability of quantized Mamba, we introduce Efficient Selectivity Reconstruction (ESR), a novel quantization simulation scheme in block-wise reconstruction that enables fast parallel scan algorithms with the non-linear quantization function. We demonstrate the effectiveness of Q-Mamba across various quantization settings, model sizes, and both generation and zero-shot tasks. In particular, for Mamba2-2.7B with W8A8H4 (8-bit weights and activations, 4-bit state caches) quantization, Q-Mamba achieves a 50% reduction in memory consumption with only a 2.13% average accuracy degradation on zero-shot tasks."},"title":{"value":"Q-Mamba: Towards more efficient Mamba models via post-training quantization"},"authors":{"value":["Tianqi Chen","Yuanteng Chen","Peisong Wang","Weixiang Xu","Zeyu Zhu","Jian Cheng"]}},"tmdate":1762757742934,"pdate":1735689600000,"externalIds":["dblp:conf/acl/ChenCWXZ025"],"tcdate":1761209772794,"writers":["~"],"signatures":["~Yuanteng_Chen1"],"forum":"LPiFfkLmvP","license":"CC BY-SA 4.0","number":647698,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1762757742934,"domain":"DBLP.org","id":"LPiFfkLmvP","version":2},{"content":{"summary":{"value":"This paper proposes R-MeeTo (Re-training Merged Token), a simple and effective framework to accelerate Vision Mamba models through token merging followed by lightweight retraining. Unlike pruning-based token reduction methods that cause severe performance drops due to the sequential dependency and information imbalance inherent to Mamba’s state space structure, R-MeeTo merges similar tokens to preserve key knowledge and then retrains the model for a few epochs to recover lost performance. The paper provides both theoretical analyses based on information bottleneck theory and extensive empirical validation. Experiments on ImageNet-1K show that R-MeeTo can recover up to 35.9% accuracy within minutes of retraining, achieving up to 1.5× inference speedup with minimal accuracy degradation across multiple Vision Mamba variants (Vim-Ti/S/B). Additional tests on VideoMamba confirm the method’s generality and scalability across different hardware and data modalities."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"See Weaknesses."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper identifies and explains a key inefficiency in token pruning for Mamba models.\n\n2. The paper provides a clear information-theoretic analysis.\n\n3. The propsed model achieves better results than the pruning baseline."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. I'm concerning that the paper is comparing with only weak baselines. For example, the Transformer baseline is still DeiT with only 81.8% base-sized model accuracy on ImageNet, which is outdated. A reasonable ViT-Base performance on ImageNet should be over 83.0% (e.g., DeiT-III [1]). Does the conclusion \"Mamba is more sensitive that Transformers for pruning\" still hold true for DeiT-III?\n\n2. The scalability is unknown. The paper only conduct experiments with base-level models with less than 100M parameters. Does it still work for larger models? I understand that your baseline architectures Vim, VideoMamba, PlainMamba do not have large models, but actually there have already been many recent studies training Mamba at a larger scale like MambaVision [2] and MambaReg [3]. Intuitively, larger models need more input information so I suspect the token pruning/merging approaches cannot work well on them.\n\n3. A clear computation–accuracy trade-off **across all model sizes** is very important for a paper on token pruning or reduction. Although I did not find such a figure or table in the paper, I roughly estimated the trade-off myself. According to your results, after merging 42% of the tokens, Vim-S achieves only 72.9% accuracy. Assuming FLOPs scale linearly with token length (a common assumption for Mamba), the corresponding FLOPs would be around 2.94G, which is still much higher than Vim-Ti’s 1.45G FLOPs, yet the accuracy is significantly lower (76.1% → 72.9%). This raises a question: why not simply reduce the model size (e.g., use Vim-Ti) to achieve better efficiency at similar or even higher accuracy? Such a trade-off appears much more favorable.\n\n4. Besides the lack of experiments on larger model sizes, another limitation of this paper is that it only focuses on a moderate sequence length. If the image resolution is increased or decreased—resulting in a much longer or shorter sequence length—would the conclusions in this paper still hold? It remains unclear whether the proposed method and its observed trends generalize across different sequence lengths.\n\n5.  The formatting of the paper needs significant improvement. Many tables are floating beside the main text, and the excessive use of \\vspace makes it difficult to visually distinguish tables from the surrounding paragraphs. It appears that the authors did not carefully adjust the layout—the current manuscript looks as if it was forcefully converted from a two-column template to the single-column ICLR format, resulting in poor readability.\n\n[1] Touvron, Hugo, Matthieu Cord, and Hervé Jégou. \"Deit iii: Revenge of the vit.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Hatamizadeh A, Kautz J. Mambavision: A hybrid mamba-transformer vision backbone[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 25261-25270.\n\n[3] Wang F, Wang J, Ren S, et al. Mamba-Reg: Vision Mamba Also Needs Registers[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 14944-14953."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762932833113,"tcdate":1761963935563,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19976/Reviewer_GQRu"],"signatures":["ICLR.cc/2026/Conference/Submission19976/Reviewer_GQRu"],"forum":"MWIv2vRbE9","number":3,"license":"CC BY 4.0","cdate":1761963935563,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19976/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762932833113,"domain":"ICLR.cc/2026/Conference","replyto":"MWIv2vRbE9","id":"lFUfvPPhAX","forumContent":{"TLDR":{"value":"Mamba models have different token pruning robustness. Marginal enrichment phenomenon is the key difference. Mamba's SSMs modules keep information in marginal tokens (end of scanning)."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Mamba","Token Efficiency"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Vision Mamba (e.g., Vim) has successfully been integrated into computer vision, and token reduction has yielded promising outcomes in Vision Transformers (ViTs). However, token reduction performs less effectively on Vision Mamba compared to ViTs. Pruning informative tokens in Mamba leads to a high loss of key knowledge tokens and a drop in performance, making it not a good solution for enhancing efficiency. Token merging, which preserves more token information than pruning, has demonstrated commendable performance in ViTs, but vanilla merging performance decreases as the reduction ratio increases either, failing to maintain the key knowledge and performance in Mamba. Re-training the model with token merging, which effectively rebuilds the key knowledge, enhances the performance of Mamba. Empirically, pruned Vims, recovered on ImageNet-1K, only drop up to 0.9% accuracy, by our proposed framework R-MeeTo in our main evaluation. We show how simple and effective the fast recovery can be achieved at minute-level, in particular, a 35.9% accuracy spike over 3 epochs of training on Vim-Ti. Moreover, Vim-Ti/S/B are re-trained within 5/7/17 minutes, and Vim-S only drop 1.3% with 1.2 $\\times$ (up to 1.5 $\\times$) speed up in inference."},"_bibtex":{"value":"@misc{\nshi2025faster,\ntitle={Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training},\nauthor={Mingjia Shi and Yuhao Zhou and Ruiji Yu and Zekai Li and Xuanlei Zhao and Zhiyuan Liang and Mengxuan Wu and Xiaojiang Peng and Panpan Zhang and Ziheng Qin and Wangbo Zhao and Kai Wang and Yang You},\nyear={2025},\nurl={https://openreview.net/forum?id=MWIv2vRbE9}\n}"},"title":{"value":"Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training"},"pdf":{"value":"/pdf/f8384c7461cf4487ea921d4d941e9ac3b0d5cd9a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"shi|faster_vision_mamba_is_rebuilt_in_minutes_via_merged_token_retraining"},"authorids":{"value":["~Mingjia_Shi1","~Yuhao_Zhou4","~Ruiji_Yu1","~Zekai_Li2","~Xuanlei_Zhao1","~Zhiyuan_Liang1","~Mengxuan_Wu1","~Xiaojiang_Peng1","~Panpan_Zhang1","~Ziheng_Qin1","~Wangbo_Zhao1","~Kai_Wang8","~Yang_You1"]},"authors":{"value":["Mingjia Shi","Yuhao Zhou","Ruiji Yu","Zekai Li","Xuanlei Zhao","Zhiyuan Liang","Mengxuan Wu","Xiaojiang Peng","Panpan Zhang","Ziheng Qin","Wangbo Zhao","Kai Wang","Yang You"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.17388v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"movahedi|selective_rotary_position_embedding"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Sajad_Movahedi:","~Timur_Carstensen1","https://dblp.org/search/pid/api?q=author:Arshia_Afzal:","https://dblp.org/search/pid/api?q=author:Frank_Hutter:","https://dblp.org/search/pid/api?q=author:Antonio_Orvieto:","https://dblp.org/search/pid/api?q=author:Volkan_Cevher:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2511.17388"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-17388,\n  publtype={informal},\n  author={Sajad Movahedi and Timur Carstensen and Arshia Afzal and Frank Hutter and Antonio Orvieto and Volkan Cevher},\n  title={Selective Rotary Position Embedding},\n  year={2025},\n  month={November},\n  cdate={1761955200000},\n  journal={CoRR},\n  volume={abs/2511.17388},\n  url={https://doi.org/10.48550/arXiv.2511.17388}\n}\n"},"abstract":{"value":"Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\\textit{RoPE}) encode positions through \\textit{fixed-angle} rotations, while in linear transformers, order is handled via input-dependent (selective) gating that decays past key-value associations. Selectivity has generally been shown to improve language-related tasks. Inspired by this, we introduce \\textit{Selective RoPE}, an \\textit{input-dependent} rotary embedding mechanism, that generalizes \\textit{RoPE}, and enables rotation in \\textit{arbitrary angles} for both linear and softmax transformers. We show that softmax attention already performs a hidden form of these rotations on query-key pairs, uncovering an implicit positional structure. We further show that in state-space models and gated linear transformers, the real part manages forgetting while the imaginary part encodes positions through rotations. We validate our method by equipping gated transformers with \\textit{Selective RoPE}, demonstrating that its input-dependent rotations improve performance in language modeling and on difficult sequence tasks like copying, state tracking, and retrieval."},"title":{"value":"Selective Rotary Position Embedding"},"authors":{"value":["Sajad Movahedi","Timur Carstensen","Arshia Afzal","Frank Hutter","Antonio Orvieto","Volkan Cevher"]}},"tmdate":1769417615665,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2511-17388"],"tcdate":1769417610120,"writers":["~"],"signatures":["~Timur_Carstensen1"],"forum":"ymWI2CwzLm","license":"CC BY-SA 4.0","number":801683,"cdate":1761955200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769417615665,"domain":"DBLP.org","id":"ymWI2CwzLm","version":2},{"content":{"summary":{"value":"This paper studies the role of MAMBA architectures on Domain Generalization benchmarks and adapts the architecture to achieve robust generalization. \nThe motivation to use MAMBA-style architectures is their linear complexity. The authors theoretically analyze conventional MAMBA for DG and make an important finding that internal hidden states in MAMBA capture domain-specific characteristics, thus hurting their generalization capabilities.\nTo address this, they propose two techniques grounded in theory. Both of them involve suppressing the domain-specific information from salient tokens. These tokens are obtained by looking at the \"attention matrix\" of the last layer and determining the top K tokens which have high attention values. To \"debias\" these, the first method mixes these tokens (derived through a mask) with style computed from another image. This way, while the content is preserved, the style is augmented, thus making the network learn domain-invariant features. \nIn the second method, they calculate the salient tokens by directly relying on the activation values and employ the same procedure as above to produce a new style image. \nThey have conducted experimental analysis on standard DG benchmarks and demonstrate empirical gains compared to the baselines."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"please see weaknesses"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"This paper comprehensively studies the generalization capabilities of MAMBA (state-space) models with respect to their ability to handle distribution shifts. The theoretical analysis is somewhat straightforward, relying on the MMD measure, but it is an important contribution to the field. The experiments follow standard protocols. The proposed methods are mathematically grounded and produce empirical gains compared to baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The main weakness is a rather weak motivation for using MAMBA for DG. It almost feels like a low-hanging fruit contribution without proper motivation for the same.\nThere is another very similar work on using MAMBA for DG called DGMAMBA that was recently published. I commend the authors for including that paper in their experimental comparison. But upon reading that paper, there are very similar ideas being proposed, especially the observation that hidden states of MAMBA carry domain-specific information.\nIn the author response phase, I would like the authors to clearly differentiate how the proposed approach is different from DG MAMBA. Furthermore, DGMAMBA and the proposed approach produce very similar numbers on at least two datasets.\nSecond, does the salient token perturbation of both M and X happen from epoch 1? Or is it after a few epochs of training? Since the method relies on ImageNet pre-trained models, maybe from epoch 1, the salient tokens can be meaningfully predicted. But what if the models are not pre-trained? How does the method change?\nSmall improvement suggestion: It would be nice to remove the overloading of $h$ - in section 3.1 it represents hidden states and in section 3.2 it denotes hypothesis."},"limitations":{"value":"The authors have sufficiently addressed the limitations."}},"nonreaders":[],"tmdate":1730879310418,"tcdate":1722323854058,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission9378/Reviewer_AMLN"],"signatures":["NeurIPS.cc/2024/Conference/Submission9378/Reviewer_AMLN"],"forum":"mAdGQ1Hh3L","number":4,"license":"CC BY 4.0","cdate":1722323854058,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission9378/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879310418,"domain":"NeurIPS.cc/2024/Conference","replyto":"mAdGQ1Hh3L","id":"TAZfSr1JEb","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Domain generalization","state space models"]},"supplementary_material":{"value":"/attachment/ce883f1aa347d3f4ebb234dc743b97b4275be824.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Domain Generalization (DG) aims to enable models to generalize to unseen target domains by learning from multiple source domains. Existing DG methods primarily rely on convolutional neural networks (CNNs), which inherently learn texture biases due to their limited receptive fields, making them prone to overfitting source domains. While some works have introduced transformer-based methods (ViTs) for DG to leverage the global receptive field, these methods incur high computational costs due to the quadratic complexity of self-attention. Recently, advanced state space models (SSMs), represented by Mamba, have shown promising results in supervised learning tasks by achieving linear complexity in sequence length during training and fast RNN-like computation during inference. Inspired by this, we investigate the generalization ability of the Mamba model under domain shifts and find that input-dependent matrices within SSMs could accumulate and amplify domain-specific features, thus hindering model generalization. To address this issue, we propose a novel SSM-based architecture with saliency-based token-aware transformation (namely START), which achieves state-of-the-art (SOTA) performances and offers a competitive alternative to CNNs and ViTs. Our START can selectively perturb and suppress domain-specific features in salient tokens within the input-dependent matrices of SSMs, thus effectively reducing the discrepancy between different domains. Extensive experiments on five benchmarks demonstrate that START outperforms existing SOTA DG methods with efficient linear complexity. Our code is available at https://github.com/lingeringlight/START."},"_bibtex":{"value":"@inproceedings{\nguo2024start,\ntitle={{START}: A Generalized State Space Model with Saliency-Driven Token-Aware Transformation},\nauthor={Jintao Guo and Lei Qi and Yinghuan Shi and Yang Gao},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=mAdGQ1Hh3L}\n}"},"title":{"value":"START: A Generalized State Space Model with Saliency-Driven Token-Aware Transformation"},"pdf":{"value":"/pdf/1e045e8d4c432c88ee7d528f95cd5406dfc62b43.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"guo|start_a_generalized_state_space_model_with_saliencydriven_tokenaware_transformation"},"authorids":{"value":["~Jintao_Guo1","~Lei_Qi1","~Yinghuan_Shi3","~Yang_Gao3"]},"authors":{"value":["Jintao Guo","Lei Qi","Yinghuan Shi","Yang Gao"]}},"version":2},{"content":{"summary":{"value":"This paper addresses the long-sequence modeling problem. A recursion in recursion strategy is proposed to balance the advantages between BB-Tree RvNNs and RvNN models. The idea is straightforward but achieves competitive performance on LRA tasks."},"soundness":{"value":"3 good"},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"See the weakness part."},"rating":{"value":"5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":"3 good"},"contribution":{"value":"3 good"},"strengths":{"value":"1. The paper is easy to follow."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. There are large numbers of related works missing. For long sequence modeling, there are at least 4 types of methods, including Linear attention, SSM, Linear RNN, and LongConv. Since these methods are implemented for the same goal, they should be included in related work as well as the experiment section.\n2. The experiments are inadequate. LRA is a toy benchmark for assessing long-sequence modeling. It is insufficient to use it as the sole indicator of effectiveness. In fact, this work focuses solely on a sub-task in LRA, making the experiments even weaker. I would encourage the author to verify the actual long sequence modeling capabilities in real-world scenarios such as language modeling, image classification, etc.\n3. Linear RNN has achieved STOA performance in many benchmarks, including LRA, language modeling, and image classification, as demonstrated in recent papers. Why should we consider non-linear RNNs, which are slow to train and perform no better than linear RNNs?\n4. The concept is straightforward and straightforward. By forming a RIR structure, it combines the advantages of the two methods. As a result, the processing time is lengthened. It would be more appealing if the processing time could be shortened. Furthermore, the competitive methods are ineffective. I don't see any standard benchmark LRA results, so it's difficult for me to justify the effectiveness of the proposed method.\n5.The maximum sequence length used in this paper for an efficient long sequence modeling method is 2K, which is a standard sequence length for transformer LLM. It would be preferable to see the proposed method in long sequence tests, such as 32K and higher. Furthermore, long sequence modeling takes much longer to process than transformer, making the method less appealing in real-world scenarios."},"limitations":{"value":"Yes."}},"nonreaders":[],"tmdate":1702411430640,"tcdate":1690437435388,"writers":["NeurIPS.cc/2023/Conference","NeurIPS.cc/2023/Conference/Submission13474/Reviewer_619V"],"signatures":["NeurIPS.cc/2023/Conference/Submission13474/Reviewer_619V"],"forum":"o6yTKfdnbA","number":7,"license":"CC BY 4.0","cdate":1690437435388,"mdate":1702411430640,"readers":["everyone"],"invitations":["NeurIPS.cc/2023/Conference/Submission13474/-/Official_Review","NeurIPS.cc/2023/Conference/-/Edit"],"domain":"NeurIPS.cc/2023/Conference","replyto":"o6yTKfdnbA","id":"98mVyxerwt","forumContent":{"venue":{"value":"NeurIPS 2023 poster"},"keywords":{"value":["Recursive Neural Networks","Long Range Arena","RvNN","Long Range Sequence Modeling","Length Generalization","LRA","Structured Encoding","Inductive Bias","Hierarchical Model","Recursive Models"]},"_bibtex":{"value":"@inproceedings{\nchowdhury2023recursion,\ntitle={Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability},\nauthor={Jishnu Ray Chowdhury and Cornelia Caragea},\nbooktitle={Thirty-seventh Conference on Neural Information Processing Systems},\nyear={2023},\nurl={https://openreview.net/forum?id=o6yTKfdnbA}\n}"},"title":{"value":"Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability"},"paperhash":{"value":"chowdhury|recursion_in_recursion_twolevel_nested_recursion_for_length_generalization_with_scalability"},"TLDR":{"value":"We introduce a nested recursive model with an efficient balanced-tree outer recursion that logarithmically scales with input length and an inefficient (but powerful) inner recursion which lineary scales with bound constant input length."},"abstract":{"value":"Binary Balanced Tree Recursive Neural Networks (BBT-RvNNs) enforce sequence composition according to a preset balanced binary tree structure. Thus, their non-linear recursion depth (which is the tree depth) is just $\\log_2 n$ ($n$ being the sequence length). Such logarithmic scaling makes BBT-RvNNs efficient and scalable on long sequence tasks such as Long Range Arena (LRA). However, such computational efficiency comes at a cost because BBT-RvNNs cannot solve simple arithmetic tasks like ListOps. On the flip side, RvNN models (e.g., Beam Tree RvNN) that do succeed on ListOps (and other structure-sensitive tasks like formal logical inference) are generally several times more expensive (in time and space) than even Recurrent Neural Networks. In this paper, we introduce a novel framework --- Recursion in Recursion (RIR) to strike a balance between the two sides - getting some of the benefits from both worlds. In RIR, we use a form of two-level nested recursion - where the outer recursion is a $k$-ary balanced tree model with another recursive model (inner recursion) implementing its cell function. For the inner recursion, we choose Beam Tree RvNNs. To adjust Beam Tree RvNNs within RIR we also propose a novel strategy of beam alignment. Overall, this entails that the total recursive depth in RIR is upper-bounded by $k \\log_k n$. Our best RIR-based model is the first model that demonstrates high ($\\geq 90\\%$) length-generalization performance on ListOps while at the same time being scalable enough to be trainable on long sequence inputs from LRA (it can reduce the memory usage of the original Beam Tree RvNN by hundreds of times). Moreover, in terms of accuracy in the LRA language tasks, it performs competitively with Structured State Space Models (SSMs) without any special initialization - outperforming Transformers by a large margin. On the other hand, while SSMs can marginally outperform RIR on LRA, they (SSMs) fail to length-generalize on ListOps. Our code is available at: https://github.com/JRC1995/BeamRecursionFamily/"},"pdf":{"value":"/pdf/6e71b9b1e74fa401ad0da7ed604e28e45fbbfe2d.pdf"},"venueid":{"value":"NeurIPS.cc/2023/Conference"},"authorids":{"value":["~Jishnu_Ray_Chowdhury2","~Cornelia_Caragea2"]},"authors":{"value":["Jishnu Ray Chowdhury","Cornelia Caragea"]}},"version":2},{"content":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Vision-Language Models; Mamba;"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"While emerging multi-modal large language models (MLLM) have demonstrated impressive advances, the quadratic complexity of their Transformer-based LLMs (3B or larger) inevitably leads to considerable computational overhead. On the other hand, the recently proposed selective state space model (i.e., Mamba) enjoys both model capacity and computational efficiency, making it an ideal component to enhance MLLM's efficiency and performance. However, recent attempts to introduce Mamba into MLLMs simply replace their LLMs with Mamba, ignoring the unique characteristics of either side. We argue that such a naive combination cannot exhibit the potential of Mamba in MLLMs. In this paper, we delve into harnessing Mamba's unique properties, and propose tailored designs from both multi-modal input and architectural perspectives to unleash its true power. First, we fully utilize Mamba's linear complexity to construct visual long sequences for a thorough perception at a minor efficiency burden. To integrate the scanning mechanism with the built visual long sequence, we devise a novel cross-stitch scanning approach to capture and fuse spatial and semantic properties simultaneously, enhancing the interaction of visual information and the vision-language alignment. Built upon these designs, we propose MambaVLM, a simple yet effective MLLM framework that exhibits highly competitive results across multiple benchmarks. Moreover, our framework is also compatible with Transformer-based LLMs (e.g., Vicuna), demonstrating remarkable training and inference efficiency. Notably, with only 0.66M data and 14 hours training on a single A800 node, our MambaVLM outperforms LLaVA-1.5 by significant margins and performs on par or even better than the 1.4B data trained Qwen-VL. The appealing results from both effectiveness and efficiency aspects indicate the promising prospects of Mamba in MLLMs."},"_bibtex":{"value":"@misc{\nchen2024unleashing,\ntitle={Unleashing the Power of Selective State Space Models in Vision-Language Models},\nauthor={Honghao Chen and Yibing Song and Shoufa Chen and Chongjian GE and Kaiqi Huang},\nyear={2024},\nurl={https://openreview.net/forum?id=0A6f1b66pE}\n}"},"title":{"value":"Unleashing the Power of Selective State Space Models in Vision-Language Models"},"pdf":{"value":"/pdf/1183a88cc6949631a83ff61a73451d323b4592a6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"chen|unleashing_the_power_of_selective_state_space_models_in_visionlanguage_models"},"authorids":{"value":["~Honghao_Chen1","~Yibing_Song1","~Shoufa_Chen1","~Chongjian_GE1","~Kaiqi_Huang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Honghao Chen","Yibing Song","Shoufa Chen","Chongjian GE","Kaiqi Huang"]}},"tmdate":1731505202370,"tcdate":1727251002717,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4294/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission4294/Authors"],"forum":"0A6f1b66pE","license":"CC BY 4.0","number":4294,"cdate":1727251002717,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/-/Withdrawn_Submission"],"mdate":1731505202370,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"0A6f1b66pE","version":2},{"content":{"summary":{"value":"The authors propose a novel module, Global-aware SSM (GSSM), which is the core of a plug-and-play block called GMamba. The key idea is to use a Discrete Fourier Transform (DFT)-based modulation mechanism to \"inject\" global information into the features before they are processed by the SSM. This frequency-domain pre-modulation guides the SSM's state updates with a true global perspective, enabling efficient global modeling with linear-logarithmic complexity"},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please check the weakness."},"rating":{"value":6},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The proposed GMamba block shows consistent and significant performance improvements when added to various backbones (e.g., ResNet, Swin, ConvNeXt) . It outperforms existing global modeling methods, including Transformer and other Mamba variants, on several dense-prediction tasks (semantic segmentation, object detection, and instance segmentation) across multiple datasets."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper's theoretical claims rest heavily on its new definition of global perception—\"global gradient dependency\". This definition requires that the Frobenius norm of the gradient of the output with respect to any input pixel is bounded by a non-zero constant $\\tau$. This is a very low bar. A gradient that is infinitesimally small but non-zero would satisfy this definition, but this may not align with an intuitive or practical understanding of \"global influence.\" The claim of \"true\" global perception is therefore only as strong as the acceptance of this new, and debatable, definition.\n\n\n2. Although the authors claim that the proposed GSSM enables efficient global image modeling, there is no related experimental data on practical inference latency or GPU memory cost. Relying only on FLOPs may not accurately reflect the module's true efficiency, as operations like 2D-DFT can have different hardware utilization profiles than standard convolutions.\n\n3. The experimental validation is extensive on dense prediction tasks (segmentation and detection). However, a standard benchmark for vision backbones is ImageNet classification. The absence of this benchmark makes it slightly more difficult to assess the module's generalizability as a fundamental building block for all vision tasks.\n\n4. Some illustrations of previous work in this paper appear to be incorrect and may mislead readers. The authors show the ViM scanning routes in Figure 1(c), but this depiction seems to be wrong (as noted in other work, e.g., [1]). This potential misrepresentation of a key baseline method is a concern.\n\n[1]. Visual mamba: A survey and new outlooks, 2024."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916633614,"tcdate":1761457821046,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission3260/Reviewer_YRTs"],"signatures":["ICLR.cc/2026/Conference/Submission3260/Reviewer_YRTs"],"forum":"96p1eBUVaN","number":1,"license":"CC BY 4.0","cdate":1761457821046,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission3260/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916633614,"domain":"ICLR.cc/2026/Conference","replyto":"96p1eBUVaN","id":"YVRWZ2OPxr","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["State Space Model","Frequency Domain Modulation","Global Image Modeling","Mathematical Definition"]},"supplementary_material":{"value":"/attachment/8db45544c11b45c2d7a82e99f05b6a7eb8bc9d57.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Despite the importance of global contextual modeling in visual tasks, a rigorous mathematical definition remains absent, and the concept is still largely described in heuristic or empirical terms. Existing methods either rely on computationally expensive attention mechanisms or are constrained by the recursive modeling nature of State Space Models (SSMs), making it challenging to achieve both efficiency and true global perception. To address this, we first propose a mathematical definition of global modeling for visual images, providing a theoretical foundation for designing globally-aware and interpretable models. Based on in-depth analysis of SSMs and frequency-domain modeling principles, we construct a complete theoretical framework that overcomes the limitations imposed by SSMs' recursive modeling mechanism from a frequency perspective, thereby adapting SSMs for global perception in image modeling. Guided by this framework, we design the Global-aware SSM (GSSM) module and formally prove that it satisfies definitional requirements of global image modeling. GSSM leverages a Discrete Fourier Transform (DFT)-based modulation mechanism, providing precise front-end control over the SSM's modeling behavior, and enabling efficient global image modeling with linear-logarithmic complexity. Building upon GSSM, we develop GMamba, a plug-and-play module that can be seamlessly integrated at any stage of Convolutional Neural Networks (CNNs). Extensive experiments across multiple tasks, including object detection, semantic segmentation, and instance segmentation, across diverse model architectures, demonstrate that GMamba consistently outperforms existing global modeling modules, validating both the effectiveness of our theoretical framework and the rigor of proposed definition. Code is available at \\url{https://github.com/Xinmu-Tantai/GMamba-GSSM}"},"_bibtex":{"value":"@inproceedings{\nhui2026enabling,\ntitle={Enabling True Global Perception in State Space Models for Visual Tasks},\nauthor={Jie Hui and Zhenxiang Zhang and Wenyu Mi and Jianji Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=96p1eBUVaN}\n}"},"title":{"value":"Enabling True Global Perception in State Space Models for Visual Tasks"},"pdf":{"value":"/pdf/0e6ef9c228142eea2c1b8a21a689032416e7f638.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"hui|enabling_true_global_perception_in_state_space_models_for_visual_tasks"},"authorids":{"value":["~Jie_Hui1","~Zhenxiang_Zhang1","~Wenyu_Mi1","~Jianji_Wang1"]},"authors":{"value":["Jie Hui","Zhenxiang Zhang","Wenyu Mi","Jianji Wang"]}},"version":2},{"content":{"TLDR":{"value":"We introduce MambaMatcher, which formulates Mamba layers to process the correlation map between image pairs to establish accurate semantic correspondences with improved performance and efficiency."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["semantic correspondence","state-space models","correlation aggregation","Mamba","feature aggregation","Similarity-aware Selective Scan"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Establishing semantic correspondences between images is a fundamental yet challenging task in computer vision. Traditional feature-metric methods enhance visual features but may miss complex inter-image relationships, while recent correlation-metric approaches attempt to model these relationships but are hindered by high computational costs due to processing 4D correlation maps. We introduce MambaMatcher, a novel method that overcomes these limitations by efficiently modeling high-dimensional correlations using selective state-space models (SSMs), treating multi-level correlation scores as states. By implementing a similarity-aware selective scan mechanism adapted from Mamba’s linear-complexity algorithm, MambaMatcher refines the 4D correlation tensor effectively without compromising feature map resolution or receptive field. Experiments on standard semantic correspondence benchmarks demonstrate that MambaMatcher achieves state-of-the-art performance without relying on large input images or computationally expensive diffusion-based feature extractors, effectively capturing rich inter-image correlations while maintaining computational efficiency."},"_bibtex":{"value":"@misc{\nkim2025selective,\ntitle={Selective State-Space Modeling of Correlation Maps for Semantic Correspondence},\nauthor={Seungwook Kim and Minsu Cho},\nyear={2025},\nurl={https://openreview.net/forum?id=GmE8ovvXaJ}\n}"},"title":{"value":"Selective State-Space Modeling of Correlation Maps for Semantic Correspondence"},"pdf":{"value":"/pdf/904fad05fa77d7d08d5defc1a4f7abaade7973d6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"kim|selective_statespace_modeling_of_correlation_maps_for_semantic_correspondence"},"authorids":{"value":["~Seungwook_Kim2","~Minsu_Cho1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Seungwook Kim","Minsu Cho"]}},"tmdate":1738735705069,"tcdate":1727294799131,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5072/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5072/Authors"],"forum":"GmE8ovvXaJ","license":"CC BY 4.0","number":5072,"cdate":1727294799131,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission5072/-/Full_Submission","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735705069,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"GmE8ovvXaJ","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2502.13145v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"liao|multimodal_mamba_decoderonly_multimodal_state_space_model_via_quadratic_to_linear_distillation"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Bencheng_Liao:","~Hongyuan_Tao2","https://dblp.org/search/pid/api?q=author:Qian_Zhang:","https://dblp.org/search/pid/api?q=author:Tianheng_Cheng:","https://dblp.org/search/pid/api?q=author:Yingyue_Li:","https://dblp.org/search/pid/api?q=author:Haoran_Yin:","","https://dblp.org/search/pid/api?q=author:Xinggang_Wang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2502.13145"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2502-13145,\n  publtype={informal},\n  author={Bencheng Liao and Hongyuan Tao and Qian Zhang and Tianheng Cheng and Yingyue Li and Haoran Yin and Wenyu Liu and Xinggang Wang},\n  title={Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation},\n  year={2025},\n  month={February},\n  cdate={1738368000000},\n  journal={CoRR},\n  volume={abs/2502.13145},\n  url={https://doi.org/10.48550/arXiv.2502.13145}\n}\n"},"abstract":{"value":"Recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance but face deployment challenges due to their quadratic computational complexity, growing Key-Value cache requirements, and reliance on separate vision encoders. We propose mmMamba, a framework for developing linear-complexity native multimodal state space models through progressive distillation from existing MLLMs using moderate academic computational resources. Our approach enables the direct conversion of trained decoder-only MLLMs to linear-complexity architectures without requiring pre-trained RNN-based LLM or vision encoders. We propose an seeding strategy to carve Mamba from trained Transformer and a three-stage distillation recipe, which can effectively transfer the knowledge from Transformer to Mamba while preserving multimodal capabilities. Our method also supports flexible hybrid architectures that combine Transformer and Mamba layers for customizable efficiency-performance trade-offs. Distilled from the Transformer-based decoder-only HoVLE, mmMamba-linear achieves competitive performance against existing linear and quadratic-complexity VLMs, while mmMamba-hybrid further improves performance significantly, approaching HoVLE's capabilities. At 103K tokens, mmMamba-linear demonstrates 20.6$\\times$ speedup and 75.8% GPU memory reduction compared to HoVLE, while mmMamba-hybrid achieves 13.5$\\times$ speedup and 60.2% memory savings. Code and models are released at https://github.com/hustvl/mmMamba"},"title":{"value":"Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation"},"authors":{"value":["Bencheng Liao","Hongyuan Tao","Qian Zhang","Tianheng Cheng","Yingyue Li","Haoran Yin","Wenyu Liu","Xinggang Wang"]}},"tmdate":1784540544512,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2502-13145"],"tcdate":1762764077113,"writers":["~"],"signatures":["~Wenyu_Liu3"],"forum":"5OahRbhJG6","license":"CC BY-SA 4.0","number":686208,"cdate":1738368000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784540544512,"domain":"DBLP.org","id":"5OahRbhJG6","version":2},{"content":{"summary":{"value":"This paper analyzes recurrent neural networks (RNNs) through a weight-space linear recurrence formulation that unifies several modern architectures — including continuous-time linear RNNs, state-space models (SSMs), and residual recurrent networks — under a single linear operator perspective.\n\nThe authors derive closed-form expressions for training dynamics and generalization in the overparameterized limit, showing that convergence properties and implicit regularization can be understood via the spectral structure of the recurrent Jacobian. They provide:\n\nA linearized weight-space recurrence model that approximates nonlinear dynamics by a low-rank operator with analytically tractable behavior.\n\nA demonstration that generalization error scales with spectral conditioning, extending kernel-based intuition from linear networks to recurrent architectures.\n\nEmpirical validation on synthetic sequence modeling and dynamical-system reconstruction tasks (mass–spring–damper, Lotka–Volterra, PEMS08 traffic), showing alignment between predicted and observed convergence trends.\n\nOverall, the paper contributes to a principled understanding of how weight-space geometry and recurrence interact to shape training efficiency and generalization."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. How sensitive are your analytical predictions to non-normal dynamics (upper-triangular Hₕ) versus the diagonal “normal” case you ultimately focus on?\n\n2. Could your framework accommodate nonlinear activation perturbations (e.g., ReLU linearization) to estimate when linear approximation fails?\n\n3. Have you compared your spectral regularization predictions to empirical spectral shrinkage observed during training (e.g., spectrum compression in Wh)?\n\n4. Could you clarify how your results relate to implicit bias analyses in linear transformers or SSMs (e.g., Merrill & Sabharwal; Orvieto et al.)?\n\n5. Are there regimes where the recurrence’s spectral radius predicts too-rapid forgetting or instability, contradicting observed dynamics?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"The paper builds on well-established analyses of linear networks and extends them naturally to recurrent settings using a spectral-decomposition framework (Schur- and SVD-based). Derivations are internally consistent and clearly documented.\n\nThe proposed linear recurrence view elegantly bridges RNNs, residual-RNNs, and diagonal SSMs, helping clarify connections between recent model families.\n\nExperiments across several dynamical-system tasks (MSD, LV, traffic flow forecasting) confirm the predicted dependence of training speed and generalization on spectral conditioning and effective recurrence length.\n\nThe inclusion of both synthetic physics systems and real-world time-series (PEMS08) demonstrates breadth and internal consistency.\n\nMathematical exposition is detailed; hyperparameters and architectures are listed (Appendix D.4–D.6). Code release and ablation details are promised."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.  The analytic results rest on linear, Gaussian assumptions; nonlinear recurrence effects and gating dynamics are only discussed qualitatively. As such, predictive power for modern gated RNNs or structured SSMs is limited.\n\n2. The analysis centers on the infinite-width, overparameterized limit; it does not quantify where the asymptotic predictions break down for finite models.\n\n3. The authors reference Saxe et al. (2014) but omit more recent theoretical works on curriculum and transfer in RNNs (e.g., Rajan, Kepple & Engleken) and on gradient-flow analyses in recurrent kernels — literature directly related to their spectral-mode interpretation.\n\n4. Although the experiments match qualitative trends, they serve mainly as demonstrations rather than quantitative tests (e.g., no variance or uncertainty estimates, small sample sizes).\n\n5. Generalization is assessed by mean-squared error only; tasks with stochastic noise or long-term dependency tests (copy, addition, character-level modeling) would strengthen claims about recurrence depth and spectral bias."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917573953,"tcdate":1762099559596,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4784/Reviewer_jyKY"],"signatures":["ICLR.cc/2026/Conference/Submission4784/Reviewer_jyKY"],"forum":"zHdKaF3ZM7","number":4,"license":"CC BY 4.0","cdate":1762099559596,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4784/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917573953,"domain":"ICLR.cc/2026/Conference","replyto":"zHdKaF3ZM7","id":"NuwNC5fYaL","forumContent":{"TLDR":{"value":"We introduce a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["physics-informed machine learning","weight-space learning","meta-learning","deep sequence modeling","linear recurrence","test-time training"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"We introduce WARP (**W**eight-space **A**daptive **R**ecurrent **P**rediction), a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into fixed-dimensional hidden states, WARP explicitly parametrizes its hidden state as the weights and biases of a distinct auxiliary neural network, and uses input differences to drive its recurrence. This brain-inspired formulation enables efficient gradient-free adaptation of the auxiliary network at test-time, in-context learning abilities, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state-of-the-art baselines on diverse classification tasks, featuring in the top three in 4 out of 6 real-world challenging datasets. Furthermore, extensive experiments across sequential image completion, multivariate time series forecasting, and dynamical system reconstruction demonstrate its expressiveness and generalization capabilities. Remarkably, a physics-informed variant of our model outperforms the next best model by more than 10x. Ablation studies confirm the architectural necessity of key components, solidifying weight-space linear RNNs as a transformative paradigm for adaptive machine intelligence."},"_bibtex":{"value":"@inproceedings{\nnzoyem2026weightspace,\ntitle={Weight-Space Linear Recurrent Neural Networks},\nauthor={Roussel Desmond Nzoyem and Nawid Keshtmand and Enrique Crespo-Fernandez and Idriss TSAYEM and Raul Santos-Rodriguez and David A.W. Barton and Tom Deakin},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zHdKaF3ZM7}\n}"},"title":{"value":"Weight-Space Linear Recurrent Neural Networks"},"pdf":{"value":"/pdf/c01f4c1dbdaf1c6e821cde510623f871a07c3404.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"nzoyem|weightspace_linear_recurrent_neural_networks"},"authorids":{"value":["~Roussel_Desmond_Nzoyem1","~Nawid_Keshtmand1","~Enrique_Crespo-Fernandez1","~Idriss_TSAYEM1","~Raul_Santos-Rodriguez1","~David_A.W._Barton1","~Tom_Deakin1"]},"authors":{"value":["Roussel Desmond Nzoyem","Nawid Keshtmand","Enrique Crespo-Fernandez","Idriss TSAYEM","Raul Santos-Rodriguez","David A.W. Barton","Tom Deakin"]}},"version":2},{"content":{"comment":{"value":"**Dear Reviewer `ic14`**,  \n\nThank you for your timely and insightful feedback. In our **general response**, we presented evaluation results on benchmarks aligned with Mamba [1] to address potential concerns about cherry-picking. In response to your request for additional benchmarks, we have **expanded our evaluation to include widely-used datasets** such as TriviaQA [2], BoolQ [3], C3 [4], RTE [5], COPA [5], MMLU [6], and CMMLU [7].  \n\nAs you correctly anticipated, the results on MMLU and CMMLU exhibit random patterns and do not achieve 1/4 (25%) accuracy. However, for less challenging tasks like TriviaQA, BoolQ, C3, and RTE, we observe **consistent performance improvements**. Among these, the improvement in BoolQ is particularly notable, aligning well with conjectures regarding the relationship between loss curves and emergent abilities in downstream tasks [8].  \n\n| **Downstream Dataset** | **TriviaQA** | **BoolQ** | **C3** | **RTE** | **COPA** |  **MMLU** | **CMMLU** |  \n|-------------------------|--------------|-----------|---------|----------|-----------|  --------------| --------------|\n| **Cosine Schedule**     | 14.30        | 54.53     | 48.44 | 52.71 | **54.0** | 23.64    | **24.94** |  \n| **Optimized Schedule**  | **14.70** ($\\uparrow$ 0.40%) | **61.41** ($\\uparrow$ 6.88%) | **50.30** ($\\uparrow$ 1.86%) | **53.79**($\\uparrow$ 1.08%) | 53.0 ($\\downarrow$ 1.0%) | **23.89** ($\\uparrow$ 0.25%) | 24.84 ($\\downarrow$ 0.10%) |  \n\nThese findings emphasize the practical benefits of our proposed multi-power law. The results demonstrate that our approach can induce a schedule that significantly improves downstream performance at a meaningful scale. \n\nThank you again for your valuable feedback, which has helped us strengthen our work.  \n\nSincerely, \n\nThe Authors\n\n[1] [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752)  \n\n[2] [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://arxiv.org/abs/1705.03551)\n\n[3] [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://arxiv.org/abs/1905.10044)\n\n[4] [Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension](https://arxiv.org/abs/1904.09679)\n\n[5] [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://arxiv.org/abs/1905.00537)\n\n[6] [Measuring Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300)\n\n[7] [CMMLU: Measuring massive multitask language understanding in Chinese](https://arxiv.org/abs/2306.09212)\n\n[8] [Understanding Emergent Abilities of Language Models from the Loss Perspective](https://arxiv.org/abs/2403.15796)"},"title":{"value":"Further Response to Reviewer  `ic14`"}},"tmdate":1733210064347,"tcdate":1733210064347,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13754/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13754/Authors"],"forum":"KnoS9XxIlK","number":42,"license":"CC BY 4.0","cdate":1733210064347,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13754/-/Official_Comment"],"mdate":1733210064347,"domain":"ICLR.cc/2025/Conference","replyto":"CxFwKayM9p","id":"sucDLDii8i","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"Loss curve prediction and optimized learning rate schedule"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large language model","Learning rate scheduler","Scaling Law","Hyperparameter optimization"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Training large models is both resource-intensive and time-consuming, making it crucial to understand the quantitative relationship between model performance and hyperparameters. In this paper, we derive an empirical law that predicts pretraining loss for large language models for every intermediate training step across various learning rate schedules, including constant, cosine, and step decay schedules. Our proposed law takes a multi-power form, combining a power law based on the sum of learning rates and additional power laws to account for a loss reduction effect as learning rate decays. We validate this law extensively on Llama-2 models of varying sizes and demonstrate that, after fitting on a few learning rate schedules, it accurately predicts the loss curves for unseen schedules of different shapes and horizons. Moreover, by minimizing the predicted final pretraining loss across learning rate schedules, we are able to find a schedule that outperforms the widely-used cosine learning rate schedule. Interestingly, this automatically discovered schedule bears some resemblance to the recently proposed Warmup-Stable-Decay (WSD) schedule (Hu et al, 2024) but achieves a slightly lower final loss. We believe these results could offer valuable insights for understanding the dynamics of pretraining and for designing learning rate schedules to improve efficiency."},"_bibtex":{"value":"@inproceedings{\nluo2025a,\ntitle={A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules},\nauthor={Kairong Luo and Haodong Wen and Shengding Hu and Zhenbo Sun and Maosong Sun and Zhiyuan Liu and Kaifeng Lyu and Wenguang Chen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=KnoS9XxIlK}\n}"},"title":{"value":"A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules"},"pdf":{"value":"/pdf/212cefb36307e11021cced0a49c98a00931e6431.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"luo|a_multipower_law_for_loss_curve_prediction_across_learning_rate_schedules"},"authorids":{"value":["~Kairong_Luo1","~Haodong_Wen1","~Shengding_Hu2","~Zhenbo_Sun1","~Zhiyuan_Liu1","~Maosong_Sun1","~Kaifeng_Lyu2","~Wenguang_Chen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Kairong Luo","Haodong Wen","Shengding Hu","Zhenbo Sun","Zhiyuan Liu","Maosong Sun","Kaifeng Lyu","Wenguang Chen"]}},"version":2},{"content":{"venue":{"value":"ICASSP (4) 2006"},"pdf":{"value":"https://ieeexplore.ieee.org/iel5/11024/34760/01661037.pdf"},"venueid":{"value":"dblp.org/conf/ICASSP/2006"},"paperhash":{"value":"tugnait|direct_fir_linear_equalization_of_doubly_selective_channels_based_on_superimposed_training"},"authorids":{"value":["~Jitendra_K._Tugnait1","https://dblp.org/search/pid/api?q=author:Shuangchi_He:"]},"html":{"value":"https://doi.org/10.1109/ICASSP.2006.1661037"},"_bibtex":{"value":"@inproceedings{DBLP:conf/icassp/TugnaitH06,\n  author={Jitendra K. Tugnait and Shuangchi He},\n  title={Direct FIR Linear Equalization of Doubly Selective Channels Based on Superimposed Training},\n  year={2006},\n  cdate={1136073600000},\n  pages={589-592},\n  url={https://doi.org/10.1109/ICASSP.2006.1661037},\n  booktitle={ICASSP (4)},\n  crossref={conf/icassp/2006}\n}\n"},"abstract":{"value":"Design of doubly-selective linear equalizers for single user frequency-selective time-varying communications channels is considered using superimposed training and without first estimating the underlying channel response. Both the time-varying channel as well as the linear equalizers are assumed to be described by a complex exponential basis expansion model (CE-BEM). A periodic (non-random) training sequence is arithmetically added (superimposed) at a low power to the information sequence at the transmitter before modulation and transmission. There is no loss in information rate. Knowledge of the superimposed training is exploited to design the FIR linear equalizer. An illustrative simulation example is presented."},"title":{"value":"Direct FIR Linear Equalization of Doubly Selective Channels Based on Superimposed Training"},"authors":{"value":["Jitendra K. Tugnait","Shuangchi He"]}},"tmdate":1742648165959,"pdate":1136073600000,"tcdate":1742648019111,"writers":["~"],"signatures":["~Jitendra_Tugnait1"],"forum":"At5K3VqIZZ","license":"CC BY-SA 4.0","number":371368,"cdate":1136073600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1742648165959,"domain":"DBLP.org","id":"At5K3VqIZZ","version":2},{"content":{"venue":{"value":"ICASSP (4) 2006"},"pdf":{"value":"https://ieeexplore.ieee.org/iel5/11024/34760/01661037.pdf"},"venueid":{"value":"dblp.org/conf/ICASSP/2006"},"paperhash":{"value":"tugnait|direct_fir_linear_equalization_of_doubly_selective_channels_based_on_superimposed_training"},"authorids":{"value":["~Jitendra_K._Tugnait1","https://dblp.org/search/pid/api?q=author:Shuangchi_He:"]},"html":{"value":"https://doi.org/10.1109/ICASSP.2006.1661037"},"_bibtex":{"value":"@inproceedings{DBLP:conf/icassp/TugnaitH06,\n  author={Jitendra K. Tugnait and Shuangchi He},\n  title={Direct FIR Linear Equalization of Doubly Selective Channels Based on Superimposed Training},\n  year={2006},\n  cdate={1136073600000},\n  pages={589-592},\n  url={https://doi.org/10.1109/ICASSP.2006.1661037},\n  booktitle={ICASSP (4)},\n  crossref={conf/icassp/2006}\n}\n"},"abstract":{"value":"Design of doubly-selective linear equalizers for single user frequency-selective time-varying communications channels is considered using superimposed training and without first estimating the underlying channel response. Both the time-varying channel as well as the linear equalizers are assumed to be described by a complex exponential basis expansion model (CE-BEM). A periodic (non-random) training sequence is arithmetically added (superimposed) at a low power to the information sequence at the transmitter before modulation and transmission. There is no loss in information rate. Knowledge of the superimposed training is exploited to design the FIR linear equalizer. An illustrative simulation example is presented."},"title":{"value":"Direct FIR Linear Equalization of Doubly Selective Channels Based on Superimposed Training"},"authors":{"value":["Jitendra K. Tugnait","Shuangchi He"]}},"tmdate":1742648033146,"pdate":1136073600000,"tcdate":1742648005653,"writers":["~"],"signatures":["~Jitendra_Tugnait1"],"forum":"KjfXztzuIk","license":"CC BY-SA 4.0","number":371121,"cdate":1136073600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1742648033146,"domain":"DBLP.org","id":"KjfXztzuIk","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2404.13257v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"shao|stssms_spatialtemporal_selective_state_of_space_model_for_traffic_forecasting"},"html":{"value":"https://doi.org/10.48550/arXiv.2404.13257"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2404-13257,\n  publtype={informal},\n  author={Zhiqi Shao and Michael G. H. Bell and Ze Wang and D. Glenn Geers and Haoning Xi and Junbin Gao},\n  title={ST-SSMs: Spatial-Temporal Selective State of Space Model for Traffic Forecasting},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2404.13257},\n  url={https://doi.org/10.48550/arXiv.2404.13257}\n}\n"},"abstract":{"value":"Traffic flow prediction, a critical aspect of intelligent transportation systems, has been increasingly popular in the field of artificial intelligence, driven by the availability of extensive traffic data. The current challenges of traffic flow prediction lie in integrating diverse factors while balancing the trade-off between computational complexity and the precision necessary for effective long-range and large-scale predictions. To address these challenges, we introduce a Spatial-Temporal Selective State Space (ST-Mamba) model, which is the first to leverage the power of spatial-temporal learning in traffic flow prediction without using graph modeling. The ST-Mamba model can effectively capture the long-range dependency for traffic flow data, thereby avoiding the issue of over-smoothing. The proposed ST-Mamba model incorporates an effective Spatial-Temporal Mixer (ST-Mixer) to seamlessly integrate spatial and temporal data processing into a unified framework and employs a Spatial-Temporal Selective State Space (ST-SSM) block to improve computational efficiency. The proposed ST-Mamba model, specifically designed for spatial-temporal data, simplifies processing procedure and enhances generalization capabilities, thereby significantly improving the accuracy of long-range traffic flow prediction. Compared to the previous state-of-the-art (SOTA) model, the proposed ST-Mamba model achieves a 61.11\\% improvement in computational speed and increases prediction accuracy by 0.67\\%. Extensive experiments with real-world traffic datasets demonstrate that the \\textsf{ST-Mamba} model sets a new benchmark in traffic flow prediction, achieving SOTA performance in computational efficiency for both long- and short-range predictions and significantly improving the overall efficiency and effectiveness of traffic management."},"title":{"value":"ST-SSMs: Spatial-Temporal Selective State of Space Model for Traffic Forecasting"},"authors":{"value":[{"fullname":"Zhiqi Shao","username":""},{"fullname":"Michael G. H. Bell","username":""},{"fullname":"Ze Wang","username":""},{"fullname":"D. Glenn Geers","username":""},{"fullname":"Haoning Xi","username":"~Haoning_Xi1"},{"fullname":"Junbin Gao","username":""}]}},"tmdate":1785210968034,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2404-13257"],"tcdate":1785210963712,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Haoning_Xi1"],"forum":"qbXmFfHCPk","license":"CC BY-SA 4.0","number":108603,"cdate":1704067200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1785210968034,"domain":"OpenReview.net/Public_Article","id":"qbXmFfHCPk","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2404.13257v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"shao|stssms_spatialtemporal_selective_state_of_space_model_for_traffic_forecasting"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zhiqi_Shao:","https://dblp.org/search/pid/api?q=author:Michael_G._H._Bell:","https://dblp.org/search/pid/api?q=author:Ze_Wang:","https://dblp.org/search/pid/api?q=author:D._Glenn_Geers:","~Haoning_Xi2","~Junbin_Gao1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2404.13257"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2404-13257,\n  publtype={informal},\n  author={Zhiqi Shao and Michael G. H. Bell and Ze Wang and D. Glenn Geers and Haoning Xi and Junbin Gao},\n  title={ST-SSMs: Spatial-Temporal Selective State of Space Model for Traffic Forecasting},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2404.13257},\n  url={https://doi.org/10.48550/arXiv.2404.13257}\n}\n"},"abstract":{"value":"Traffic flow prediction, a critical aspect of intelligent transportation systems, has been increasingly popular in the field of artificial intelligence, driven by the availability of extensive traffic data. The current challenges of traffic flow prediction lie in integrating diverse factors while balancing the trade-off between computational complexity and the precision necessary for effective long-range and large-scale predictions. To address these challenges, we introduce a Spatial-Temporal Selective State Space (ST-Mamba) model, which is the first to leverage the power of spatial-temporal learning in traffic flow prediction without using graph modeling. The ST-Mamba model can effectively capture the long-range dependency for traffic flow data, thereby avoiding the issue of over-smoothing. The proposed ST-Mamba model incorporates an effective Spatial-Temporal Mixer (ST-Mixer) to seamlessly integrate spatial and temporal data processing into a unified framework and employs a Spatial-Temporal Selective State Space (ST-SSM) block to improve computational efficiency. The proposed ST-Mamba model, specifically designed for spatial-temporal data, simplifies processing procedure and enhances generalization capabilities, thereby significantly improving the accuracy of long-range traffic flow prediction. Compared to the previous state-of-the-art (SOTA) model, the proposed ST-Mamba model achieves a 61.11\\% improvement in computational speed and increases prediction accuracy by 0.67\\%. Extensive experiments with real-world traffic datasets demonstrate that the \\textsf{ST-Mamba} model sets a new benchmark in traffic flow prediction, achieving SOTA performance in computational efficiency for both long- and short-range predictions and significantly improving the overall efficiency and effectiveness of traffic management."},"title":{"value":"ST-SSMs: Spatial-Temporal Selective State of Space Model for Traffic Forecasting"},"authors":{"value":["Zhiqi Shao","Michael G. H. Bell","Ze Wang","D. Glenn Geers","Haoning Xi","Junbin Gao"]}},"tmdate":1785208683170,"pdate":1704067200000,"tcdate":1727474044615,"writers":["~"],"signatures":["~Junbin_Gao1"],"forum":"FsDEEhS9MD","license":"CC BY-SA 4.0","number":95154,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1785208683170,"domain":"DBLP.org","id":"FsDEEhS9MD","version":2},{"content":{"summary":{"value":"This paper tries to prove the theoretical possibility of training a transformer model in almost linear time, more specifically, calculating both the forward and backward passes of the attention layer in almost linear time, while having a small error of the order of 1/poly(n), where n represents the context length (or sequence length of the model). To do this, they use results from Alman & Song (2023,2024) as well as reordering the gradient calculations to stay in almost linear time.\n\nAlman, Josh, and Zhao Song. \"Fast attention requires bounded entries.\" Advances in Neural Information Processing Systems 36 (2023): 63117-63135.\n\nAlman, Josh, and Zhao Song. \"The fine-grained complexity of gradient computation for training large language models.\" Advances in Neural Information Processing Systems 37 (2024): 61419-61454."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- Were any practical experiments done, as claimed in the abstract?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":1},"strengths":{"value":"- The paper extends the results from Alman & Song to the full transformer (including multihead, MLP layers, and residual connections) to attempt to show that it is possible to train a transformer in almost linear time."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The paper is not very well-written and at times hard to follow or read. Additionally, the paper is difficult to understand without reading a substantial portion of the appendix. All proofs of the paper are relegated to the appendix, and the informal versions of the theorems in the paper state that they show that different parts of the model can be calculated in near-linear time without any intuition of strong reasoning as to why. It is hard to say whether the paper is the main paper or the appendix itself.\n- The paper claims multiple times that the results either \"transcend\" or represent a significant leap forward; however, most of the paper seems to be an application of two papers, which they cite and a reordering of the gradient calculations such that the operations stay near-linear.\n- In the abstract, the authors claim that they have done numerical experiments to show that their theory works; however, these are nowhere to be found. In addition, the two papers that reduce the calculation of attention to near linear time (forward and backwards) state that this could be applicable to highly quantised models (4 bits or less), but in non-quantised the U and V matrices would be infeasibly large (the $k_1$, would be in the order of millions, or billions, which would make them slower than quadratic calculations when n = 128k.\n- In multiple proofs in the appendix, the authors claim poly(n) / poly(n) <= 1/poly(n) (the poly(n)s are not derived from the same place and are in practice different) because the matrices can be written in log(n) bits; this assumption is very strong and not justified. Without it, the paper does not show that the error does not grow.\n- The last two weaknesses can be explained by a perceived misunderstanding of the related works, the central part of getting the attention operation to be near linear time with a small error is that the weights of the weight matrices can be bounded by $o(\\sqrt{log(n)})$ this however never shows in this paper (outside of lemma C.13)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925058577,"tcdate":1761959942364,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14692/Reviewer_rmB9"],"signatures":["ICLR.cc/2026/Conference/Submission14692/Reviewer_rmB9"],"forum":"ymidNjoH4V","number":2,"license":"CC BY 4.0","cdate":1761959942364,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14692/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925058577,"domain":"ICLR.cc/2026/Conference","replyto":"ymidNjoH4V","id":"0u0UJI157G","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Attention","Transformers","LLM","Training","Theory"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The computational complexity of the self-attention mechanism in popular transformer architectures poses significant challenges for training and inference, and becomes the bottleneck for long inputs. Is it possible to significantly reduce the quadratic time complexity of computing the gradients in multi-layer transformer models? This paper proves that a novel fast approximation method can calculate the gradients in almost linear time $n^{1+o(1)}$ where $n$ is the input sequence length, while it maintains a polynomially small approximation error $1 / \\mathrm{poly}(n)$ across the entire model. Our theory holds for general loss functions and when the multi-layer transformer model contains many practical sub-modules, such as residual connection, causal mask, and multi-head attention. We further validate our approach through numerical experiments, demonstrating both its high approximation fidelity and substantial speedups in practice.\nBy improving the efficiency of gradient computation, we hope that this work will facilitate more effective training and deployment of long-context language models based on our theoretical results."},"_bibtex":{"value":"@misc{\ncao2025training,\ntitle={Training Multi-Layer Transformers in Almost Linear Time},\nauthor={Yang Cao and Yingyu Liang and Zhizhou Sha and Zhenmei Shi and Zhao Song},\nyear={2025},\nurl={https://openreview.net/forum?id=ymidNjoH4V}\n}"},"title":{"value":"Training Multi-Layer Transformers in Almost Linear Time"},"pdf":{"value":"/pdf/dfc6682a4e19acfeac53e184cbc213d1ac235391.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"cao|training_multilayer_transformers_in_almost_linear_time"},"authorids":{"value":["~Yang_Cao26","~Yingyu_Liang1","~Zhizhou_Sha1","~Zhenmei_Shi1","~Zhao_Song3"]},"authors":{"value":["Yang Cao","Yingyu Liang","Zhizhou Sha","Zhenmei Shi","Zhao Song"]}},"version":2},{"content":{"summary":{"value":"In this paper, the authors investigate the problem of time series imputation. Since the authors find that the DDPM-based models are usually not capable of achieving sequence modeling with low time complexity and cannot handle the inter-variables and temporal dependence. The authors integrate Mamba and develop the bidirectional model for time series imputation and forecasting. The authors evaluate the proposed method on several datasets and achieve good performance."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"N.A."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"N.A."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tThe contribution is limited. This paper looks like a combination of the diffusion model and Mamba. Specifically, the advantage of low time complexity comes from Mamba, and the technique of changing the SSM-based block to a bidirectional one is not very new. A similar idea has been used in bi-LSTM.  \n2.\tIn addition, this article mainly solves the imputation problem, but the author considers time series forecasting, which makes the whole work seem to be pieced together.  \n3.\tFor fair comparison, authors should consider common imputation benchmarks such as the healthcare dataset in PhysioNet Challenge 2012 and air quality dataset.  \n4.\tAlthough the author mainly emphasizes that this method is better than other methods based on the diffusion model, this is not enough. The author should consider more time series imputation methods such as [1,2,3,4].  \n\n\n[1] Filling the Gaps: Multivariate Time Series Imputation by Graph Neural Networks.  \n[2] Gatgpt: A pre-trained large language model with graph attention network for spatiotemporal imputation.  \n[3] Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observations.  \n[4] Multi-Variate Time Series Forecasting on Variable Subsets."}},"nonreaders":[],"tmdate":1731427590226,"tcdate":1730895109467,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2397/Reviewer_2h3v"],"signatures":["ICLR.cc/2025/Conference/Submission2397/Reviewer_2h3v"],"forum":"j1OucVFZMJ","number":5,"license":"CC BY 4.0","cdate":1730895109467,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2397/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427590226,"domain":"ICLR.cc/2025/Conference","replyto":"j1OucVFZMJ","id":"GFmLF6htFY","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["diffusion","time series impuation","state space model"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Probabilistic time series imputation has been widely applied in real-world scenarios due to its ability to estimate uncertainty of imputation results. Meanwhile, denoising diffusion probabilistic models (DDPMs) have achieved great success in probabilistic time series imputation tasks with its power to model complex distributions. However, current DDPM-based probabilistic time series imputation methodologies are confronted with two types of challenges: 1) \\textit{ The backbone modules of the denoising parts are not capable of achieving sequence modeling with low time complexity.} 2) \\textit{ The architecture of denoising modules can not handle the inter-variable and bidirectional dependencies in the time series imputation problem effectively.} To address the first challenge, we integrate the computational efficient state space model, namely Mamba, as the backbone denosing module for DDPMs. To tackle the second challenge, we carefully devise several SSM-based blocks for bidirectional modeling and inter-variable relation understanding. Experimental results demonstrate that our approach can achieve state-of-the-art time series imputation results on multiple datasets, different missing scenarios and missing ratios."},"_bibtex":{"value":"@misc{\ngao2025diffimp,\ntitle={DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone},\nauthor={Hongfan Gao and Wangmeng Shen and Xiangfei Qiu and Ronghui Xu and Jilin Hu and Bin Yang},\nyear={2025},\nurl={https://openreview.net/forum?id=j1OucVFZMJ}\n}"},"title":{"value":"DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone"},"pdf":{"value":"/pdf/26faf0d7208528fa5a326d36fadf37ed808e0f38.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"gao|diffimp_efficient_diffusion_model_for_probabilistic_time_series_imputation_with_bidirectional_mamba_backbone"},"authorids":{"value":["~Hongfan_Gao1","~Wangmeng_Shen1","~Xiangfei_Qiu1","~Ronghui_Xu2","~Jilin_Hu1","~Bin_Yang4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hongfan Gao","Wangmeng Shen","Xiangfei Qiu","Ronghui Xu","Jilin Hu","Bin Yang"]}},"version":2},{"content":{"summary":{"value":"The authors propose a new method called PUM-Net thataugments a Mamba SSM with a dual-memory system to improve long-context performance with two different types of memory."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. Can you provide latency and throughput benchmarks comparing PUM-Net directly against the vanilla Mamba baseline, not just attention models.\n2. How does the brute-force search for the internal memory scale as sequence length (and thus, chunk count) increases?\n3. What is the cost and workflow to update the static external memory?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. It shows better perplexity on the 64k sequence than the Mamba baseline, exceeding its 4k training length\n2. It demonstrates much higher throughput and lower memory use compared to attention-based long-context models (NSA, Flash-Attention)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The claims regarding efficiency and throughput are misleading due to the omission of a direct speed comparison with the vanilla Mamba baseline. The proposed PUM-Net block is substantially more complex than a standard Mamba block—it involves a two-stage process and consumes more memory (Fig. 5, right). This complexity represents a significant trade-off, not a \"free\" upgrade, and almost certainly results in slower performance.\n2. the external knowledge is static, meaning it will inevitably become stale. The evaluation performance will inevitably determined by how well this external knowledge is trained.\n3. Since the memory creation process is not simple and relied on additional models, I am not sure about the scalability and universality of the method."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926744209,"tcdate":1762743005164,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16691/Reviewer_4tNR"],"signatures":["ICLR.cc/2026/Conference/Submission16691/Reviewer_4tNR"],"forum":"15KpLriUTU","number":4,"license":"CC BY 4.0","cdate":1762743005164,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16691/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926744209,"domain":"ICLR.cc/2026/Conference","replyto":"15KpLriUTU","id":"LM4mPgsN8B","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["State Space Model","Long Context","Internal Memory","External Memory"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Recent Mamba-family State Space Models (SSMs) leverage linear recurrent dynamics to achieve efficiency in long-sequence modeling. However, their exponential decay kernels inevitably cause rapid forgetting, making it difficult to recall information once sequence lengths far exceed the training horizon. Inspired by advances in Transformer-based architectures such as Native Sparse Attention (NSA), which employ internal chunk memory to alleviate long-term forgetting, we extend Mamba with a chunk-wise internal long-term memory that improves the retrieval of distant context. While this enhances in-context recall, a more fundamental challenge for long-context models is the ability to access and integrate vast external world knowledge without compromising efficiency. Existing retrieval-augmented generation (RAG) approaches attempt to address this by appending retrieved documents to queries, which substantially increases training cost and fails to fully integrate internal and external memory representations. To overcome these limitations, we propose the Plastic Unified Memory Network (PUM-Net), a unified dual-memory architecture that, for the first time, enables joint pre-training over both dynamic internal memory and static, pre-encoded external knowledge. This plastic unification allows external memory to refine internal states during training, enabling bidirectional interaction without inflating sequence length, thereby supporting more effective long-context modeling and achieving substantial improvements across long-range challenging benchmarks."},"_bibtex":{"value":"@misc{\nliu2026pumnet,\ntitle={{PUM}-Net: Plastic Unified Memory Network with Associative Interaction for Long-Context State Space Models},\nauthor={Yuchen Liu and Yinghao Zhang and Deshu Chen and Hongwei Zhang and Yichi Zhang and Zhijian Zhou and Xin Guo and Yuan Cheng and Chao Qu and Yuan Qi},\nyear={2026},\nurl={https://openreview.net/forum?id=15KpLriUTU}\n}"},"title":{"value":"PUM-Net: Plastic Unified Memory Network with Associative Interaction for Long-Context State Space Models"},"pdf":{"value":"/pdf/cf765fd6e34b8efc3d037179daea5b3566db079a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"liu|pumnet_plastic_unified_memory_network_with_associative_interaction_for_longcontext_state_space_models"},"authorids":{"value":["~Yuchen_Liu26","~Yinghao_Zhang2","~Deshu_Chen2","~Hongwei_Zhang3","~Yichi_Zhang3","~Zhijian_Zhou2","~Xin_Guo2","~Yuan_Cheng1","~Chao_Qu3","~Yuan_Qi2"]},"authors":{"value":["Yuchen Liu","Yinghao Zhang","Deshu Chen","Hongwei Zhang","Yichi Zhang","Zhijian Zhou","Xin Guo","Yuan Cheng","Chao Qu","Yuan Qi"]}},"version":2},{"content":{"TLDR":{"value":"This proposal explores implementing a Mamba architecture for efficient protein sequence generation, aiming to balance performance with reduced computational resource demands and making the architecture more suited to biological dynamics."},"venue":{"value":"THU 2024 Fall AML Submission"},"pdf":{"value":"/pdf/8d5ee7449a5695662b51c83289c76642879f2be4.pdf"},"keywords":{"value":["protein sequence generation","Mamba architecture","efficient modeling","protein language models","biological dynamics"]},"venueid":{"value":"tsinghua.edu.cn/THU/2024/Fall/AML/Submission"},"paperhash":{"value":"jiang|[proposalml]protein_sequence_generation_model"},"authorids":{"value":["~pan_jiang1","zeeshanzulfiqar79@outlook.com","ivanyazykov.mipt@gmail.com"]},"abstract":{"value":"This proposal outlines a novel approach for protein sequence generation, a key area in computational biology focused on generating sequences with specific functional and structural characteristics. Despite recent advances with deep learning models, challenges remain in handling long sequences and capturing biological dynamics. We propose implementing a Mamba architecture that leverages selective state-space updates to achieve efficient protein sequence generation. With linear computational complexity and effective handling of long-range dependencies, this architecture is designed to be more suited to biological dynamics while reducing resource requirements. A comprehensive evaluation on the UniRef50 dataset will demonstrate its potential to deliver competitive performance, providing a viable solution for research environments with limited computational resources."},"_bibtex":{"value":"@inproceedings{\njiang2024proposalmlprotein,\ntitle={[Proposal-{ML}]Protein Sequence Generation Model},\nauthor={pan jiang and Zeeshan Zulfiqar and Ivan Iazykov},\nbooktitle={Submitted to Tsinghua University Course: Advanced Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=I2FbCxyYVS},\nnote={under review}\n}"},"title":{"value":"[Proposal-ML]Protein Sequence Generation Model"},"authors":{"value":["pan jiang","Zeeshan Zulfiqar","Ivan Iazykov"]}},"tmdate":1730803172830,"tcdate":1730298136808,"writers":["tsinghua.edu.cn/THU/2024/Fall/AML","tsinghua.edu.cn/THU/2024/Fall/AML/Submission52/Authors"],"signatures":["tsinghua.edu.cn/THU/2024/Fall/AML/Submission52/Authors"],"forum":"I2FbCxyYVS","license":"CC BY-SA 4.0","number":52,"cdate":1730298136808,"readers":["everyone"],"invitations":["tsinghua.edu.cn/THU/2024/Fall/AML/-/Submission","tsinghua.edu.cn/THU/2024/Fall/AML/-/Post_Submission"],"mdate":1730803172830,"odate":1730803170633,"domain":"tsinghua.edu.cn/THU/2024/Fall/AML","id":"I2FbCxyYVS","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2404.09498v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"xie|fusionmamba_dynamic_feature_enhancement_for_multimodal_image_fusion_with_mamba"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xinyu_Xie:","https://dblp.org/search/pid/api?q=author:Yawen_Cui:","https://dblp.org/search/pid/api?q=author:Chio-In_Ieong:","https://dblp.org/search/pid/api?q=author:Tao_Tan:","https://dblp.org/search/pid/api?q=author:Xiaozhi_Zhang:","~Xubin_Zheng2","https://dblp.org/search/pid/api?q=author:Zitong_Yu:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2404.09498"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2404-09498,\n  publtype={informal},\n  author={Xinyu Xie and Yawen Cui and Chio-In Ieong and Tao Tan and Xiaozhi Zhang and Xubin Zheng and Zitong Yu},\n  title={FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2404.09498},\n  url={https://doi.org/10.48550/arXiv.2404.09498}\n}\n"},"abstract":{"value":"Multimodal image fusion aims to integrate information from different imaging techniques to produce a comprehensive, detail-rich single image for downstream vision tasks. Existing methods based on local convolutional neural networks (CNNs) struggle to capture global features efficiently, while Transformer-based models are computationally expensive, although they excel at global modeling. Mamba addresses these limitations by leveraging selective structured state space models (S4) to effectively handle long-range dependencies while maintaining linear complexity. In this paper, we propose FusionMamba, a novel dynamic feature enhancement framework that aims to overcome the challenges faced by CNNs and Vision Transformers (ViTs) in computer vision tasks. The framework improves the visual state-space model Mamba by integrating dynamic convolution and channel attention mechanisms, which not only retains its powerful global feature modeling capability, but also greatly reduces redundancy and enhances the expressiveness of local features. In addition, we have developed a new module called the dynamic feature fusion module (DFFM). It combines the dynamic feature enhancement module (DFEM) for texture enhancement and disparity perception with the cross-modal fusion Mamba module (CMFM), which focuses on enhancing the inter-modal correlation while suppressing redundant information. Experiments show that FusionMamba achieves state-of-the-art performance in a variety of multimodal image fusion tasks as well as downstream experiments, demonstrating its broad applicability and superiority."},"title":{"value":"FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba"},"authors":{"value":["Xinyu Xie","Yawen Cui","Chio-In Ieong","Tao Tan","Xiaozhi Zhang","Xubin Zheng","Zitong Yu"]}},"tmdate":1753978165325,"pdate":1704067200000,"externalIds":["dblp:journals/corr/abs-2404-09498"],"tcdate":1753978148901,"writers":["~"],"signatures":["~Xubin_Zheng2"],"forum":"fregRfd6Tk","license":"CC BY-SA 4.0","number":605107,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1753978165325,"domain":"DBLP.org","id":"fregRfd6Tk","version":2},{"content":{"venue":{"value":"Vis. Intell. 2024"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/s44267-024-00072-9.pdf"},"venueid":{"value":"dblp.org/journals/VISINTELLIGENCE/2024"},"paperhash":{"value":"xie|fusionmamba_dynamic_feature_enhancement_for_multimodal_image_fusion_with_mamba"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xinyu_Xie:","https://dblp.org/search/pid/api?q=author:Yawen_Cui:","https://dblp.org/search/pid/api?q=author:Tao_Tan:","~Xubin_Zheng2","https://dblp.org/search/pid/api?q=author:Zitong_Yu:"]},"html":{"value":"https://doi.org/10.1007/s44267-024-00072-9"},"_bibtex":{"value":"@article{DBLP:journals/visintelligence/XieCTZY24,\n  author={Xinyu Xie and Yawen Cui and Tao Tan and Xubin Zheng and Zitong Yu},\n  title={FusionMamba: dynamic feature enhancement for multimodal image fusion with Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={Vis. Intell.},\n  volume={2},\n  number={1},\n  pages={37},\n  url={https://doi.org/10.1007/s44267-024-00072-9}\n}\n"},"abstract":{"value":"Multimodal image fusion aims to integrate information from different imaging techniques to produce a comprehensive, detail-rich single image for downstream vision tasks. Existing methods based on local convolutional neural networks (CNNs) struggle to capture global features efficiently, while Transformer-based models are computationally expensive, although they excel at global modeling. Mamba addresses these limitations by leveraging selective structured state space models (S4) to effectively handle long-range dependencies while maintaining linear complexity. In this paper, we propose FusionMamba, a novel dynamic feature enhancement framework that aims to overcome the challenges faced by CNNs and Vision Transformers (ViTs) in computer vision tasks. The framework improves the visual state-space model Mamba by integrating dynamic convolution and channel attention mechanisms, which not only retains its powerful global feature modeling capability, but also greatly reduces redundancy and enhances the expressiveness of local features. In addition, we have developed a new module called the dynamic feature fusion module (DFFM). It combines the dynamic feature enhancement module (DFEM) for texture enhancement and disparity perception with the cross-modal fusion Mamba module (CMFM), which focuses on enhancing the inter-modal correlation while suppressing redundant information. Experiments show that FusionMamba achieves state-of-the-art performance in a variety of multimodal image fusion tasks as well as downstream experiments, demonstrating its broad applicability and superiority."},"title":{"value":"FusionMamba: dynamic feature enhancement for multimodal image fusion with Mamba"},"authors":{"value":["Xinyu Xie","Yawen Cui","Tao Tan","Xubin Zheng","Zitong Yu"]}},"tmdate":1753978160074,"pdate":1704067200000,"externalIds":["dblp:journals/visintelligence/XieCTZY24"],"tcdate":1753978149080,"writers":["~"],"signatures":["~Xubin_Zheng2"],"forum":"4XD63gU1B1","license":"CC BY-SA 4.0","number":605109,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1753978160074,"domain":"DBLP.org","id":"4XD63gU1B1","version":2},{"content":{"summary":{"value":"This work introduces a frequency-selective state-space module that leverages adaptive spectral features for effective object detection and speech recognition. The core idea can be viewed as combining a learnable spectral module, such as SincNet or Garbor filter, with a Mamba-style module that enables adaptive state-space modeling."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"* Based on my understanding, [1] is highly relevant to your work. Comparing your method with [1] would help clarify the advantage of modeling spectral bands, which [1] does not address. Have you attempted such a comparison, that is, spectral band learning + adaptive state-space modeling versus frequency-feature learning + adaptive state-space modeling?\n\n[1] RFMamba: Frequency-Aware State Space Model for RF-Based Human-Centric Perception  - ICLR 24"},"rating":{"value":4},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"* The proposed method yields empirical performance improvements in radar object detection and segmentation tasks and demonstrates greater robustness than the baselines in the audio keyword detection experiments."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* [1] combines frequency features with adaptive state-space modeling, which appears to be the most relevant prior work. This work attempts to distinguish itself from [1] by learning the frequency bands; however, it is unclear what new benefits this actually provides. The advantage of this distinction is not clearly explained. An empirical comparison with [1] also seems to be missing; if I am wrong, please correct me.\n\n\n* The proposed method seems tailored to specific tasks  (e.g., radar object detection vs. speech), according to the experiment section. This raises concerns about its generalizability and consistency to other datasets, particularly depending on how the data should be processed for spectral representation.\n\n[1] RFMamba: Frequency-Aware State Space Model for RF-Based Human-Centric Perception  - ICLR 24"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942581499,"tcdate":1762274559421,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission23270/Reviewer_mX8w"],"signatures":["ICLR.cc/2026/Conference/Submission23270/Reviewer_mX8w"],"forum":"SyTuymSbBJ","number":3,"license":"CC BY 4.0","cdate":1762274559421,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission23270/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942581499,"domain":"ICLR.cc/2026/Conference","replyto":"SyTuymSbBJ","id":"Ra6uDlfvvF","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We propose a state-space model for learning spectral representations of time series data inspired by discrete Fourier transforms."},"keywords":{"value":["frequency spectrum","state space models","spectral operators","band-selective kernels","radar object detection","audio keyword recognition","sequence models"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"We introduce the first state space model (FSSM) with frequency selective spectral operators, parameterizing a family of stable, causal, band-selective kernels whose spectral weights are conditioned on the end task. This yields a representation that adapts its characteristics per task domain while retaining linear-time inference and memory. The key novelty is the trainable spectral front-end through which the model can adapt frequency weighting and inter-bin window size. We show the effectiveness of our learned spectral representations on two independent domains: radar object detection and speech keyword recognition, outperforming state of the art frequency based methods in both domains while maintaining competitive throughput and computational overhead. We further show the robustness of our approach under input perturbations, demonstrating the value of stabilized sequential operators in spectral representation learning."},"_bibtex":{"value":"@misc{\nsen2026fssm,\ntitle={{FSSM}: Frequency-Selective State Space Models for Spectral Representation Learning},\nauthor={Anuvab Sen and Mir Sayeed Mohammad and Saibal Mukhopadhyay},\nyear={2026},\nurl={https://openreview.net/forum?id=SyTuymSbBJ}\n}"},"title":{"value":"FSSM: Frequency-Selective State Space Models for Spectral Representation Learning"},"pdf":{"value":"/pdf/c79f7107ce97de50e6f86b3e654576f0e1200e0c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"sen|fssm_frequencyselective_state_space_models_for_spectral_representation_learning"},"authorids":{"value":["~Anuvab_Sen1","~Mir_Sayeed_Mohammad1","~Saibal_Mukhopadhyay2"]},"authors":{"value":["Anuvab Sen","Mir Sayeed Mohammad","Saibal Mukhopadhyay"]}},"version":2},{"content":{"TLDR":{"value":"State Space Models (MAMBA) achieve comparable performance to Transformers in molecular generation while offering improved computational efficiency, particularly for larger models."},"venue":{"value":"AIDrugX Poster"},"pdf":{"value":"/pdf/599c0fd30083d607ed030095678c5f43071f2959.pdf"},"keywords":{"value":["state space architecture","transformer architecture","drug discovery","de novo generation"]},"venueid":{"value":"NeurIPS.cc/2024/Workshop/AIDrugX"},"paperhash":{"value":"lombard|molecular_generation_with_state_space_sequence_models"},"authorids":{"value":["~Anri_Lombard1","shane.acton@alumni.uct.ac.za","~Ulrich_Armel_Mbou_Sob1","~Jan_Buys1"]},"abstract":{"value":"Molecular generation is a critical task in drug discovery but current approaches often struggle with efficiency and scalability when dealing with complex molecular structures. \nThis paper aims to address these challenges by training and evaluating models for molecular generation using the MAMBA State Space Model architecture.\nWe develop models with 20M and 90M parameters trained on the MOSES and ZINC datasets, respectively, using the Sequential Attachment-based Fragment Embedding (SAFE) representation. \nWe compare MAMBA models against the prevailing Transformer architecture in terms of generation quality and computational efficiency.\nOur findings suggest that MAMBA models can achieve performance comparable to Transformers in generating valid, unique, and diverse molecules. Generation from both architectures can achieve close to perfect validity and uniqueness scores, although MAMBA models require more conservative sampling parameters or regeneration steps to achieve these results. MAMBA models consistently demonstrates lower perplexity and reduced GPU power consumption (up to 30\\% reduction) compared to Transformer models. \nThese results indicate that State Space Models may offer a computationally efficient alternative for molecular generation tasks, potentially enabling more efficient processing of larger datasets and complex molecular structures. \nThe efficiency gains of MAMBA models become more pronounced with longer sequences, suggesting that this architecture could enable the modeling and generation of more complex molecules. This capability could significantly expand the scope of AI-driven molecular design in drug discovery."},"_bibtex":{"value":"@inproceedings{\nlombard2024molecular,\ntitle={Molecular Generation with State Space Sequence Models},\nauthor={Anri Lombard and Shane Acton and Ulrich Armel Mbou Sob and Jan Buys},\nbooktitle={NeurIPS 2024 Workshop on AI for New Drug Modalities},\nyear={2024},\nurl={https://openreview.net/forum?id=1ib5oyTQIb}\n}"},"title":{"value":"Molecular Generation with State Space Sequence Models"},"authors":{"value":["Anri Lombard","Shane Acton","Ulrich Armel Mbou Sob","Jan Buys"]}},"tmdate":1733034241997,"pdate":1728807695683,"tcdate":1727869352607,"writers":["NeurIPS.cc/2024/Workshop/AIDrugX","NeurIPS.cc/2024/Workshop/AIDrugX/Submission138/Authors"],"signatures":["NeurIPS.cc/2024/Workshop/AIDrugX/Submission138/Authors"],"forum":"1ib5oyTQIb","license":"CC BY 4.0","number":138,"cdate":1727869352607,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Workshop/AIDrugX/-/Submission","NeurIPS.cc/2024/Workshop/AIDrugX/-/Post_Submission","NeurIPS.cc/2024/Workshop/AIDrugX/-/Edit","NeurIPS.cc/2024/Workshop/AIDrugX/Submission138/-/Revision"],"mdate":1733034241997,"odate":1733034241985,"domain":"NeurIPS.cc/2024/Workshop/AIDrugX","id":"1ib5oyTQIb","version":2},{"content":{"comment":{"value":"**5. Robustness and state-norm diagnostics.**\nWe also added robustness evaluation under true test-time input perturbations: Gaussian noise, spikes, block masks, amplitude shifts, and last-value-hold corruptions. The regularizer does not improve robust forecasting MSE; robust MSE remains essentially unchanged. However, it strongly improves internal Mamba stability diagnostics under perturbations.\n\nUnder perturbed inputs, the baseline has mean sampled LMI violation 0.02345, while the LMI-regularized model has mean violation 0.00512, roughly a 78% reduction. The mean max SSM state norm decreases from 0.26651 to 0.25588, and the worst observed state norm decreases from approximately 0.6065 to 0.5387. Thus, the current method improves internal passivity/stability diagnostics under perturbations without degrading clean or robust forecasting accuracy.\n\nWe will not claim robust-MSE improvement. Instead, we will state that the method reduces sampled Mamba-core violations and state excursions under corrupted inputs.\n\n**6. Router-stability experiment.**\nBecause SST includes a router between the Mamba long-range expert and the short-range branch, we also tested an energy-aware router penalty that discourages high Mamba weight on samples with high local LMI violation. This experiment is mostly neutral. It preserves clean MSE and slightly reduces Mamba usage in all paired runs, but the effect is very small:\n\n$$\\Delta m_{\\mathrm{weight}}\\approx -2.2\\times 10^{-5}$$\n\nThe correlation between Mamba weight and LMI violation remains close to zero. We therefore do not present this as a main positive result; instead, we will include it as an appendix ablation and note that stronger perturbation-aware router training is future work.\n\n**7. Improving accessibility and modeling clarity.**\nWe will revise the introduction to delay technical notation and first explain the model intuitively: selective SSMs evolve through linear state dynamics whose coefficients are modulated by token-dependent gates. We will then introduce the distinction between $x(t)$, the selection/scheduling signal, and $u(t)$, the port input appearing in the passivity supply rate. We will also clarify that $A_{\\mathrm{eff}}$ denotes the effective state matrix obtained after substituting the current gating/selection values into the parameter maps. Finally, we will explain Eq. (1) as a continuous-time abstraction of selective SSMs and connect it later to the discrete Mamba recurrence used in the experiments.\n\n**Summary.**\nIn response to your review, we will revise the paper to:\n\n1. Clarify the novelty relative to classical passivity/ISS/switched-system theory.\n2. Add a discrete-time Mamba LMI derivation.\n3. Add SST/Mamba experiments over 28 dataset-horizon pairs.\n4. Include $P=I$, diagonal-$Q$, and projection ablations.\n5. Report robustness and state-norm diagnostics under input corruptions.\n6. Improve the introduction and modeling explanations.\n\nWe believe these changes substantially strengthen both the practical relevance and the positioning of the theoretical contribution."},"title":{"value":"[Part II] Response to Reviewer 5yPE"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1778859502682,"tcdate":1778859502682,"writers":["TMLR","TMLR/Paper7659/Authors"],"signatures":["TMLR/Paper7659/Authors"],"forum":"7Vav53cDeN","number":4,"license":"CC BY 4.0","cdate":1778859502682,"readers":["everyone"],"invitations":["TMLR/Paper7659/-/Official_Comment"],"mdate":1778859502682,"domain":"TMLR","replyto":"wDGq0w7i9D","id":"q05zMr9QZ0","forumContent":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"supplementary_material":{"value":"/attachment/62e66b6a21ab636b3ab233ff3c733762ba104f8b.pdf"},"abstract":{"value":"Selective State-Space Models (SSMs) such as Mamba have become central to long-sequence modeling. Still, their stability is poorly understood: their state-space coefficients are modulated online by a token-dependent gating signal, making the recurrence neither linear time-invariant nor classically nonlinear. We study continuous-time selective SSMs through passivity, dissipativity, and Input-to-State Stability (ISS), explicitly separating the selection signal $x(\\cdot)$ from the driving input $u(\\cdot)$. We obtain four results: exponential forgetting under strict dissipativity; a canonical $\\mathrm{AUC}_{\\mathrm{loc}}$ quadratic storage for the frozen-selection subsystem that accommodates discontinuous gating; a parametric LMI together with universal kernel constraints and \"irreversible forgetting\" under universal quadratic storage; and sufficient conditions for global ISS uniformly over admissible selection schedules. We then bridge to practice by deriving a sampled block LMI for the Mamba selective-scan core, which is used as a differentiable training-time regularizer. Across seven standard time-series datasets and four prediction horizons, the regularizer reduces sampled Mamba-core LMI violations by roughly 92% in 28/28 pairs at a clean-MSE cost of less than 0.018%. It improves internal Mamba passivity and state-norm diagnostics under injected perturbations. Our results turn classical control-theoretic tools into verifiable structural and training criteria for selective SSMs, while honestly scoping which guarantees transfer to a deep selective-scan architecture."},"_bibtex":{"value":"@article{\nzubic2026regularity,\ntitle={Regularity and Stability Properties of Selective {SSM}s with Discontinuous Gating},\nauthor={Nikola Zubic and Davide Scaramuzza},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=7Vav53cDeN},\nnote={}\n}"},"title":{"value":"Regularity and Stability Properties of Selective SSMs with Discontinuous Gating"},"changes_since_last_submission":{"value":"Accepted after rebuttal revision \"as is\"."},"pdf":{"value":"/pdf/348c094a1371853757a1a70f1e2b4549fe4adfb2.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"zubic|regularity_and_stability_properties_of_selective_ssms_with_discontinuous_gating"},"authorids":{"value":["~Nikola_Zubic1","~Davide_Scaramuzza1"]},"assigned_action_editor":{"value":"~Yoshinobu_Kawahara1"},"authors":{"value":["Nikola Zubic","Davide Scaramuzza"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2502.20764v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"wang|visual_attention_exploration_in_visionbased_mamba_models"},"authorids":{"value":["~Junpeng_Wang1","~Chin-Chia_Michael_Yeh1","https://dblp.org/search/pid/api?q=author:Uday_Singh_Saini:","https://dblp.org/search/pid/api?q=author:Mahashweta_Das:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2502.20764"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2502-20764,\n  publtype={informal},\n  author={Junpeng Wang and Chin-Chia Michael Yeh and Uday Singh Saini and Mahashweta Das},\n  title={Visual Attention Exploration in Vision-Based Mamba Models},\n  year={2025},\n  month={February},\n  cdate={1738368000000},\n  journal={CoRR},\n  volume={abs/2502.20764},\n  url={https://doi.org/10.48550/arXiv.2502.20764}\n}\n"},"abstract":{"value":"State space models (SSMs) have emerged as an efficient alternative to transformer-based models, offering linear complexity that scales better than transformers. One of the latest advances in SSMs, Mamba, introduces a selective scan mechanism that assigns trainable weights to input tokens, effectively mimicking the attention mechanism. Mamba has also been successfully extended to the vision domain by decomposing 2D images into smaller patches and arranging them as 1D sequences. However, it remains unclear how these patches interact with (or attend to) each other in relation to their original 2D spatial location. Additionally, the order used to arrange the patches into a sequence also significantly impacts their attention distribution. To better understand the attention between patches and explore the attention patterns, we introduce a visual analytics tool specifically designed for vision-based Mamba models. This tool enables a deeper understanding of how attention is distributed across patches in different Mamba blocks and how it evolves throughout a Mamba model. Using the tool, we also investigate the impact of different patch-ordering strategies on the learned attention, offering further insights into the model's behavior."},"title":{"value":"Visual Attention Exploration in Vision-Based Mamba Models"},"authors":{"value":["Junpeng Wang","Chin-Chia Michael Yeh","Uday Singh Saini","Mahashweta Das"]}},"tmdate":1747344564987,"pdate":1735689600000,"tcdate":1747323835552,"writers":["~"],"signatures":["~Junpeng_Wang1"],"forum":"oqh1MkadEP","license":"CC BY-SA 4.0","number":493030,"cdate":1738368000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747344564987,"domain":"DBLP.org","id":"oqh1MkadEP","version":2},{"content":{"venue":{"value":"MICCAI (8) 2024"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-031-72111-3_48.pdf"},"venueid":{"value":"dblp.org/conf/MICCAI/2024"},"paperhash":{"value":"xu|polypmamba_polyp_segmentation_with_visual_mamba"},"authorids":{"value":["~Zhongxing_Xu1","~Feilong_Tang2","https://dblp.org/search/pid/api?q=author:Zhe_Chen:","https://dblp.org/search/pid/api?q=author:Zheng_Zhou:","https://dblp.org/search/pid/api?q=author:Weishan_Wu:","https://dblp.org/search/pid/api?q=author:Yuyao_Yang:","https://dblp.org/search/pid/api?q=author:Yu_Liang:","https://dblp.org/search/pid/api?q=author:Jiyu_Jiang:","https://dblp.org/search/pid/api?q=author:Xuyue_Cai:","~Jionglong_Su1"]},"html":{"value":"https://doi.org/10.1007/978-3-031-72111-3_48"},"_bibtex":{"value":"@inproceedings{DBLP:conf/miccai/XuTCZWYLJCS24,\n  author={Zhongxing Xu and Feilong Tang and Zhe Chen and Zheng Zhou and Weishan Wu and Yuyao Yang and Yu Liang and Jiyu Jiang and Xuyue Cai and Jionglong Su},\n  title={Polyp-Mamba: Polyp Segmentation with Visual Mamba},\n  year={2024},\n  cdate={1704067200000},\n  pages={510-521},\n  url={https://doi.org/10.1007/978-3-031-72111-3_48},\n  booktitle={MICCAI (8)},\n  crossref={conf/miccai/2024-8}\n}\n"},"abstract":{"value":"Accurate segmentation of polyps is crucial for efficient colorectal cancer detection during the colonoscopy screenings. State Space Models, exemplified by Mamba, have recently emerged as a promising approach, excelling in long-range interaction modeling with linear computational complexity. However, previous methods do not consider the cross-scale dependencies of different pixels and the consistency in feature representations and semantic embedding, which are crucial for polyp segmentation. Therefore, we introduce Polyp-Mamba, a novel unified framework aimed at overcoming the above limitations by integrating multi-scale feature learning with semantic structure analysis. Specifically, our framework includes a Scale-Aware Semantic module that enables the embedding of multi-scale features from the encoder to achieve semantic information modeling across both intra- and inter-scales, rather than the single-scale approach employed in prior studies. Furthermore, the Global Semantic Injection module is deployed to inject scale-aware semantics into the corresponding decoder features, aiming to fuse global and local information and enhance pyramid feature representation. Experimental results across five challenging datasets and six metrics demonstrate that our proposed method not only surpasses state-of-the-art methods but also sets a new benchmark in the field, underscoring the Polyp-Mamba framework’s exceptional proficiency in the polyp segmentation tasks."},"title":{"value":"Polyp-Mamba: Polyp Segmentation with Visual Mamba"},"authors":{"value":["Zhongxing Xu","Feilong Tang","Zhe Chen","Zheng Zhou","Weishan Wu","Yuyao Yang","Yu Liang","Jiyu Jiang","Xuyue Cai","Jionglong Su"]}},"tmdate":1752920197994,"pdate":1704067200000,"tcdate":1741145206008,"writers":["~"],"signatures":["~Zhongxing_Xu1"],"forum":"FA3mnlmYo9","license":"CC BY-SA 4.0","number":351536,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1752920197994,"domain":"DBLP.org","id":"FA3mnlmYo9","version":2},{"content":{"summary":{"value":"The paper follows the meta continual learning (MCL) framework as outlined by Lee et al., 2024. The authors meta-train sequential models on offline meta-training sequences to enhance their sequence modelling capability. The authors propose using Mamba as the sequential model instead of transformers or attention-free variants to alleviate high computational costs while still achieving satisfactory performance. Additionally, the authors introduce a selective regularization technique for meta-training, which enhances the association between query tokens and previously correlated input tokens. Experimental results demonstrate that Mamba achieves improved generalization and robustness compared to transformer variants in MCL, while using less memory for inference."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"I am open to discussion and willing to reconsider my score if my major concerns can be adequately addressed.\n\n\n**Claims on the Effectiveness of the Proposed Regularization Technique**\n\n- For example, lines 326-329 state:\n  > We apply this regularization to MambaCL and other sequence prediction models (weighted by a scalar λ) together with the MCL objective in Eq. (7), which improves the meta-training stability and convergence for all models.\n\n- The authors do not fully support their claims about \"improving the meta-training stability and convergence for all models.\" Specifically, there are no experiments showing learning curves (or similar alternatives) for all models during meta-training to compare results with and without this technique. \n\n- A seemingly related empirical evidence is presented in Figure 4. However, the results appear to pertain to a *single* model, and it is unclear, based on the figure caption and the text in lines 481-485, which specific model (i.e., Mamba, transformers) was used in this ablation study. Although the experiment demonstrates the sensitivity of meta-testing performance to the regularization strength, it lacks comprehensive evidence across multiple models to support the authors claim.\n\n\n**Experiment Implementation Details**\n\n- In the paper, it is mentioned: \n  > Following Lee et al., 2024, we set the initial learning rate to 1 × 10⁻⁴...\n\n- Cloud the authors please provide some motivations for using the same hyperparameters as in Lee et al., 2024, given that the meta-training setups differ? Specifically, the authors used a pre-trained CLIP backbone as a visual encoder and included the proposed regularization loss across all models. \n\n- Moreover, were these hyperparameters adjusted for different model architectures based on some meta-validation sets, e.g., for linear transformers and Mamba? If not, wouldn't using fixed hyperparameters for all experiments and models potentially lead to sub-optimal results? If these hyperparameters are not optimal for every models, this could produce misleading results and potentially invalidate the observations.\n\n**Meta-Overfitting in Figures 3a and 3b**\n\n- The authors observed that transformers and their variants seem to suffer from severe meta-overfitting based on the results in Figures 3a and 3b. However, the potential underlying causes for this overfitting are quite unclear. Specifically:\n\n  - As previously mentioned, based on the current description of the implementation details, it's unclear whether this overfitting is due to the use of improper hyperparameters, such as learning rates.\n\n  - Additionally, it is undetermined whether this overfitting is influenced by the use of regularization terms for all models during meta-training. Would removing this regularization loss for transformers significantly reduce meta-overfitting?\n\n- Could the authors please provide some insights into why Mamba did not suffer from the same degree of overfitting?\n\n- While the occurrence of meta-overfitting is expected, the degree of overfitting—particularly in relation to the number of training tasks and training shots used in meta-training—exhibited by transformers and their variants in Figures 3a and 3b is somewhat surprising. Specifically, in Figure 3b, adding more training shots per class even, and almost monotonically, decreased the classification accuracy on the queries.\n\n\n**Robustness in Figure 3c**\n\n- It is somewhat unclear how the authors performed the input noise perturbation. Specifically, what does $ x_i$ in line 473 refer to? Is it the original input image to the CLIP encoder, or the extracted image embeddings that serve as inputs to the sequential learning models?\n\n- I find it very interesting that Mamba exhibits excellent robustness to input noise, even with a standard deviation as large as 10. Could the authors potentially discuss some potential reasons behind Mamba's extreme robustness to large input noise?\n\n**General Comments on MCL**\n\n- Some important challenges in the MCL setup for continual learning include: 1) its application to long continual learning sequences, 2) the requirement for offline training datasets (meta-training), and 3) generalization to unseen long OOD meta-testing tasks. These challenges cannot be resolved simply by switching from transformers or their variants to Mamba.\n\n- Are there any differences on the problem formulation and the meta-training setups between the ones in the paper and the one in MetaICL: Learning to Learn In Context, Min et al., NAACL 2022?"},"rating":{"value":3},"details_of_ethics_concerns":{"value":"n/a"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The paper is well-structured and easy to follow.\n- The authors clearly explained the issue of increased compute complexity with using transformers for MCL."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"In general:\n\n- The paper shows limited novelty. The problem formulation, specifically the recasting of the continual learning problem as a sequential modelling problem in recurrent models, mirrors the previous work by Lee et al., 2024. From the technical side, the authors propose a new selective regularization technique for meta-training and claim it improves training stability and convergence. While the technique itself is novel, there are several questionable aspects regarding this technique and the authors' claims. I cannot fully credit the novelty of this technique until these issues are addressed.\n\n- Although the authors claim better generalization and robustness when using Mamba instead of transformers based on empirical results, these results appear somewhat questionable. Furthermore, there is a lack of new insights and detailed analysis; for instance, the authors did not delve deeper into the underlying mechanisms that led to these results. This deeper analysis is crucial, especially if the primary motivation of the paper is to use Mamba (or any different model architecture) instead of transformers for the same problem settings.\n\nPlease kindly refer to the questions for more details."}},"nonreaders":[],"tmdate":1731427182299,"tcdate":1731137178245,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9/Reviewer_Zeh1"],"signatures":["ICLR.cc/2025/Conference/Submission9/Reviewer_Zeh1"],"forum":"1TXDtnDIsV","number":3,"license":"CC BY 4.0","cdate":1731137178245,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427182299,"domain":"ICLR.cc/2025/Conference","replyto":"1TXDtnDIsV","id":"2CEx1dKhXB","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Continual Learning","Sequence Modelling"]},"primary_area":{"value":"transfer learning, meta learning, and lifelong learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Continual learning (CL) aims to efficiently learn and accumulate knowledge from a data stream with different distributions. By formulating CL as a sequence prediction task, meta-continual learning (MCL) enables to meta-learn an efficient continual learner based on the recent advanced sequence models, e.g., Transformers. Although attention-free models (e.g., Linear Transformers) can ideally match CL's essential objective and efficiency requirements, they usually perform not well in MCL. Considering that the attention-free Mamba achieves excellent performances matching Transformers' on general sequence modeling tasks, in this paper, we aim to answer a question -- Can attention-free Mamba perform well on MCL? By formulating Mamba with a selective state space model (SSM) for MCL tasks, we propose to meta-learn Mamba as a continual learner, referred to as MambaCL. By incorporating a selectivity regularization, we can effectively train MambaCL. Through comprehensive experiments across various CL tasks, we also explore how Mamba and other models perform in different MCL scenarios. Our experiments and analyses highlight the promising performance and generalization capabilities of Mamba in MCL."},"_bibtex":{"value":"@misc{\nzhao2025learning,\ntitle={Learning Mamba as a Continual Learner},\nauthor={Chongyang Zhao and Dong Gong},\nyear={2025},\nurl={https://openreview.net/forum?id=1TXDtnDIsV}\n}"},"title":{"value":"Learning Mamba as a Continual Learner"},"pdf":{"value":"/pdf/65bb972e176ce4ef1a4d56629ad29b0e78847ac1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhao|learning_mamba_as_a_continual_learner"},"authorids":{"value":["~Chongyang_Zhao1","~Dong_Gong1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chongyang Zhao","Dong Gong"]}},"version":2},{"content":{"summary":{"value":"Deep selective state-space models have achieved promising results on sequence modelling tasks. Yet, theoretical understanding of these models remain limited. This paper proposes to study how the embedding of different tokens evolve as they are processed through Mamba layers. In particular, they show that they either converge to 0 or diverge to infinity, based on some criteria that they provide. Importantly, this divergence speed depends on the position of the token. Motivated on their findings, the authors propose some modifications to the standard Mamba architecture that improve its performance."},"soundness":{"value":4},"confidence":{"value":3},"questions":{"value":"See above."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"Given the very little theoretical understanding of these models, such a theoretical contribution is valuable to the community. Despite its theoretical focus, the paper is easy to follow and one can grasp the main insights without having to delve into the math. The empirical improvements the theory motivates seem to be significant.\n\n(Score increased from 5 to 8 after the discussion period)"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"My main concern with the paper is that its conclusions remain limited compared to what the analysis could, in my opinion, deliver. Here are a few questions that would help, once answered, to improve this point:\n- How does the token dynamics in Mamba compare to the ones of Transformers? Are there some qualitative differences? Does it lead to some insights on the different inductive biases of these architectures.\n- Is it possible to extend the analysis to the bidirectional case, i.e. without the causal masking (the implicit attention matrix would not be lower triangular anymore)?\n- Similar question than the previous point but this time to extend the analysis to gated RNNs such as LSTMs or GRUs, whose simplified versions have some similarity with Mamba.\n\nWhile it is totally fine to restrain the theoretical analysis to the 1D, simulations assessing whether the conclusions of the analysis on the 1D case also hold in higher dimension are missing. One example of such simulations would be to initialize a Mamba layer in the same way as it is usually done in practice, give it random inputs, and observe how tokens evolve. It might be interesting to play with how correlated the random inputs are and with the magnitude of $a$.\n\nFinally, I remain suspicious of the empirical results given that no learning rate tuning was reported. Without such tuning I find the results hard to trust."}},"nonreaders":[],"tmdate":1733151460009,"tcdate":1729528670968,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5608/Reviewer_jxmT"],"signatures":["ICLR.cc/2025/Conference/Submission5608/Reviewer_jxmT"],"forum":"qtTIP5Gjc5","number":1,"license":"CC BY 4.0","cdate":1729528670968,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5608/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733151460009,"domain":"ICLR.cc/2025/Conference","replyto":"qtTIP5Gjc5","id":"1U1sOyC6t6","forumContent":{"TLDR":{"value":"We describe the dynamical properties of tokens in a deep selective state-space model, and based on that, we suggest improvements to the model by excluding negatively impactful scenarios and reordering tokens using importance scores."},"venue":{"value":"ICLR 2025 Spotlight"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Selective state-space model","continuous-time limit","dynamical system","asymptotic behavior","token reordering"]},"supplementary_material":{"value":"/attachment/f01b2b77a31c531086f0fefd1143d89ad8415a39.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity.  We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance.  For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training.  Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance.  Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications."},"_bibtex":{"value":"@inproceedings{\nvo2025demystifying,\ntitle={Demystifying the Token Dynamics of Deep Selective State Space Models},\nauthor={Thieu Vo and Duy-Tung Pham and Xin T. Tong and Tan Minh Nguyen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=qtTIP5Gjc5}\n}"},"title":{"value":"Demystifying the Token Dynamics of Deep Selective State Space Models"},"pdf":{"value":"/pdf/abc1efe1f910ee48b743f7ba26000b1d5e622f85.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"vo|demystifying_the_token_dynamics_of_deep_selective_state_space_models"},"authorids":{"value":["~Thieu_Vo1","~Duy-Tung_Pham1","~Xin_T._Tong2","~Tan_Minh_Nguyen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Thieu Vo","Duy-Tung Pham","Xin T. Tong","Tan Minh Nguyen"]}},"version":2},{"content":{"summary":{"value":"The paper proposes to reformulate dynamic graph learning as a global event sequence generation task. \nIt introduces a generative transformer-based model, GET, which models temporal interactions  as sequential events with causal masking and global attention. Experiments on several temporal graph benchmarks show competitive performance and improved speed."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see the weakness above"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"(1) The paper offers a clear and unified perspective by formulating dynamic graph learning  as a global event sequence generation problem, connecting temporal graph models with  sequence modeling and transformer architectures. \n\n(2) The proposed GET model achieves strong empirical performance and improved inference efficiency  on several benchmarks, showing its effectiveness in capturing long-range temporal dependencies."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The methodological novelty of the paper is limited.  Framing dynamic graph learning as global event sequence  resembles prior work  in transformer-based models (e.g., TCL, DyGFormer). Moreover, although the model is described as ``generative``,  its training objective remains discriminative: it is still based on negative sampling and softmax scoring without explicit probabilistic modeling of event likelihoods.  \n\n2. The paper claims that inference complexity shifts from a multiplicative to an additive form. However, the prediction of destination nodes still relies on negative sampling. In practice, the candidate set size $K$ can easily reach tens of thousands, making the computational cost still substantial. More importantly, different negative sampling strategies may lead to unstable evaluation results, yet the paper does not report any sensitivity analysis with respect to  different negative sampling strategies.\n\n3. Key hyperparameters such as the event sequence length $L$, the candidate set size $K$,  and the loss trade-off coefficient $\\lambda$ in the objective function are used without justification or discussion.  These parameters directly affect both the model's computational complexity  and its learning behavior, yet the paper provides no analysis or sensitivity study  to show how varying them impacts performance or stability.\n\n4. The theoretical analysis in section 5 is conceptually appealing but lacks rigor. Specifically, the theoretical “proof” of the strict inclusion relations is heuristic rather than formal. The function spaces are never formally defined, the “simulation via attention mask” argument confuses neighborhood masking with message passing and the causal attention analogy does not capture the recursive dynamics of memory-based models. The counter-example illustrating “strictness” is descriptive but non-constructive, lacking any formal mapping or quantitative verification. Therefore, this section should be interpreted as an intuition example, not a rigorous proof of expressiveness containment. \n\n5. The paper claims that GET changes the inference complexity of discriminative dynamic graph models (e.g. TGN) from multiplicative  $O(K \\cdot T_{\\text{propagate}})$ to additive $O(L^2 d + Kd)$, yet this comparison relies on unrealistic assumptions.  First, this characterization does not accurately reflect how models such as TGN  are implemented and deployed in practice. In standard TGN, each node maintains a memory state $m_v(t)$  that summarizes its interaction history.  During inference, edge probabilities are computed through a decoder  $f([m_u(t), m_v(t)])$, without re-running neighborhood propagation per candidate.  Thus, the actual inference complexity is approximately $O(Kd)$ rather than  $O(K \\cdot T_{\\text{propagate}})$.   Therefore, the comparison made in the paper implicitly assumes a worst-case setting  that repeatedly executes message passing for each candidate.  This exaggerates the efficiency gap between GET and memory-based models.   Second, The claim that inference cost is dominated by $K$ is only valid in the short-history regime ($L$ small).  When event sequences become long or graphs are temporally dense, the quadratic attention complexity $O(L^2 d)$  can surpass the linear term $O(Kd)$, thereby weakening the claimed additive advantage."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919729591,"tcdate":1761896325919,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7662/Reviewer_u5FF"],"signatures":["ICLR.cc/2026/Conference/Submission7662/Reviewer_u5FF"],"forum":"786oOfRVXO","number":3,"license":"CC BY 4.0","cdate":1761896325919,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7662/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919729591,"domain":"ICLR.cc/2026/Conference","replyto":"786oOfRVXO","id":"GasIcdLGbD","forumContent":{"TLDR":{"value":"A generative Transformer-based model for dynamic graph learning with structural-temporal context and efficient sequence prediction."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Graph Learning","Temporal Graph","Transformer","Link Prediction","Dynamic Graph"]},"supplementary_material":{"value":"/attachment/7801304ce9ec41fd5628c04e9a899fbba0d682b1.zip"},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"The prevailing paradigms in dynamic graph learning, which rely on local neighbor propagation or node-specific memory updates, are fundamentally limited in their ability to model global interaction patterns and to scale inference efficiently. Rethinking these challenges, we propose GET (Global Event Transformer), an event sequence modeling framework that reformulates dynamic graph learning as global event sequence modeling. By processing the entire interaction history as a unified stream of events, GET is designed to capture dependencies beyond the local receptive fields of prior methods, while its \"encode once, score many\" design removes the candidate-wise inference bottleneck of discriminative models. GET flexibly incorporates structural priors through lightweight GNN encoders and memory modules, which provide local structural and temporal cues that the global Transformer then reasons over. Our main focus in this paper is dynamic link prediction under standard Temporal Graph Benchmark (TGB) protocols, complemented by preliminary short-horizon generative evaluations on small/medium graphs. Extensive experiments on five large-scale TGB benchmarks and six additional datasets show that GET achieves strong competitive performance while delivering substantially faster inference throughput (up to 21.4 × on tgbl-wiki) compared to strong baselines."},"_bibtex":{"value":"@misc{\nchen2026get,\ntitle={{GET}: Rethinking Dynamic Graph Learning as Global Event Sequence Generation},\nauthor={Libin Chen and Xiyuan Wang and Xiaohui Zhang and Yanbo Wang and Luyao Wang and Hongfu Liu and Jing Chen and Muhan Zhang},\nyear={2026},\nurl={https://openreview.net/forum?id=786oOfRVXO}\n}"},"title":{"value":"GET: Rethinking Dynamic Graph Learning as Global Event Sequence Generation"},"pdf":{"value":"/pdf/25d8ba81cbc55477976c27c98b7feff8976f3264.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"chen|get_rethinking_dynamic_graph_learning_as_global_event_sequence_generation"},"authorids":{"value":["~Libin_Chen1","~Xiyuan_Wang1","~Xiaohui_Zhang9","~Yanbo_Wang2","~Luyao_Wang4","~Hongfu_Liu5","~Jing_Chen37","~Muhan_Zhang1"]},"authors":{"value":["Libin Chen","Xiyuan Wang","Xiaohui Zhang","Yanbo Wang","Luyao Wang","Hongfu Liu","Jing Chen","Muhan Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper provides a theoretical investigation into the capabilities of State Space Models (SSMs) compared to Transformers in their ability to dynamically extract tokens based on the input. The authors prove that SSMs combined with Feedforward Neural Network (FNN) layers can achieve performance comparable to Transformers in three cases: input copying, associative recall, and nonparametric regression."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- Could the authors clarify the differences between the results in this paper and those presented in [A] (see weaknesses)?\n\n- Could the authors discuss the difficulty of generalizing these results to selective SSMs?\n\n- In Line 405, could the authors explain the origin of the expression inside the big-O notation?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The paper establishes a rigorous theoretical foundation, demonstrating that SSMs combined with FNNs can emulate the dynamic token selection mechanism of Transformers. This theoretical analysis bridges a gap in understanding the potential of SSMs in sequence modeling.\n\n- The paper evaluates the proposed method across multiple tasks—input copying, associative recall, and nonparametric regression—showing that SSMs can achieve performance on par with Transformers."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The paper is challenging to read due to dense technical details. It would be helpful if the authors could include a figure or remark after each theorem to illustrate the reasoning and provide an intuitive explanation.\n\n- The input-independent SSMs used in this paper are somewhat outdated, given recent developments in selective SSMs (e.g., Mamba) and their variants. Additionally, the authors compare a combination of SSM and FNN with only the self-attention component of Transformers. This comparison may not be entirely fair, as the strength of a Transformer block lies in the combination of self-attention and FNN layers.\n\n- A deeper comparison of the results presented here with those in [A] is necessary. In fact, the authors of [A] consider a more general version of selective SSMs, called Generalized State Space Models (GSSMs), which includes SSMs with FNN as a subclass. The results in [A] show that GSSMs do not outperform Transformers on tasks such as input copying, whereas the current paper suggests a different perspective.\n\nReference: [A] Jelassi, Samy, et al. \"Repeat after me: Transformers are better than state space models at copying.\" arXiv preprint arXiv:2402.01032 (2024)."}},"nonreaders":[],"tmdate":1731428942416,"tcdate":1730733454512,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11158/Reviewer_uwcy"],"signatures":["ICLR.cc/2025/Conference/Submission11158/Reviewer_uwcy"],"forum":"QFgbJOYJSE","number":3,"license":"CC BY 4.0","cdate":1730733454512,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11158/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428942416,"domain":"ICLR.cc/2025/Conference","replyto":"QFgbJOYJSE","id":"HlKP7cG54Q","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["State Space Model","Transformer","Nonparametric regression"]},"supplementary_material":{"value":"/attachment/3ce47472b340b4d3facb435e7d5b788e63a48c46.zip"},"primary_area":{"value":"learning theory"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Deep neural networks based on state space models (SSMs) are attracting significant attention in sequence modeling since their computational cost is much smaller than that of Transformers. While the capabilities of SSMs have been demonstrated through experiments in various tasks, theoretical understanding of SSMs is still limited. In particular, most theoretical studies discuss the capabilities of SSM layers without nonlinear layers, and there is a lack of discussion on their combination with nonlinear layers. In this paper, we explore the capabilities of SSMs combined with fully connected neural networks, and show that they are comparable to Transformers in extracting the essential tokens depending on the input. As concrete examples, we consider two synthetic tasks, which are challenging for a single SSM layer, and demonstrate that SSMs combined with nonlinear layers can efficiently solve these tasks.  Furthermore, we study the nonparametric regression task, and prove that the ability of SSMs is equivalent to that of Transformers in estimating functions belonging to a certain class."},"_bibtex":{"value":"@inproceedings{\nnishikawa2025state,\ntitle={State Space Models are Provably Comparable to Transformers in Dynamic Token Selection},\nauthor={Naoki Nishikawa and Taiji Suzuki},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=QFgbJOYJSE}\n}"},"title":{"value":"State Space Models are Provably Comparable to Transformers in Dynamic Token Selection"},"pdf":{"value":"/pdf/7955d5853157c68ef049e331bb8268ff810d6fd7.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"nishikawa|state_space_models_are_provably_comparable_to_transformers_in_dynamic_token_selection"},"authorids":{"value":["~Naoki_Nishikawa1","~Taiji_Suzuki1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Naoki Nishikawa","Taiji Suzuki"]}},"version":2},{"content":{"venue":{"value":"ICLR 2025 Conference Desk Rejected Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["mamba","transformer","mixture-of-experts","time series prediction"]},"primary_area":{"value":"learning on time series and dynamical systems"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We propose the MambaFormer-MOE, a mamba-based mixture-of-experts (MOEs) model for multivariate time series prediction. There are three major features of our model. 1. We propose a temporal modeling module based-on the Mamba architecture to model temporal correlations with linear complexity. 2. We propose a cross-variate correlation modeling mechanism based-on self-attention to equip Mamba with multivariate time series prediction capability. 3. We propose a MOE mechanism that has experts that specialize in mixing the variates in different ways. It makes the model generalizable to multivariate time series from different domains. Our empirical results demonstrate that our model has SOTA prediction performance on various multivariate time series datasets."},"_bibtex":{"value":"@misc{\nli2024mambaformermoe,\ntitle={MambaFormer-{MOE}: Mamba-Transformer-based Mixture-of-Experts for Time Series Prediction},\nauthor={Weijian Li and Han Liu},\nyear={2024},\nurl={https://openreview.net/forum?id=55Ruu7bF4b}\n}"},"title":{"value":"MambaFormer-MOE: Mamba-Transformer-based Mixture-of-Experts for Time Series Prediction"},"pdf":{"value":"/pdf/ec5f4992f801fa4bfd051442cebde2acf9fae278.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"li|mambaformermoe_mambatransformerbased_mixtureofexperts_for_time_series_prediction"},"authorids":{"value":["~Weijian_Li2","~Han_Liu4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Weijian Li","Han Liu"]}},"tmdate":1729018045176,"tcdate":1727513296256,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13854/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13854/Authors"],"forum":"55Ruu7bF4b","license":"CC BY 4.0","number":13854,"cdate":1727513296256,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/-/Desk_Rejected_Submission"],"mdate":1729018045176,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"55Ruu7bF4b","version":2},{"content":{"summary":{"value":"This paper studies MAMBA's limitations in copying tasks. The authors start by introducing MAMBA basics and how MAMBA simulates linear attention. Then, the authors argue that, similar to linear attention, the dot-products between the current position and target positions should be larger than other positions in order to copy the target positions. Therefore, scaling MAMBA can improve copying performance as scaling is beneficial to linear attention.  Furthermore, the authors discuss how Chain-of-Thought can improve MAMBA in dynamic programming, but gains are limited by the length of the answer."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Typos:\nfI: In →In ? line 397\nA(n) should be An. line 420.\n\nI'm happy to discuss the paper in case of misunderstanding."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The topic is interesting. Models don't (currently) have an explicit blackboard to store variables for copying, only approximately through state vectors based on probabilistic predictions."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The presentation is not good. Notations are not concise enough for readers. One solution is, the authors can provide a notation list and figure out a way to share notations with literature. Figure 1 is a good idea. The authors could integrate equations into the figure.  I also feel some texts and equations are redundant in the main texts. For example, preliminaries (introduction to MAMBA) are too long and only Eq 1 is closely related to this paper. The paper should be more dense and concise.\n\n2. Motivation is not clear. The authors claim that their work focuses on token-level tasks (line 115) in contrast to existing works. But all the discussions and experiments are based on copying a sequence. This can be seen in definition 1 \"Then the output of COPY operation is a sequence of vectors o1,o2,...,oN with oi = vpos(i) where pos(i) ∈ Si is the position we want to copy\".  Furthermore, it is not clear how copying connects to dynamic programming. Does the authors study in the setting that the model only do copy for DP?  Can you try some toke-level copying tasks like LAMBADA.\n\n3. To me, the findings are not novel and not distinguished from existing works. Most of the paper focuses on introducing existing works. \n\n4. The main claim about the bottleneck of MAMBA is not discussed in this paper.  While we can see this from empirical studies, it is not clear how we can take inspiration from the assumption the authors make.  This is important as the authors claim they explain the limitations from theoretical standpoints."}},"nonreaders":[],"tmdate":1732552912173,"tcdate":1731171532168,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6956/Reviewer_PqMr"],"signatures":["ICLR.cc/2025/Conference/Submission6956/Reviewer_PqMr"],"forum":"UYarAv7rUx","number":4,"license":"CC BY 4.0","cdate":1731171532168,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6956/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732552912173,"domain":"ICLR.cc/2025/Conference","replyto":"UYarAv7rUx","id":"yKQZCgbOc3","forumContent":{"TLDR":{"value":"This paper explores the expressive power of Mamba."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Expressive Power","Chain of Thought"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Transformers have been the cornerstone of current Large Language Models (LLMs); however, its linear growth in overhead during inference with respect to sequence length poses challenges for modeling long sequences. In this context, Mamba has gradually attracted attention due to its constant-level size during inference and existing empirical results have shown that it can perform comparably to Transformers in sequence modeling while offering significant savings. However, one may ask that, can Mamba always enjoy the ``free lunch\"? In this paper, we focus on analyzing the expressive ability of Mamba from a theoretical standpoint. First, inspired by the connection between Mamba and linear attention, we investigate potential shortcomings of the Mamba when performing the COPY operation. Our results indicate that Mamba with constant size may encounter bottlenecks when handling COPY, while it can achieve perfect performance when the size scales linearly with sequence length. Based on this observation, we analyze Mamba's ability to tackle DP problems when equipped with Chain of Thought (CoT). Our findings suggest that to solve arbitrary DP problems, the total cost of Mamba is comparable to standard and efficient Transformers. However, similar to efficient Transformers, when facing DP problems with favorable properties such as locality, Mamba can provide savings in overhead. Our results contribute to a deeper understanding of Mamba."},"_bibtex":{"value":"@misc{\nren2025can,\ntitle={Can Mamba Always Enjoy the ''Free Lunch''?},\nauthor={Ruifeng Ren and Zhicong Li and Yong Liu},\nyear={2025},\nurl={https://openreview.net/forum?id=UYarAv7rUx}\n}"},"title":{"value":"Can Mamba Always Enjoy the \"Free Lunch\"?"},"pdf":{"value":"/pdf/b4552a0f966e40cf480dfcacf744f021f4b8b521.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ren|can_mamba_always_enjoy_the_free_lunch"},"authorids":{"value":["~Ruifeng_Ren3","~Zhicong_Li1","~Yong_Liu7"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Ruifeng Ren","Zhicong Li","Yong Liu"]}},"version":2},{"content":{"summary":{"value":"The paper introduces HRVMamba, a high-resolution visual state space model tailored for dense prediction tasks. It builds on the Mamba framework, a hardware-efficient state space model (SSM) known for linear computational complexity. The authors highlight limitations in previous visual Mamba models—namely, insufficient inductive bias, long-range forgetting, and low-resolution outputs. To overcome these, HRVMamba incorporates the Dynamic Visual State Space (DVSS) block, combining multi-scale convolutional kernels and deformable convolution for enhanced local and long-range feature extraction. The HRVMamba model employs a multi-resolution parallel structure inspired by HRNet, preserving high-resolution representations and facilitating multi-scale feature learning."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"Please refer to the weakness part."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- Empirical results indicate that HRVMamba outperforms contemporary CNNs, ViTs, and SSMs on benchmarks, delivering competitive performance with fewer computational resources.\n- The figures in the paper are clean and aesthetically pleasing, which enhances readability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Limited Novelty: The HRVMamba model mainly combines existing methods, including the VSS block, DWConv, DCN, and the HRNet architecture. This integration-based approach may not meet ICLR’s high standards for innovation.\n- Concern for `Limitation 1`: While the paper addresses the lack of 2D inductive bias in previous visual Mamba models, it raises concerns about whether introducing such bias could restrict the **scaling ability** of HRVMamba. Vision Transformers (ViTs) have demonstrated that reduced inductive bias can facilitate better scaling, so incorporating strong inductive biases might limit HRVMamba's scalability and performance on larger-scale models.\n- Concern for `Limitation 2`: The paper uses Deformable Convolutions (DCN) to mitigate the long-range forgetting issue observed in previous visual Mamba models. However, there is a concern about whether DCN can effectively address this problem as the sequence length scales up. The efficacy of DCN for maintaining high-level feature relationships over significantly longer sequences remains uncertain, raising questions about its robustness as a scalable solution for long-range dependencies.\n- The paper references preprints and arXiv versions of significant works, such as Mamba (COLM), Vision Mamba (ICML), and VMamba (NeurIPS). The authors should update these citations to their final published versions to reflect the current state of the literature."}},"nonreaders":[],"tmdate":1731427847365,"tcdate":1730710521632,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3460/Reviewer_YccU"],"signatures":["ICLR.cc/2025/Conference/Submission3460/Reviewer_YccU"],"forum":"4UxXe3JZta","number":3,"license":"CC BY 4.0","cdate":1730710521632,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3460/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427847365,"domain":"ICLR.cc/2025/Conference","replyto":"4UxXe3JZta","id":"myDZwnZtlN","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Dense Prediction","Human pose estimation","Semantic segmentation"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recently, State Space Models (SSMs) with efficient hardware-aware designs, \\ie, Mamba, have demonstrated significant potential in computer vision tasks due to their linear computational complexity with respect to token length and their global receptive field. However, Mamba's performance on dense prediction tasks, including human pose estimation and semantic segmentation, has been constrained by three key challenges: insufficient inductive bias, long-range forgetting, and low-resolution output representation.\nTo address these challenges, we introduce the Dynamic Visual State Space (DVSS) block, which utilizes multi-scale convolutional kernels to extract local features across different scales and enhance inductive bias, and employs deformable convolution to mitigate the long-range forgetting problem while enabling adaptive spatial aggregation based on input and task-specific information. By leveraging the multi-resolution parallel design proposed in HRNet, we introduce High-Resolution Visual State Space Model (HRVMamba) based on the DVSS block, which preserves high-resolution representations throughout the entire process while promoting effective multi-scale feature learning.\nExtensive experiments highlight HRVMamba's impressive performance on dense prediction tasks, achieving competitive results against existing benchmark models without bells and whistles.\nWe will make the source code publicly accessible."},"_bibtex":{"value":"@misc{\nzhang2025hrvmamba,\ntitle={{HRVM}amba: High-Resolution Visual State Space Model for Dense Prediction},\nauthor={Hao Zhang and Yongqiang Ma and Wenqi Shao and Ping Luo and Nanning Zheng and Kaipeng Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=4UxXe3JZta}\n}"},"title":{"value":"HRVMamba: High-Resolution Visual State Space Model for Dense Prediction"},"pdf":{"value":"/pdf/43b8895c02f64825cd898821c60038dc64c54e1f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhang|hrvmamba_highresolution_visual_state_space_model_for_dense_prediction"},"authorids":{"value":["~Hao_Zhang56","~Yongqiang_Ma3","~Wenqi_Shao2","~Ping_Luo2","~Nanning_Zheng1","~Kaipeng_Zhang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hao Zhang","Yongqiang Ma","Wenqi Shao","Ping Luo","Nanning Zheng","Kaipeng Zhang"]}},"version":2},{"content":{"summary":{"value":"LASP-2 is proposed as an improvement upon LASP of Sun et al. 2024. Like LASP, LASP-2 implements linear attention based on sequence parallelism, but with better communication and computation efficiency. Central to the proposed idea is a single all-gather communication of the memory state matrix with dimension independent of the sequence or chunk length. The overall cost scales linearly with the number of devices involved in the SP communication group. An extension is also given to handle both linear and standard attention modules. Experiments are conducted using sequences as long as  2048K to validate the proposed methods."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Given the above weakness concerns, the only significant difference between LASP and LASP-2 seems to be the allgather operation (and the subsequent update on the aggregated state) in the latter that allows every device in the system to have access to the complete set of memory states for entire chunks. \n\nIn light of this, ablation on varying sizes of gathering would be desired to verify whether the operation of (6) (and (8) in the case of masking) is indeed responsible for the improved throughput for length greater than 1024K in Fig. 3."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper is well written with nice tutorial on linear attention.\n \nPractical merits are high.  \n\nThe allgather operation for making the state information of all chunks available to every device in the system seems unique and appears to be the reason behind the performance advantage observed for very long sequences."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The claim here is that relative to LASP of Sun et al. 2024, the proposed method, coined LASP-2, offers better communication efficiency since it is the state matrix that gets transferred whose dimension does not depend on the sequence or chunk length, as opposed to K and V matrices that depend on the chunk length. But this is not true. The communication requirement is the same with the original LASP algorithm that communicates the KV activation. \n\nLikewise, the claim above and below eq (5) about concurrent computations for each chunk being unique to LASP-2 is unfounded. It is easy to see that identical operations are already done in LASP."}},"nonreaders":[],"tmdate":1732586032784,"tcdate":1730822504194,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9000/Reviewer_Npbh"],"signatures":["ICLR.cc/2025/Conference/Submission9000/Reviewer_Npbh"],"forum":"c6TDOPEQ0e","number":4,"license":"CC BY 4.0","cdate":1730822504194,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9000/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732586032784,"domain":"ICLR.cc/2025/Conference","replyto":"c6TDOPEQ0e","id":"WjnvumUn7B","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Sequence parallelism","Distributed Training"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Linear sequence modeling approaches, such as linear attention, provide advantages like linear-time training and constant-memory inference over sequence lengths. However, existing sequence parallelism (SP) methods are either not optimized for their right-product-first feature or use ring-style communication as in LASP, which results in lower computation parallelism, limits their scalability for longer sequences in distributed systems. In this paper, we introduce LASP-2, a new SP approach designed to enhance both communication and computation efficiency in linear (attention) transformer models with very-long input sequences. Compared to LASP, LASP-2 rethinks the minimal communication requirement for SP on linear attention, reorganizes the whole communication-computation order of LASP. In this way, only one single all-gather collective communication is needed on intermediate memory states, whose sizes are independent of the sequence length, leading to significant improvements of both communication and computation parallelism, as well as their overlap. Additionally, we extend LASP-2 to LASP-2H by applying similar communication redesign to standard attention modules, offering an efficient SP solution for hybrid models that combine linear and standard attention layers. Our evaluation on a Linear-Llama3 model, a variant of Llama3 with linear attention replacing standard attention, demonstrates the effectiveness of LASP-2 and LASP-2H. Specifically, LASP-2 achieves throughput improvements of 15.2\\% over LASP and 36.6\\% over Ring Attention, with a sequence length of 2048K across 64 GPUs."},"_bibtex":{"value":"@misc{\nsun2025lasp,\ntitle={{LASP}-2: Rethinking Sequence Parallelism for Linear Attention and its Hybrid},\nauthor={Weigao Sun and Disen Lan and Xiaoye Qu and Yu Cheng},\nyear={2025},\nurl={https://openreview.net/forum?id=c6TDOPEQ0e}\n}"},"title":{"value":"LASP-2: Rethinking Sequence Parallelism for Linear Attention and its Hybrid"},"pdf":{"value":"/pdf/8630b47f770e67ce27a7e41f57f3e623b469634f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sun|lasp2_rethinking_sequence_parallelism_for_linear_attention_and_its_hybrid"},"authorids":{"value":["~Weigao_Sun1","~Disen_Lan1","~Xiaoye_Qu1","~Yu_Cheng1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Weigao Sun","Disen Lan","Xiaoye Qu","Yu Cheng"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.10696v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"misra|on_the_lowshot_transferability_of_[v]mamba"},"authorids":{"value":["~Diganta_Misra1","~Jay_Gala1","~Antonio_Orvieto3"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.10696"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-10696,\n  publtype={informal},\n  author={Diganta Misra and Jay Gala and Antonio Orvieto},\n  title={On the low-shot transferability of [V]-Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.10696},\n  url={https://doi.org/10.48550/arXiv.2403.10696}\n}\n"},"abstract":{"value":"The strength of modern large-scale neural networks lies in their ability to efficiently adapt to new tasks with few examples. Although extensive research has investigated the transferability of Vision Transformers (ViTs) to various downstream tasks under diverse constraints, this study shifts focus to explore the transfer learning potential of [V]-Mamba. We compare its performance with ViTs across different few-shot data budgets and efficient transfer methods. Our analysis yields three key insights into [V]-Mamba's few-shot transfer performance: (a) [V]-Mamba demonstrates superior or equivalent few-shot learning capabilities compared to ViTs when utilizing linear probing (LP) for transfer, (b) Conversely, [V]-Mamba exhibits weaker or similar few-shot learning performance compared to ViTs when employing visual prompting (VP) as the transfer method, and (c) We observe a weak positive correlation between the performance gap in transfer via LP and VP and the scale of the [V]-Mamba model. This preliminary analysis lays the foundation for more comprehensive studies aimed at furthering our understanding of the capabilities of [V]-Mamba variants and their distinctions from ViTs."},"title":{"value":"On the low-shot transferability of [V]-Mamba"},"authors":{"value":["Diganta Misra","Jay Gala","Antonio Orvieto"]}},"tmdate":1754034342725,"pdate":1704067200000,"tcdate":1727719233142,"writers":["~"],"signatures":["~Jay_Gala1"],"forum":"mbW0VyVzde","license":"CC BY-SA 4.0","number":122283,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1754034342725,"domain":"DBLP.org","id":"mbW0VyVzde","version":2},{"content":{"summary":{"value":"The paper introduces Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), enhancing Mamba multi-modal large language models (MLLM) by improving their ability to extract fine-grained visual information. EMMA uses a pixel-wise alignment module for better structural alignment and a multi-scale feature fusion (MFF) module to maintain visual information during cross-modal alignment. Extensive experiments demonstrate that EMMA achieves lower latency and faster performance compared to other Mamba-based MLLMs, with superior cross-modal alignment and reduced hallucination. The code for this model will be made available."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Currently, models use ViT as the image encoder. Is it possible to build an MLLM entirely based on Mamba? What potential advantages and challenges might this approach entail?\n\n2. In transformer-based LLMs, is there a similar issue with pixel-level alignment? Would the method proposed in the paper also be applicable to transformer architectures?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Well Written: The paper is clearly articulated, making the complex concepts accessible and easy to understand for the reader.\n\n2. Motivation Well Clarified: The authors firstly identify the imbalance in the quality of visual and textual latent features in MLLM Mamba models. They propose a pixel-wise alignment approach to autoregressively enhance the learning and processing of structured visual features, leading to improved cross-modal alignment between visual and textual latents.\n\n3. Experiments Solid: The paper presents comprehensive experiments conducted on a variety of multi-modal benchmarks. These experiments include thorough comparisons with current state-of-the-art models based on both Mamba and transformer architectures, demonstrating the robustness of the proposed approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Why is there an imbalance in the quality of visual and textual latents in the Mamba LLM, where coarse visual features are ineffectively aligned with higher-quality textual features? How does this differ from transformer-based models? Do transformer-based models face the same issue? \n\n2. Could you provide a more objective, quantitative analysis of the loss of fine-grained visual cues in the LLM, as the current visualizations with just a few images seem insufficient? Is this phenomenon common across MLLMs, or is it specific to the use of the Mamba LLM?\n\n3. Why do the current results show that the Mamba LLM still lags behind the transformer architecture? In this case, does the speed advantage gained by using the Mamba architecture sufficiently compensate for the performance loss?"}},"nonreaders":[],"tmdate":1731429337628,"tcdate":1730565403325,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10276/Reviewer_T8Kc"],"signatures":["ICLR.cc/2025/Conference/Submission10276/Reviewer_T8Kc"],"forum":"Ev4iw23gdI","number":2,"license":"CC BY 4.0","cdate":1730565403325,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10276/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429337628,"domain":"ICLR.cc/2025/Conference","replyto":"Ev4iw23gdI","id":"ymQcjGNqrS","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Multimodal models","State space models","Efficient architectures","Mamba","Computational Efficiency","Multimodal Alignment"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Mamba-based architectures have shown to be a promising new direction for deep learning models owing to their competitive performance and sub-quadratic deployment speed. However, current Mamba multi-modal large language models (MLLM) are insufficient in extracting visual features, leading to imbalanced cross-modal alignment between visual and textural latents, negatively impacting performance on multi-modal tasks. In this work, we propose Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), which enables the MLLM to extract fine-grained visual information. Specifically, we propose a pixel-wise alignment module to autoregressively optimize the learning and processing of spatial image-level features along with textual tokens, enabling structural alignment at the image level. In addition, to prevent the degradation of visual information during the cross-model alignment process, we propose a multi-scale feature fusion (MFF) module to combine multi-scale visual features from intermediate layers, enabling hierarchical alignment at the feature level. Extensive experiments are conducted across a variety of multi-modal benchmarks. Our model shows lower latency than other Mamba-based MLLMs and is nearly four times faster than transformer-based MLLMs of similar scale during inference. Due to better cross-modal alignment, our model exhibits lower degrees of hallucination and enhanced sensitivity to visual details, which manifests in superior performance across diverse multi-modal benchmarks. Code provided at https://github.com/xingyifei2016/EMMA."},"_bibtex":{"value":"@inproceedings{\nxing2025emma,\ntitle={{EMMA}: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment},\nauthor={Yifei Xing and Xiangyuan Lan and Ruiping Wang and Dongmei Jiang and Wenjun Huang and Zheng Qingfang and Yaowei Wang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=Ev4iw23gdI}\n}"},"title":{"value":"EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment"},"pdf":{"value":"/pdf/3866b69574c95327cb1f7af1da4822dbdcca4a54.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"xing|emma_empowering_multimodal_mamba_with_structural_and_hierarchical_alignment"},"authorids":{"value":["~Yifei_Xing1","~Xiangyuan_Lan4","~Ruiping_Wang1","~Dongmei_Jiang2","~Wenjun_Huang3","~Zheng_Qingfang1","~Yaowei_Wang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yifei Xing","Xiangyuan Lan","Ruiping Wang","Dongmei Jiang","Wenjun Huang","Zheng Qingfang","Yaowei Wang"]}},"version":2},{"content":{"summary":{"value":"This paper studies sequence models (SSMs and masked attention) through a **dynamical-systems lens** and argues that **task requirements are reflected in the spectrum of state-transition eigenvalues**. Using the DSF formulation, causal attention is written as a recurrence, so eigenvalue magnitudes capture how much the past state is retained (near 1) or forgotten (near 0). The authors empirically profile eigenvalue distributions across architectures (S4, LRU, Mamba-2; softmax, linear, norm attention), layers, and tasks (ListOps, IMDb, CIFAR-10, MQAR, WikiText).\n\nMain findings are:\n\n* **Spectral signatures align with task needs.** Long-memory tasks concentrate mass near 1; tasks needing selective forgetting show peaks near 0.\n* **Architectural choices shift spectra in intuitive ways.** Adding explicit **gates** reduces the need for implicit gating in the dynamics; adding **convolution** offloads local context and shifts mass toward 0; **normalization** choices in norm attention trade retention vs selectivity; making **Mamba-2 pseudo-LTI** (constant discretization) yields S4-like spectra and helps ListOps.\n* The spectrum can therefore serve as a **diagnostic/design tool** to steer models toward retention or selectivity."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. **Quantification.** Can you report correlations (or simple regressions) between task performance and spectral mass in specific bins (e.g., ([0.9,1]) vs ([0,0.1])) across runs/layers? This would strengthen the predictive value of the metric.\n2. **Dynamics over training.** Do spectra evolve monotonically toward the final pattern? Showing trajectories (epochs vs mass in each bin) could clarify causality.\n3. **Complex phases.** Did you analyze the angles of complex eigenvalues (oscillatory modes)? Any task where phase structure matters?\n4. **Head and seed variance.** The appendix includes more heads; could you summarize variance across heads/seeds with statistical testing?\n5. **Capacity control.** When comparing architectures, are parameter counts and FLOPs matched closely? If not, can you add matched-capacity comparisons?\n6. **When spectra mislead.** Are there counterexamples where spectra look favorable but performance lags (or vice versa)? Understanding failure modes would bound the tool’s applicability."},"rating":{"value":4},"details_of_ethics_concerns":{"value":"No concerns."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"* **Unifying perspective.** A clear DSF view that lets attention and SSMs be compared apples-to-apples through the same spectral metric.\n* **Comprehensive empirics.** Multi-task, multi-model, per-layer analyses with ablations (gating, conv, normalization, depth, pseudo-LTI). This breadth gives credibility to the main trend claims.\n* **Actionable insights.** Concrete “design → spectrum → behavior” links (e.g., conv shifts mass to 0; constant discretization moves Mamba-2 toward retention). Practitioners can use this to target spectra for a task.\n* **Clarity of interpretation.** The near-0 vs near-1 story is intuitive and connects to long-standing control/SSM theory, helping demystify why models succeed or fail on LRA-style tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* **Incremental novelty.** Much of the narrative (poles near the unit circle ↔ long memory; selective forgetting ↔ small eigenvalues) echoes prior SSM/RNN work; S5 and related papers already emphasize spectral placement for retention and task performance. The contribution is more **systematization** than new theory.\n* **Limited quantification.** The link between spectra and performance is presented mostly as qualitative alignment. No formal predictive model (e.g., correlation between “mass in [0.9,1]” and accuracy/perplexity) or causal intervention is provided.\n* **Coarse metric.** Binning only by **magnitude** discards **phase** information of complex eigenvalues (oscillations) and ignores distributional shape beyond coarse buckets. This may hide important structure.\n* **Scale and controls.** It’s unclear how sensitive results are to model size, parameter budget, or training recipe. Using one head for plots can mask head-to-head variance; significance tests across seeds are limited in the main text.\n* **Scope.** Benchmarks are mostly small/medium. It remains uncertain how well these spectral diagnostics extrapolate to large-scale language models."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762940623208,"tcdate":1760657588001,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21047/Reviewer_8FeC"],"signatures":["ICLR.cc/2026/Conference/Submission21047/Reviewer_8FeC"],"forum":"ALf0xHCCaP","number":1,"license":"CC BY 4.0","cdate":1760657588001,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission21047/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762940623208,"domain":"ICLR.cc/2026/Conference","replyto":"ALf0xHCCaP","id":"sdQE2PXsaH","forumContent":{"TLDR":{"value":"This paper shows that eigenvalues offer a metric for memory retention and selective forgetting in sequence models, guiding model design choices to meet task requirements."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Sequence models","dynamical systems","memory dynamics"]},"supplementary_material":{"value":"/attachment/a34ab7106844c8b1974675fdf300085c394ec367.pdf"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Although softmax attention drives state-of-the-art performance for sequence models, its quadratic complexity limits scalability, motivating linear alternatives such as state space models (SSMs). While these alternatives improve efficiency, their fundamental differences in information processing remain poorly understood. In this work, we leverage the recently proposed dynamical systems framework to represent softmax, norm and linear attention as dynamical systems, enabling a structured comparison with SSMs by analyzing their respective eigenvalue spectra. Since eigenvalues capture essential aspects of dynamical system behavior, we conduct an extensive empirical analysis across diverse sequence models and benchmarks. We first show that eigenvalues influence essential aspects of memory and long-range dependency modeling, revealing spectral signatures that align with task requirements. Building on these insights, we then investigate how architectural modifications in sequence models impact both eigenvalue spectra and task performance.  This correspondence further strengthens the position of eigenvalue analysis as a principled metric for interpreting, understanding, and ultimately improving the capabilities of sequence models."},"_bibtex":{"value":"@misc{\nrickenbach2026tasklevel,\ntitle={Task-Level Insights from Eigenvalues across Sequence Models},\nauthor={Rahel Rickenbach and Jelena Trisovic and Alexandre Didier and Jerome Sieber and Melanie Zeilinger},\nyear={2026},\nurl={https://openreview.net/forum?id=ALf0xHCCaP}\n}"},"title":{"value":"Task-Level Insights from Eigenvalues across Sequence Models"},"pdf":{"value":"/pdf/ef15be1e4ca44ec34adb76dbbbb5e30448f4997c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"rickenbach|tasklevel_insights_from_eigenvalues_across_sequence_models"},"authorids":{"value":["~Rahel_Rickenbach1","~Jelena_Trisovic1","~Alexandre_Didier1","~Jerome_Sieber1","~Melanie_Zeilinger1"]},"authors":{"value":["Rahel Rickenbach","Jelena Trisovic","Alexandre Didier","Jerome Sieber","Melanie Zeilinger"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2510.14510v5"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"wu|enhancing_time_series_forecasting_through_selective_representation_spaces_a_patch_perspective"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Xingjian_Wu:","https://dblp.org/search/pid/api?q=author:Xiangfei_Qiu:","~Hanyin_Cheng1","","https://dblp.org/search/pid/api?q=author:Jilin_Hu:","https://dblp.org/search/pid/api?q=author:Chenjuan_Guo:","https://dblp.org/search/pid/api?q=author:Bin_Yang_0002:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2510.14510"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2510-14510,\n  publtype={informal},\n  author={Xingjian Wu and Xiangfei Qiu and Hanyin Cheng and Zhengyu Li and Jilin Hu and Chenjuan Guo and Bin Yang},\n  title={Enhancing Time Series Forecasting through Selective Representation Spaces: A Patch Perspective},\n  year={2025},\n  month={October},\n  cdate={1759276800000},\n  journal={CoRR},\n  volume={abs/2510.14510},\n  url={https://doi.org/10.48550/arXiv.2510.14510}\n}\n"},"abstract":{"value":"Time Series Forecasting has made significant progress with the help of Patching technique, which partitions time series into multiple patches to effectively retain contextual semantic information into a representation space beneficial for modeling long-term dependencies. However, conventional patching partitions a time series into adjacent patches, which causes a fixed representation space, thus resulting in insufficiently expressful representations. In this paper, we pioneer the exploration of constructing a selective representation space to flexibly include the most informative patches for forecasting. Specifically, we propose the Selective Representation Space (SRS) module, which utilizes the learnable Selective Patching and Dynamic Reassembly techniques to adaptively select and shuffle the patches from the contextual time series, aiming at fully exploiting the information of contextual time series to enhance the forecasting performance of patch-based models. To demonstrate the effectiveness of SRS module, we propose a simple yet effective SRSNet consisting of SRS and an MLP head, which achieves state-of-the-art performance on real-world datasets from multiple domains. Furthermore, as a novel plug-and-play module, SRS can also enhance the performance of existing patch-based models. The resources are available at https://github.com/decisionintelligence/SRSNet."},"title":{"value":"Enhancing Time Series Forecasting through Selective Representation Spaces: A Patch Perspective"},"authors":{"value":["Xingjian Wu","Xiangfei Qiu","Hanyin Cheng","Zhengyu Li","Jilin Hu","Chenjuan Guo","Bin Yang"]}},"tmdate":1784461857455,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2510-14510"],"tcdate":1769515670749,"writers":["~"],"signatures":["~Zhengyu_Li9"],"forum":"dKZ2ILjsLM","license":"CC BY-SA 4.0","number":807674,"cdate":1759276800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784461857455,"domain":"DBLP.org","id":"dKZ2ILjsLM","version":2},{"content":{"venue":{"value":"NeurIPS 2025 spotlight"},"keywords":{"value":["Time Series Forecasting"]},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Time Series Forecasting has made significant progress with the help of Patching technique, which partitions time series into multiple patches to effectively retain contextual semantic information into a representation space beneficial for modeling long-term dependencies. However, conventional patching partitions a time series into adjacent patches, which causes a fixed representation space, thus resulting in insufficiently expressful representations. In this paper, we pioneer the exploration of constructing a selective representation space to flexibly include the most informative patches for forecasting. Specifically, we propose the Selective Representation Space (SRS) module, which utilizes the learnable Selective Patching and Dynamic Reassembly techniques to adaptively select and shuffle the patches from the contextual time series, aiming at fully exploiting the information of contextual time series to enhance the forecasting performance of patch-based models. To demonstrate the effectiveness of SRS module, we propose a simple yet effective SRSNet consisting of SRS and an MLP head, which achieves state-of-the-art performance on real-world datasets from multiple domains. Furthermore, as a novel plug-and-play module, SRS can also enhance the performance of existing patch-based models. The resources are available at https://github.com/decisionintelligence/SRSNet."},"_bibtex":{"value":"@inproceedings{\nwu2025enhancing,\ntitle={Enhancing Time Series Forecasting through Selective Representation Spaces: A Patch Perspective},\nauthor={Xingjian Wu and Xiangfei Qiu and Hanyin Cheng and Zhengyu Li and Jilin Hu and Chenjuan Guo and Bin Yang},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=BirE0jYKt0}\n}"},"title":{"value":"Enhancing Time Series Forecasting through Selective Representation Spaces: A Patch Perspective"},"pdf":{"value":"/pdf/e7b262f7c9ef17e0b1608ff32b4895bb051e725b.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"wu|enhancing_time_series_forecasting_through_selective_representation_spaces_a_patch_perspective"},"authorids":{"value":["~Xingjian_Wu1","~Xiangfei_Qiu1","~Hanyin_Cheng1","~Zhengyu_Li9","~Jilin_Hu1","~Chenjuan_Guo1","~Bin_Yang4"]},"authors":{"value":["Xingjian Wu","Xiangfei Qiu","Hanyin Cheng","Zhengyu Li","Jilin Hu","Chenjuan Guo","Bin Yang"]}},"tmdate":1783627463402,"pdate":1758216513551,"tcdate":1744009643932,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission1041/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission1041/Authors"],"forum":"BirE0jYKt0","license":"CC BY 4.0","number":1041,"cdate":1744009643932,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission1041/-/Full_Submission","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission1041/-/Camera_Ready_Revision","NeurIPS.cc/2025/Conference/-/Desk_Rejected_Submission"],"mdate":1783627463402,"odate":1761704724967,"domain":"NeurIPS.cc/2025/Conference","id":"BirE0jYKt0","version":2},{"content":{"summary":{"value":"This paper presents a novel model architecture designed to handle the complexities of both long-sequence processing. This is achieved by integrating S4 global attention with local attention mechanisms to form a hierarchical structure. The S4 layer is utilized at the bottom to capture long dependencies, while the local attention layers above aim to simplify attention complexity and expedite computation. The model outperforms the S4 alone and traditional Transformers in specific tasks."},"presentation":{"value":"4 excellent"},"contribution":{"value":"2 fair"},"soundness":{"value":"3 good"},"strengths":{"value":"Strengths:\n1. This paper describes a novel model that integrates S4 global attention and local (window-based or chunk-based) attention to address both long-range dependencies in language modeling. The hierarchical structure, with S4 at the bottom and local attention on top, aims to balance complexity and computation speed. \n2. The model has shown improvements over S4 and traditional Transformers in long-range and text generation tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Weaknesses:\n1. S4 and Transformer are widely-used models, combining them together brings somehow incremental novelty contributions. \n2. The comparisons with alternative methods for capturing global context, such as RNN-like mechanisms or other efficient attention mechanisms, are incomplete and lack essential details. Only S4 is compared in table 1.  Some recent long-sequence modeling studies:\n  a. LongNet: Scaling Transformers to 1,000,000,000 Tokens\n  b. Long Range Language Modeling via Gated State Spaces"},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"Questions:\n1. T5-base is not implemented with the same setting with the proposed model. It would be more convincing to report apple-to-apple comparisons with the same setting, like the same pre-trained datasets."},"rating":{"value":"5: marginally below the acceptance threshold"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636391833,"tcdate":1698941440862,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission4247/Reviewer_6SGN"],"signatures":["ICLR.cc/2024/Conference/Submission4247/Reviewer_6SGN"],"forum":"xuxYaBMd9F","number":4,"license":"CC BY 4.0","cdate":1698941440862,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission4247/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636391833,"domain":"ICLR.cc/2024/Conference","replyto":"xuxYaBMd9F","id":"fn1LipzVnP","forumContent":{"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["long sequence","language models"]},"supplementary_material":{"value":"/attachment/548d541393786fc0ab6fd9f2c0d8120e1b84277a.zip"},"primary_area":{"value":"representation learning for computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Transformer models have achieved superior performance in various natural language processing tasks. However, the quadratic computational cost of the attention mechanism limits its practicality for long sequences. There are existing attention variants that improve the computational efficiency, but they have limited ability to effectively compute global information. In parallel to Transformer models, state space models (SSMs) are tailored for long sequences, but they are not flexible enough to capture complicated local information. We propose SPADE, short for State Space Augmented Transformer. Specifically, we augment a SSM into the bottom layer of SPADE, and we employ efficient local attention methods for the other layers. The SSM augments global information, which complements the lack of long-range dependency issue in local attention methods. Experimental results on the Long Range Arena benchmark and language modeling tasks demonstrate the effectiveness of the proposed method. To further demonstrate the scalability of SPADE, we pre-train large encoder-decoder models and present fine-tuning results on natural language understanding and natural language generation tasks. Our code and pre-trained model checkpoints will be publicly available."},"_bibtex":{"value":"@misc{\nzuo2024efficient,\ntitle={Efficient Long Sequence Modeling via State Space Augmented Transformer},\nauthor={Simiao Zuo and Xiaodong Liu and Jian Jiao and Denis X Charles and Eren Manavoglu and Tuo Zhao and Jianfeng Gao},\nyear={2024},\nurl={https://openreview.net/forum?id=xuxYaBMd9F}\n}"},"title":{"value":"Efficient Long Sequence Modeling via State Space Augmented Transformer"},"pdf":{"value":"/pdf/7c669b0f790ca0a854abdea1582f9384eeefbee5.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"zuo|efficient_long_sequence_modeling_via_state_space_augmented_transformer"},"authorids":{"value":["~Simiao_Zuo1","~Xiaodong_Liu1","~Jian_Jiao2","~Denis_X_Charles1","~Eren_Manavoglu1","~Tuo_Zhao1","~Jianfeng_Gao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Simiao Zuo","Xiaodong Liu","Jian Jiao","Denis X Charles","Eren Manavoglu","Tuo Zhao","Jianfeng Gao"]}},"version":2},{"content":{"venue":{"value":"ICML 2024 Poster"},"abstract":{"value":"Transformers with linear attention allow for efficient parallel training but can simultaneously be formulated as an RNN with 2D (matrix-valued) hidden states, thus enjoying linear-time inference complexity. However, linear attention generally underperforms ordinary softmax attention. Moreover, current implementations of linear attention lack I/O-awareness and are thus slower than highly optimized implementations of softmax attention. This work describes a hardware-efficient algorithm for linear attention that trades off memory movement against parallelizability. The resulting implementation, dubbed FlashLinearAttention, is faster than FlashAttention-2 as a standalone layer even on short sequence lengths (e.g., 1K). We then generalize this algorithm to a more expressive variant of linear attention with data-dependent gates. When used as a replacement for the standard attention layer in Transformers, the resulting gated linear attention (GLA) Transformer is found to perform competitively against the LLaMA-architecture Transformer as well recent linear-time-inference baselines such as RetNet and Mamba on moderate-scale language modeling experiments. GLA Transformer is especially effective at length generalization, enabling a model trained on 2K to generalize to sequences longer than 20K without significant perplexity degradations. For training speed, the GLA Transformer has higher throughput than a similarly-sized Mamba model."},"_bibtex":{"value":"@inproceedings{\nyang2024gated,\ntitle={Gated Linear Attention Transformers with Hardware-Efficient Training},\nauthor={Songlin Yang and Bailin Wang and Yikang Shen and Rameswar Panda and Yoon Kim},\nbooktitle={Forty-first International Conference on Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=ia5XvxFUJT}\n}"},"title":{"value":"Gated Linear Attention Transformers with Hardware-Efficient Training"},"pdf":{"value":"/pdf/c83a4c1aed45c267e44d9f8c94b16e5a902bca84.pdf"},"venueid":{"value":"ICML.cc/2024/Conference"},"paperhash":{"value":"yang|gated_linear_attention_transformers_with_hardwareefficient_training"},"authorids":{"value":["~Songlin_Yang1","~Bailin_Wang3","~Yikang_Shen1","~Rameswar_Panda1","~Yoon_Kim1"]},"authors":{"value":["Songlin Yang","Bailin Wang","Yikang Shen","Rameswar Panda","Yoon Kim"]}},"tmdate":1719287294506,"pdate":1714610498280,"tcdate":1706873049711,"writers":["ICML.cc/2024/Conference","ICML.cc/2024/Conference/Submission9608/Authors"],"signatures":["ICML.cc/2024/Conference/Submission9608/Authors"],"forum":"ia5XvxFUJT","license":"CC BY 4.0","number":9608,"cdate":1706873049711,"readers":["everyone"],"invitations":["ICML.cc/2024/Conference/-/Submission","ICML.cc/2024/Conference/-/Post_Submission","ICML.cc/2024/Conference/-/Edit","ICML.cc/2024/Conference/Submission9608/-/Camera_Ready_Revision"],"mdate":1719287294506,"odate":1717693102818,"domain":"ICML.cc/2024/Conference","id":"ia5XvxFUJT","version":2},{"content":{"venue":{"value":"https://arxiv.org/pdf/2312.06635.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"yang|gated_linear_attention_transformers_with_hardwareefficient_training"},"authorids":{"value":["~Songlin_Yang1","~Bailin_Wang3","~Yikang_Shen1","~Rameswar_Panda1","~Yoon_Kim1"]},"abstract":{"value":"Transformers with linear attention allow for efficient parallel training but can simultaneously be formulated as an RNN with 2D (matrix-valued) hidden states, thus enjoying linear-time inference complexity. However, linear attention generally underperforms ordinary softmax attention. Moreover, current implementations of linear attention lack I/O-awareness and are thus slower than highly optimized implementations of softmax attention. This work describes a hardware-efficient algorithm for linear attention that trades off memory movement against parallelizability. The resulting implementation, dubbed FLASHLINEARATTENTION, is faster than FLASHATTENTION-2(Dao, 2023) as a standalone layer even at short sequence lengths (e.g., 1K). We then generalize this algorithm to a more expressive variant of linear attention with data-dependent gates. When used as a replacement for the standard attention layer in Transformers, the resulting gated linear attention (GLA) Transformer is found to perform competitively against the LLaMA-architecture Transformer (Touvron et al., 2023) as well recent linear-time-inference baselines such as RetNet(Sun et al., 2023a) and Mamba (Gu & Dao, 2023) on moderate-scale language modeling experiments. GLA Transformer is especially effective at length generalization, enabling a model trained on 2K to generalize to 28K on PG19 without significant perplexity degradations. For training speed, the GLA Transformer has higher throughput than a similarly-sized Mamba model."},"title":{"value":"Gated Linear Attention Transformers with Hardware-Efficient Training"},"authors":{"value":["Songlin Yang","Bailin Wang","Yikang Shen","Rameswar Panda","Yoon Kim"]}},"tmdate":1710262735602,"pdate":1702313920132,"tcdate":1710262735602,"writers":["~Songlin_Yang1","~Bailin_Wang3","~Yikang_Shen1","~Rameswar_Panda1","~Yoon_Kim1"],"signatures":["~Songlin_Yang1"],"forum":"GeAww2SG9p","license":"CC BY 4.0","number":25431,"cdate":1710262735602,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1710262735602,"domain":"OpenReview.net/Archive","id":"GeAww2SG9p","version":2},{"content":{"summary":{"value":"This paper shows that many sequence models (including attention, SSMs, and recurrent models) can be viewed as linear time-varying dynamical systems. This is helpful for answering some questions about the differences and similarities between these architectures."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"It would help to add additional exposition on the results in Fig. 2. What are the takeaways? It looks like exponentiating improves performance, but this isn’t stated explicitly."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"**Importance of unifying framework**. The present state of sequence model research is quite muddy, with tons of new architectures being proposed. On the surface these architectures often look quite complicated and differentiated, when in reality they are closely related to one another.  Because of this, I think unifying frameworks are useful for helping push progress in the space. \n\n**The role of normalization in S6 and Linear Attention.** The paper includes a very interesting discussion of the parameterization of the normalization in S6 and Linear attention."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Presentation and clarity.** The introduction and abstract do not provide any details on how the Dynamical Systems Framework works and assumes a reader is familiar with dynamical systems. To improve the presentation and accessibility, I would recommend including a high-level discussion of what the DSF is and what challenges it addresses. \n\n**Theoretical Novelty.** In the first paragraph of the introduction the authors state, “Although these models show great promise in boosting efficiency, current comparisons with attention are merely empirical.” This isn’t correct: there have been many theoretical studies comparing these new recurrent architectures with Softmax attention. To name a few:\n\n- [Arora *et al.](https://arxiv.org/abs/2402.18668)* include a theoretical analysis grounded in results from communication complexity that highlights differences in models ability to perform associative recall.\n- Using tools from circuit complexity, [Merill *et al.*](https://arxiv.org/pdf/2404.08819) show that Attention and SSMs are in the same complexity class ($\\text{TC}^0$).\n- Using tools from communication complexity, [Bhattamishra *et al.*](https://arxiv.org/pdf/2406.09347)  provide separation results between attention and recurrent architectures on numerous tasks.\n\n**Empirical Novelty.** The main empirical result in this paper is that increased state size (across architectures) leads to improved performance on MQAR and that separable attention matches Softmax attention with sufficiently large recurrent size. This result has already been shown theoretically and empirically in [Arora *et al.*](https://arxiv.org/pdf/2402.18668) (ICML ‘24) - see Figure 2, Theorem 3.1, and Section 3. The work under review builds upon Arora *et al.* by including a few additional architectures to the analysis. But this prior work should be mentioned in the related work and/or the introduction given the significant overlap in results . \n\n**Claims around Lemma 2.** Some of the language around Lemma 2 is a bit incautious. The authors state that because of Lemma 2, “the larger the state expansion n, the more expressivity the architecture has”. In other words, they are claiming that increasing state expansion strictly increases expressivity. To support this claim, the theorem would need to additionally show that a dynamical system of state dimension $N$ ***cannot*** recover all dynamical systems with state dimension $\\hat{N}$.  Additionally, I think its important to add the qualification that this holds assuming everything else about the architecture is held constant. It’s of course possible for an architecture with smaller state size to admit solutions that a larger state size model cannot represent if the parameterizations are different. \n\n**Minor typos and clarifications.**\n\n- Line 32: “framework that allows to evaluate” → “allows us to evaluate”\n- Line 61: “We use $\\sigma(\\cdot)$ to denote is the sigmoid function.\n- In the list of questions answered by the paper (Line 40), I would include hyperlinks to the relevant sections."},"limitations":{"value":"Yes the authors discuss limitations."}},"nonreaders":[],"tmdate":1730880102971,"tcdate":1720470236668,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission20302/Reviewer_GS8T"],"signatures":["NeurIPS.cc/2024/Conference/Submission20302/Reviewer_GS8T"],"forum":"iF7MnXnxRw","number":2,"license":"CC BY 4.0","cdate":1720470236668,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission20302/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730880102971,"domain":"NeurIPS.cc/2024/Conference","replyto":"iF7MnXnxRw","id":"ZJMOhutwO7","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["deep learning architectures","softmax attention","linear attention","state space models","recurrent neural networks","dynamical system models"]},"supplementary_material":{"value":"/attachment/477b9fea44e7cd97ea957e64806a11ced418ad14.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Softmax attention is the principle backbone of foundation models for various artificial intelligence applications, yet its quadratic complexity in sequence length can limit its inference throughput in long-context settings. To address this challenge, alternative architectures such as linear attention, State Space Models (SSMs), and Recurrent Neural Networks (RNNs) have been considered as more efficient alternatives. While connections between these approaches exist, such models are commonly developed in isolation and there is a lack of theoretical understanding of the shared principles underpinning these architectures and their subtle differences, greatly influencing performance and scalability. In this paper, we introduce the Dynamical Systems Framework (DSF), which allows a principled investigation of all these architectures in a common representation. Our framework facilitates rigorous comparisons, providing new insights on the distinctive characteristics of each model class. For instance, we compare linear attention and selective SSMs, detailing their differences and conditions under which both are equivalent. We also provide principled comparisons between softmax attention and other model classes, discussing the theoretical conditions under which softmax attention can be approximated. Additionally, we substantiate these new insights with empirical validations and mathematical arguments. This shows the DSF's potential to guide the systematic development of future more efficient and scalable foundation models."},"_bibtex":{"value":"@inproceedings{\nsieber2024understanding,\ntitle={Understanding the Differences in Foundation Models: Attention, State  Space Models, and Recurrent Neural Networks},\nauthor={Jerome Sieber and Carmen Amo Alonso and Alexandre Didier and Melanie Zeilinger and Antonio Orvieto},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=iF7MnXnxRw}\n}"},"title":{"value":"Understanding the Differences in Foundation Models: Attention, State  Space Models, and Recurrent Neural Networks"},"pdf":{"value":"/pdf/53cb725e5aa20518f58b8b6b46d806687bae9108.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"sieber|understanding_the_differences_in_foundation_models_attention_state_space_models_and_recurrent_neural_networks"},"authorids":{"value":["~Jerome_Sieber1","~Carmen_Amo_Alonso3","~Alexandre_Didier1","~Melanie_Zeilinger1","~Antonio_Orvieto3"]},"authors":{"value":["Jerome Sieber","Carmen Amo Alonso","Alexandre Didier","Melanie Zeilinger","Antonio Orvieto"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","Quantization"]},"supplementary_material":{"value":"/attachment/0924a0296e7a1a9ad7158edce07c33777608192e.pdf"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSMs), such as Mamba, have recently demonstrated the potential to match or even surpass Transformers in language understanding tasks, making them a promising alternative for designing Large Language Models (LLMs). \nConcurrently, model quantization, especially Post-Training Quantization (PTQ), has been proven effective in reducing memory usage and inference latency in LLMs.\nIn this paper, we explore post-training quantization for Mamba (\\textbf{Q-Mamba}) by turning both linear projections and state caches into low-bit integers for efficient inference. \nAfter a theoretical analysis of the causes of outliers in states, we propose \\textbf{Decoupled Scale Quantization (DSQ)}, which mitigates outliers in both the state and channel dimensions by applying separate quantization scales.\nTo preserve the selective ability of quantized Mamba, we introduce \\textbf{Efficient Selectivity Reconstruction (ESR)}, a block-wise reconstruction method that involves a novel quantization simulation scheme, enabling fast parallel scan algorithms with the non-linear quantization function.\nWe demonstrate the effectiveness of Q-Mamba across various quantization settings, model sizes, and both generation and zero-shot tasks. \nIn particular, for Mamba2-2.7B with W8A8H4 quantization, Q-Mamba achieves a 50\\% reduction in memory consumption with only a 2.13\\% average accuracy degradation on zero-shot tasks."},"_bibtex":{"value":"@misc{\ntianqi2025qmamba,\ntitle={Q-Mamba: Towards more efficient Mamba models via Post-Training Quantization},\nauthor={Chen Tianqi and Yuanteng Chen and Weixiang Xu and Zeyu Zhu and Peisong Wang and Jian Cheng},\nyear={2025},\nurl={https://openreview.net/forum?id=AY1S52vr0a}\n}"},"title":{"value":"Q-Mamba: Towards more efficient Mamba models via Post-Training Quantization"},"pdf":{"value":"/pdf/1c7e10c8b02e5d50be6957f056d845dde2575456.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"tianqi|qmamba_towards_more_efficient_mamba_models_via_posttraining_quantization"},"authorids":{"value":["~Chen_Tianqi1","~Yuanteng_Chen1","~Weixiang_Xu2","~Zeyu_Zhu1","~Peisong_Wang1","~Jian_Cheng7"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chen Tianqi","Yuanteng Chen","Weixiang Xu","Zeyu Zhu","Peisong Wang","Jian Cheng"]}},"tmdate":1738735642991,"tcdate":1726810658890,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2045/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission2045/Authors"],"forum":"AY1S52vr0a","license":"CC BY 4.0","number":2045,"cdate":1726810658890,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission2045/-/Full_Submission","ICLR.cc/2025/Conference/Submission2045/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735642991,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"AY1S52vr0a","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2503.24067v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"li|transmamba_flexibly_switching_between_transformer_and_mamba"},"authorids":{"value":["~Yixing_Li1","~Ruobing_Xie1","https://dblp.org/search/pid/api?q=author:Zhen_Yang:","https://dblp.org/search/pid/api?q=author:Xingwu_Sun:","~Shuaipeng_Li1","https://dblp.org/search/pid/api?q=author:Weidong_Han_0006:","https://dblp.org/search/pid/api?q=author:Zhanhui_Kang:","https://dblp.org/search/pid/api?q=author:Yu_Cheng:","https://dblp.org/search/pid/api?q=author:Cheng-Zhong_Xu_0001:","~Di_Wang25","~Jie_Jiang3"]},"html":{"value":"https://doi.org/10.48550/arXiv.2503.24067"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-24067,\n  publtype={informal},\n  author={Yixing Li and Ruobing Xie and Zhen Yang and Xingwu Sun and Shuaipeng Li and Weidong Han and Zhanhui Kang and Yu Cheng and Cheng-Zhong Xu and Di Wang and Jie Jiang},\n  title={TransMamba: Flexibly Switching between Transformer and Mamba},\n  year={2025},\n  month={March},\n  cdate={1740787200000},\n  journal={CoRR},\n  volume={abs/2503.24067},\n  url={https://doi.org/10.48550/arXiv.2503.24067}\n}\n"},"abstract":{"value":"Transformers are the cornerstone of modern large language models, but their quadratic computational complexity limits efficiency in long-sequence processing. Recent advancements in Mamba, a state space model (SSM) with linear complexity, offer promising efficiency gains but suffer from unstable contextual learning and multitask generalization. This paper proposes TransMamba, a novel framework that unifies Transformer and Mamba through shared parameter matrices (e.g., QKV and CBx), and thus could dynamically switch between attention and SSM mechanisms at different token lengths and layers. We design the Memory converter to bridge Transformer and Mamba by converting attention outputs into SSM-compatible states, ensuring seamless information flow at TransPoints where the transformation happens. The TransPoint scheduling is also thoroughly explored for further improvements. We conducted extensive experiments demonstrating that TransMamba achieves superior training efficiency and performance compared to baselines, and validated the deeper consistency between Transformer and Mamba paradigms, offering a scalable solution for next-generation sequence modeling."},"title":{"value":"TransMamba: Flexibly Switching between Transformer and Mamba"},"authors":{"value":["Yixing Li","Ruobing Xie","Zhen Yang","Xingwu Sun","Shuaipeng Li","Weidong Han","Zhanhui Kang","Yu Cheng","Cheng-Zhong Xu","Di Wang","Jie Jiang"]}},"tmdate":1768986022722,"pdate":1735689600000,"tcdate":1747292716179,"writers":["~"],"signatures":["~Shuaipeng_Li1"],"forum":"Faqi5AUSG8","license":"CC BY-SA 4.0","number":475601,"cdate":1740787200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1768986022722,"domain":"DBLP.org","id":"Faqi5AUSG8","version":2},{"content":{"summary":{"value":"The authors introduced Mamba architecture to solve the problems of tabular data. They showed the effectiveness of the proposed method by comparing its performance with neural networks-based algorithms and tree-based methods. And they confirmed the superiority of a sequence and passing method by comparing various pooling methods."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please check the above weaknesses."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The authors proposed the Mamba architecture to solve tabular problems. It showed good performance overall, especially in classification performance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The proposed method seems to be nothing more than using the Mamba structure on tabular data. Although there are some structural suggestions, it is difficult to say that the idea is novel overall.\nAnd when I check the experimental results, it seems difficult to say that the performance has been greatly improved.Additional discussions and experiments are needed to see what advantages can be taken advantage of by applying the Mamba structure to tabular data, what differential performance improvements can be made compared to existing methods, and what can be considered if a Mamba structure or a new structure based on Mamba is proposed in the future."}},"nonreaders":[],"tmdate":1731427504630,"tcdate":1730647728286,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1879/Reviewer_2WJ3"],"signatures":["ICLR.cc/2025/Conference/Submission1879/Reviewer_2WJ3"],"forum":"wElgE9qBb5","number":3,"license":"CC BY 4.0","cdate":1730647728286,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1879/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427504630,"domain":"ICLR.cc/2025/Conference","replyto":"wElgE9qBb5","id":"xaHLICPuLJ","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Tabular Deep Learning","Mamba","Sequential Models","SSM","Recurrent Neural Networks"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The analysis of tabular data has traditionally been dominated by gradient-boosted decision trees (GBDTs), known for their proficiency with mixed categorical and numerical features. However, recent deep learning innovations are challenging this dominance. We introduce Mambular, an adaptation of the Mamba architecture optimized for tabular data. We extensively benchmark Mambular against state-of-the-art models, including neural networks and tree-based methods, and demonstrate its competitive performance across diverse datasets.\nAdditionally, we explore various adaptations of Mambular to understand its effectiveness for tabular data. We investigate different pooling strategies, feature interaction mechanisms, and bi-directional processing. Our analysis shows that interpreting features as a sequence and passing them through Mamba layers results in surprisingly performant models.  The results highlight Mambular’s potential as a versatile and powerful architecture for tabular data analysis, expanding the scope of deep learning applications in this domain.\n The source code is available at: https://anonymous.4open.science/r/mamba-tabular-485F/"},"_bibtex":{"value":"@misc{\nthielmann2025mambular,\ntitle={Mambular: A Sequential Model for Tabular Deep Learning},\nauthor={Anton Frederik Thielmann and Manish Kumar and Christoph Weisser and Arik Reuter and Benjamin S{\\\"a}fken and Soheila Samiee},\nyear={2025},\nurl={https://openreview.net/forum?id=wElgE9qBb5}\n}"},"title":{"value":"Mambular: A Sequential Model for Tabular Deep Learning"},"pdf":{"value":"/pdf/4cfe83c12311b6ae99930d6ccfd4ada19aab102f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"thielmann|mambular_a_sequential_model_for_tabular_deep_learning"},"authorids":{"value":["~Anton_Frederik_Thielmann1","~Manish_Kumar4","~Christoph_Weisser1","~Arik_Reuter1","~Benjamin_Säfken1","~Soheila_Samiee1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Anton Frederik Thielmann","Manish Kumar","Christoph Weisser","Arik Reuter","Benjamin Säfken","Soheila Samiee"]}},"version":2},{"content":{"comment":{"value":"> This is the most critical part of my comment. Except CIFAR 100, other datasets are not very standard to test pruning efficacy. The accuracy results are given as a passing remark. Bigger benchmarks (something like ImageNet) would be the best choices for the effectiveness of pruning.\n\nogbg-molhiv is a standard benchmark in the OGB datasets used for graph neural networks [3, 4], and PIQA is a standard benchmark for LLMs [5, 6].\n\nDue to time and resource constraints, we have not run on even larger datasets. We want to note that we are the first to model the relationship among NN parameters with exchangeability and to perform partial computation during inference exploiting the property. We have demonstrated that the ExPrune algorithm generally works for 3 model families across 3 problem domains, with the largest model size containing 7.6B parameters, and the current paper is already packed with theory, algorithm, and a comprehensive evaluation. Scaling the algorithm to even larger models and larger datasets could be another interesting problem to be solved in another academic paper.\n\n[1] Fontana, Matteo, Gianluca Zeni, and Simone Vantini. \"Conformal prediction: a unified review of theory and new challenges.\" Bernoulli 29, no. 1 (2023): 1-23.\n\n[2] Bloem-Reddy, Benjamin, and Yee Whye Teh. \"Probabilistic symmetries and invariant neural networks.\" Journal of Machine Learning Research 21, no. 90 (2020): 1-61.\n\n[3] Xu, Keyulu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. \"How powerful are graph neural networks?.\" International Conference on Learning Representations (2019).\n\n[4] Kipf, T. N. \"Semi-supervised classification with graph convolutional networks.\" International Conference on Learning Representations (2017).\n\n[5] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" In First conference on language modeling. 2024.\n\n[6] Jiang, Albert Q., Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot et al. \"Mixtral of experts.\" arXiv preprint arXiv:2401.04088 (2024)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764722951001,"tcdate":1764722951001,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13259/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13259/Authors"],"forum":"2ssFZn7ISk","number":23,"license":"CC BY 4.0","cdate":1764722951001,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13259/-/Official_Comment"],"mdate":1764722951001,"domain":"ICLR.cc/2026/Conference","replyto":"SaIm2OpIUS","id":"WCq5F4LQsJ","forumContent":{"TLDR":{"value":"We formalize the symmetry in neural networks using statistical exchangeability, and exploit it to dynamically prune neural networks on a per-input basis."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["exchangeability","parameter space symmetry","dynamic pruning","efficient inference"]},"primary_area":{"value":"probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)"},"abstract":{"value":"Modern neural networks (NN) contain an ever-growing number of parameters, substantially increasing the memory and computational cost of inference. Researchers have explored various ways to reduce the inference cost of NNs by reducing the model size before deployment and dynamically pruning the inference computation at runtime. In this work, we present ExPrune, a general, dynamic pruning optimization that enables multi-granularity partial computation on a per-input basis. ExPrune requires no change to the model architecture or the training algorithm. ExPrune is based on our theoretical results that the relationship between certain model parameters and intermediate values can be described by a statistical property called \\textit{exchangeability}. By identifying exchangeable parameters and values in the model, we are able to first partially evaluate the network, analyze the statistics of the partial results, and make pruning decisions on the fly.\nBecause ExPrune is theory grounded, it generalizes across model architectures in different problem domains. We evaluate ExPrune on one computer vision models, one graph model and one language model. ExPrune provides 10.98--17.33\\% reduction in FLOPs with negligible accuracy drop and 21.61--27.16\\% reduction in FLOPs with at most 1\\% accuracy drop. We also demonstrate that ExPrune composes with static magnitude pruning. On models that have been aggressively statically pruned, ExPrune still provides additional 10.24--11.11\\% reduction in FLOPs with negligible accuracy drop and 13.91--14.39\\% reduction in FLOPs with at most 1\\% accuracy drop."},"_bibtex":{"value":"@misc{\nyi2026exchangeability,\ntitle={Exchangeability in Neural Networks and its Application to Dynamic Pruning},\nauthor={Pu Luke Yi and Tianlang Chen and Yifan Yang and Sara Achour},\nyear={2026},\nurl={https://openreview.net/forum?id=2ssFZn7ISk}\n}"},"title":{"value":"Exchangeability in Neural Networks and its Application to Dynamic Pruning"},"pdf":{"value":"/pdf/8db7e99fdf68cd693be28f1698f014d48233d76a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yi|exchangeability_in_neural_networks_and_its_application_to_dynamic_pruning"},"authorids":{"value":["~Pu_Luke_Yi1","~Tianlang_Chen2","~Yifan_Yang34","~Sara_Achour1"]},"authors":{"value":["Pu Luke Yi","Tianlang Chen","Yifan Yang","Sara Achour"]}},"version":2},{"content":{"summary":{"value":"MambaVLM is a highly efficient multi-modal large language model framework that integrates Mamba’s linear complexity with a novel cross-stitch scanning approach to improve both visual information interaction and vision-language alignment. Achieving competitive benchmark results with only 0.66 million data points and 14 hours of training on a single A800 node, MambaVLM significantly outperforms LLaVA-1.5 and rivals the performance of Qwen-VL, demonstrating Mamba’s potential in enhancing MLLM efficiency and effectiveness."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please see the section on weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The proposed method in the paper performs very well, achieving better performance than LLaVA 1.5 with only half the training time.\n2. The approach is ingenious, using Mamba for long-context vision-language modeling is a promising avenue worth exploring.\n3. The paper is written with a clear structure."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The performance comparison with the original LLaVA is somewhat unfair, as the method in the paper uses two visual encoders. It would be better if a version with only ViT-CLIP could be provided.\n2. The method description in the paper is unclear; perhaps I missed where it explains how Mamba-VLM + Vicuna is implemented. It seems that if Vicuna is used, only the Mamba projector is related to Mamba. Of course, I also understand that the performance of VLMs is highly dependent on the performance of the LLM, and Mamba as an LLM is still relatively weak."}},"nonreaders":[],"tmdate":1731428096935,"tcdate":1730573521913,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4294/Reviewer_ApMf"],"signatures":["ICLR.cc/2025/Conference/Submission4294/Reviewer_ApMf"],"forum":"0A6f1b66pE","number":1,"license":"CC BY 4.0","cdate":1730573521913,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4294/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428096935,"domain":"ICLR.cc/2025/Conference","replyto":"0A6f1b66pE","id":"6f6yKR3u3C","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Vision-Language Models; Mamba;"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"While emerging multi-modal large language models (MLLM) have demonstrated impressive advances, the quadratic complexity of their Transformer-based LLMs (3B or larger) inevitably leads to considerable computational overhead. On the other hand, the recently proposed selective state space model (i.e., Mamba) enjoys both model capacity and computational efficiency, making it an ideal component to enhance MLLM's efficiency and performance. However, recent attempts to introduce Mamba into MLLMs simply replace their LLMs with Mamba, ignoring the unique characteristics of either side. We argue that such a naive combination cannot exhibit the potential of Mamba in MLLMs. In this paper, we delve into harnessing Mamba's unique properties, and propose tailored designs from both multi-modal input and architectural perspectives to unleash its true power. First, we fully utilize Mamba's linear complexity to construct visual long sequences for a thorough perception at a minor efficiency burden. To integrate the scanning mechanism with the built visual long sequence, we devise a novel cross-stitch scanning approach to capture and fuse spatial and semantic properties simultaneously, enhancing the interaction of visual information and the vision-language alignment. Built upon these designs, we propose MambaVLM, a simple yet effective MLLM framework that exhibits highly competitive results across multiple benchmarks. Moreover, our framework is also compatible with Transformer-based LLMs (e.g., Vicuna), demonstrating remarkable training and inference efficiency. Notably, with only 0.66M data and 14 hours training on a single A800 node, our MambaVLM outperforms LLaVA-1.5 by significant margins and performs on par or even better than the 1.4B data trained Qwen-VL. The appealing results from both effectiveness and efficiency aspects indicate the promising prospects of Mamba in MLLMs."},"_bibtex":{"value":"@misc{\nchen2024unleashing,\ntitle={Unleashing the Power of Selective State Space Models in Vision-Language Models},\nauthor={Honghao Chen and Yibing Song and Shoufa Chen and Chongjian GE and Kaiqi Huang},\nyear={2024},\nurl={https://openreview.net/forum?id=0A6f1b66pE}\n}"},"title":{"value":"Unleashing the Power of Selective State Space Models in Vision-Language Models"},"pdf":{"value":"/pdf/1183a88cc6949631a83ff61a73451d323b4592a6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"chen|unleashing_the_power_of_selective_state_space_models_in_visionlanguage_models"},"authorids":{"value":["~Honghao_Chen1","~Yibing_Song1","~Shoufa_Chen1","~Chongjian_GE1","~Kaiqi_Huang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Honghao Chen","Yibing Song","Shoufa Chen","Chongjian GE","Kaiqi Huang"]}},"version":2},{"content":{"comment":{"value":"> **Q2: The source of the model’s strong cross-domain performance remains unclear. To what extent does it stem from the carefully designed 1D-ResNet and frame-difference backbone, and to what extent from the TTT adaptation layer?**\n\n**A11:** We analyze each key module and alternative backbone in the ablation study. When frame stem is not used for preprocessing, that is, simply using convolution operations on the input video frames to align the output feature dimension with the input dimension of the backbone network, the performance of the model is the worst (MAE: 14.12). When extracting inter-frame differences as the input feature, the performance is improved (MAE: 10.44). In the replacement of the backbone network, the performance is improved when using the 1D-ResNet designed in this paper (MAE: 1.94) compared with using a single-layer 1D-CNN (MAE: 2.61) and not using any backbone network (MAE: 3.38). In the comparison with different TTA variants, we use Tent (MAE: 8.67) or SAR (MAE: 7.70) methods to adapt the pre-trained models containing fusion stem and 1D-ResNet to the target domain, and the results are significantly worse than those of the models containing TTT layers. This indicates that each module has played a certain role. Fine-grained time difference features are of great significance for rPPG signal modeling. 1D-ResNet focuses on the rPPG representation in skin pixels and further extracts deep information. The TTT layer enhances the generalization ability of the model through training during the testing process. The results can be found in our Common Response and Section 4.4.\n\nThank you again for your valuable feedback and insights.\n\n[1] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R´e. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474–1487, 2020.\n\n[2] Albert Gu, Karan Goel, and Christopher R´e. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.\n\n[3] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R´e. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572–585, 2021.\n\n[4] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First conference on language modeling, 2024.\n\n[5] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024.\n\n[6] Jingda Du, Si-Qi Liu, Bochao Zhang, and Pong C Yuen. Dual-bridging with adversarial noise generation for domain adaptive rppg estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10355–10364, 2023.\n\n[7] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.\n\n[8] Haodong Li, Hao Lu, and Ying-Cong Chen. Bi-tta: Bidirectional test-time adapter for remote physiological measurement. In European Conference on Computer Vision, pp. 356–374. Springer, 2024.\n\n[9] Pei-Kai Huang, Tzu-Hsien Chen, Ya-Ting Chan, Kuan-Wen Chen, Shih-Yu Yang, Yen-Chun Chou, and Chiou-Ting Hsu. Fully test-time rppg estimation via synthetic signal-guided feature learning. Pattern Recognition, 170:112102, 2026.\n\n[10] Antonio D’Innocente, Francesco Cappio Borlino, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. One-shot unsupervised cross-domain detection. In European Conference on Computer Vision, pp. 732–748. Springer, 2020.\n\n[11] Marvin Klingner, Mouadh Ayache, and Tim Fingscheidt. Continual batchnorm adaptation (cbna) for semantic segmentation. IEEE Transactions on Intelligent Transportation Systems, 23(11): 20899–20911, 2022."},"title":{"value":"Response to Reviewer jTxE (4/4)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763848828341,"tcdate":1763847522740,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7347/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7347/Authors"],"forum":"l4KbvGkQSw","number":9,"license":"CC BY 4.0","cdate":1763847522740,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7347/-/Official_Comment"],"mdate":1763848828341,"domain":"ICLR.cc/2026/Conference","replyto":"a6m3fcmk1j","id":"DqODdCxmxV","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"Measure heart rate using the test-time training method in cross-domain scenarios."},"keywords":{"value":["rPPG","Heart rate measurement","Test-Time Training","Cross-domain"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Remote photoplethysmography (rPPG), a contactless technology for measuring physiological signals, holds significant promise for smart healthcare and affective computing. However, a key challenge for existing deep learning methods is the paradox between maintaining high measurement accuracy and ensuring low computational cost, especially in cross-domain scenarios. To address this, we propose PhysTTT, a novel and lightweight framework for heart rate measurement that integrates multiple 1D-CNNs with residual structures and a Test-Time Training (TTT) layer. Multi-time frame differences fusion and 1D-CNNs extract spatio-temporal features from facial video sequences by modeling subtle brightness variations, the TTT layer compresses the context information into a learnable vector space, enhancing the temporal modeling capability. Crucially, the TTT mechanism enables the model to adapt to unseen data distributions during inference, significantly boosting cross-domain generalization. Extensive experiments demonstrate that PhysTTT achieves state-of-the-art accuracy in both in-domain and cross-domain evaluations, offering an optimal balance of high performance, strong generalization, and low computational cost. Our code is publicly available at https://anonymous.4open.science/r/PhysTTT-B605/."},"_bibtex":{"value":"@misc{\nyao2026physttt,\ntitle={Phys{TTT}: Accurate and Lightweight Cross-Domain Heart Rate Measurement with Test-Time Training},\nauthor={Yancheng Yao and Yifei Liu and Shangwei Zhu and Yu Wang and Le Yuan and Shaohua Liu},\nyear={2026},\nurl={https://openreview.net/forum?id=l4KbvGkQSw}\n}"},"title":{"value":"PhysTTT: Accurate and Lightweight Cross-Domain Heart Rate Measurement with Test-Time Training"},"pdf":{"value":"/pdf/1458f1a5a8a0703f0f751dfcaf1c542ccfc6ba6f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yao|physttt_accurate_and_lightweight_crossdomain_heart_rate_measurement_with_testtime_training"},"authorids":{"value":["~Yancheng_Yao1","~Yifei_Liu15","~Shangwei_Zhu1","~Yu_Wang132","~Le_Yuan2","~Shaohua_Liu3"]},"authors":{"value":["Yancheng Yao","Yifei Liu","Shangwei Zhu","Yu Wang","Le Yuan","Shaohua Liu"]}},"version":2},{"content":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Multivariate Time Series","Time Series Modeling","Time Series Forecasting","Time Series Classification"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Modeling multivariate time series is a well-established problem with a wide range of applications from healthcare to financial markets. It, however, is challenging as it requires methods to (1) have high expressive power of representing complicated dependencies along the time axis to capture both long-term progression and seasonal patterns, (2) capture the inter-variate dependencies when it is informative, (3) dynamically model the dependencies of variate and time dimensions, and (4) have efficient training and inference for very long sequences. Traditional State Space Models (SSMs) are classical approaches for univariate time series modeling due to their simplicity and expressive power to represent linear dependencies. They, however, have fundamentally limited expressive power to capture non-linear dependencies, are slow in practice, and fail to model the inter-variate information flow. Despite recent attempts to improve the expressive power of SSMs by using deep structured SSMs, the existing methods are either limited to univariate time series, fail to model complex patterns (e.g., seasonal patterns), fail to dynamically model the dependencies of variate and time dimensions, and/or are input-independent.  We present Chimera, an expressive variation of the 2-dimensional SSMs with careful design of parameters to maintain high expressive power while keeping the training complexity linear. Using two SSM heads with different discretization processes and input-dependent parameters, Chimera is provably able to learn long-term progression, seasonal patterns, and desirable dynamic autoregressive processes. To improve the efficiency of complex 2D recurrence, we present a fast training using a new 2-dimensional parallel selective scan. Our experimental evaluation shows the superior performance of Chimera on extensive and diverse benchmarks, including ECG and speech time series classification, long-term and short-term time series forecasting, and time series anomaly detection."},"_bibtex":{"value":"@inproceedings{\nbehrouz2024chimera,\ntitle={Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models},\nauthor={Ali Behrouz and Michele Santacatterina and Ramin Zabih},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=ncYGjx2vnE}\n}"},"title":{"value":"Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models"},"pdf":{"value":"/pdf/293e7ef70612d586ad3576a085191e54b2c0eb16.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"behrouz|chimera_effectively_modeling_multivariate_time_series_with_2dimensional_state_space_models"},"authorids":{"value":["~Ali_Behrouz1","~Michele_Santacatterina2","~Ramin_Zabih1"]},"authors":{"value":["Ali Behrouz","Michele Santacatterina","Ramin Zabih"]}},"tmdate":1730873963583,"pdate":1727288061051,"tcdate":1715746410382,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission14135/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission14135/Authors"],"forum":"ncYGjx2vnE","license":"CC BY 4.0","number":14135,"cdate":1715746410382,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/-/Submission","NeurIPS.cc/2024/Conference/-/Post_Submission","NeurIPS.cc/2024/Conference/Submission14135/-/Revision","NeurIPS.cc/2024/Conference/-/Edit","NeurIPS.cc/2024/Conference/Submission14135/-/Camera_Ready_Revision"],"mdate":1730873963583,"odate":1730873963565,"domain":"NeurIPS.cc/2024/Conference","id":"ncYGjx2vnE","version":2},{"content":{"summary":{"value":"This paper proposes the MoVE framework for efficient multivariate time series forecasting. MoVE decouples temporal modeling from cross-variable modeling into two expert modules: a Temporal Expert based on CycleNet for periodic modeling, and a Cross-Variable Expert (RCVTE) for cross-variable modeling. These two experts are dynamically gated and fused to adapt to different types of time series. The authors also introduce Period-Aware Local Curriculum Learning (PALCL) and Robust Cross-Variable Attention (RCVA) to enhance training stability and generalization capabilities. Experiments on multiple benchmark datasets validate the performance of MoVE, achieving new state-of-the-art results on the ETTh1, Electricity, and Solar datasets."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"See the weaknesses."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. By decoupling temporal modeling from cross-variable modeling through the MoE framework, the model's interpretability and adaptability are improved.\n2. The introduction of PALCL and RCVA effectively alleviates the attention collapse problem, enhancing training stability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The experimental results only achieved first place in 7 out of 16 cases, which is not very good.\n2. Figure 1 is very schematic and does not well illustrate the overall architecture of the model.\n3. MoVE has limited innovation, with an architecture that is very similar to the SST[1]. Moreover, there is a lack of relevant performance comparisons.\n4. The core modules of MoVE (such as RCVTE and PALCL) are, to some extent, combinations and improvements of existing methods (CycleNet, iTransformer), lacking entirely original modeling mechanisms.\n[1] SST: Multi-Scale Hybrid Mamba-Transformer Experts for Long-Short Range Time Series Forecasting"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919373160,"tcdate":1761893087979,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7230/Reviewer_goo3"],"signatures":["ICLR.cc/2026/Conference/Submission7230/Reviewer_goo3"],"forum":"lDG61tzAyQ","number":1,"license":"CC BY 4.0","cdate":1761893087979,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7230/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919373160,"domain":"ICLR.cc/2026/Conference","replyto":"lDG61tzAyQ","id":"Cvzr7NCF2J","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Time-Series Forecasting; Periodicity; MoE; Transformer;"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Multivariate time-series forecasting presents the dual challenge of modeling intricate temporal dynamics and complex cross-variable dependencies. Prevailing approaches often prioritize one aspect at the expense of the other, leading to suboptimal performance. To address this limitation, we introduce MoVE, a novel framework that synergistically integrates temporal and cross-variable modeling within a unified architecture. MoVE employs two specialized experts. A Temporal Expert for capturing long-range dependencies and a lightweight Cross-Variable Expert for modeling robust cross-variable interactions. By decoupling these components within a Mixture-of-Experts framework and optimizing them collaboratively, MoVE dynamically adapts to diverse forecasting scenarios. Extensive experiments demonstrate that our framework achieves superior performance, establishing a new paradigm for effective multivariate timer series forecasting."},"_bibtex":{"value":"@misc{\nliang2026move,\ntitle={Mo{VE}: Synergistic Integration of Temporal and Cross-Variable Experts for Efficient Multivariate Time Series Forecasting},\nauthor={Dizhen Liang},\nyear={2026},\nurl={https://openreview.net/forum?id=lDG61tzAyQ}\n}"},"title":{"value":"MoVE: Synergistic Integration of Temporal and Cross-Variable Experts for Efficient Multivariate Time Series Forecasting"},"pdf":{"value":"/pdf/8dfa65c20b014dbd07eb38f2f766369efdc1ad91.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"liang|move_synergistic_integration_of_temporal_and_crossvariable_experts_for_efficient_multivariate_time_series_forecasting"},"authorids":{"value":["~Dizhen_Liang1"]},"authors":{"value":["Dizhen Liang"]}},"version":2},{"content":{"summary":{"value":"This work presents a new module by combining eidetic memory, fading memory, and long-term eidetic memory (through an innovative selection operation). The proposed new module has strong sequence modeling capacity and high inference efficiency. The proposed new architecture achieves perplexity comparable to Transformers and SSMs with promising long-sequence ability."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"* For the hybrid model baseline, what is the architecture (activation, layer ordering, FFN size, and attention-Mamba ratio)?\n* In previous work, usually hybrid models (e.g., Griffin, Jamba, Samba, Zamba) can slightly outperform pure Transformer or pure Mamba. Why does it perform worse than Transformer or Mamba in this work?\n* How is the proposed model's performance on real-world retrieval tasks like phonebook lookup and needle-in-a-haystack?\n* What is the impact of window length for the sliding window attention?\n* How is the profiling of the \"KV Cache\" of the proposed method in terms of sequence length?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"* This work provides analysis for both attention and Mamba and proposes \"B'MOJO-F\" by combining sliding window attention and Mamba.\n* In order to further improve the memorization/recall ability, the Innovation Selection is proposed to compensate for the lossy memory in fading memory and add important token information to the long-term eidetic memory. The Long-term eidetic memory can increase as the input sequence length increases but could be way more efficient than standard attention.\n* The resulting architecture shows better memory efficiency on Associative Recall tasks than Mamba, although it still underperforms Transformers."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* Lack of sufficient ablation study. This work claims that it leverages 4 kinds of memory: short-term eidetic memory 'in-context,' permanent structural memory 'in-weights,' fading memory 'in-state,' and long-term eidetic memory 'in-storage'. Specifically, how does the short-term eidetic memory impact the capacity for recall and language modeling? Additionally, in tables 1 & 2, for \"BMoJo (Fading + Eidetic)\", which eidetic memory is this referring to? It seems that adding this eidetic memory does not improve performance.\n* It seems that the new architecture achieves similar benchmark average accuracy and perplexity compared to pure Transformer. Although the authors claim that it is 10% faster than Mistral, there are many acceleration methods for pure Transformer models like GQA and kernel fusion which may mitigate the gap. What is the main advantage of the proposed architecture against Transformer?"},"limitations":{"value":"Please check the above."}},"nonreaders":[],"tmdate":1730879860884,"tcdate":1721004560263,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission16642/Reviewer_majr"],"signatures":["NeurIPS.cc/2024/Conference/Submission16642/Reviewer_majr"],"forum":"RnQdRY1h5v","number":4,"license":"CC BY 4.0","cdate":1721004560263,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission16642/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879860884,"domain":"NeurIPS.cc/2024/Conference","replyto":"RnQdRY1h5v","id":"j8KZ4lhPLT","forumContent":{"TLDR":{"value":"We introduce a novel hybrid architecture (State Space Models + Attention)  that efficiently processes past information using both fading and eidetic memory."},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Sequence Models","Language Models","State Space Models","Hybrid Architectures"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"We describe a family of architectures to support transductive inference by allowing memory to grow to a finite but a-priori unknown bound while making efficient use of finite resources for inference. Current architectures use such resources to represent data either eidetically over a finite span ('context' in Transformers), or fading over an infinite span (in State Space Models, or SSMs). Recent hybrid architectures have combined eidetic and fading memory, but with limitations that do not allow the designer or the learning process to seamlessly modulate the two, nor to extend the eidetic memory span. We leverage ideas from Stochastic Realization Theory to develop a class of models called B'MOJO to seamlessly combine eidetic and fading memory within an elementary composable module. The overall architecture can be used to implement models that can access short-term eidetic memory 'in-context,' permanent structural memory 'in-weights,' fading memory 'in-state,' and long-term eidetic memory 'in-storage' by natively incorporating retrieval from an asynchronously updated memory. We show that Transformers, existing SSMs such as Mamba, and hybrid architectures such as Jamba are special cases of B'MOJO and describe a basic implementation that can be stacked and scaled efficiently in hardware. We test B'MOJO on transductive inference tasks, such as associative recall, where it outperforms existing SSMs and Hybrid models; as a baseline, we test ordinary language modeling where B'MOJO achieves perplexity comparable to similarly-sized Transformers and SSMs up to 1.4B parameters, while being up to 10% faster to train. Finally, we test whether models trained inductively on a-priori bounded sequences (up to 8K tokens) can still perform transductive inference on sequences many-fold longer. B'MOJO's ability to modulate eidetic and fading memory results in better inference on longer sequences tested up to 32K tokens, four-fold the length of the longest sequences seen during training."},"_bibtex":{"value":"@inproceedings{\nzancato2024bmojo,\ntitle={B'{MOJO}: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory},\nauthor={Luca Zancato and Arjun Seshadri and Yonatan Dukler and Aditya Golatkar and Yantao Shen and Benjamin Bowman and Matthew Trager and Alessandro Achille and Stefano Soatto},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=RnQdRY1h5v}\n}"},"title":{"value":"B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory"},"pdf":{"value":"/pdf/7e6d66c19548b38081f7dab7696bb96a6f9916eb.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"zancato|bmojo_hybrid_state_space_realizations_of_foundation_models_with_eidetic_and_fading_memory"},"authorids":{"value":["~Luca_Zancato1","~Arjun_Seshadri1","~Yonatan_Dukler1","~Aditya_Golatkar1","~Yantao_Shen2","~Benjamin_Bowman1","~Matthew_Trager2","~Alessandro_Achille1","~Stefano_Soatto3"]},"authors":{"value":["Luca Zancato","Arjun Seshadri","Yonatan Dukler","Aditya Golatkar","Yantao Shen","Benjamin Bowman","Matthew Trager","Alessandro Achille","Stefano Soatto"]}},"version":2},{"content":{"venue":{"value":"TIME 2018"},"pdf":{"value":"https://drops.dagstuhl.de/storage/00lipics/lipics-vol120-time2018/LIPIcs.TIME.2018.16/LIPIcs.TIME.2018.16.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2018"},"paperhash":{"value":"kernberger|on_the_expressive_power_of_hybrid_branchingtime_logics"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Daniel_Kernberger:","~Martin_Lange1"]},"html":{"value":"https://doi.org/10.4230/LIPIcs.TIME.2018.16"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/KernbergerL18,\n  author={Daniel Kernberger and Martin Lange},\n  title={On the Expressive Power of Hybrid Branching-Time Logics},\n  year={2018},\n  cdate={1514764800000},\n  pages={16:1-16:18},\n  url={https://doi.org/10.4230/LIPIcs.TIME.2018.16},\n  booktitle={TIME},\n  crossref={conf/time/2018}\n}\n"},"abstract":{"value":"Hybrid branching-time logics are a powerful extension of branching-time logics like CTL, CTL^* or even the modal mu-calculus through the addition of binders, jumps and variable tests. Their expressiveness is not restricted by bisimulation-invariance anymore. Hence, they do not retain the tree model property, and the finite model property is equally lost. Their satisfiability problems are typically undecidable, their model checking problems (on finite models) are decidable with complexities ranging from polynomial to non-elementary time. In this paper we study the expressive power of such hybrid branching-time logics beyond some earlier results about their invariance under hybrid bisimulations. In particular, we aim to extend the hierarchy of non-hybrid branching-time logics CTL, CTL^+, CTL^* and the modal mu-calculus to their hybrid extensions. We show that most separation results can be transferred to the hybrid world, even though the required techniques become a bit more involved. We also present some collapse results for restricted classes of models that are especially worth investigating, namely linear, tree-shaped and finite models."},"title":{"value":"On the Expressive Power of Hybrid Branching-Time Logics"},"authors":{"value":["Daniel Kernberger","Martin Lange"]}},"tmdate":1748273793378,"pdate":1514764800000,"tcdate":1748273747599,"writers":["~"],"signatures":["~Martin_Lange1"],"forum":"fwYkatbJu6","license":"CC BY-SA 4.0","number":551825,"cdate":1514764800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1748273793378,"domain":"DBLP.org","id":"fwYkatbJu6","version":2},{"content":{"venue":{"value":"IEEE Trans. Geosci. Remote. Sens. 2024"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/36/10354519/10643108.pdf"},"venueid":{"value":"dblp.org/journals/TGRS/2024"},"paperhash":{"value":"fu|ssumamba_spatialspectral_selective_state_space_model_for_hyperspectral_image_denoising"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Guanyiman_Fu:","~Fengchao_Xiong1","~Jianfeng_Lu3","~Jun_Zhou1"]},"html":{"value":"https://doi.org/10.1109/TGRS.2024.3446812"},"_bibtex":{"value":"@article{DBLP:journals/tgrs/FuXLZ24,\n  author={Guanyiman Fu and Fengchao Xiong and Jianfeng Lu and Jun Zhou},\n  title={SSUMamba: Spatial-Spectral Selective State Space Model for Hyperspectral Image Denoising},\n  year={2024},\n  cdate={1704067200000},\n  journal={IEEE Trans. Geosci. Remote. Sens.},\n  volume={62},\n  pages={1-14},\n  url={https://doi.org/10.1109/TGRS.2024.3446812}\n}\n"},"abstract":{"value":"Denoising is a crucial preprocessing step for hyperspectral images (HSIs) due to noise arising from intraimaging mechanisms and environmental factors. Long-range spatial-spectral correlation modeling is beneficial for HSI denoising but often comes with high complexity. Based on the state space model (SSM), Mamba is known for its remarkable long-range dependency modeling capabilities and computational efficiency. Building on this, we introduce a memory-efficient spatial-spectral UMamba (SSUMamba) for HSI denoising, with the spatial-spectral continuous scan (SSCS) Mamba being the core component. SSCS Mamba alternates the row, column, and band in six different orders to generate the sequence and uses the bidirectional SSM to exploit long-range spatial-spectral dependencies. In each order, the images are rearranged between adjacent scans to ensure spatial-spectral continuity. In addition, 3-D convolutions are embedded into the SSCS Mamba to enhance local spatial-spectral modeling. Experiments demonstrate that SSUMamba achieves superior denoising results with lower memory consumption per batch compared with transformer-based methods. The source code is available at: https://github.com/lronkitty/SSUMamba ."},"title":{"value":"SSUMamba: Spatial-Spectral Selective State Space Model for Hyperspectral Image Denoising"},"authors":{"value":["Guanyiman Fu","Fengchao Xiong","Jianfeng Lu","Jun Zhou"]}},"tmdate":1731467993645,"pdate":1704067200000,"tcdate":1727751279263,"writers":["~"],"signatures":["~Jianfeng_Lu3"],"forum":"JbZJgQA3PQ","license":"CC BY-SA 4.0","number":126968,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1731467993645,"domain":"DBLP.org","id":"JbZJgQA3PQ","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2505.23214v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"xu|samamba_adaptive_state_space_modeling_with_hierarchical_vision_for_infrared_small_target_detection"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Wenhao_Xu:","https://dblp.org/search/pid/api?q=author:ShuChen_Zheng:","https://dblp.org/search/pid/api?q=author:Changwei_Wang_0001:","https://dblp.org/search/pid/api?q=author:Zherui_Zhang:","https://dblp.org/search/pid/api?q=author:Chuan_Ren:","https://dblp.org/search/pid/api?q=author:Rongtao_Xu:","~Shibiao_Xu1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2505.23214"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2505-23214,\n  publtype={informal},\n  author={Wenhao Xu and ShuChen Zheng and Changwei Wang and Zherui Zhang and Chuan Ren and Rongtao Xu and Shibiao Xu},\n  title={SAMamba: Adaptive State Space Modeling with Hierarchical Vision for Infrared Small Target Detection},\n  year={2025},\n  month={May},\n  cdate={1746057600000},\n  journal={CoRR},\n  volume={abs/2505.23214},\n  url={https://doi.org/10.48550/arXiv.2505.23214}\n}\n"},"abstract":{"value":"Infrared small target detection (ISTD) is vital for long-range surveillance in military, maritime, and early warning applications. ISTD is challenged by targets occupying less than 0.15% of the image and low distinguishability from complex backgrounds. Existing deep learning methods often suffer from information loss during downsampling and inefficient global context modeling. This paper presents SAMamba, a novel framework integrating SAM2's hierarchical feature learning with Mamba's selective sequence modeling. Key innovations include: (1) A Feature Selection Adapter (FS-Adapter) for efficient natural-to-infrared domain adaptation via dual-stage selection (token-level with a learnable task embedding and channel-wise adaptive transformations); (2) A Cross-Channel State-Space Interaction (CSI) module for efficient global context modeling with linear complexity using selective state space modeling; and (3) A Detail-Preserving Contextual Fusion (DPCF) module that adaptively combines multi-scale features with a gating mechanism to balance high-resolution and low-resolution feature contributions. SAMamba addresses core ISTD challenges by bridging the domain gap, maintaining fine-grained details, and efficiently modeling long-range dependencies. Experiments on NUAA-SIRST, IRSTD-1k, and NUDT-SIRST datasets show SAMamba significantly outperforms state-of-the-art methods, especially in challenging scenarios with heterogeneous backgrounds and varying target scales. Code: https://github.com/zhengshuchen/SAMamba."},"title":{"value":"SAMamba: Adaptive State Space Modeling with Hierarchical Vision for Infrared Small Target Detection"},"authors":{"value":["Wenhao Xu","ShuChen Zheng","Changwei Wang","Zherui Zhang","Chuan Ren","Rongtao Xu","Shibiao Xu"]}},"tmdate":1769485164871,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2505-23214"],"tcdate":1769485129405,"writers":["~"],"signatures":["~Shibiao_Xu2"],"forum":"wJdbtBobm0","license":"CC BY-SA 4.0","number":806097,"cdate":1746057600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769485164871,"domain":"DBLP.org","id":"wJdbtBobm0","version":2},{"content":{"summary":{"value":"The paper investigates the challenges faced by State Space Models (SSMs), such as Mamba, on recall-based tasks compared to Transformer architectures. The authors propose an initialization technique that encourages SSM layers to behave more similarly to self-attention. They empirically show that this initialization significantly improves Mamba's ability to learn memory-related tasks."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"* The plots showing attention patterns should be explained more clearly. What inputs were used to generate these visualizations? How dependent are they on the seed used for training? How do these patterns evolve over time? I think that this last question would be especially interesting given that the authors try to align SSM and self-attention at initialization, and I wonder whether this causes the dynamics to overall behave similarly, or rather that they simply start close and gradually diverge.\n* I find the results in Figure 3 strange: first, it seems that the initialization has relatively small effect on Mamba 1(with accuracy going at best up to 50%) while the Figure 2 shows every different behavior (>90%). Am I misunderstanding something? Second, even the results for Mamba 2 seem to vary greatly based on seed, which is quite undesirable and detracts from the conclusions of the paper.\n* Why do the authors believe there is such a large difference between Mamba 1 and Mamba 2? They mention this briefly on line 307, but I think this question deserves more attention.\n* Similarly, why should Mamba perform better than Linear Attention, given that it was initialized to mimic its behavior?\n\nMinor:\n* Line 258: \"who suggest these weights should not be strictly equal\" — It may be useful to provide some motivation for this.\n* Figure 8 mentions Mamba 1/2, but I believe only Mamba 2 is shown.\n* l.485: I believe that the authors use a fixed T_i to initialize all S_j (since L and M are different) but this is not clear."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper is well-written and addresses a fundamental limitation of SSMs compared to Transformers, which is still not fully understood. The proposed approach of initializing SSM layers to mimic self-attention is well-motivated, and the observed attention patterns suggest notable similarities in the behavior of the two. The authors present extensive experiments across various tasks and architectural configurations, confirming that the initialization generally improves performance. They also provide comparisons with linear attention and pretrained Mamba models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While the authors motivate the problem well, I find that they do not spend enough time discussing the broader implications of their work. From a theoretical perspective, the initialization approach is mainly heuristic, and does not provide a fully satisfactory understanding of why SSMs struggle with such tasks. On the empirical side, it is unclear whether the proposed initialization could improve performance on language-based tasks. Additionally, I have some concerns about the robustness and clarity of the presented results (see questions below). The differing behaviors between Mamba and Mamba2 are also somewhat concerning, given that the motivation for the initialization does not distinguish between the two. To sum up, I would like the authors to clarify the main theoretical and empirical takeaways of their work."}},"nonreaders":[],"tmdate":1731428295437,"tcdate":1730650042526,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11529/Reviewer_ZnsJ"],"signatures":["ICLR.cc/2025/Conference/Submission11529/Reviewer_ZnsJ"],"forum":"iVy7aRMb0K","number":2,"license":"CC BY 4.0","cdate":1730650042526,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11529/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428295437,"domain":"ICLR.cc/2025/Conference","replyto":"iVy7aRMb0K","id":"0M7vrdU1zW","forumContent":{"TLDR":{"value":"Mamba learns to recall more easily when we initialize it to be more like linear attention"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","SSM","sequence models","language models","initialization","linear attention"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with respect to their input sequence length. But in practice, state space models have fairly large state sizes, and we conjecture that they should be able to perform much better at these tasks than previously reported. We investigate whether their poor copying and recall performance could be due in part to training difficulties rather than fundamental capacity constraints. Based on observations of their \"attention'' maps, we propose a structured initialization technique that allows state space layers to more readily mimic attention. Across a variety of architecture settings, our initialization makes it substantially easier for Mamba to learn to copy and do associative recall from scratch."},"_bibtex":{"value":"@misc{\ntrockman2025mimetic,\ntitle={Mimetic Initialization Helps State Space Models Learn to Recall},\nauthor={Asher Trockman and Hrayr Harutyunyan and J Zico Kolter and Sanjiv Kumar and Srinadh Bhojanapalli},\nyear={2025},\nurl={https://openreview.net/forum?id=iVy7aRMb0K}\n}"},"title":{"value":"Mimetic Initialization Helps State Space Models Learn to Recall"},"pdf":{"value":"/pdf/5b9624e012b1653fd84db25266eba2a30fcf0588.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"trockman|mimetic_initialization_helps_state_space_models_learn_to_recall"},"authorids":{"value":["~Asher_Trockman1","~Hrayr_Harutyunyan1","~J_Zico_Kolter1","~Sanjiv_Kumar1","~Srinadh_Bhojanapalli1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Asher Trockman","Hrayr Harutyunyan","J Zico Kolter","Sanjiv Kumar","Srinadh Bhojanapalli"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2408.01129v4"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"qu|a_survey_of_mamba"},"authorids":{"value":["~Haohao_Qu1","~Liangbo_Ning1","~Rui_An2","~Wenqi_Fan1","https://dblp.org/search/pid/api?q=author:Tyler_Derr:","https://dblp.org/search/pid/api?q=author:Hui_Liu_0031:","https://dblp.org/search/pid/api?q=author:Xin_Xu_0002:","~Qing_Li5"]},"html":{"value":"https://doi.org/10.48550/arXiv.2408.01129"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2408-01129,\n  publtype={informal},\n  author={Haohao Qu and Liangbo Ning and Rui An and Wenqi Fan and Tyler Derr and Hui Liu and Xin Xu and Qing Li},\n  title={A Survey of Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2408.01129},\n  url={https://doi.org/10.48550/arXiv.2408.01129}\n}\n"},"abstract":{"value":"As one of the most representative DL techniques, Transformer architecture has empowered numerous advanced models, especially the large language models (LLMs) that comprise billions of parameters, becoming a cornerstone in deep learning. Despite the impressive achievements, Transformers still face inherent limitations, particularly the time-consuming inference resulting from the quadratic computation complexity of attention calculation. Recently, a novel architecture named Mamba, drawing inspiration from classical state space models (SSMs), has emerged as a promising alternative for building foundation models, delivering comparable modeling abilities to Transformers while preserving near-linear scalability concerning sequence length. This has sparked an increasing number of studies actively exploring Mamba's potential to achieve impressive performance across diverse domains. Given such rapid evolution, there is a critical need for a systematic review that consolidates existing Mamba-empowered models, offering a comprehensive understanding of this emerging model architecture. In this survey, we therefore conduct an in-depth investigation of recent Mamba-associated studies, covering three main aspects: the advancements of Mamba-based models, the techniques of adapting Mamba to diverse data, and the applications where Mamba can excel. Specifically, we first review the foundational knowledge of various representative deep learning models and the details of Mamba-1&2 as preliminaries. Then, to showcase the significance of Mamba for AI, we comprehensively review the related studies focusing on Mamba models' architecture design, data adaptability, and applications. Finally, we present a discussion of current limitations and explore various promising research directions to provide deeper insights for future investigations."},"title":{"value":"A Survey of Mamba"},"authors":{"value":["Haohao Qu","Liangbo Ning","Rui An","Wenqi Fan","Tyler Derr","Hui Liu","Xin Xu","Qing Li"]}},"tmdate":1784644326409,"pdate":1704067200000,"tcdate":1731567152102,"writers":["~"],"signatures":["~Qing_Li5"],"forum":"B5RjAkNAYp","license":"CC BY-SA 4.0","number":239324,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784644326409,"domain":"DBLP.org","id":"B5RjAkNAYp","version":2},{"content":{"comment":{"value":"### 2. Equations\n\nWe realized that we missed adding a label to the Mamba equations in the main text, sorry for this.\n\n* **Eq(1) if $\\bar{A}$, $\\bar{B}$, $\\bar{C}$, $\\bar{D}$ are learnable matrices, what do $\\bar{B}(x)$ and $\\bar{C}(x)$\n mean?**\n\nMamba makes $\\bar{B}(x)$ and $\\bar{C}(x)$ input ($x$)-dependent  so the model can “select” what context to propagate forward. We adopted this notation from [1] which we have cited in the main text and manuscript as we believe it makes it explicit that this is done. \n\n* **Based on Eq(1) it seems that Mamba is not significant to RNN. Can the author explain the difference?**\n\nMamba has similarities to RNNs and the authors of Mamba explicitly make this comparison. Under specific initialization, algorithm 2 in the Mamba paper [2] has the form of a gated RNN. In addition to the more general formulation of selective propagation, the authors introduced a “hardware-aware” algorithm for efficient training. We adapted their optimized code from the official repository [3]\n\n* **Eq (1) what's the difference between $x$, $s$ and $a$. Why don't use the same notation? Is $P$ the same as $\\pi_\\theta$?**\n\n$x$ is the SMILES sequence, $s$ is the state (all intermediate token sequences), and $a$ is the action (the token selected at a given time-step). State and Action are used in RL and we introduced this at the beginning of the section. $P$ is not the same as $\\pi_\\theta$. $P$ is the probability of generating the SMILES sequence, $x$.  $\\pi_\\theta$ denotes the network that generates the SMILES. We write this in the following sentence in the text:\n\n\"where $\\pi_\\theta$ is the Mamba backbone and referred to as the Agent to match RL terminology and $a_t$ and $s_t$ are the token selected and token sequence so far, at time-step t, respectively.\"\n\n* **Eq (2) and (3) same question: what's the difference between $R(x)$ and $R(a, s)$?**\n\n$R(x)$ is the return associated with a SMILES sequence, i.e., the scalar property value. $R(a, s)$ is the state-transition reward in RL, i.e., “what is the reward associated with taking the action in the given state?” In our setting, the action is the token selected and the state is the token sequence so far. In Appendix B.5, we mention that all intermediate rewards are 0 because there is only a reward for the final SMILES sequence.\n\n* **Eq (3) Are the authors defining $R$ or $\\pi_{Augmented}(x)$ here? If it’s $\\pi_{Augmented}(x)$, how is $R$ defined?**\n\nWe are defining $\\pi_{Augmented}(x)$. $R(x)$ is any arbitrary reward function that takes as input the SMILES, $x$, and returns the scalar reward. For example, in the experiments comparing with GEAM, $R$ is the docking, QED, SA score function:\n\n$R(x) = \\widehat{DS}(x) \\times QED(x) \\times \\widehat{SA}(x) \\in [0, 1]$\n\n### 3. Line plots\n\nThank you for this suggestion, we are thinking of how to use this to convey the information better without adding too many plots.\n\n\n\n\\\n\\\n[1] https://jameschen.io/jekyll/update/2024/02/12/mamba.html\n\n[2] https://arxiv.org/abs/2312.00752\n\n[3] https://github.com/state-spaces/mamba"},"title":{"value":"Response to Reviewer f1gd (continued)"}},"tmdate":1732723333613,"tcdate":1732717946383,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1831/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission1831/Authors"],"forum":"Hbpzrh7JbN","number":32,"license":"CC BY 4.0","cdate":1732717946383,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1831/-/Official_Comment"],"mdate":1732723333613,"domain":"ICLR.cc/2025/Conference","replyto":"syIr44S9Ef","id":"D5EJCMi1Sn","forumContent":{"TLDR":{"value":"state-of-the-art sample efficiency using mamba for generative molecular design"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["generative design","drug discovery","sample efficiency","language models","reinforcement learning","scaling"]},"supplementary_material":{"value":"/attachment/4735cc8666aadddbad349c1f2433393b9784f551.pdf"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Generative molecular design for drug discovery has very recently achieved a wave\nof experimental validation, with language-based backbones being the most common\narchitectures employed. The most important factor for downstream success is\nwhether an *in silico* oracle is well correlated with the desired end-point. To this end,\ncurrent methods use cheaper proxy oracles with higher throughput before evaluating\nthe most promising subset with high-fidelity oracles. The ability to directly optimize\nhigh-fidelity oracles would greatly enhance generative design and be expected to\nimprove hit rates. However, current models are not efficient enough to consider such\na prospect, exemplifying the sample efficiency problem. In this work, we introduce\n**Saturn**, which leverages the Augmented Memory algorithm and demonstrates the\nfirst application of the Mamba architecture for generative molecular design. We\nelucidate *how* experience replay with data augmentation improves sample efficiency\nand *how* Mamba synergistically exploits this mechanism. Saturn outperforms 22\nmodels on multi-parameter optimization tasks relevant to drug discovery and may\npossess sufficient sample efficiency to consider the prospect of directly optimizing\nhigh-fidelity oracles."},"_bibtex":{"value":"@misc{\nguo2025saturn,\ntitle={Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation},\nauthor={Jeff Guo and Philippe Schwaller},\nyear={2025},\nurl={https://openreview.net/forum?id=Hbpzrh7JbN}\n}"},"title":{"value":"Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation"},"pdf":{"value":"/pdf/c0929a667bd3f510fb8322a9d402f95c457e38e1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"guo|saturn_sampleefficient_generative_molecular_design_using_memory_manipulation"},"authorids":{"value":["~Jeff_Guo1","~Philippe_Schwaller1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jeff Guo","Philippe Schwaller"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2604.10597v3"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"ma|corey_entropyguided_runtime_chunk_scheduling_for_selective_scan_kernels"},"html":{"value":"https://doi.org/10.48550/arXiv.2604.10597"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2604-10597,\n  publtype={informal},\n  author={Bo Ma and Jinsong Wu and Hongjiang Wei and Wei Qi Yan},\n  title={COREY: Entropy-Guided Runtime Chunk Scheduling for Selective Scan Kernels},\n  year={2026},\n  month={April},\n  cdate={1775001600000},\n  journal={CoRR},\n  volume={abs/2604.10597},\n  url={https://doi.org/10.48550/arXiv.2604.10597}\n}\n"},"abstract":{"value":"Mamba selective state space models (SSMs) provide linear-time sequence modeling but remain sensitive to selective-scan chunk scheduling. We present COREY, a \\emph{concept-and-feasibility} runtime scheduler that maps fixed-bin activation entropy to chunk size. We evaluate COREY in three tiers: a prototype cost model, real-checkpoint kernel timing, and routed end-to-end ablations on modern GPUs. At the kernel level, a calibrated rule, \\(H_{\\mathrm{ref}}=\\log K\\), recovers the locally optimal chunk and matches a one-time static oracle, yielding \\(4.41\\times\\) lower latency than an unoptimized baseline on a consumer GPU and \\(3.90\\times\\)--\\(4.04\\times\\) lower latency on a data-center accelerator. Routing this choice into a patched live scan kernel closes the engineering loop without improving end-to-end speed: in unified routed ablations, the best static chunk outperforms all entropy-guided and proxy schedulers. Sampled-histogram COREY adds \\(+4.6\\%\\) overhead; a guarded fallback to Static-512 reduces this to \\(+1.3\\%\\); and a lightweight sequence-length-keyed table further reduces it to \\(+0.7\\%\\). However, both remain slower than the static oracle because they retain scheduling cost. On an 80-prompt LongBench subset, passive and routed inference are exactly output-equivalent, with \\(100\\%\\) greedy-token agreement and zero metric deltas. A mixed-regime study shows that a single sequence-length rule matches the per-regime chunk oracle for balanced serving. COREY is therefore validated as a quality-preserving scheduling prototype, but current entropy statistics are not a robust throughput win over static chunk tuning on measured SSM checkpoint workloads. SourceCode: https://github.com/mabo1215/COREY_Transformer/."},"title":{"value":"COREY: Entropy-Guided Runtime Chunk Scheduling for Selective Scan Kernels"},"authors":{"value":[{"fullname":"Bo Ma","username":""},{"fullname":"Jinsong Wu","username":"~Jinsong_Wu1"},{"fullname":"Hongjiang Wei","username":""},{"fullname":"Wei Qi Yan","username":""}]}},"tmdate":1787369197939,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2604-10597"],"tcdate":1787369190798,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jinsong_Wu1"],"forum":"D3KAr5SCQK","license":"CC BY-SA 4.0","number":135619,"cdate":1775001600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1787369197939,"domain":"OpenReview.net/Public_Article","id":"D3KAr5SCQK","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2405.01726v7"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"fu|ssumamba_spatialspectral_selective_state_space_model_for_hyperspectral_image_denoising"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Guanyiman_Fu:","~Fengchao_Xiong1","~Jianfeng_Lu3","~Jun_Zhou1","~Yuntao_Qian1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2405.01726"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2405-01726,\n  publtype={informal},\n  author={Guanyiman Fu and Fengchao Xiong and Jianfeng Lu and Jun Zhou and Yuntao Qian},\n  title={SSUMamba: Spatial-Spectral Selective State Space Model for Hyperspectral Image Denoising},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2405.01726},\n  url={https://doi.org/10.48550/arXiv.2405.01726}\n}\n"},"abstract":{"value":"Denoising is a crucial preprocessing step for hyperspectral images (HSIs) due to noise arising from intra-imaging mechanisms and environmental factors. Long-range spatial-spectral correlation modeling is beneficial for HSI denoising but often comes with high computational complexity. Based on the state space model (SSM), Mamba is known for its remarkable long-range dependency modeling capabilities and computational efficiency. Building on this, we introduce a memory-efficient spatial-spectral UMamba (SSUMamba) for HSI denoising, with the spatial-spectral continuous scan (SSCS) Mamba being the core component. SSCS Mamba alternates the row, column, and band in six different orders to generate the sequence and uses the bidirectional SSM to exploit long-range spatial-spectral dependencies. In each order, the images are rearranged between adjacent scans to ensure spatial-spectral continuity. Additionally, 3D convolutions are embedded into the SSCS Mamba to enhance local spatial-spectral modeling. Experiments demonstrate that SSUMamba achieves superior denoising results with lower memory consumption per batch compared to transformer-based methods. The source code is available at https://github.com/lronkitty/SSUMamba."},"title":{"value":"SSUMamba: Spatial-Spectral Selective State Space Model for Hyperspectral Image Denoising"},"authors":{"value":["Guanyiman Fu","Fengchao Xiong","Jianfeng Lu","Jun Zhou","Yuntao Qian"]}},"tmdate":1762322387924,"pdate":1704067200000,"tcdate":1727751279376,"writers":["~"],"signatures":["~Jianfeng_Lu3"],"forum":"cmFMcPmEOZ","license":"CC BY-SA 4.0","number":126973,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1762322387924,"domain":"DBLP.org","id":"cmFMcPmEOZ","version":2},{"content":{"summary":{"value":"The paper revisits the task of node affinity prediction in temporal graphs, which aims to forecast the future affinity distribution of each node rather than just predicting future links. The authors observe that simple heuristics like Persistent Forecast (PF) and Exponential Moving Average (EMA) often outperform existing Temporal Graph Neural Networks (TGNNs). To address this, they theoretically show that such heuristics can be viewed as linear State Space Models (SSMs) and propose a new TGNN architecture called NAVIS, which integrates a learnable linear SSM with a “virtual global state” to capture global temporal dynamics. The paper also replaces the cross-entropy loss with a ranking-based LambdaLoss to better align with the affinity prediction objective. Extensive experiments on multiple temporal graph benchmarks demonstrate that NAVIS consistently outperforms both heuristic and neural baselines."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see the weakness above"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper explains why existing TGNNs fail to capture simple temporal dependencies and provides a theoretical basis for improving them.\n\n2. The proposed NAVIS model is conceptually simple yet achieves strong and consistent performance across benchmarks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. There exists a conceptual gap between the theoretical analysis and the modeling assumptions used in practice.\nTheorem2 argues that standard RNN/LSTM/GRU units cannot reproduce the persistent forecast (PF, i.e., $h_i = x_i$) because their activation functions restrict the output range. While mathematically valid, this result relies on an assumption that the inputs are unbounded and not normalized. If the inputs are bounded, or if a fixed linear readout is added to counteract activation saturation, the expressive limitation may no longer hold. \n\n2. The design and motivation of the global virtual state $g$ are intuitive but lack precise statistical or identifiability analysis.\nThe paper only states that $g$ is ``an aggregation of recent affinity vectors in a buffer`` and claims that ``using recent vectors works well in practice``.\nHowever, the following points are not clarified:\n(1) What's the specific aggregation operator (mean, exponentially weighted average or  attention？);\n(2) There is no statistical or identifiability analysis showing under what conditions this aggregation can truly extract a global trend rather than just averaging noise. \n(3) Without (2), it is unclear whether it  models a global latent factor or merely performs temporal smoothing of the previous global affinity distribution.\nSince $g$ is central to the claim that heuristics $\\approx$ linear SSMs but still require global trend modeling, a deeper ablation across different aggregators and buffer lengths is necessary.\n\n3. Although the authors acknowledge that parameter size grows as $O(N^2)$ and propose truncation by retaining the top-$a$ affinities for each node, there is no quantitative evaluation of the trade-off between $a$, accuracy, and efficiency. In highly dynamic graphs, where new neighbors emerge or distributions shift, truncation based solely on previous affinity magnitudes may omit critical new relations. \n\n4. The main comparisons involve TGNNs and heuristic methods, showing consistent improvements, but no experiments include modern nonlinear, selective SSMs or DyG-Mamba[1,2] under the same protocol. Without such baselines, the claim that ``linear SSM-based TGNNs outperform more complex temporal models`` remains context-dependent and potentially overstated.\n\n5. The authors convert several link-prediction datasets into affinity-prediction datasets by temporal aggregation over daily, yearly, or term-based intervals. This transformation smooths the data and amplifies temporal continuity, which inherently favors linear smoothing models like NAVIS. In scenarios with bursty or non-stationary behaviors, such as cold-start or short-term spikes, the benefit may vanish. Evaluation on finer time resolutions or non-aggregated event streams would better test model robustness to abrupt dynamics.\n\n6. While Table7 reports ablations for three core components, the paper omits finer interpretability analyses: (1) distributions and temporal trajectories of the learned gates $z_h$ and $z_s$; (2) feature-attribution or permutation-importance tests for $g$; and (3) quantitative evidence that $g$ captures regime shifts earlier than node-local dynamics. These analyses would strengthen the claim that the virtual global state encodes shared temporal trends.\n\n7. The paper reports only NDCG@10 and trains on top-20 samples. Metrics such as Recall, MRR should be included for a more complete assessment of affinity prediction quality.\n\n[1] DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models. \n\n[2] DyG-Mamba: Continuous State Space Modeling on Dynamic Graphs"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360295929,"tcdate":1761968527979,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission3619/Reviewer_uBoh"],"signatures":["ICLR.cc/2026/Conference/Submission3619/Reviewer_uBoh"],"forum":"6UvkemEgK3","number":3,"license":"CC BY 4.0","cdate":1761968527979,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission3619/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360295929,"domain":"ICLR.cc/2026/Conference","replyto":"6UvkemEgK3","id":"94QLEUFUK9","forumContent":{"TLDR":{"value":"This paper revisits node affinity prediction on temporal graphs and explains why simple heuristics often beat modern TGNNs."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Node affinity","Temporal graph networks","dynamic graphs","graph neural netowrks"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Node affinity prediction is a common task that is widely used in temporal graph learning with applications in social and financial networks, recommender systems, and more. Recent works have addressed this task by  adapting state-of-the-art dynamic link property prediction models to node affinity prediction. However, simple heuristics, such as persistent forecast or moving average, outperform these models.\nIn this work, we analyze the challenges in training current Temporal Graph Neural Networks for node affinity prediction and suggest appropriate solutions. Combining the solutions, we develop NAVIS - Node Affinity prediction model using VIrtual State, by exploiting the equivalence between heuristics and state space models. While promising, training NAVIS is non-trivial. Therefore, we further introduce a novel loss function for node affinity prediction. We evaluate NAVIS  on TGB and show that it outperforms the state of the art, including heuristics."},"_bibtex":{"value":"@inproceedings{\nfeldman2026revisting,\ntitle={Revisting Node Affinity Prediction In Temporal Graphs},\nauthor={Or Feldman and Krishna Sri Ipsit Mantri and Moshe Eliasof and Chaim Baskin},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=6UvkemEgK3}\n}"},"title":{"value":"Revisting Node Affinity Prediction In Temporal Graphs"},"pdf":{"value":"/pdf/0257bfc5a4e5df3bebf07fd351c5bf6601828c16.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"feldman|revisting_node_affinity_prediction_in_temporal_graphs"},"authorids":{"value":["~Or_Feldman1","~Krishna_Sri_Ipsit_Mantri1","~Moshe_Eliasof1","~Chaim_Baskin1"]},"authors":{"value":["Or Feldman","Krishna Sri Ipsit Mantri","Moshe Eliasof","Chaim Baskin"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a statistical framework for anomaly detection in medical images using diffusion models, emphasizing reliable decision-making through selective inference. The authors propose a Diffusion Model-based Anomalous Region Detection test, which quantifies the statistical reliability of detected anomalies by computing selective p-values. This approach allows for controlled error rates, supporting clinical applications where rigorous reliability is essential. They implement selective inference with a piecewise-linear reconstruction error function, leveraging polyhedral geometry, and validate their method through numerical experiments and brain imaging tests. The work aims to bridge the gap between generative AI and statistical rigor in medical diagnostics."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- Could you clarify the reasoning behind describing the diffusion model as “detecting” anomalies?\n- Could you provide details on the computational time required for the DMAD-test on a standard dataset? \n- Could you clarify whether the supplementary code provided is fully executable, and, if so, whether it includes detailed instructions for setup and testing?\n- Have you considered testing on broader datasets, such as DeepLesion or other modalities?\n- Given the computational complexity and reliance on high-level statistical inference, do you foresee any specific challenges in deploying this method in real-time clinical settings?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"This approach leverages the unique combination of polyhedral and piecewise-linear functions in constructing a selective p-value, which allows the method to yield statistically sound inferences in an area where conventional anomaly detection techniques might lack formal reliability measures. The use of selective inference in the context of diffusion-based image generation is particularly novel, as the approach traditionally applies to feature selection in simpler models.\n\nThe authors provide a thorough theoretical foundation for their test. Their experiments demonstrate both the effectiveness and robustness of the method, detailing performance under various types of noise distributions and comparing it against other statistical tests like Bonferroni correction and permutation testing. Additionally, the stepwise construction and conditioning processes for selective inference are well-documented, highlighting the method’s robustness and adaptability in different testing scenarios.\n\nThe paper is well-structured, with a logical progression from the theoretical underpinnings to the experimental validation. The core concepts, including the their test, selective p-value, and the role of selective inference, are clearly defined. Figures illustrating the architecture and reconstruction process enhance reader comprehension. The piecewise-linear and polyhedral properties are described in a way that allows technically proficient readers to understand the model’s behavior and interpretability.\n\nThis work has some potential for impact. By providing statistical reliability measures via selective inference, the authors address a key limitation in deploying AI for clinical diagnostics: the need for transparent and error-controlled anomaly detection. The methodology’s ability to maintain controlled false-positive rates aligns with clinical requirements, supporting better trust and broader adoption of generative AI in medical applications. This is a meaningful contribution."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The statement \"the diffusion model detects anomalies\" (p.4) is somewhat misleading, as the detection is more accurately based on comparing input and reconstructed images to highlight discrepancies. This phrasing implies the model actively detects anomalies, while in practice, it passively reconstructs images, with anomalies inferred from reconstruction errors. A more accurate description could clarify the diffusion model's role and prevent misinterpretation of the model's capabilities.\n\nMany plots, notably Figures 2, 3, and 5, lack essential elements like axis labels and legends, limiting their interpretability. For complex analyses like these, well-labeled figures are critical for readers to follow and validate the findings. \n\nGiven the complexity of the statistical testing framework, a visual illustration of the DMAD-test would be beneficial. Many statistical papers can be challenging to interpret without sufficient diagrams, and a schematic representation of the test could improve comprehension, particularly for readers unfamiliar with selective inference. This could be placed early on to guide readers through the methodology.\n\nWhile computational resources are mentioned, there is no discussion of computational time, which is critical for assessing the method’s feasibility in practical applications. Adding specific timing metrics, ideally with a breakdown by test stage, would help readers gauge whether this approach could realistically be applied to clinical workflows.\n\nThe experiments focus on brain images with tumors, limiting the generalizability of the findings. Testing on a broader dataset, such as DeepLesion (https://nihcc.app.box.com/v/DeepLesion), or including non-tumor regions within volumes could provide a more comprehensive evaluation. Additionally, validating on datasets with a mix of normal and abnormal slices would assess the model's robustness and applicability to varied clinical scenarios.\n\nThe supplementary material mentions code, but it is unclear if this is executable code or pseudocode. Given the complexity of the statistical framework, accessible and runnable code would support reproducibility and encourage other researchers to build on this work. \n\nWhile the theoretical equations appear solid, some expressions, especially those in the SI framework, are intricate. Although no clear mistakes are present, a supplementary derivation or walkthrough of the core equations would support readers who may struggle with verifying them. This could be presented as an appendix to guide readers through critical steps and foster better understanding."}},"nonreaders":[],"tmdate":1732865482448,"tcdate":1730014451666,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9961/Reviewer_NqFB"],"signatures":["ICLR.cc/2025/Conference/Submission9961/Reviewer_NqFB"],"forum":"WrdLgVY5ZH","number":1,"license":"CC BY 4.0","cdate":1730014451666,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9961/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732865482448,"domain":"ICLR.cc/2025/Conference","replyto":"WrdLgVY5ZH","id":"i7Skq6a2Yz","forumContent":{"TLDR":{"value":"A new statistical test for anomaly detection using diffusion model is proposed based on selective inference framework."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["diffusion models","anomaly detection","statistical test","selective inference"]},"supplementary_material":{"value":"/attachment/e402240773ffde5dcf4d2e90a554e9300adc5b7c.zip"},"primary_area":{"value":"probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Advancements in AI image generation, particularly diffusion models, have progressed rapidly. However, the absence of an established framework for quantifying the reliability of AI-generated images hinders their use in critical decision-making tasks, such as medical image diagnosis. In this study, we address the task of detecting anomalous regions in medical images using diffusion models and propose a statistical method to quantify the reliability of the detected anomalies. The core concept of our method involves a selective inference framework, wherein statistical tests are conducted under the condition that the images are produced by a diffusion model. With our approach, the statistical significance of anomaly detection results can be quantified in the form of a $p$-value, enabling decision-making with controlled error rates, as is standard in medical practice. We demonstrate the theoretical soundness and practical effectiveness of our statistical test through numerical experiments on both synthetic and brain image datasets."},"_bibtex":{"value":"@misc{\nkatsuoka2025statistical,\ntitle={Statistical Test on Diffusion Model-based Anomaly Detection by Selective Inference},\nauthor={Teruyuki Katsuoka and Tomohiro Shiraishi and Daiki Miwa and Vo Nguyen Le Duy and Ichiro Takeuchi},\nyear={2025},\nurl={https://openreview.net/forum?id=WrdLgVY5ZH}\n}"},"title":{"value":"Statistical Test on Diffusion Model-based Anomaly Detection by Selective Inference"},"pdf":{"value":"/pdf/132c70472c4079a9dea7b8b8fb89b5d782b2be2a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"katsuoka|statistical_test_on_diffusion_modelbased_anomaly_detection_by_selective_inference"},"authorids":{"value":["~Teruyuki_Katsuoka1","~Tomohiro_Shiraishi1","~Daiki_Miwa1","~Vo_Nguyen_Le_Duy1","~Ichiro_Takeuchi1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Teruyuki Katsuoka","Tomohiro Shiraishi","Daiki Miwa","Vo Nguyen Le Duy","Ichiro Takeuchi"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a novel text-to-image model named LinFusion, addressing the challenge of generating high-resolution visual content with diffusion models. To optimize this, the authors propose to ultilize the popular linear attention and present methods for normalization-aware and non-causal operations, achieving performance on par with or even superior\nto the original diffusion model while significantly reducing time and memory complexity. Experiments demonstrate that LinFusion achieves comparable or superior performance to the original Stable Diffusion on tasks like zero-shot cross-resolution generation, with excellent results on MS COCO."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please referring to the Weaknesses above."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"- The paper presents an efficient text-to-image model, LinFusion, which innovatively addresses the computational inefficiencies inherent in high-resolution image generation with diffusion models.\n- Two notable innovations of LinFusion are normalization-aware mamba and non-causal mamba, which significantly improving the model's performance.\n- The authors have conducted an extensive set of experiments, demonstrating LinFusion's effectiveness across various resolutions and showcasing its superior capability in generating ultra-high-resolution images like 16K on a single GPU.\n- The writing is clear and methodical, effectively guiding readers through the complex technical details while maintaining a focus on the practical implications of the research.\n- The paper stands out for its thorough experimental validation, which not only benchmarks LinFusion against existing models but also integrates it with various components and pipelines, highlighting its versatility and compatibility in real-world applications."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- While LinFusion demonstrates significant improvements in computational efficiency, mamba2 is designed for language models. Could you give more comparison with state-of-the-art linear attention methods[1,2,3] in computer vision.\n\n- The results of the experiment are unconvincing. Could provide a more holistic assessment of LinFusion's performance across different aspects of image generation., such as HPSv2，T2I_Combench，DPG？\n\n- Could the linear attention combined with the MM-DiT blocks，which are popular in state-of-the-art diffusion models，SD3 and FLUX?\n\n- The training cost of this method is low. Is the training sufficient? Will longer training or increasing the training resolution (1k or 2k) further improve the model effect?\n\n[1] Cai, Han, et al. \"Efficientvit: Multi-scale linear attention for high-resolution dense prediction.\" arXiv preprint arXiv:2205.14756 (2022).\n\n[2] Zhu, Lianghui, et al. \"DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention.\" arXiv preprint arXiv:2405.18428 (2024).\n\n[3] Zhu, Lianghui, et al. \"Vision mamba: Efficient visual representation learning with bidirectional state space model.\" arXiv preprint arXiv:2401.09417 (2024)."}},"nonreaders":[],"tmdate":1731427443416,"tcdate":1729928627249,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1552/Reviewer_cB7B"],"signatures":["ICLR.cc/2025/Conference/Submission1552/Reviewer_cB7B"],"forum":"D2as3jDmRA","number":2,"license":"CC BY 4.0","cdate":1729928627249,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1552/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427443416,"domain":"ICLR.cc/2025/Conference","replyto":"D2as3jDmRA","id":"MV7V8JRgtr","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We propose a generalized linear attention mechanism, with which we use to replace the self-attention layers in existing diffusion models. The resultant model can generate ultrahigh resolution images such as 16K on a single GPU."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Linear Attention","Diffusion Models","Image Generation"]},"supplementary_material":{"value":"/attachment/811ef1e443954192e59ab90bc85f34658c822b9d.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Modern diffusion models, particularly those utilizing a Transformer-based UNet for denoising, rely heavily on self-attention operations to manage complex spatial relationships, thus achieving impressive generation performance. However, this existing paradigm faces significant challenges in generating high-resolution visual content due to its quadratic time and memory complexity with respect to the number of spatial tokens. To address this limitation, we aim at a novel linear attention mechanism as an alternative in this paper. \nSpecifically, we begin our exploration from recently introduced models with linear complexity, e.g., Mamba2, RWKV6, Gated Linear Attention, etc, and identify two key features—attention normalization and non-causal inference—that enhance high-resolution visual generation performance. Building on these insights, we introduce a generalized linear attention paradigm, which serves as a low-rank approximation of a wide spectrum of popular linear token mixers. To save the training cost and better leverage pre-trained models, we initialize our models and distill the knowledge from pre-trained StableDiffusion (SD). We find that the distilled model, termed LinFusion, achieves performance on par with or superior to the original SD after only modest training, while significantly reducing time and memory complexity. Extensive experiments on SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion enables satisfactory and efficient zero-shot cross-resolution generation, accommodating ultra-resolution images like 16K on a single GPU. Moreover, it is highly compatible with pre-trained SD components and pipelines, such as ControlNet, IP-Adapter, DemoFusion, DistriFusion, etc, requiring no adaptation efforts."},"_bibtex":{"value":"@misc{\nliu2025linfusion,\ntitle={LinFusion: 1 {GPU}, 1 Minute, 16K Image},\nauthor={Songhua Liu and Weihao Yu and Zhenxiong Tan and Xinchao Wang},\nyear={2025},\nurl={https://openreview.net/forum?id=D2as3jDmRA}\n}"},"title":{"value":"LinFusion: 1 GPU, 1 Minute, 16K Image"},"pdf":{"value":"/pdf/ce1c9b46b6efa0b82f1f80f589d63b4a863c6854.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"liu|linfusion_1_gpu_1_minute_16k_image"},"authorids":{"value":["~Songhua_Liu2","~Weihao_Yu2","~Zhenxiong_Tan1","~Xinchao_Wang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Songhua Liu","Weihao Yu","Zhenxiong Tan","Xinchao Wang"]}},"version":2},{"content":{"summary":{"value":"This paper investigates how Mamba perform in trajectory optimization in offline RL with ablation analysis on mamba's data input structures and architectural structures and shows Mamba DT can achieve SOTA performance with less parameters."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"please see weakness."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper writing is good, the visualizations look good.\n2. The input concatenation experiments provides useful practical insight also for other sequence-based decision-making models \n3. The paper provides a detailed analysis of how various components of Mamba, such as the hidden attention mechanism and different residual structures, influence performance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Finding 3 is not very surprising on the tested MDP environment, since they by definition should focus only on recent states. It will be interesting to explore how this mechanism might perform in environments with long-term dependencies where the Markov property does not hold strictly.\n2. Only tested on standard Atari and mujoco tasks. How would mamba perform on tasks that requires long horizon planning skills? such as maze navigation or tasks with delayed rewards."},"limitations":{"value":"please see weakness"}},"nonreaders":[],"tmdate":1730879217015,"tcdate":1721113078563,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission8092/Reviewer_qC8w"],"signatures":["NeurIPS.cc/2024/Conference/Submission8092/Reviewer_qC8w"],"forum":"yWSxjlFsmX","number":2,"license":"CC BY 4.0","cdate":1721113078563,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission8092/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879217015,"domain":"NeurIPS.cc/2024/Conference","replyto":"yWSxjlFsmX","id":"OLd6YCKqkx","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"TLDR":{"value":"An investigation of Mamba in offline RL."},"keywords":{"value":["Offline RL; Trajectory Optimization; Mamba"]},"primary_area":{"value":"reinforcement_learning"},"abstract":{"value":"Transformer-based trajectory optimization methods have demonstrated exceptional performance in offline Reinforcement Learning (offline RL). Yet, it poses challenges due to substantial parameter size and limited scalability, which is particularly critical in sequential decision-making scenarios where resources are constrained such as in robots and drones with limited computational power. Mamba, a promising new linear-time sequence model, offers performance on par with transformers while delivering substantially fewer parameters on long sequences. As it remains unclear whether Mamba is compatible with trajectory optimization, this work aims to conduct comprehensive experiments to explore the potential of Decision Mamba (dubbed DeMa) in offline RL from the aspect of data structures and essential components with the following insights: (1) Long sequences impose a significant computational burden without contributing to performance improvements since DeMa's focus on sequences diminishes approximately exponentially. Consequently, we introduce a Transformer-like DeMa as opposed to an RNN-like DeMa. (2) For the components of DeMa, we identify the hidden attention mechanism as a critical factor in its success, which can also work well with other residual structures and does not require position embedding. Extensive evaluations demonstrate that our specially designed DeMa is compatible with trajectory optimization and surpasses previous methods, outperforming Decision Transformer (DT) with higher performance while using 30\\% fewer parameters in Atari, and exceeding DT with only a quarter of the parameters in MuJoCo."},"_bibtex":{"value":"@inproceedings{\ndai2024is,\ntitle={Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?},\nauthor={Yang Dai and Oubo Ma and Longfei Zhang and Xingxing Liang and Shengchao Hu and Mengzhu Wang and Shouling Ji and Jincai Huang and Li Shen},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=yWSxjlFsmX}\n}"},"title":{"value":"Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?"},"pdf":{"value":"/pdf/e8f05bc8b78365623dc8f45e047f65b46390a923.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"dai|is_mamba_compatible_with_trajectory_optimization_in_offline_reinforcement_learning"},"authorids":{"value":["~Yang_Dai2","~Oubo_Ma1","~Longfei_Zhang3","~Xingxing_Liang1","~Shengchao_Hu1","~Mengzhu_Wang1","~Shouling_Ji1","~Jincai_Huang1","~Li_Shen1"]},"authors":{"value":["Yang Dai","Oubo Ma","Longfei Zhang","Xingxing Liang","Shengchao Hu","Mengzhu Wang","Shouling Ji","Jincai Huang","Li Shen"]}},"version":2},{"content":{"summary":{"value":"This paper provides an efficient method for time series imputation problems with a self-supervised training manner. The authors implemented the state-space model based Mamba architecture as the backbone for diffusion models. The authors validate their model on different real-world datasets."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. You have mentioned your model is a probabilistic model but the analysis of probabilistic imputation is not clear. Can the authors provide more details about why your model is capable of achieving probabilistic impuation and the implementation of the probabilistic time series imputation?\n\n2. Can the author provide more details about the choice of mask ratios among the datasets and how the inference time is calculated among different methods?\n\n3. The model is a conditional dpm. Can the authors provide more details about the conditions and how the conditions are used in the model?\n\n4. A straightforward conclusion is that different distance has different impacts on one timestamp. Can the authors provide more details about the inter-channel dependencies and how the model addresses the distance issue?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. This paper proposes an effective method for the time series data and the architecture details (bidirectional modeling and channel dependency) is well-suited for the time series data.\n2. For probablistic time series imputation, the proposed method has a significant improvement in the CRPS-sum metric."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The implementation details of probabilistic time series imputation is not clearly explained in the paper. \n\n2. The diffimp seems like a condition diffusion model but the conditions are not clearly explained in the paper. \n\n3. While modeling inter-channel dependencies, it is clear that the dependency is related to the distance of the timestep, is diffimp capable of addressing the distance issue?"}},"nonreaders":[],"tmdate":1732458058649,"tcdate":1730156860914,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2397/Reviewer_ZcoV"],"signatures":["ICLR.cc/2025/Conference/Submission2397/Reviewer_ZcoV"],"forum":"j1OucVFZMJ","number":1,"license":"CC BY 4.0","cdate":1730156860914,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2397/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732458058649,"domain":"ICLR.cc/2025/Conference","replyto":"j1OucVFZMJ","id":"jKZ39S3dn6","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["diffusion","time series impuation","state space model"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Probabilistic time series imputation has been widely applied in real-world scenarios due to its ability to estimate uncertainty of imputation results. Meanwhile, denoising diffusion probabilistic models (DDPMs) have achieved great success in probabilistic time series imputation tasks with its power to model complex distributions. However, current DDPM-based probabilistic time series imputation methodologies are confronted with two types of challenges: 1) \\textit{ The backbone modules of the denoising parts are not capable of achieving sequence modeling with low time complexity.} 2) \\textit{ The architecture of denoising modules can not handle the inter-variable and bidirectional dependencies in the time series imputation problem effectively.} To address the first challenge, we integrate the computational efficient state space model, namely Mamba, as the backbone denosing module for DDPMs. To tackle the second challenge, we carefully devise several SSM-based blocks for bidirectional modeling and inter-variable relation understanding. Experimental results demonstrate that our approach can achieve state-of-the-art time series imputation results on multiple datasets, different missing scenarios and missing ratios."},"_bibtex":{"value":"@misc{\ngao2025diffimp,\ntitle={DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone},\nauthor={Hongfan Gao and Wangmeng Shen and Xiangfei Qiu and Ronghui Xu and Jilin Hu and Bin Yang},\nyear={2025},\nurl={https://openreview.net/forum?id=j1OucVFZMJ}\n}"},"title":{"value":"DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone"},"pdf":{"value":"/pdf/26faf0d7208528fa5a326d36fadf37ed808e0f38.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"gao|diffimp_efficient_diffusion_model_for_probabilistic_time_series_imputation_with_bidirectional_mamba_backbone"},"authorids":{"value":["~Hongfan_Gao1","~Wangmeng_Shen1","~Xiangfei_Qiu1","~Ronghui_Xu2","~Jilin_Hu1","~Bin_Yang4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hongfan Gao","Wangmeng Shen","Xiangfei Qiu","Ronghui Xu","Jilin Hu","Bin Yang"]}},"version":2},{"content":{"summary":{"value":"The paper proposes COMBA (Cross-Batch Aggregation), a framework extending [1] that aims to scale state-space models (SSMs) for large-scale graph learning. The central idea is to partition a large graph into smaller subgraphs (batches) and introduce a cross-batch aggregation mechanism that enables information exchange among batches with shared nodes. Similar to [1], COMBA applies SSM layers over the sequence of multi-hop node representations, treating the hop dimension as the sequential axis.\n\n\n\n\n[1] Behrouz, Ali, and Farnoosh Hashemi. \"Graph mamba: Towards learning on graphs with state space models.\" Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining. 2024."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See Weaknesses."},"rating":{"value":0},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":1},"strengths":{"value":"- scaling state-space models to large-scale graphs is an important questions to study\n- the proposed method shows competitive performance to previous graph state space models"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- the major weakness of this paper is its presentation. The notation is highly ambiguous, and many key implementation details are missing. I strongly encourage the authors to substantially revise the notation and exposition, as the current version is extremely difficult to follow. For example, \n    - in equation (1), $A$ is the adjancency matrix and $\\text{Gen}(G, A)=A^k$ is used to represent the binary matrices that indicate whether two nodes are connected by a path of exactly length $k$. It is fairly uncommon to denote this by $A^k$, which usally means the powers of a matrix.\n    - Line 203-204: it defines $A_{b_m}=\\\\{A\\^1\\_\\{b\\_m\\}, ..., A\\^\\{k\\}\\_\\{b\\_m\\}\\\\}$, but then it writes $A\\^k\\_\\{b\\_m\\}=\\text{Gen}(G_\\{b\\_m\\}, A_\\{b\\_m\\}^k)$. It does not make sense that $A^k_{b_m}$ appears on both ends. This definition is also contradictive to previous definition $\\text{Gen}(G, A)=A^k$. \n    - Line 217 uses $\\mathcal{Z}\\_\\{n, k\\}$ to denotes representations of node $n$ at $k$-th hop. But later at equation (9) a different notation $Z\\_\\{b_\\{m\\}, k\\}(n)$ is used. \n    - It is unclear how Section 4.1 and Section 4.2 are related. Section 4.2 begins with  $Z^{(0)}$ initialized from the original node features and then performs a cross-batch computation, which appears conceptually disconnected from the procedure described in Section 4.1.\n    - $\\hat{m}$ in equation (9) is never defined.\n    - crucially, how to choose the set of batches of nodes chosen is not mentioned in the main text. \n- Some of relevant literature (state space models for graphs) is absent, e.g., [2-5]\n\n\n[2] Pan, Zhenyu, et al. \"Hetegraph-mamba: Heterogeneous graph learning via selective state space model.\" arXiv preprint arXiv:2405.13915 (2024).\n\n[3] Chen, Dexiong, Till Hendrik Schulz, and Karsten Borgwardt. \"Learning long range dependencies on graphs via random walks.\" arXiv preprint arXiv:2406.03386 (2024).\n\n[4] Huang, Yinan, Siqi Miao, and Pan Li. \"What Can We Learn from State Space Models for Machine Learning on Graphs?.\" arXiv preprint arXiv:2406.05815 (2024).\n\n[5] Zhao, Gongpei, et al. \"Grassnet: State space model meets graph neural network.\" arXiv preprint arXiv:2408.08583 (2024)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762942227165,"tcdate":1761943506093,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22461/Reviewer_2K7R"],"signatures":["ICLR.cc/2026/Conference/Submission22461/Reviewer_2K7R"],"forum":"E4Pm2M5m09","number":4,"license":"CC BY 4.0","cdate":1761943506093,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22461/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762942227165,"domain":"ICLR.cc/2026/Conference","replyto":"E4Pm2M5m09","id":"3RYPM9sHvz","forumContent":{"TLDR":{"value":"In this paper, we propose COMBA to tackle large graph learning using state space models, with graph context gating and cross batch aggregation."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Graph neural networks","State space models","Batch learning","Homogeneous graphs","Classification"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"State space models (SSMs) have recently emerged for modeling long-range dependency in sequence data, with much simplified computational costs than modern alternatives, such as transformers. Advancing SMMs to graph structured data, especially for large graphs, is a significant challenge because SSMs are sequence models and the shear graph volumes make it very expensive to convert graphs as sequences for effective learning. In this paper, we propose COMBA to tackle large graph learning using state space models, with two key innovations: graph context gating and cross batch aggregation. Graph context refers to different hops of neighborhood for each node, and graph context gating allows COMBA to use such context to learn best control of neighbor aggregation. For each graph context, COMBA samples nodes as batches, and train a graph neural network (GNN), with information being aggregated cross batches, allowing COMBA to scale to large graphs. Our theoretical study asserts that cross-batch aggregation guarantees lower error than training GNN without aggregation. Experiments on benchmark networks demonstrate significant performance gains compared to baseline approaches. Code and benchmark datasets will be released for public access."},"_bibtex":{"value":"@misc{\nshen2025comba,\ntitle={{COMBA}: Cross Batch Aggregation for Learning Large Graphs with Context Gating State Space Models},\nauthor={Jiajun Shen and Yufei Jin and Yi He and Xingquan Zhu},\nyear={2025},\nurl={https://openreview.net/forum?id=E4Pm2M5m09}\n}"},"title":{"value":"COMBA: Cross Batch Aggregation for Learning Large Graphs with Context Gating State Space Models"},"pdf":{"value":"/pdf/58113009b83859a859d68373caa03caa11a889f6.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"shen|comba_cross_batch_aggregation_for_learning_large_graphs_with_context_gating_state_space_models"},"authorids":{"value":["~Jiajun_Shen6","~Yufei_Jin1","~Yi_He4","~Xingquan_Zhu1"]},"authors":{"value":["Jiajun Shen","Yufei Jin","Yi He","Xingquan Zhu"]}},"version":2},{"content":{"venue":{"value":"Crossref"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-032-04549-2_3.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"li|swift_statespace_wavelet_integrated_forecasting_technology_for_enhanced_time_series_prediction"},"html":{"value":"https://doi.org/10.1007/978-3-032-04549-2_3"},"abstract":{"value":"Time series forecasting remains a fundamental challenge in data science. We introduce SWIFT, a novel neural architecture that synergistically combines selective state space models (Mamba) with multi-scale dilated convolutions for enhanced time series forecasting. Our approach incorporates: (1) a Selective Temporal State Space module extending Mamba with time series-specific gating; (2) a Multi-Scale Dilated Convolutional Network with adaptive receptive fields; and (3) a Feature Interaction Bridge facilitating cross-pathway information exchange. Experiments on six benchmark datasets demonstrate SWIFT outperforms other common methods, achieving 6.5% average improvement in MSE and 5.8% in MAE across various prediction horizons."},"title":{"value":"SWIFT: State-Space Wavelet Integrated Forecasting Technology for Enhanced Time Series Prediction"},"authors":{"value":[{"fullname":"Wei Li","username":""}]}},"tmdate":1789099433308,"pdate":1767225600000,"externalIds":["doi:10.1007/978-3-032-04549-2_3"],"tcdate":1769341802233,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Wei_Li133"],"forum":"qu3OqL1D6j","license":"CC BY-SA 4.0","number":38136,"cdate":1757435548822,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/-/Edit","OpenReview.net/Public_Article/-/Author_Removal"],"mdate":1789099433308,"domain":"OpenReview.net/Public_Article","id":"qu3OqL1D6j","version":2},{"content":{"venue":{"value":"TIME 2024"},"pdf":{"value":"https://drops.dagstuhl.de/storage/00lipics/lipics-vol318-time2024/LIPIcs.TIME.2024.16/LIPIcs.TIME.2024.16.pdf"},"venueid":{"value":"dblp.org/conf/TIME/2024"},"paperhash":{"value":"alsmann|realtime_higherorder_recursion_schemes"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Eric_Alsmann:","~Florian_Bruse1"]},"html":{"value":"https://doi.org/10.4230/LIPIcs.TIME.2024.16"},"_bibtex":{"value":"@inproceedings{DBLP:conf/time/AlsmannB24,\n  author={Eric Alsmann and Florian Bruse},\n  title={Real-Time Higher-Order Recursion Schemes},\n  year={2024},\n  cdate={1704067200000},\n  pages={16:1-16:20},\n  url={https://doi.org/10.4230/LIPIcs.TIME.2024.16},\n  booktitle={TIME},\n  crossref={conf/time/2024}\n}\n"},"abstract":{"value":"Higher-Order Recursion Schemes (HORS) have long been studied as a tool to model functional programs. Model-checking the tree generated by a HORS of order k against a parity automaton is known to be k-EXPTIME-complete. This paper introduces timed HORS, a real-time version of HORS in the sense of Alur/Dill'90, to be model-checked against a pair of a parity automaton and a timed automaton. We show that adding dense linear time to the notion of recursion schemes adds one exponential to the cost of model-checking, i.e., model-checking a timed HORS of order k can be done in (k+1)-EXPTIME. This is shown by an adaption of the region-graph construction known from the model-checking of timed CTL. We also obtain a hardness result for k = 1, but we strongly conjecture that it holds for all k. This result is obtained by encoding runs of 2-EXPTIME Turing machines into the trees generated by timed HORS."},"title":{"value":"Real-Time Higher-Order Recursion Schemes"},"authors":{"value":["Eric Alsmann","Florian Bruse"]}},"tmdate":1748085533474,"pdate":1704067200000,"tcdate":1748085511030,"writers":["~"],"signatures":["~Florian_Bruse1"],"forum":"JtjDELD2kb","license":"CC BY-SA 4.0","number":550599,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1748085533474,"domain":"DBLP.org","id":"JtjDELD2kb","version":2},{"content":{"summary":{"value":"This paper explores the capabilities of Mamba state-space models (SSMs) in comparison to Transformer large language models (LLMs) in various downstream learning tasks. Despite Mamba's success in some areas, the paper identifies challenges and limitations in achieving performance parity with Transformers on standard benchmarks, particularly in in-context learning (ICL), mixed-precision fine-tuning (MPFT), and parameter-efficient fine-tuning (PEFT). The study demonstrates that while Mamba models have robust recurrent dynamics and can achieve significant speed and memory efficiency gains through fine-tuning techniques, their downstream learning improvements still lag behind those of Transformers."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see weakness."},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. This paper is well-written and easy to read.\n\n2. The study shows that Mamba’s recurrent dynamics are robust to small input changes, which is validated both theoretically and empirically. This robustness ensures stability in training and fine-tuning processes.\n\n3. Despite initial shortcomings in ICL performance, extensive experiments demonstrate that mamba models exhibit strong potential for improvement through efficient fine-tuning. The models can achieve up to 81.5% of the ICL performance improvement, highlighting their adaptability with appropriate tuning methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I appreciate the authors for providing a theoretical analysis to demonstrate the controllability of implementing AMP on Mamba blocks, and the experiments indicate that PEFT is also suitable for Mamba. However, I have several concerns:\n\n1. The authors define the Mamba process as a generalized operation: $x_t=F_{\\theta}(x_{t-1},u_t)$, but the actual output of Mamba is $y_t = \\bar{C_t}x_t$.Therefore, the theoretical analysis provided in the paper pertains to the stability of the hidden state under small perturbations. Is it possible to extend this analysis directly to the output $y_t$? Since the stability of the hidden state does not necessarily imply the stability of the output.\n2. Theorem 1 ensures the feasibility of implementing LoRA on Mamba blocks but focuses on the $W$ matrix, neglecting the consideration of the most crucial transition matrix $\\bar{A_t}$ in Mamba. Does this mean that the $\\bar{A_t}$ matrix was not subjected to LoRA during fine-tuning? If so, is it possible to consider applying PEFT to the $\\bar{A_t}$ matrix as well?\n3. As an empirically-driven paper, would it be possible to include more backbones for comparison in future versions? Currently, the only baseline for comparison is Pythia."},"limitations":{"value":"The authors raise some limitaions, for example, the lower-precision method is not explored in this paper, but the authors claime to solve them in the future."}},"nonreaders":[],"tmdate":1730879575625,"tcdate":1720510627472,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission12737/Reviewer_Uruo"],"signatures":["NeurIPS.cc/2024/Conference/Submission12737/Reviewer_Uruo"],"forum":"C3t6GMPnC5","number":1,"license":"CC BY 4.0","cdate":1720510627472,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission12737/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879575625,"domain":"NeurIPS.cc/2024/Conference","replyto":"C3t6GMPnC5","id":"upDzvF0y5e","forumContent":{"TLDR":{"value":"Mamba has stable recurrent dynamics for efficient fine-tuning, which allows better in-context learning."},"venue":{"value":"Submitted to NeurIPS 2024"},"keywords":{"value":["Mamba","state-space models","in-context learning","large language models","LLMs","SSMs"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Mamba state-space models (SSMs) have recently outperformed state-of-the-art (SOTA) Transformer large language models (LLMs) in various tasks and been widely adapted. However, Mamba’s downstream learning capabilities remain either unexplored–e.g., mixed-precision (MPFT) and parameter-efficient fine-tuning (PEFT)–or under-evaluated–e.g., in-context learning (ICL). For the latter, recent works reported Mamba’s ICL rivals SOTA Transformer LLMs using non-standard benchmarks. In contrast, we show that on standard benchmarks, pretrained Mamba models achieve only 38% of the ICL performance improvements (over zero-shot) of comparable Transformers.\n\nEnabling MPFT and PEFT in Mamba architectures is challenging due to recurrent dynamics and highly customized CUDA kernels, respectively. However, we prove that Mamba’s recurrent dynamics are robust to small input changes using dynamical systems theory. Empirically, we show that performance changes in Mamba’s inference and fine-tuning due to mixed-precision align with Transformer LLMs. Furthermore, we prove that targeting key memory buffers in Mamba’s customized CUDA kernels for low-rank adaptation regularizes SSM parameters, thus achieving parameter efficiency while retaining speedups. We show that combining MPFT and PEFT enables up to 2.15 times more tokens-per-second and 65.5% reduced memory-per-token compared to full Mamba fine-tuning, while achieving up to 81.5% of the ICL performance improvements (over zero-shot) of comparably fine-tuned Transformers."},"_bibtex":{"value":"@misc{\nanonymous2024mamba,\ntitle={Mamba State-Space Models Can Be Strong Downstream Learners},\nauthor={Anonymous},\nyear={2024},\nurl={https://openreview.net/forum?id=C3t6GMPnC5}\n}"},"title":{"value":"Mamba State-Space Models Can Be Strong Downstream Learners"},"pdf":{"value":"/pdf/f7d8e4232cef7c095a61b273456dcfa13215eb1e.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"halloran|mamba_statespace_models_can_be_strong_downstream_learners"},"authorids":{"value":["~John_Timothy_Halloran1","~Manbir_S_Gulati1","~Paul_F_Roysdon1"]},"authors":{"value":["John Timothy Halloran","Manbir S Gulati","Paul F Roysdon"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2503.13004v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"liu|tfdm_timevariant_frequencybased_point_cloud_diffusion_with_mamba"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jiaxu_Liu_0002:","~Li_Li16","https://dblp.org/search/pid/api?q=author:Hubert_P._H._Shum:","~Toby_P._Breckon1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2503.13004"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-13004,\n  publtype={informal},\n  author={Jiaxu Liu and Li Li and Hubert P. H. Shum and Toby P. Breckon},\n  title={TFDM: Time-Variant Frequency-Based Point Cloud Diffusion with Mamba},\n  year={2025},\n  month={March},\n  cdate={1740787200000},\n  journal={CoRR},\n  volume={abs/2503.13004},\n  url={https://doi.org/10.48550/arXiv.2503.13004}\n}\n"},"abstract":{"value":"Diffusion models currently demonstrate impressive performance over various generative tasks. Recent work on image diffusion highlights the strong capabilities of Mamba (state space models) due to its efficient handling of long-range dependencies and sequential data modeling. Unfortunately, joint consideration of state space models with 3D point cloud generation remains limited. To harness the powerful capabilities of the Mamba model for 3D point cloud generation, we propose a novel diffusion framework containing dual latent Mamba block (DM-Block) and a time-variant frequency encoder (TF-Encoder). The DM-Block apply a space-filling curve to reorder points into sequences suitable for Mamba state-space modeling, while operating in a latent space to mitigate the computational overhead that arises from direct 3D data processing. Meanwhile, the TF-Encoder takes advantage of the ability of the diffusion model to refine fine details in later recovery stages by prioritizing key points within the U-Net architecture. This frequency-based mechanism ensures enhanced detail quality in the final stages of generation. Experimental results on the ShapeNet-v2 dataset demonstrate that our method achieves state-of-the-art performance (ShapeNet-v2: 0.14\\% on 1-NNA-Abs50 EMD and 57.90\\% on COV EMD) on certain metrics for specific categories while reducing computational parameters and inference time by up to 10$\\times$ and 9$\\times$, respectively. Source code is available in Supplementary Materials and will be released upon accpetance."},"title":{"value":"TFDM: Time-Variant Frequency-Based Point Cloud Diffusion with Mamba"},"authors":{"value":["Jiaxu Liu","Li Li","Hubert P. H. Shum","Toby P. Breckon"]}},"tmdate":1761390612141,"pdate":1735689600000,"tcdate":1751990946634,"writers":["~"],"signatures":["~Li_Li16"],"forum":"1vwgt9Vmuu","license":"CC BY-SA 4.0","number":566381,"cdate":1740787200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1761390612141,"domain":"DBLP.org","id":"1vwgt9Vmuu","version":2},{"content":{"summary":{"value":"This paper introduces a model-based algorithm for episodic MDP whose transition probability is embedded in a given reproducing kernel Hilbert space. The assumption of RKHS-embedding of transition probability can handle non-linear relationships in transition probability. The proposed algorithm (RKHS-RL) estimates the transition core using ridge regression based on the collected data, and constructs an optimistic action-value function based on the infinite dimensional confidence ball. The proposed method achieves a dimension independent regret bound of $O(H \\sqrt{T})$ where $H$ is the horizon length, $T$ is the total number of interaction between the agent and the environment. Furthermore, the authors confirm the performance of finite-dimensional RKHS-RL through experiments in a simple tabular setting."},"soundness":{"value":"3 good"},"confidence":{"value":"3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked."},"questions":{"value":"1. How can the proposed method be applied to a continuous state-action space? The current planning approach seems challenging to implement in a continuous state-action space.\n\n2. How was it possible to achieve a tight regret bound with respect to H when compared to the regret bound of Yang & Wang (2020)?\n\n3. The regret bound of RKHS-RL does not include the effective dimension of the kernel. In that case, does the regret not depend on dimension even when using a finite-dimensional kernel? What is the key technique to eliminate dimension dependence when compared to KernelMatrixRL in Yang & Wang (2020) ?\n\n4. In the finite-dimensional case of RKHS-RL, how is the regularity bound in Assumption 2 defined? Can we assume a bound independent of the size of the state-action space in finite-dimensional case?\n\n5. Why does Figure 1-(b) show negative values for regret?\n\n6. How does the following inequality in the Proof of Lemma 2 hold? \n: $$ H || \\Phi\\_{n,h} \\tilde{\\circ} (M^* - M'\\_n) ||_1 \\le H || \\Phi\\_{n,h} \\tilde{\\circ} (M^* - M'\\_n) ||_2 $$"},"rating":{"value":"6: marginally above the acceptance threshold"},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":"3 good"},"contribution":{"value":"3 good"},"strengths":{"value":"- The motivation for the problem addressed in this paper is well-explained, drawing from the literature and related work. Additionally, the organization of the paper is appropriately structured to facilitate understanding.\n\n- The RKHS-embedding of transition probability is useful in modeling non-linear transition probabilities for state-action pairs. The author has demonstrated that the regret bound of the proposed method can achieve dimension-free sub-linear regret."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The computational complexity of the proposed method has not been addressed. In “Introduction”, the authors mentioned that the proposed model can handle infinite state and action spaces. However, it seems that the computation in Algorithms 1 and 2 is heavily influenced by the size of the state-action space. It would be helpful to specify how efficient the proposed method is both statistically and computationally compared to previous algorithms, particularly for infinite state spaces.\n\n- The proposed algorithm appears similar to KernelMatrixRL (Yang & Wang, 2020). While the authors mentioned that the RKHS-embedding setting poses significant challenges compared to approach of Yang & Wang (2020), it would be beneficial to explain in detail what specific challenges arise and how they were addressed using mathematical techniques in the paper.\n\n- There is a gap between the settings discussed in the paper and the simulations. The proposed method can handle settings that previous approaches, such as tabular MDPs, linear MDPs, and parametric MDPs, cannot. However the experiments were conducted in a simple tabular setting. Additionally, it is anticipated that Q-learning or SARSA would perform significantly better in the current experiments. It would be valuable to include comparisons with other baseline algorithms."}},"nonreaders":[],"tmdate":1699636886241,"tcdate":1698751262866,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission7398/Reviewer_oU2p"],"signatures":["ICLR.cc/2024/Conference/Submission7398/Reviewer_oU2p"],"forum":"U0c2IaQhHk","number":2,"license":"CC BY 4.0","cdate":1698751262866,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission7398/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636886241,"domain":"ICLR.cc/2024/Conference","replyto":"U0c2IaQhHk","id":"do4xoQXTUe","forumContent":{"TLDR":{"value":"We propose a novel approach that leverages reproducing kernel Hilbert spaces (RKHSs) to tackle the curse of dimensionality problem on online reinforcement learning with continuous state and action spaces."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["online reinforcement learning","reproducing kernel Hilbert space","embedding learning"]},"supplementary_material":{"value":"/attachment/174cbee6b952a4fd69e0215a9f6b5252cf6668db.zip"},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Reinforcement Learning (RL) is a powerful tool for solving complex decision-making problems. However, existing RL approaches suffer from the curse of dimensionality when dealing with large or continuous state and action spaces. This paper introduces a non-parametric online RL algorithm called RKHS-RL that overcomes these challenges by utilizing reproducing kernels and the RKHS-embedding assumption. The proposed algorithm can handle both finite and infinite state and action spaces, as well as nonlinear relationships in transition probabilities. The RKHS-RL algorithm estimates the transition core using ridge regression and balances exploration and exploitation through infinite-dimensional confidence balls. The paper provides theoretical guarantees, demonstrating that RKHS-RL achieves a sublinear regret bound of $\\tilde{\\mathcal{O}}(H\\sqrt{T})$, where $T$ denotes the time step of the algorithm and $H$ represents the horizon of the Markov Decision Process (MDP), making it an effective approach for RL problems."},"_bibtex":{"value":"@misc{\nlin2024exploring,\ntitle={Exploring the State and Action Space in Reinforcement Learning with Infinite-Dimensional Confidence Balls},\nauthor={Yucong Lin and Yicheng Teng and Jingda Wu and Junwei Lu},\nyear={2024},\nurl={https://openreview.net/forum?id=U0c2IaQhHk}\n}"},"title":{"value":"Exploring the State and Action Space in Reinforcement Learning with Infinite-Dimensional Confidence Balls"},"pdf":{"value":"/pdf/1487a0c240655d4ee5a50dbf77ab9b30fde3b879.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"lin|exploring_the_state_and_action_space_in_reinforcement_learning_with_infinitedimensional_confidence_balls"},"authorids":{"value":["~Yucong_Lin2","~Yicheng_Teng1","~Jingda_Wu1","~Junwei_Lu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yucong Lin","Yicheng Teng","Jingda Wu","Junwei Lu"]}},"version":2},{"content":{"TLDR":{"value":"Correspondences Learning with Selective State Spaces"},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Correspondence learning","image matching","feature matching"]},"supplementary_material":{"value":"/attachment/2d09b8e2610b05f0cb43cedcc37982b32e359fa9.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Two-view correspondence learning aims to discern true and false correspondences between image pairs by recognizing their underlying different information. Previous methods either treat the information equally or fail to discard the superfluous information of false correspondences, tending to be invalid in practical scenarios. Therefore, inspired by Mamba's inherent competence of selectivity, we propose MambaMatch as a Mamba-based correspondence filter to selectively mine information from true correspondences and to dispose of the potentially interfering information of false correspondences. Specifically, the selection is achieved by adaptively adjusting model parameters in a high-dimensional latent space, which also avoids attention leakage and implements context compression, ensuring the precise and efficient exploitation of pertinent information. Meanwhile, channel awareness is tailored to serve as a complementary aspect of comprehensive information acquisition. Moreover, we design a novel local-context enhancement module to capture reasonable local context that is crucial for correspondence pruning. Extensive experiments demonstrate that our approach outperforms existing state-of-the-art methods on several visual tasks while saving time and space costs."},"_bibtex":{"value":"@misc{\nfang2024mambamatch,\ntitle={MambaMatch: Learning Two-View Correspondences with Selective State Spaces},\nauthor={Xiang Fang and Shihua Zhang and Jiayi Ma},\nyear={2024},\nurl={https://openreview.net/forum?id=YuJdtpPV4n}\n}"},"title":{"value":"MambaMatch: Learning Two-View Correspondences with Selective State Spaces"},"pdf":{"value":"/pdf/63484318bfc95ae154ff8a779ed5675dcfd84a4f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"fang|mambamatch_learning_twoview_correspondences_with_selective_state_spaces"},"authorids":{"value":["~Xiang_Fang2","~Shihua_Zhang2","~Jiayi_Ma2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xiang Fang","Shihua Zhang","Jiayi Ma"]}},"tmdate":1731588122596,"tcdate":1727355680771,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6894/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission6894/Authors"],"forum":"YuJdtpPV4n","license":"CC BY 4.0","number":6894,"cdate":1727355680771,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission6894/-/Full_Submission","ICLR.cc/2025/Conference/-/Withdrawn_Submission"],"mdate":1731588122596,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"YuJdtpPV4n","version":2},{"content":{"summary":{"value":"The paper introduces a novel model (GateLoop) for long-sequence modeling. The model employs a state space model (SSM) with data-controlled state transitions to incorporate the inputs at multiple steps, hence overcoming known issues of standard linear recurrent models in language modeling (such as associative recall). To increase efficiency, they propose a parallelized version of the proposed method using associative scan computation, which improved complexity to $O(l \\log l)$, where $l$ is the sequence length. The model is parametrized according to recent developments in signal propagation for linear recurrent units, i.e. by parametrizing the state transitions in polar form. Experimental evidence of the effectiveness of the method is provided on WikiText103."},"presentation":{"value":"2 fair"},"contribution":{"value":"1 poor"},"soundness":{"value":"2 fair"},"strengths":{"value":"1. Relevance. The paper addresses the recently growing interest in finding alternatives to Transformers for language modeling, which has been shown to underperform in tasks involving very long sequences. Hence, the proposed method can be seen as a step toward improving and generalizing this class of models. \n2. Preliminary experiments are promising, outperforming existing baselines. \n3. The exposition is overall clear, and the paper is well-written, with a clear story and motivation."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. One of my main concerns is the experimental evidence provided and its methodology.  \n\n    a. **Evidence**: In the abstract, it is stated that \"GateLoop empirically outperforms existing models for auto-regressive language modeling\", but the evidence is provided on only one task. I believe that for a paper with no theory that claims practical advantages, empirical verification should be exhaustive. \n\n    b. **Methodology**. The baselines are reported from an existing paper, and it is unclear if the comparison is fair. For instance, are the models trained with the same number of steps? Were the hyperparameters tuned consistently with the reported baselines?\n2. **Motivation**. In its current form, the paper does not provide enough justification for the proposed method. Why should it perform better than H3 and its variants? The differences to H3 should be made clearer. H3 also has multiplicative interaction to effectively compare the representations of different tokens in the sequence. The authors only say that \"[H3] employs SSMs in conjunction with data-controlled element-wise gating\". It would be helpful if the differences were made more explicit. \nMinor:\n3. The model's description in section 3 should be made clearer. In particular, it is my understanding that the $k_n$ and $v_n$ values depend both on the input and weight matrices. That should be made more explicit. This can be also done for instance by outlining the algorithm/pseudo code of the forward pass of the proposed layer. \n4. The quantity $l$ is used without definition in the abstract.\n5. works of Gul et al --> work of. \"existing only utilize a special case\". \"An Recurrent Neural Network\". Missing reference: \"we instead resort to an associative scan outlined in section?\""},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"See weaknesses."},"rating":{"value":"3: reject, not good enough"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636802801,"tcdate":1698400745618,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6897/Reviewer_kCvT"],"signatures":["ICLR.cc/2024/Conference/Submission6897/Reviewer_kCvT"],"forum":"02Ug9N8DCI","number":1,"license":"CC BY 4.0","cdate":1698400745618,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6897/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636802801,"domain":"ICLR.cc/2024/Conference","replyto":"02Ug9N8DCI","id":"2jQRpYdNEP","forumContent":{"venue":{"value":"Submitted to ICLR 2024"},"TLDR":{"value":"GateLoop generalizes linear recurrent models (e.g. S4, S5, LRU, RetNet) using data-controlled state-transitions."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Data-controlled","Linear Recurrence","Sequence Modeling","GateLoop","Linear","RNN","State Space Model","SSM","S4","S5","LRU","RetNet","generalization"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Linear Recurrence has proven to be a powerful tool for modeling long sequences efficiently. In this work, we show that existing models fail to take full advantage of its potential. Motivated by this finding, we develop GateLoop, a foundational sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet, by employing data-controlled state transitions. \nUtilizing this theoretical advance, GateLoop empirically outperforms existing models for auto-regressive language modeling. Our method comes with a low-cost $O(l)$ recurrent mode and an efficient $O(l \\log_{2} l)$ parallel mode making use of highly optimized associative scan implementations. Furthermore, we derive an $O(l^2)$ surrogate-attention mode, revealing remarkable implications for Transformer and recently proposed architectures. \nSpecifically, we prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention. \nWhile many existing models solely rely on data-controlled cumulative sums for context aggregation, our findings suggest that incorporating data-controlled complex cumulative products may be a crucial step towards more powerful sequence models."},"_bibtex":{"value":"@misc{\nkatsch2024gateloop,\ntitle={GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling},\nauthor={Tobias Katsch},\nyear={2024},\nurl={https://openreview.net/forum?id=02Ug9N8DCI}\n}"},"title":{"value":"GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling"},"pdf":{"value":"/pdf/539cf74fff3ea51ab4ed275e6f34e07c55215962.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"katsch|gateloop_fully_datacontrolled_linear_recurrence_for_sequence_modeling"},"authorids":{"value":["~Tobias_Katsch1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tobias Katsch"]}},"version":2},{"content":{"TLDR":{"value":"MambaTG jointly models historical-sequence evolution and candidate-history compatibility for effective future link prediction across repeated and unseen edges."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Temporal graph learning; future link prediction"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Future link prediction on temporal graphs involves both repeated interactions between previously connected node pairs and first-time interactions between pairs with no prior interaction. While repeated-edge prediction can exploit direct pairwise evidence, unseen-edge prediction must generalize beyond historical recurrence. Existing temporal graph models often exploit signals tied to recurring interactions or historical overlap, which are highly informative for repeated edges but become limited or unavailable for unseen edges, contributing to substantial performance degradation on unseen-edge prediction. Based on this observation, we identify two complementary signals for modeling sequential dynamics beyond historical recurrence: (1) historical-sequence evolution, which captures dependencies among the source node's chronologically ordered interactions, and (2) candidate--history compatibility, which captures how well a candidate fits the source node's evolving history. Based on the decomposition, we propose MambaTG, a candidate-aware Mamba-based framework that jointly models these two signals. Mamba provides a natural mechanism for this design: its selective recurrence captures the evolution of ordered interaction histories, while its input-dependent state transitions allow the history state to adapt to different candidate inputs. Extensive experiments on 13 temporal graph datasets against 13 representative baselines show that MambaTG achieves strong and consistent performance across both repeated and unseen edges while remaining computationally efficient."},"_bibtex":{"value":"@inproceedings{\nanonymous2026mambatg,\ntitle={Mamba{TG}: Candidate-Aware State Space Modeling for Future Link Prediction on Temporal Graphs},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=v38OXwNBs3},\nnote={under review}\n}"},"title":{"value":"MambaTG: Candidate-Aware State Space Modeling for Future Link Prediction on Temporal Graphs"},"pdf":{"value":"/pdf/97f797cf07ca2fe58d7df0e63f2db4f6e40cbd1a.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791229332745,"tcdate":1789446618412,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission19628/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission19628/Authors"],"forum":"v38OXwNBs3","license":"CC BY 4.0","number":19628,"cdate":1789446618412,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission19628/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791229332745,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"v38OXwNBs3","version":2},{"content":{"summary":{"value":"The paper proposes Set-Sequence, a model for large-scale time series prediction. It first uses a Set module to extract a permutation-invariant population summary, then combines it with individual features in a Sequence module (e.g., Transformer or RNN) for prediction. Experiments show improved accuracy, efficiency, and interpretability across multiple tasks."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1) How is the proposed setting fundamentally different from classical multivariate time series modeling (multivariate time series feature selection, reweighting), beyond computational efficiency?\n\n2) If the model does not explicitly capture pairwise or structured dependencies, can it truly model cross-entity interactions?\n\n3) Does restricting the dataset to the top 500 equities introduce survivorship or selection bias, and how robust are the results under a less-filtered universe?\n\n4) How does the exchangeability assumption affect performance when dependencies are localized or non-exchangeable?\n\n5) Why does the model underperform attention-based baselines despite its claimed expressivity advantages?\n\nWould exploring alternative pooling functions (e.g., quantile or group-wise) change the expressivity–efficiency trade-off?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"The paper presents a clear and modular architecture that integrates a permutation-invariant Set module with standard sequence models, offering linear scalability and theoretical grounding through connections to Deep Sets. Its empirical evaluation—spanning large-scale financial datasets such as CRSP equities and CoreLogic mortgages—demonstrates strong predictive and economic performance. However, the validation remains limited to financial domains, and the interpretability analysis, while suggestive of meaningful latent structure, is primarily correlational and not deeply explored."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1) The motivation is partially unclear: while the paper aims to handle large cross-sections of time series efficiently, it is not well distinguished from the classical setting of multivariate time series modeling. If each entity were viewed as a “variable,” the proposed Set-Sequence formulation would resemble feature aggregation rather than feature selection. However, the paper does not clarify this distinction nor position its contribution relative to multivariate temporal modeling, leaving its conceptual novelty somewhat ambiguous.\n\n2) While the paper motivates itself by the need to capture cross-entity interactions, it does not explicitly model pairwise or structured dependencies as in graph or attention-based methods. Instead, it addresses a different, more specific challenge: efficiently summarizing population-level context under exchangeability when the number of entities is extremely large. This makes the contribution one of scalability and global aggregation, rather than relational modeling in the strict sense.\n\n3) The dataset includes only the top 500 market-cap equities per period and excludes missing returns, which may introduce survivorship or selection bias. It is recommended to include a less-filtered baseline (e.g., fixed universe or delisted stocks) and year-by-year sensitivity analyses to ensure robustness.\n\n4)  The model assumes exchangeability and uses pooled moments, which may fail under strong non-exchangeable or localized dependencies. A small synthetic experiment comparing Set-Seq with attention or graph models under such conditions would clarify its expressive limits.\n\n5) The paper demonstrates the expressive capability of the model; however, in practical testing, its performance shows a clear gap compared to attention-based mechanisms. The paper does not clearly explain the cause of this discrepancy and only highlights the model’s superiority in terms of time complexity.\n\n6)  The paper mainly uses mean and polynomial pooling. Including or emphasizing ablations on alternative pooling methods (e.g., quantile, robust, or group-wise pooling) would better illustrate expressivity–cost trade-offs."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922782070,"tcdate":1761711065971,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11754/Reviewer_1etJ"],"signatures":["ICLR.cc/2026/Conference/Submission11754/Reviewer_1etJ"],"forum":"c1A3DCni3K","number":1,"license":"CC BY 4.0","cdate":1761711065971,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11754/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922782070,"domain":"ICLR.cc/2026/Conference","replyto":"c1A3DCni3K","id":"jW2WsTA2Zq","forumContent":{"TLDR":{"value":"When numerous time series are exchangeable, the Set-Sequence model efficiently captures cross-sectional dependencies"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Set modeling","Sequence modeling","High-dimensional time series","Cross-sectional dependencies","Interpretable AI","Synthetic data","Mortgage risk prediction","Exchangeability","Time series prediction","Financial AI","Deep learning"]},"supplementary_material":{"value":"/attachment/325e76334150d1a8a8e96c235d088931acd8c9ae.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Many prediction problems across science and engineering, especially in finance and economics, involve large cross-sections of individual time series, where each unit (e.g., a loan, stock, or customer) is driven by unit-level features and latent cross-sectional dynamics. While sequence models have advanced per-unit temporal prediction, capturing cross-sectional effects often still relies on hand-crafted summary features. We propose Set-Sequence, a model that learns cross-sectional structure directly, enhancing expressivity and eliminating manual feature engineering. At each time step, a permutation-invariant Set module summarizes the unit set; a Sequence module then models each unit’s dynamics conditioned on both its features and the learned summary. The architecture accommodates unaligned series, supports varying numbers of units at inference, integrates with standard sequence backbones (e.g., Transformers), and scales linearly in cross-sectional size. Across a synthetic contagion task and two large-scale real-world applications—equity portfolio optimization and loan risk prediction—Set-Sequence significantly outperforms strong baselines, delivering higher Sharpe ratios, improved AUCs, and interpretable cross-sectional summaries."},"_bibtex":{"value":"@misc{\nepstein2026a,\ntitle={A Set-Sequence Model for Time Series},\nauthor={Elliot L Epstein and Apaar Sadhwani and Kay Giesecke},\nyear={2026},\nurl={https://openreview.net/forum?id=c1A3DCni3K}\n}"},"title":{"value":"A Set-Sequence Model for Time Series"},"pdf":{"value":"/pdf/af39a1344d262612bd1bf64c15a33dbcc3a7f6ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"epstein|a_setsequence_model_for_time_series"},"authorids":{"value":["~Elliot_L_Epstein1","~Apaar_Sadhwani1","~Kay_Giesecke1"]},"authors":{"value":["Elliot L Epstein","Apaar Sadhwani","Kay Giesecke"]}},"version":2},{"content":{"comment":{"value":"(Continued from the previous response)\n\n**Parallelization**\nThe reviewer notes that the k-hop parallelization “applies equally to any linear/polynomial GNN.” We agree that the ability to unfold a linear recurrence is well known; indeed, this is precisely the mechanism that underpins all modern SSMs [3,4,5].  However, the novelty in our work does not lie in observing that a linear recurrence can be parallelized; it lies in embedding the linear SSM recurrence as the graph propagation mechanism itself, in a way that existing SSM-based graph models do not.\n- Our MP-SSM is, to our knowledge, the first graph architecture that integrates the SSM principle of linear, tied-weight recurrence directly into the message passing mechanism, not as a post-hoc module.\n- This yields a model that is simultaneously permutation-equivariant, graph-native, and parallelizable, while maintaining the SSM structure essential for long-range dynamics.\n\nAll these benefits cannot be straightforwardly achieved by simply using linear GNNs or spectral-filter-based models, nor by applying SSM modules to sequences extracted from graphs. We also kindly refer the reviewer to our detailed response to **Reviewer erdw**, which explains the challenges, such as the inevitable loss of structural information, faced by graph models (like S4G [2]) that attempt to directly apply SSM frameworks to sequences derived from the graph.\nConcluding, parallelization is not merely a “trick,” but a natural byproduct of MP-SSM’s design, which unifies graph diffusion and SSM-like linear recurrence into a single update equation. This contrasts with prior GraphSSM models in the literature, which either apply spatial diffusion GNNs and SSMs in a decoupled manner or first extract sequences from the graph before applying SSM layers, as covered in our Introduction and  Related Work Sections.  Following your comment, we further highlighted this point in Appendix B to better reflect it. Thank you.\n\n[2] Song, Yunchong, et al. \"Breaking the Bottleneck on Graphs with Structured State Spaces.\" Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024.\n\n[3] Gu, Albert, Karan Goel, and Christopher Ré. \"Efficiently modeling long sequences with structured state spaces.\" ICLR 2022.\n\n[4] Orvieto, Antonio, et al. \"Resurrecting recurrent neural networks for long sequences.\" International Conference on Machine Learning. PMLR, 2023.\n\n[5] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" First conference on language modeling. 2024.\n\n**(2) ON THE THEORETICAL NOVELTY**\n\nThe reviewer suggests that the theoretical contribution is modest because the derivations rely on linearity. However, the novelty does not lie in taking derivatives per se, but in what the linearity enables: **exact sensitivity analysis and quantitative lower bounds.**\nTo our knowledge, our MP-SSM is the first GNN architecture for which:\n- The node-to-node Jacobian can be computed exactly (Theorem 3.4),\n- Local sensitivity in the deep regime can be approximated precisely (Theorem 3.6),\n- Minimum sensitivity admits topology-dependent lower bounds linked to oversquashing (Corollary 3.7 and Remark 3.8),\n- Global sensitivity admits non-vanishing lower bounds for any connected graph (Theorem 3.11).\n\nPrior linear GNNs or spectral filters do not provide such guarantees. Moreover, MP-SSM's design is what makes these derivations meaningful: the recurrence and weight-sharing structure directly connect the analytical results to deep-graph learning phenomena (vanishing gradients, oversquashing).\n\n**On the ReLU comparison**\nOur ReLU-based comparison in Theorem 3.13 was meant as an illustrative but formal and quantitative example. We agree that using $\\sqrt(2)$ scaling reduces vanishing in a ReLU-GCN, but the point was to quantify the intrinsic difference between repeated nonlinear diffusion and repeated linear diffusion with a single nonlinearity. Applying appropriate scaling to both ReLU-GCN and MP-SSM models would still lead to the same exponential gap."},"title":{"value":"Response - part 2"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763647323276,"tcdate":1763647323276,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20107/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission20107/Authors"],"forum":"u8Vn63eG2K","number":10,"license":"CC BY 4.0","cdate":1763647323276,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20107/-/Official_Comment"],"mdate":1763647323276,"domain":"ICLR.cc/2026/Conference","replyto":"0vXBtgybbJ","id":"r7Nn8H0LKi","forumContent":{"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"TLDR":{"value":"We revisit GSSMs via sensitivity analysis. We integrate modern SSMs into message passing, yielding a parallel model with Jacobians for exact flow bounds, enabling stable long-range propagation and strong results on static and temporal graphs."},"keywords":{"value":["graph neural networks","state space models"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"The recent success of State-Space Models (SSMs) in sequence modeling has inspired their extension to graphs, giving rise to Graph State-Space Models (GSSMs). While effective, existing approaches often rely on sequentializations or spectral decompositions that lack permutation equivariance, message-passing compatibility, and computational efficiency. Moreover, they typically target either static or temporal graphs in isolation and, crucially, provide only loose or qualitative results on information propagation, offering no exact guarantees on challenges such as vanishing gradients and over-squashing.\nIn this work, we revisit the design of GSSMs through the lens of sensitivity analysis. We introduce a principled integration of modern SSM computation into the Message-Passing Neural Network framework, yielding a unified architecture that is computationally efficient, permutation equivariant, and supports fast parallelism. Our formulation admits closed-form Jacobian computations, enabling an exact sensitivity analysis of node-to-node dependencies and rigorous lower bounds on information flow, contrasting sharply with prior heuristic approaches. These theoretical insights clarify when and how stable long-range propagation can be achieved. Finally, we validate our model across a wide range of benchmarks, including node classification, graph property prediction, long-range reasoning, and spatiotemporal forecasting, where it achieves strong empirical performance while preserving the simplicity of message passing."},"_bibtex":{"value":"@misc{\nanonymous2026a,\ntitle={A Sensitivity Analysis of State-Space Models on~Graphs},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=u8Vn63eG2K}\n}"},"title":{"value":"A Sensitivity Analysis of State-Space Models on Graphs"},"pdf":{"value":"/pdf/f42baaf45f99ad40ccc8a68901e607e7d69ae1a3.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"ceni|a_sensitivity_analysis_of_statespace_models_on_graphs"},"authorids":{"value":["~Andrea_Ceni1","~Alessio_Gravina1","~Carola-Bibiane_Schönlieb1","~Claudio_Gallicchio1","~Davide_Bacciu1","~Moshe_Eliasof1"]},"authors":{"value":["Andrea Ceni","Alessio Gravina","Carola-Bibiane Schönlieb","Claudio Gallicchio","Davide Bacciu","Moshe Eliasof"]}},"version":2},{"content":{"summary":{"value":"Advancements in state space models (SSMs), particularly a model called Mamba, have demonstrated efficient performance in supervised learning, offering linear complexity in training and rapid computation during inference, similar to RNNs. This paper explores the potential of the Mamba model for DG and identifies a challenge: input-dependent matrices in SSMs can exacerbate domain-specific features, thus impairing generalization.\n\nTo overcome this limitation, the authors introduce a novel SSM-based architecture featuring a saliency-based token-aware transformation, termed START. This architecture effectively reduces domain discrepancies by selectively altering and suppressing domain-specific features within the salient tokens of input-dependent matrices. START achieves state-of-the-art performance in DG, presenting a viable and competitive alternative to both CNNs and ViTs."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Please refer to the weakness."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. This passage discusses a theoretical analysis of the Mamba model's performance in domain generalization. The analysis reveals that the Mamba model's input-dependent matrices have a tendency to accumulate domain-specific features during their recurrent processing. This accumulation could potentially limit the model’s ability to generalize effectively across different, unseen domains. Essentially, the matrices that should help the model adapt to new data instead make it more sensitive to the specific characteristics of the data it was trained on, thereby impeding its general applicability.\n\n2. This statement outlines the proposal of a new state space model (SSM)-based architecture, which incorporates a saliency-driven, token-aware transformation aimed at addressing the issues previously identified with the Mamba model in domain generalization tasks. This novel architecture is designed to serve as an effective alternative to both convolutional neural networks (CNNs) and vision transformers (ViTs). It boasts an ability to generalize excellently across diverse domains while maintaining the efficient linear complexity characteristic of state space models, making it a promising option for handling domain shifts in a computationally efficient manner."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"It would be better to report computation costs like inference time."},"limitations":{"value":"This paper discusses the limitations."}},"nonreaders":[],"tmdate":1730879310589,"tcdate":1721104667378,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission9378/Reviewer_qvXD"],"signatures":["NeurIPS.cc/2024/Conference/Submission9378/Reviewer_qvXD"],"forum":"mAdGQ1Hh3L","number":3,"license":"CC BY 4.0","cdate":1721104667378,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission9378/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879310589,"domain":"NeurIPS.cc/2024/Conference","replyto":"mAdGQ1Hh3L","id":"d53nGxcuZv","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Domain generalization","state space models"]},"supplementary_material":{"value":"/attachment/ce883f1aa347d3f4ebb234dc743b97b4275be824.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Domain Generalization (DG) aims to enable models to generalize to unseen target domains by learning from multiple source domains. Existing DG methods primarily rely on convolutional neural networks (CNNs), which inherently learn texture biases due to their limited receptive fields, making them prone to overfitting source domains. While some works have introduced transformer-based methods (ViTs) for DG to leverage the global receptive field, these methods incur high computational costs due to the quadratic complexity of self-attention. Recently, advanced state space models (SSMs), represented by Mamba, have shown promising results in supervised learning tasks by achieving linear complexity in sequence length during training and fast RNN-like computation during inference. Inspired by this, we investigate the generalization ability of the Mamba model under domain shifts and find that input-dependent matrices within SSMs could accumulate and amplify domain-specific features, thus hindering model generalization. To address this issue, we propose a novel SSM-based architecture with saliency-based token-aware transformation (namely START), which achieves state-of-the-art (SOTA) performances and offers a competitive alternative to CNNs and ViTs. Our START can selectively perturb and suppress domain-specific features in salient tokens within the input-dependent matrices of SSMs, thus effectively reducing the discrepancy between different domains. Extensive experiments on five benchmarks demonstrate that START outperforms existing SOTA DG methods with efficient linear complexity. Our code is available at https://github.com/lingeringlight/START."},"_bibtex":{"value":"@inproceedings{\nguo2024start,\ntitle={{START}: A Generalized State Space Model with Saliency-Driven Token-Aware Transformation},\nauthor={Jintao Guo and Lei Qi and Yinghuan Shi and Yang Gao},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=mAdGQ1Hh3L}\n}"},"title":{"value":"START: A Generalized State Space Model with Saliency-Driven Token-Aware Transformation"},"pdf":{"value":"/pdf/1e045e8d4c432c88ee7d528f95cd5406dfc62b43.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"guo|start_a_generalized_state_space_model_with_saliencydriven_tokenaware_transformation"},"authorids":{"value":["~Jintao_Guo1","~Lei_Qi1","~Yinghuan_Shi3","~Yang_Gao3"]},"authors":{"value":["Jintao Guo","Lei Qi","Yinghuan Shi","Yang Gao"]}},"version":2},{"content":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Rejected by TMLR"},"abstract":{"value":"Sequence models based on linear state spaces (SSMs) have recently emerged as a promising choice of architecture for modeling long range dependencies across various modalities. However, they invariably rely on discretization of a continuous state space, which complicates their presentation and understanding. In this work, we dispose of the discretization step, and propose a model based on vanilla Diagonal Linear RNNs ($\\mathrm{DLR}$). We empirically show that, despite being conceptually much simpler, $\\mathrm{DLR}$ is as performant as previously-proposed SSMs on a variety of tasks and benchmarks including Long Range Arena and raw speech classification. Moreover, we characterize the expressivity of SSMs (including $\\mathrm{DLR}$) and attention-based models via a suite of $13$ synthetic sequence-to-sequence tasks involving interactions over tens of thousands of tokens, ranging from simple operations, such as shifting an input sequence, to detecting co-dependent visual features over long spatial ranges in flattened images. We find that while SSMs report near-perfect performance on tasks that can be modeled via $\\textit{few}$ convolutional kernels, they struggle on tasks requiring $\\textit{many}$ such kernels and especially when the desired sequence manipulation is $\\textit{context-dependent}$. For example, $\\mathrm{DLR}$ learns to perfectly shift a $0.5M$-long input by an arbitrary number of positions but fails when the shift size depends on context. Despite these limitations, $\\mathrm{DLR}$ reaches high performance on two higher-order reasoning tasks $\\mathrm{ListOpsSubTrees}$ and $\\mathrm{PathfinderSegmentation}\\text{-}\\mathrm{256}$ with input lengths $8K$ and $65K$ respectively, and gives encouraging performance on $\\mathrm{PathfinderSegmentation}\\text{-}\\mathrm{512}$ with input length $262K$ for which attention is not a viable choice.\n---"},"_bibtex":{"value":"@article{\ngupta2023simplifying,\ntitle={Simplifying and Understanding State Space Models with Diagonal Linear {RNN}s},\nauthor={Ankit Gupta and Harsh Mehta and Jonathan Berant},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2023},\nurl={https://openreview.net/forum?id=YrBFZ2egXv},\nnote={Rejected}\n}"},"title":{"value":"Simplifying and Understanding State Space Models with Diagonal Linear RNNs"},"pdf":{"value":"/pdf/32f06181c229adce4bed59d2d1b44f1869dcb7a7.pdf"},"venueid":{"value":"TMLR/Rejected"},"paperhash":{"value":"gupta|simplifying_and_understanding_state_space_models_with_diagonal_linear_rnns","readers":["everyone"]},"authorids":{"readers":["everyone"],"value":["~Ankit_Gupta3","~Harsh_Mehta1","~Jonathan_Berant1"]},"assigned_action_editor":{"value":"~Tao_Qin1"},"authors":{"readers":["everyone"],"value":["Ankit Gupta","Harsh Mehta","Jonathan Berant"]}},"tmdate":1726599503820,"tcdate":1687484518979,"writers":["TMLR","TMLR/Paper1319/Authors"],"signatures":["TMLR/Paper1319/Authors"],"forum":"YrBFZ2egXv","number":1319,"license":"CC BY 4.0","cdate":1687484518979,"mdate":1726599503820,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/Paper1319/-/Revision","TMLR/-/Under_Review","TMLR/-/Edit","TMLR/-/Rejected","TMLR/-/Authors_Release"],"odate":1688443657256,"domain":"TMLR","id":"YrBFZ2egXv","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2409.12033v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"montagna|topological_deep_learning_with_statespace_models_a_mamba_approach_for_simplicial_complexes"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Marco_Montagna:","~Simone_Scardapane1","https://dblp.org/search/pid/api?q=author:Lev_Telyatnikov:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2409.12033"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2409-12033,\n  publtype={informal},\n  author={Marco Montagna and Simone Scardapane and Lev Telyatnikov},\n  title={Topological Deep Learning with State-Space Models: A Mamba Approach for Simplicial Complexes},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2409.12033},\n  url={https://doi.org/10.48550/arXiv.2409.12033}\n}\n"},"abstract":{"value":"Graph Neural Networks based on the message-passing (MP) mechanism are a dominant approach for handling graph-structured data. However, they are inherently limited to modeling only pairwise interactions, making it difficult to explicitly capture the complexity of systems with $n$-body relations. To address this, topological deep learning has emerged as a promising field for studying and modeling higher-order interactions using various topological domains, such as simplicial and cellular complexes. While these new domains provide powerful representations, they introduce new challenges, such as effectively modeling the interactions among higher-order structures through higher-order MP. Meanwhile, structured state-space sequence models have proven to be effective for sequence modeling and have recently been adapted for graph data by encoding the neighborhood of a node as a sequence, thereby avoiding the MP mechanism. In this work, we propose a novel architecture designed to operate with simplicial complexes, utilizing the Mamba state-space model as its backbone. Our approach generates sequences for the nodes based on the neighboring cells, enabling direct communication between all higher-order structures, regardless of their rank. We extensively validate our model, demonstrating that it achieves competitive performance compared to state-of-the-art models developed for simplicial complexes."},"title":{"value":"Topological Deep Learning with State-Space Models: A Mamba Approach for Simplicial Complexes"},"authors":{"value":["Marco Montagna","Simone Scardapane","Lev Telyatnikov"]}},"tmdate":1731318302242,"pdate":1704067200000,"tcdate":1731318277506,"writers":["~"],"signatures":["~Simone_Scardapane1"],"forum":"MZLGW0fDsJ","license":"CC BY-SA 4.0","number":180685,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1731318302242,"domain":"DBLP.org","id":"MZLGW0fDsJ","version":2},{"content":{"venue":{"value":"CVPR 2026"},"abstract":{"value":"Accurate trajectory prediction is vital for safe autonomous driving, yet existing approaches struggle to balance modeling power and computational efficiency. Attention-based architectures incur quadratic complexity with increasing agents, while recurrent models struggle to capture long-range dependencies and fine-grained local dynamics. Building upon this, we present FoSS, a dual-branch framework that unifies frequency-domain reasoning with linear-time sequence modeling. The frequency-domain branch performs a discrete Fourier transform to decompose trajectories into amplitude components encoding global intent and phase components capturing local variations, followed by a progressive helix reordering module that preserves spectral order; two selective state-space submodules, Coarse2Fine-SSM and SpecEvolve-SSM, refine spectral features with $\\mathcal{O}(N)$ complexity. In parallel, a time-domain dynamic selective SSM reconstructs self-attention behavior in linear time to retain long-range temporal context. A cross-attention layer fuses temporal and spectral representations, while learnable queries generate multiple candidate trajectories, and a weighted fusion head expresses motion uncertainty. Experiments on Argoverse 1 and Argoverse 2 benchmarks demonstrate that FoSS achieves state-of-the-art accuracy while reducing computation by 22.5\\% and parameters by over 40\\%. Comprehensive ablations confirm the necessity of each component."},"_bibtex":{"value":"@inproceedings{\nhuang2026foss,\ntitle={Fo{SS}: Modeling Long-Range Dependencies and Multimodal Uncertainty in Trajectory Prediction via Fourier{\\textendash}State Space Integration},\nauthor={Yizhou Huang and Genze Jiang and Yihua Cheng and Kezhi Wang},\nbooktitle={Conference on Computer Vision and Pattern Recognition 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=icsrG0ijaA}\n}"},"title":{"value":"FoSS: Modeling Long-Range Dependencies and Multimodal Uncertainty in Trajectory Prediction via Fourier–State Space Integration"},"pdf":{"value":"https://openaccess.thecvf.com/content/CVPR2026/papers/Huang_FoSS_Modeling_Long-Range_Dependencies_and_Multimodal_Uncertainty_in_Trajectory_Prediction_CVPR_2026_paper.pdf"},"venueid":{"value":"thecvf.com/CVPR/2026/Conference"},"paperhash":{"value":"huang|foss_modeling_longrange_dependencies_and_multimodal_uncertainty_in_trajectory_prediction_via_fourierstate_space_integration"},"authorids":{"value":["~Yizhou_Huang3","~Genze_Jiang1","~Yihua_Cheng1","~Kezhi_Wang1"]},"authors":{"value":["Yizhou Huang","Genze Jiang","Yihua Cheng","Kezhi Wang"]}},"tmdate":1789656466562,"pdate":1789656437158,"tcdate":1765220258537,"writers":["thecvf.com/CVPR/2026/Conference","thecvf.com/CVPR/2026/Conference/Submission31101/Authors"],"signatures":["thecvf.com/CVPR/2026/Conference/Submission31101/Authors"],"forum":"icsrG0ijaA","license":"CC BY 4.0","number":31101,"cdate":1765220258537,"readers":["everyone"],"invitations":["thecvf.com/CVPR/2026/Conference/-/Submission","thecvf.com/CVPR/2026/Conference/Submission31101/-/Full_Submission","thecvf.com/CVPR/2026/Conference/-/Post_Submission","thecvf.com/CVPR/2026/Conference/-/Edit","thecvf.com/CVPR/2026/Conference/-/Compute_Flag"],"mdate":1789656466562,"odate":1789656437158,"domain":"thecvf.com/CVPR/2026/Conference","id":"icsrG0ijaA","version":2},{"content":{"TLDR":{"value":"We introduce a training-aware, calibration-free framework for joint reduction of the state order and step-size scheduling rank of Mamba selective state-space models."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["selective state-space models","Mamba","model compression","model order reduction","low-rank approximation"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Modeling long sequences, such as text and audio, requires computationally efficient architectures; Mamba addresses this need using selective state-space models (SSMs). However, two structural dimensions intrinsic to these selective SSMs—the state order and the step-size scheduling dimension—directly affect per-step cost and parameter count. We introduce a training-aware framework that uses structural regularization during full-order training to enable joint reduction of both dimensions. A per-layer finite-horizon output-error bound motivates a scale-invariant, decay-aware state score and a factorized low-rank surrogate for the step-size map. After training, we perform one-shot reduction using state selection and truncated SVD to construct a model with the requested dimensions from the learned parameters alone, without calibration data or post-compression fine-tuning. On Selective Copying, SC09, and The Pile, the jointly reduced models outperform vanilla post-hoc reduction in reported point estimates at matched reduction levels. In isolated state reduction, our calibration-free method outperforms the adapted calibration-based baselines on Selective Copying and SC09 and remains competitive on The Pile."},"_bibtex":{"value":"@inproceedings{\nanonymous2026trainingaware,\ntitle={Training-Aware, Calibration-Free Model Reduction for Selective State Space Models},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=rLRq4RvB7I},\nnote={under review}\n}"},"title":{"value":"Training-Aware, Calibration-Free Model Reduction for Selective State Space Models"},"pdf":{"value":"/pdf/7981993a584250a21e4e213596c3a48b15b4ea38.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791233494554,"tcdate":1789742753112,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission44307/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission44307/Authors"],"forum":"rLRq4RvB7I","license":"CC BY 4.0","number":44307,"cdate":1789742753112,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission44307/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791233494554,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"rLRq4RvB7I","version":2},{"content":{"summary":{"value":"This paper proposes a method to use Mamba as a multi-agent reinforcement learning (MARL) policy. The proposed method is called SP-Mamba. The standard method for allowing policies to memorise previous states is to use recurrent neural networks (RNNs) however, this work proposed Mamba as it has been shown to outperform RNNs in other tasks. The authors show that through a self-predictive loss, where the policy predicts the next encoded observation, they are able to stabilise the learning of SP-Mamba and outperform both strong baselines and a naive mamba implementation (without the self-predictive loss). Additionally, ablations are performed to find the best variational autoencoder and the best combination of objectives to train the VAE’s self-predictive loss."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"* On line 53 the authors say that MAT has stringent assumptions, what are these assumptions? \n\n* On line 61 it is mentioned that SSMs are difficult to train for decision-making tasks, why is S5 [7] not discussed, where an SSM is used for partially observable single-agent RL tasks?\n\n* There is a mistake on line 92, a should be u\n\n* Why is the critic not trained using Mamba also, it seems strange that a GRU is used, but it is not discussed why in a paper about using Mamba instead of a GRU.\n\n* Q3 on line 301/302 is vague. Effective in terms of what metrics and in comparison to which baseline?\n\n* On line 363-365 you discuss that multiple categorical variables with a size of 32 x 32 will lose too much information, but were smaller sizes considered and experimented with?\n\n* On line 406-409 the hyperparameters of the various VAEs are discussed, how were these chosen, are they standard values from previous work? \n\nReferences:\n\n[7] Lu, C., Schroecker, Y., Gu, A., Parisotto, E., Foerster, J., Singh, S. and Behbahani, F., 2024. Structured state space models for in-context reinforcement learning. Advances in Neural Information Processing Systems, 36."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The idea to use Mamba with a self-predictive loss is a novel approach to stabilising Mamba in MARL. Additionally, the way the self-predictive loss is constructed by focusing on reward maximisation rather than reconstruction, as is common in world model training, is a useful insight. Finally, the ablations performed are thorough and well-considered as they ablate useful features of the architecture."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The main issue I have with this work is that the claims made throughout the paper do not align with the results.\n* In the introduction, the authors mention “Substantial experiments demonstrate…”. Substantial experiments were not performed. Only six scenarios were tested, from a single environment suite (SMAC). This is not considered substantial and is in fact around the average for a MARL paper [1].\n* In the conclusion, they say “scalable and efficient approach for handling complex multi-agent environments”. It is not explained how this work is scalable and in what dimension it can scale e.g. number of agents, timesteps or model size. Additionally, it should not be claimed that this work can handle complex MARL environments as it is only tested on 6 tasks from a single environment suite, from these results, one cannot assume it will generalise to arbitrary complex environments.\n\nAdditionally, I find the following to be weaknesses of this work\n* It seems suspect that the state-of-the-art (SOTA) method in cooperative MARL is referenced in this work - Multi-Agent Transformer [6] - however it is not used as a baseline. It would make sense to include this not only because it is SOTA, but also because it is a transformer-based policy, which is more similar to Mamba than the RNN-based policies used in recurrent PPO, QMIX and RODE. MAT also tests on SMAC and indeed on all the same tasks used in this work and in all cases MAT significantly outperforms the results reported in this work.\n* To expand on this point, the results do not seem to align with previous work which has tested PPO and QMIX and in most cases significantly underperforms the results from multiple other independent works [4,5,6].\n* There seems to be a lack of hyperparameter tuning which may significantly affect the results. This is cited as a strength in the conclusion of the paper, but I do not think it is reasonable to expect the hyperparameters that work well for SP-Mamba to also perform well in PPO. Additionally, it is not clear how hyperparameters were chosen. Was there some empirical evidence to support the choice of the hyperparameters selection? If so, this should be included in the paper.\n* It is concerning that this work only tests on 6 tasks from a single suite. Gorsane et al. [1] recommend at least two distinct environment suites. This calls the significance of the results into question. Additionally [1] and [2] recommend evaluating with at least 10 seeds, this work only evaluates using 4 seeds.\n* It is no longer recommended to test on SMAC v1 as it suffers from a lack of stochasticity and partial observability [3]. Instead, SMAC v2 [3] should be used.\n* Given that Mamba should improve the memory capabilities of the approach it would have been interesting to see how it performs in more memory-intensive environments.\n\n\nReferences:\n\n[1] Gorsane, R., Mahjoub, O., de Kock, R.J., Dubb, R., Singh, S. and Pretorius, A., 2022. Towards a standardised performance evaluation protocol for cooperative marl. Advances in Neural Information Processing Systems, 35, pp.5510-5521.\n\n[2] Agarwal, R., Schwarzer, M., Castro, P.S., Courville, A.C. and Bellemare, M., 2021. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, pp.29304-29320.\n\n[3] Ellis, B., Cook, J., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J. and Whiteson, S., 2024. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36.\n\n[4] Rashid, T., Samvelyan, M., De Witt, C.S., Farquhar, G., Foerster, J. and Whiteson, S., 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178), pp.1-51.\n\n[5] Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A. and Wu, Y., 2022. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35, pp.24611-24624.\n\n[6] Wen, M., Kuba, J., Lin, R., Zhang, W., Wen, Y., Wang, J. and Yang, Y., 2022. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35, pp.16509-16521."}},"nonreaders":[],"tmdate":1731428104047,"tcdate":1730457214138,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4326/Reviewer_RK1D"],"signatures":["ICLR.cc/2025/Conference/Submission4326/Reviewer_RK1D"],"forum":"7ZyFjPUeJp","number":2,"license":"CC BY 4.0","cdate":1730457214138,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4326/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428104047,"domain":"ICLR.cc/2025/Conference","replyto":"7ZyFjPUeJp","id":"fDQdBL3ubN","forumContent":{"TLDR":{"value":"This paper introduces a novel framework that leverages the Mamba model to predict agents' future observations, aiding in more stable and informed decision-making."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Sequence model","state space model","Mamba","multi-agent reinforcement learning","self-predictive representation learning"]},"supplementary_material":{"value":"/attachment/e863238325a5f1f99ae7fff9af63ffa5f65d267a.zip"},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"In multi-agent reinforcement learning (MARL), agents must collaborate to achieve team goals while only having access to limited local observations. This partial observability, coupled with the dynamic presence of other agents, renders the environment non-stationary for each agent, complicating the policy training. A critical challenge in this setting is the efficient utilization of historical information for decision-making. Building on the hypothesis that self-predictive features can improve policy learning, we introduce the self-predictive Mamba, a novel framework that integrates the Mamba model with self-predictive representation learning for decentralized policy optimization. Self-predictive Mamba leverages a unique policy architecture where the Mamba model is trained to predict future observations, aiding in more stable and informed decision-making. Substantial experiments demonstrate that self-predictive Mamba significantly outperforms the widely used recurrent neural network (RNN)-based MARL policies and surpasses those naively employing the Mamba model."},"_bibtex":{"value":"@misc{\nfeng2025selfpredictive,\ntitle={Self-predictive Mamba: Improving Multi-agent Reinforcement Learning with Self-predictive Encoding},\nauthor={Zhaohan Feng and Runqing Wang and Boxuan Zhang and Jian Sun and Fang Deng and Gang Wang},\nyear={2025},\nurl={https://openreview.net/forum?id=7ZyFjPUeJp}\n}"},"title":{"value":"Self-predictive Mamba: Improving Multi-agent Reinforcement Learning with Self-predictive Encoding"},"pdf":{"value":"/pdf/ffb48d14a6e3feb32e46714c83431ac04ab00547.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"feng|selfpredictive_mamba_improving_multiagent_reinforcement_learning_with_selfpredictive_encoding"},"authorids":{"value":["~Zhaohan_Feng1","~Runqing_Wang1","~Boxuan_Zhang3","~Jian_Sun7","~Fang_Deng1","~Gang_Wang17"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zhaohan Feng","Runqing Wang","Boxuan Zhang","Jian Sun","Fang Deng","Gang Wang"]}},"version":2},{"content":{"summary":{"value":"This paper extends the work presented in “HELM: Hierarchical Encoding for mRNA Language Modeling” [1] by replacing the hierarchical encoding in Euclidean space with one in hyperbolic space. Hyperbolic geometry is leveraged as an inductive bias for hierarchical language modeling of mRNA sequences. Following the architecture described in [1], the proposed HyperHELM model employs a Euclidean backbone, with only the final layer replaced by a hyperbolic prototypical classifier. This design preserves hardware efficiency while exploiting the representational advantages of hyperbolic geometry. The paper investigates three approaches to hyperbolic learning: hyperbolic multinomial logistic regression (MLR), distance-to-prototype learning, and prototype classifiers based on hyperbolic entailment cones. It is the first work to explore hyperbolic language modeling for mRNA sequence data, demonstrating clear benefits over Euclidean space-based models across multiple downstream tasks.\n\n[1] Mehdi Yazdani-Jahromi, Mangal Prakash, Tommaso Mansi, Artem Moskalev, and Rui Liao. HELM: Hierarchical Encoding for mRNA Language Modeling. In *International Conference on Learning Representations*, 2025."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"- **Availability:** Will the code, pretrained model weights, and downstream task dataset splits be made publicly available if the paper is accepted?\n- **Reproducibility:** Please comment on the reproducibility of the work. Are sufficient details provided to replicate the experiments?\n- **Downstream task results:** In Table 1, please clarify what the reported numbers represent. Are these results from single runs, or do they correspond to mean values across multiple experiments or cross-validation folds?\n- **Baselines**: Can you compare HyperHELM to simpler baselines? I think having proper baselines would improve the overall presentation of the results.\n- **HXE loss parameter:** Have you experimented with different values of the hyperparameter $ \\alpha $ in the HXE loss? It appears to be an important factor, yet it was fixed at 0.2 for all experiments. Please elaborate on the reasoning behind this choice and whether sensitivity analysis was performed."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- Overall, the paper is well written, with a clear central idea and an effective flow that guides the reader through the motivation and methodology.\n- The paper presents **HyperHELM**, the first mRNA language model pretrained using masked language modeling in hyperbolic space. While the concept of hierarchical encoding for mRNA sequence data was previously introduced in [1], the application of hyperbolic geometry to RNA sequence modeling is novel.\n- HyperHELM is evaluated on ten diverse downstream tasks spanning multiple organisms and covering various mRNA-related properties, such as protein production, expression levels, degradation, and abundance. It achieves the best performance on nine out of ten datasets, underscoring the advantages of hierarchy-aware learning in hyperbolic space.\n- Unlike the Euclidean space-based HELM, which struggles with longer RNA transcripts, HyperHELM maintains or even improves its performance as sequence length increases. This finding is consistent with prior reports of hyperbolic models in other domains and is particularly important, as longer mRNA sequences are often underrepresented in datasets and pose challenges for existing models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Novelty:** While the idea of using hyperbolic geometry has not previously been applied to mRNA sequence modeling, it has been successfully explored across many areas of machine learning. The application of hyperbolic geometry to mRNA sequence encoding is well motivated; however, the theoretical framework used in the paper does not contribute new insights to the broader machine learning field. As such, the paper’s novelty lies primarily in its application rather than in methodological innovation.\n- **Reproducibility:** The paper does not mention whether the code or pretrained model weights will be made publicly available. For application-oriented work, this should be considered essential. Moreover, the dataset splits used for downstream tasks are not provided. This omission is particularly concerning, given that labeled mRNA datasets are typically small (often fewer than 5,000 samples). Repeating experiments with different random splits can lead to significant variation in performance.\n- **Downstream task results**: Table 1 reports Spearman’s rank correlation and accuracy for ten tasks, but only single numerical values are provided, without clarification on whether these represent mean results or single runs. Combined with the lack of details on dataset splits, this makes the results potentially unreliable. For such small datasets, it is standard practice to perform cross-validation or multiple runs with different splits, reporting both mean and standard deviation to ensure robustness and statistical validity.\n- **Baselines**: HyperHELM is compared to five baselines on four downstream tasks, but only two baselines on the remaining six. The chosen baselines include RNA-FM, SpliceBERT, and CodonBERT. Since the authors use a simple probing setup (with frozen language model parameters), comparisons to RNA-FM, a model pretrained exclusively on non-coding RNAs, are not particularly informative. Similarly, SpliceBERT was pretrained on pre-mRNAs, making mRNA data effectively out-of-distribution. While these models may perform well when fine-tuned on new RNA types, such comparisons are not meaningful under a probing-only setup. Additionally, simpler baselines such as linear or logistic regression are missing and would help contextualize the gains of the proposed approach.\n- **References**: Several references appear unrelated to the core topic (for example, *Xu et al.* in line 131). There also appears to be an overrepresentation of citations from a single research group, some of which are only tangentially related and not essential for understanding the paper. A more balanced and focused selection of references would strengthen the scholarly framing.\n\nI recommend **rejecting** the paper.\n\n1. The paper lacks methodological novelty in machine learning, as it mainly extends an existing framework (HELM) by replacing Euclidean space with hyperbolic space.\n2. Reproducibility is limited due to the absence of publicly available code, pretrained weights, and dataset splits, raising concerns about result reliability.\n\nWhile the paper is clearly written and the motivation to use hyperbolic geometry for mRNA sequence modeling is well justified, the contribution is primarily applied rather than methodological. The theoretical and algorithmic components of hyperbolic learning are already well established in prior literature, and the paper does not introduce new learning principles or modeling techniques.\n\nEmpirically, the reported results on downstream tasks are promising - HyperHELM outperforms Euclidean baselines on 9 out of 10 datasets and shows robustness to longer RNA sequences. However, the absence of code, pretrained weights, and dataset splits makes it difficult to verify or reproduce these findings. Moreover, the use of small datasets without clear information on data splits or repeated trials reduces the statistical reliability of the reported improvements.\n\nOverall, the work is a solid application of existing hyperbolic methods to a new biological domain, but it falls short of the novelty and reproducibility standards expected for acceptance at a top-tier conference. The paper, in this form, is more suitable for presentation at a workshop than a main conference."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916241159,"tcdate":1761128284950,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2449/Reviewer_5EoM"],"signatures":["ICLR.cc/2026/Conference/Submission2449/Reviewer_5EoM"],"forum":"QSoZeaeuCw","number":1,"license":"CC BY 4.0","cdate":1761128284950,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2449/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916241159,"domain":"ICLR.cc/2026/Conference","replyto":"QSoZeaeuCw","id":"MVjnozJsRk","forumContent":{"TLDR":{"value":"We guide mRNA language modeling by embedding the codon hierarchy in hyperbolic space and using the resulting embedding as a classifier during masked language modeling pre-training."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Hyperbolic embeddings","Hierarchical modeling","mRNA language models","mRNA property prediction"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"abstract":{"value":"Language models are increasingly applied to biological sequences like proteins and mRNA, yet their default Euclidean geometry may mismatch the hierarchical structures inherent to biological data. While hyperbolic geometry provides a better alternative for accommodating hierarchical data, it has yet to find a way into language modeling for mRNA sequences. In this work, we introduce HyperHELM, a framework that implements masked language model pre-training in hyperbolic space for mRNA sequences. Using a hybrid design with hyperbolic layers atop Euclidean backbone, HyperHELM aligns learned representations with the biological hierarchy defined by the relationship between mRNA and amino acids. Across multiple multi-species datasets, it outperforms Euclidean baselines on 9 out of 10 tasks involving property prediction, with 10\\% improvement on average, and excels in out-of-distribution generalization to long and low-GC content sequences; for antibody region annotation, it surpasses hierarchy-aware Euclidean models by 3\\% in annotation accuracy. Our results highlight hyperbolic geometry as an effective inductive bias for hierarchical language modeling of mRNA sequences."},"_bibtex":{"value":"@misc{\nspengler2026hyperhelm,\ntitle={Hyper{HELM}: Hyperbolic Hierarchy Encoding for m{RNA} Language Modeling},\nauthor={Max van Spengler and Artem Moskalev and Tommaso Mansi and Mangal Prakash and Rui Liao},\nyear={2026},\nurl={https://openreview.net/forum?id=QSoZeaeuCw}\n}"},"title":{"value":"HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling"},"pdf":{"value":"/pdf/4257de12604e90761b61d4a8cb16387b6abf6d48.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"spengler|hyperhelm_hyperbolic_hierarchy_encoding_for_mrna_language_modeling"},"authorids":{"value":["~Max_van_Spengler1","~Artem_Moskalev1","~Tommaso_Mansi5","~Mangal_Prakash1","~Rui_Liao2"]},"authors":{"value":["Max van Spengler","Artem Moskalev","Tommaso Mansi","Mangal Prakash","Rui Liao"]}},"version":2},{"content":{"summary":{"value":"The paper introduces Mamba Neural Operator (MambaNO) for solving Partial Differential Equations (PDEs) efficiently. Unlike existing methods, which are computationally expensive and often neglect global and local feature integration, MambaNO offers O(N) complexity. It balances global and local integration through a state-space model and alias-free architecture. MambaNO demonstrates accurate approximation of operators for PDEs and achieves state-of-the-art performance on benchmarks with fewer parameters and better efficiency."},"soundness":{"value":3},"confidence":{"value":1},"questions":{"value":"1. Could the author clarify the main applications of this type neural operator? And do they face the similar constraint of LLM models? In addition, why autoregressive models like MAMBA is more helpful than standard ML models? What are the benefits of causal dependency in these applications. \n2. Is the main application about function approximation problem or generic PDE solving? If it includes generic PDE solving, the author should also provide experiments on that."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper is in general well-motivated and provides an promising MAMBA-based alias-free framework for solving PDE. \n2. The paper is well-written with nice visualization with some experimental verifications."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Since alias-free is an important property of the framework, it should move it from the appendix to the main body and provide a succinct summarization. Also, the paper should clarify the meaning of alias-free in the main body. Does it mean zero alias error? Can we view them as the reconstruction error for function approximation. \n2. The plots (figure 1,2,3) need better explanation of the setting. It is unclear from the plots what tasks they refer to and what does the those colorful patches/separation mean. \n3. Since MAMBA is a work that mainly address the system concerns, like memory and computation saving via kernal computation, the paper should make very clear comparison with the original MAMBA to explain their design choices. How does the MAMBA techniques improves the neural operator settings and why other methods cannot achieve it. E.g. Are there memory benefit. Furthermore, the paper makes several extensions to the original MAMBA work in Figure 1, like down/up sampling, conv integration, and the autoencoder style architecture. The paper should makes better justification why they adopt this way of stacking modules. \n4. Since the integration of MAMBA as the neural operator is not unique, the author should perform ablation study on the alternative ways of stacking MAMBA layers. Further ablation studies like varying the different MAMBA module inherent parameters will be helpful."},"limitations":{"value":"1. In general for Neurips submission, the author should frame the paper towards generic compute scientist than specialist in this field. So, they should give clear definition, application and motivation of problem. The current intro and background are written in either very vague (related work) or techinical (mamba operator setting) manner.\n2. I am not an expert in neural operator learning. I would leave the judgment of effectiveness (e.g. figure 3) and novelty to other reviewers."}},"nonreaders":[],"tmdate":1730879645333,"tcdate":1721684100828,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission13583/Reviewer_oeYD"],"signatures":["NeurIPS.cc/2024/Conference/Submission13583/Reviewer_oeYD"],"forum":"gUEBXGV8JM","number":4,"license":"CC BY 4.0","cdate":1721684100828,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission13583/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879645333,"domain":"NeurIPS.cc/2024/Conference","replyto":"gUEBXGV8JM","id":"NMPQQyRdyS","forumContent":{"TLDR":{"value":"A neural operator that introduces a novel Mamba integration."},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Mamba","Neural Operator"]},"supplementary_material":{"value":"/attachment/b716cee883d8506aa057cb8752c87763b208ea82.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Benefiting from the booming deep learning techniques, neural operators (NO) are considered as an ideal alternative to break the traditions of solving Partial Differential Equations (PDE) with expensive cost.\nYet with the remarkable progress, current solutions concern little on the holistic function features--both global and local information-- during the process of solving PDEs.\nBesides, a meticulously designed kernel integration to meet desirable performance often suffers from a severe computational burden, such as GNO with $O(N(N-1))$, FNO with $O(NlogN)$, and Transformer-based NO with $O(N^2)$.\nTo counteract the dilemma, we propose a mamba neural operator with $O(N)$ computational complexity, namely MambaNO.\nFunctionally, MambaNO achieves a clever balance between global integration, facilitated by state space model of Mamba that scans the entire function, and local integration, engaged with an alias-free architecture. We prove a property of continuous-discrete equivalence to show the capability of\nMambaNO in approximating operators arising from universal PDEs to desired accuracy. MambaNOs are evaluated on a diverse set of benchmarks with possibly multi-scale solutions and set new state-of-the-art scores, yet with fewer parameters and better efficiency."},"_bibtex":{"value":"@inproceedings{\nzheng2024aliasfree,\ntitle={Alias-Free Mamba Neural Operator},\nauthor={Jianwei Zheng and LiweiNo and Ni Xu and Junwei Zhu and XiaoxuLin and Xiaoqin Zhang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=gUEBXGV8JM}\n}"},"title":{"value":"Alias-Free Mamba Neural Operator"},"pdf":{"value":"/pdf/a1a1561a826925c5f0083b9694af193271f8b359.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"zheng|aliasfree_mamba_neural_operator"},"authorids":{"value":["~Jianwei_Zheng2","~LiweiNo1","~Ni_Xu1","~Junwei_Zhu3","~XiaoxuLin1","~Xiaoqin_Zhang4"]},"authors":{"value":["Jianwei Zheng","LiweiNo","Ni Xu","Junwei Zhu","XiaoxuLin","Xiaoqin Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper presents a new model architecture on top of the Mamba model. Specifically, it maintains a session-specific and an external key-value memory. The authors propose some techniques for retrieving and integrating information from there two memories efficiently. Empirical results show that the proposed model can outperform both vanilla Mamba and Transformer models, while being more efficient in training and inference than vanilla attention and Native Sparse Attention."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"- There is an extra period in Line 142.\n- Can you provide the loss curves for the transformer baseline model as well?\n- Can you provide the results of Figure 5, but with a larger model (e.g., up to 10B parameters)? Also, can you provide the numbers in this figure for the vanilla Mamba model as well?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"- The method for integrating session-specific and external knowledge is novel, and some of the empirical results show that it gives considerable performance gains over the vanilla Mamba model.\n- The method is also more efficient than one variant of sparse attention in both training and inference."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The results reported in Table 1 and Figure 2 is strange and unconvincing. For such large values of perplexity (some of the values are even worse than random guessing), the model is outputting gibberish. Thus, this results do not support the claim that PUM-Net has better length generalization than the baselines. On the positive side, the perplexity is lower in general. However, this might be a result of the fact that PUM-Net has access to an external corpus, which the baselines do not have.\n- Since PUM-Net contains both attention and Mamba layers, it can be viewed as a kind of Transformer-Mamba hybrid architecture. Thus, I think it would be fair to include hybrid architectures as baselines as well.\n- According to Appendix G, PUM-Net has linear-time complexity. However, during decoding, each token needs to attend to the entire set of the session-specific memory states. Is this part linear in terms of complexity?\n- It is unclear from Figure 4 that PUM-Net has superior passkey retrieval abilities. The number of green cells appears to be similar to the baselines.\n- Most of the empirical results in the paper is based on a 130M model, which is way too small compared to the ones used in real applications. Moreover, it seems in Section 4.3 that the authors have also finetuned a 2.8B Mamba model. Why are the experiments prior to this section conducted with the 130M model instead of the larger 2.8B model?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926745315,"tcdate":1761732034369,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16691/Reviewer_CVkt"],"signatures":["ICLR.cc/2026/Conference/Submission16691/Reviewer_CVkt"],"forum":"15KpLriUTU","number":2,"license":"CC BY 4.0","cdate":1761732034369,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16691/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926745315,"domain":"ICLR.cc/2026/Conference","replyto":"15KpLriUTU","id":"ipUzd1VoFq","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["State Space Model","Long Context","Internal Memory","External Memory"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Recent Mamba-family State Space Models (SSMs) leverage linear recurrent dynamics to achieve efficiency in long-sequence modeling. However, their exponential decay kernels inevitably cause rapid forgetting, making it difficult to recall information once sequence lengths far exceed the training horizon. Inspired by advances in Transformer-based architectures such as Native Sparse Attention (NSA), which employ internal chunk memory to alleviate long-term forgetting, we extend Mamba with a chunk-wise internal long-term memory that improves the retrieval of distant context. While this enhances in-context recall, a more fundamental challenge for long-context models is the ability to access and integrate vast external world knowledge without compromising efficiency. Existing retrieval-augmented generation (RAG) approaches attempt to address this by appending retrieved documents to queries, which substantially increases training cost and fails to fully integrate internal and external memory representations. To overcome these limitations, we propose the Plastic Unified Memory Network (PUM-Net), a unified dual-memory architecture that, for the first time, enables joint pre-training over both dynamic internal memory and static, pre-encoded external knowledge. This plastic unification allows external memory to refine internal states during training, enabling bidirectional interaction without inflating sequence length, thereby supporting more effective long-context modeling and achieving substantial improvements across long-range challenging benchmarks."},"_bibtex":{"value":"@misc{\nliu2026pumnet,\ntitle={{PUM}-Net: Plastic Unified Memory Network with Associative Interaction for Long-Context State Space Models},\nauthor={Yuchen Liu and Yinghao Zhang and Deshu Chen and Hongwei Zhang and Yichi Zhang and Zhijian Zhou and Xin Guo and Yuan Cheng and Chao Qu and Yuan Qi},\nyear={2026},\nurl={https://openreview.net/forum?id=15KpLriUTU}\n}"},"title":{"value":"PUM-Net: Plastic Unified Memory Network with Associative Interaction for Long-Context State Space Models"},"pdf":{"value":"/pdf/cf765fd6e34b8efc3d037179daea5b3566db079a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"liu|pumnet_plastic_unified_memory_network_with_associative_interaction_for_longcontext_state_space_models"},"authorids":{"value":["~Yuchen_Liu26","~Yinghao_Zhang2","~Deshu_Chen2","~Hongwei_Zhang3","~Yichi_Zhang3","~Zhijian_Zhou2","~Xin_Guo2","~Yuan_Cheng1","~Chao_Qu3","~Yuan_Qi2"]},"authors":{"value":["Yuchen Liu","Yinghao Zhang","Deshu Chen","Hongwei Zhang","Yichi Zhang","Zhijian Zhou","Xin Guo","Yuan Cheng","Chao Qu","Yuan Qi"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["attention","vision transformer","dynamic network"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Existing research largely attributes the global sequence modeling capability of Transformers to the explicit computation of attention weights, a process that inherently incurs quadratic computational complexity. In this work, we offer a different perspective: we demonstrate that attention can be mathematically reframed as a Multi-Layer Perceptron (MLP) equipped with dynamically predicted parameters. Through this lens, we explain attention's global modeling power not as explicit token-wise aggregation, but as an implicit process where dynamically generated parameters act as a compressed representation of the global context. Inspired by this insight, we investigate a fundamental question: can we achieve Transformer-level sequence global modeling entirely through dynamic parameterization while maintaining linear complexity, effectively replacing explicit attention? To explore this, we design various dynamic parameter prediction strategies and integrate them into standard network layers. Extensive empirical studies on vision models demonstrate that dynamic parameterization can indeed serve as a highly effective, linear-complexity alternative to explicit attention, offering a promising paradigm for efficient sequence modeling."},"_bibtex":{"value":"@inproceedings{\nanonymous2026lineartime,\ntitle={Linear-Time Global Visual Modeling without Explicit Attention},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KGft5lIHeq},\nnote={under review}\n}"},"title":{"value":"Linear-Time Global Visual Modeling without Explicit Attention"},"pdf":{"value":"/pdf/f1224614bbdfce45c0f15c4954d4aa8fffda02a4.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791221132820,"tcdate":1788012484628,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission6154/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission6154/Authors"],"forum":"KGft5lIHeq","license":"CC BY 4.0","number":6154,"cdate":1788012484628,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/Submission6154/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791221132820,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"KGft5lIHeq","version":2},{"content":{"summary":{"value":"This paper introduces ProteinAdapter, a Mamba-based approach for protein representation learning. It leverages the concept of integrating 3D structure and 1D sequence information using a Mamba-based Pre-alignment Module and multiple Mamba Fusion Blocks, which combine embeddings from pre-trained Protein Language Models and Protein Structure Models. The authors design a Multi-Scale Predictor for downstream tasks based on the characteristics of proteins. The paper validates the ProteinAdapter on benchmarks such as PEER and ATOM3D in both single-task and multi-task learning compared to previous works."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"* Why are all the tasks from the PEER Benchmark in single-task learning not included in Table 1?\n* Regarding efficiency, could the authors provide data on training time and inference time?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"* This represents an innovative approach to utilizing Mamba as the foundation for fusion module, which is known for its computational efficiency. By incorporating frozen pre-trained models and a Mamba-based fusion module, the overall model achieves a high level of efficiency.\n* Results in Table 6 demonstrate it achieving better performances with fewer trainable parameters and higher training efficiency.\n* The concept of a multi-scale predictor based on the biological properties of proteins is well motivated."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The experimental setup of the model contains significant omissions. According to 4.2, if the structure input is replaced with the output from layer of sequence embedding, the resulting data fails to demonstrate the effectiveness of the fusion module in integrating real structural and sequence embeddings, rendering the experiment invalid. It would be essential to align the data with a 3D dataset such as the Protein Data Bank (PDB), or generating pseudo structure using pre-trained structure prediction model as written in 4.2, and re-evaluate the benchmark for \"ProteinAdapter (w/o structure)\" to enhance credibility.\n* If the original configuration, which includes both sequence and structural inputs, is maintained, the auxiliary tasks in the multi-task learning framework and the Fold Classification task in Table 2 are structure-related tasks, raising concerns about data leakage. This is evident in Table 5, where the performance gains of the two tasks are not on the same scale.\n* The contributions and innovations of this work are not particularly substantial, as the patterns of integrating 3D and 1D data are not breakthroughing (as several 3D + 1D models compared within the text).\n* This paper contains several writing issues, such as \"bold\" being misspelled as \"blod\" in all figure captions, and two arrows missing in the merge section of Figure 3. Additionally, the description of the dataset is quite vague, such as it does not clearly state the statistical information of the dataset and not fully listing its sources."}},"nonreaders":[],"tmdate":1731428247945,"tcdate":1729922242022,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9568/Reviewer_eQZK"],"signatures":["ICLR.cc/2025/Conference/Submission9568/Reviewer_eQZK"],"forum":"jqx5XI4Yr3","number":2,"license":"CC BY 4.0","cdate":1729922242022,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9568/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428247945,"domain":"ICLR.cc/2025/Conference","replyto":"jqx5XI4Yr3","id":"yBT1HzuNVt","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"TLDR":{"value":"The first Mamba-based adapter exploring the parameter efficient fine-tuning of pre-trained large protein models."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Protein Representation","Structured State Space Models"]},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The study of proteins is crucial in various scientific disciplines, but understanding their intricate multi-level relationships remains challenging. Inspired by the sequence and structure understanding of Large Protein Models (LPMs), we introduce a new ProteinAdapter, to efficiently transfer the broad knowledge encapsulated in multiple LPMs, e.g., ESM-1b, to task-specific insights. ProteinAdapter could largely save labor-intensive analysis on the 3D position and the amino acid order. Specifically, (1) with a modest number of additional parameters, ProteinAdapter facilitates multi-level protein representation learning by integrating both sequence and geometric structure embeddings from LPMs; (2) based on the learned embedding, we further scale up the proposed ProteinAdapter to various tasks with a unified Multi-Scale Predictor, which optimally harnesses the learned embeddings through task-specific attention. Albeit simple, the proposed method is scalable to multiple downstream tasks without bells and whistles. Extensive experiments on over 20 tasks show that ProteinAdapter outperforms state-of-the-art methods under both single-task and multi-task scenarios."},"_bibtex":{"value":"@misc{\nwang2024proteinadapter,\ntitle={ProteinAdapter: Adapting Pre-trained Large Protein Models for Efficient Protein Representation Learning},\nauthor={Chao Wang and Zhedong Zheng and Yifan Sun and Hehe Fan and Yi Yang},\nyear={2024},\nurl={https://openreview.net/forum?id=jqx5XI4Yr3}\n}"},"title":{"value":"ProteinAdapter: Adapting Pre-trained Large Protein Models for Efficient Protein Representation Learning"},"pdf":{"value":"/pdf/94b4ab19fcbc6f52e6476d8c84e16970dfea192b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"wang|proteinadapter_adapting_pretrained_large_protein_models_for_efficient_protein_representation_learning"},"authorids":{"value":["~Chao_Wang31","~Zhedong_Zheng1","~Yifan_Sun2","~Hehe_Fan1","~Yi_Yang22"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chao Wang","Zhedong Zheng","Yifan Sun","Hehe Fan","Yi Yang"]}},"version":2},{"content":{"summary":{"value":"This work proposes an extension of the Mamba State Space architecture that is inspired by contemporary neuroscientific models of the hippocampus. The architecture introduces additional pathways that feed into an adapted SSM module. The experiments evaluate the architecture on multiple benchmark tasks in comparison to Mamba and include an ablation study into the role of the individual modules."},"soundness":{"value":1},"confidence":{"value":3},"questions":{"value":"The introduction states that the \"trial-and-error process [of AI design], while powerful, frequently lacks a foundation in first principles and leaves interpretability a persistent challenge\". However, is that not how evolutionary search processes work? I am not sure what \"first principles\" went into the design of the hippocampus."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"Establishing potential architectural links between neural circuitry and state-space models is thought-provoking and may lead to new insights for the design of next-generation architectures. The work is well motivated and makes interesting connections to the neuroscientific literature\nDespite the more complex architecture, Table 2 demonstrates GPU-based training with comparable efficiency to Mamba."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The manuscript is using a model of the hippocampus as \"blueprint\" for the proposed machine learning architecture but it does not clearly separate between sources of inspiration, hypothesis, and empirical validation of the resulting approach. \n\nSection 3 makes it difficult to understand what is established Mamba practice and what is novel. The mix of neuroscientific language (e.g. \"mossy fiber\") and machine learning terminology (e.g. \"projection\") makes it hard to discern what is description and what is motivation or inspiration. \n\nLeaving aside the fact that the evoked contemporary theories of hippocampal computation are far from settled, even if we knew exactly how the hippocampus worked, it would not follow that the same architecture would be a good idea to implement on a modern GPU. It is indeed unlikely that the von Neumann cache hierachy illustrated in Figure 3 suits hippocampal computation (and vice versa). In fact, section 3.3 notes that the models recurrent relations are difficult to parallize and require custom kernels. This means that the performance gains of a proposed architecture must justify the downside of difficult parallelization. However, the presented experimental results, while promising, do not provide a rigorous comparison with prior work. The only baseline in Mamba but there is no comparison with alternative hybrid approaches like Jamba (Lenz+2025) and their respective baselines.\n\nMoreover, the ablation study does not shed much light on the function of the circuit. Notably, removing the DG pathway slightly improved performance but it is not clear why. The offered hypothesis that the module is not required for simple signal filtering is one possibility but further experimentation would be required to firmly establish when including the module is a good idea.\n\nFinally, Section 4.3.3. teases that a \"success of this approach has opened a new research avenue\" but leaves the  \"specifics of which [to be] detailed in a forthcoming publication\". Without further evidence, it is not possible to validate this claim and I recommend removing it until it can be fully explained in the planned publication.\n\n**Additional comments**\n\n- The work cites  Mamba as Gu & Dao, 2023 but it was published in 2024. Please ensure that you cite the published version over preprints.\n- Not to get too hung up on words but to \"propose a paradigm shift\" (in the Kuhn sense?) might be overstating the case for what amounts to an architectural innovation of an existing deep learning model.\n- I recommend to consider changing the title to be more descriptive of the work. It is currently impossible to guess from the title what the work is about."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922047087,"tcdate":1761770001077,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10846/Reviewer_qnSj"],"signatures":["ICLR.cc/2026/Conference/Submission10846/Reviewer_qnSj"],"forum":"clCsSQ5rKg","number":2,"license":"CC BY 4.0","cdate":1761770001077,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10846/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922047087,"domain":"ICLR.cc/2026/Conference","replyto":"clCsSQ5rKg","id":"AOZi4uv5dM","forumContent":{"TLDR":{"value":"We reframe Mamba as an incomplete model of the hippocampus and demonstrate that a more neuro-faithful architecture unlocks superior algorithmic reasoning, emergent robustness, and profound biological fidelity."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["State Space Models","Computational Neuroscience","Hippocampal Modeling","Interpretability","Architectural Priors"]},"supplementary_material":{"value":"/attachment/a2ecf359bc475d91326c5bba6dcd339f63cbdc98.zip"},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Grounded in the evolutionary principle that resource constraints favor structural solutions for complex computation, we propose a neuro-centric framework that reframes the success of State Space Models (SSMs), e.g. Mamba, as an unconscious convergence to an incomplete model of the hippocampus—the brain's canonical circuit for sophisticated computations such as pattern separation and completion. Motivated by this, we introduce NeuroMamba (NeuMa), a novel architecture that consciously and faithfully implements the canonical hippocampal circuit, including the dentate gyrus (DG), Cornu Ammonis 3 (CA3), and Cornu Ammonis 1 (CA1), using foundational SSM blocks. Enabled by custom kernels, our design bridges the gap between biological plausibility and practical efficiency. Experiments demonstrate that NeuMa achieves superior performance and learning efficiency on synthetic benchmarks. More critically, it exhibits profound biological fidelity by spontaneously replicating the \"orthogonalized state machine\" dynamics of the biological hippocampus. Finally, we validate its capacity for real-world scientific discovery by developing a generative agent for piezoelectric catalysis that achieves superior performance in this complex, low-resource domain, thereby showcasing a new path for AI architecture design rooted in neuroscience."},"_bibtex":{"value":"@misc{\nyang2026neuma,\ntitle={NeuMa, Born to Work},\nauthor={Yuanda Yang and HAORAN ZHANG and Yuehui Li},\nyear={2026},\nurl={https://openreview.net/forum?id=clCsSQ5rKg}\n}"},"title":{"value":"NeuMa, Born to Work"},"pdf":{"value":"/pdf/50462e129b9287a504163b7223ce661545a64d8f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yang|neuma_born_to_work"},"authorids":{"value":["~Yuanda_Yang1","~HAORAN_ZHANG45","~Yuehui_Li1"]},"authors":{"value":["Yuanda Yang","HAORAN ZHANG","Yuehui Li"]}},"version":2},{"content":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["diffusion","time series impuation","state space model"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Probabilistic time series imputation has been widely applied in real-world scenarios due to its ability to estimate uncertainty of imputation results. Meanwhile, denoising diffusion probabilistic models (DDPMs) have achieved great success in probabilistic time series imputation tasks with its power to model complex distributions. However, current DDPM-based probabilistic time series imputation methodologies are confronted with two types of challenges: 1) \\textit{ The backbone modules of the denoising parts are not capable of achieving sequence modeling with low time complexity.} 2) \\textit{ The architecture of denoising modules can not handle the inter-variable and bidirectional dependencies in the time series imputation problem effectively.} To address the first challenge, we integrate the computational efficient state space model, namely Mamba, as the backbone denosing module for DDPMs. To tackle the second challenge, we carefully devise several SSM-based blocks for bidirectional modeling and inter-variable relation understanding. Experimental results demonstrate that our approach can achieve state-of-the-art time series imputation results on multiple datasets, different missing scenarios and missing ratios."},"_bibtex":{"value":"@misc{\ngao2025diffimp,\ntitle={DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone},\nauthor={Hongfan Gao and Wangmeng Shen and Xiangfei Qiu and Ronghui Xu and Jilin Hu and Bin Yang},\nyear={2025},\nurl={https://openreview.net/forum?id=j1OucVFZMJ}\n}"},"title":{"value":"DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone"},"pdf":{"value":"/pdf/26faf0d7208528fa5a326d36fadf37ed808e0f38.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"gao|diffimp_efficient_diffusion_model_for_probabilistic_time_series_imputation_with_bidirectional_mamba_backbone"},"authorids":{"value":["~Hongfan_Gao1","~Wangmeng_Shen1","~Xiangfei_Qiu1","~Ronghui_Xu2","~Jilin_Hu1","~Bin_Yang4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hongfan Gao","Wangmeng Shen","Xiangfei Qiu","Ronghui Xu","Jilin Hu","Bin Yang"]}},"tmdate":1738735648433,"tcdate":1726924547106,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2397/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission2397/Authors"],"forum":"j1OucVFZMJ","license":"CC BY 4.0","number":2397,"cdate":1726924547106,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission2397/-/Full_Submission","ICLR.cc/2025/Conference/Submission2397/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735648433,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"j1OucVFZMJ","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.13338v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"gao|diffimp_efficient_diffusion_model_for_probabilistic_time_series_imputation_with_bidirectional_mamba_backbone"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Hongfan_Gao:","https://dblp.org/search/pid/api?q=author:Wangmeng_Shen:","https://dblp.org/search/pid/api?q=author:Xiangfei_Qiu:","~Ronghui_Xu2","https://dblp.org/search/pid/api?q=author:Jilin_Hu:","https://dblp.org/search/pid/api?q=author:Bin_Yang_0002:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.13338"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-13338,\n  publtype={informal},\n  author={Hongfan Gao and Wangmeng Shen and Xiangfei Qiu and Ronghui Xu and Jilin Hu and Bin Yang},\n  title={DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.13338},\n  url={https://doi.org/10.48550/arXiv.2410.13338}\n}\n"},"abstract":{"value":"Probabilistic time series imputation has been widely applied in real-world scenarios due to its ability to estimate uncertainty of imputation results. Meanwhile, denoising diffusion probabilistic models (DDPMs) have achieved great success in probabilistic time series imputation tasks with its power to model complex distributions. However, current DDPM-based probabilistic time series imputation methodologies are confronted with two types of challenges: 1)~\\textit{~The backbone modules of the denoising parts are not capable of achieving sequence modeling with low time complexity.} 2)~\\textit{The architecture of denoising modules can not handle the inter-variable and bidirectional dependencies in the time series imputation problem effectively.} To address the first challenge, we integrate the computational efficient state space model, namely Mamba, as the backbone denosing module for DDPMs. To tackle the second challenge, we carefully devise several SSM-based blocks for bidirectional modeling and inter-variable relation understanding. Experimental results demonstrate that our approach can achieve state-of-the-art time series imputation results on multiple datasets, different missing scenarios and missing ratios."},"title":{"value":"DiffImp: Efficient Diffusion Model for Probabilistic Time Series Imputation with Bidirectional Mamba Backbone"},"authors":{"value":["Hongfan Gao","Wangmeng Shen","Xiangfei Qiu","Ronghui Xu","Jilin Hu","Bin Yang"]}},"tmdate":1735049085270,"pdate":1704067200000,"tcdate":1735049074931,"writers":["~"],"signatures":["~Ronghui_Xu2"],"forum":"YXIjxcndBv","license":"CC BY-SA 4.0","number":259375,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1735049085270,"domain":"DBLP.org","id":"YXIjxcndBv","version":2},{"content":{"summary":{"value":"This paper introduces Mesanet, which is a linear transformer based on the Mesa optimal regression update rule. The model is inspired by test-time training and the fact that the state update of linear transformers can be seen as an online learning objective. The model is sound, and the chunkwise parallel training makes the model efficient to train; however, the training is more time-consuming than other linear transformers such as GLA and Deltanet due to the solver update. The empirical results are extensive, and the model is evaluated on many different downstream language modeling tasks and length generalization."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"My most concerns are mentioned above and I mainly have 3 small questions:\n\n**1)** What is the effect of convoulution on q,k,v in Mesanet as it is a cruical component for Deltanet and Gated Deltanet and Mamba2. Also, since it has conections to the TTT paradigm as stated in: https://kexue.fm/archives/11320\n\n**2)** Why does $\\Lambda_t = \\frac{(1-\\gamma_t)}{\\beta_t}$ structured as so?\n\n**3)** How are forget gates $\\beta_t$ and $\\gamma_t$ defined and extracted? Specifically, what nonlinearities and feature mappings are used for them? Are they similar to those in Mamba-2, or do they use a temperatured sigmoid as in GLA or GSA?\n\nLastly, I am happy to increase my score if above presentation issues and clarifications regarding the paper are resolved."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"The paper have several strengths:\n\n- **Model design**: The model design of Mesanet and the connection to test time training is elegant.\n\n- **Experimental setup:** The experimental setup for pre-training in language modeling tasks are sound and followed by the stablished setups such as [1,2].\n\n- **Fairness of comparison:** The paper tunes the learning rate for all different models which isa significant plus point as studies such as Gated Deltanet [3] specifically use the same learning rate for all models, which can indeed be not fair comparison between baselines.\n\n- **Chunk-wise parallel support for training:** In general the training paradigm and chunk wise form is designed nicely and helps parallel training.\n\n\n\n\n\n\n\n\n\n\n\n------\n\n###  References\n\n[1] Gated Linear Attention Transformers with Hardware-Efficient Training: Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim\n\n[2] Parallelizing Linear Transformers with the Delta Rule over Sequence Length: Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, Yoon Kim\n\n[3] Gated Delta Networks: Improving Mamba2 with Delta Rule: Songlin Yang, Jan Kautz, Ali Hatamizadeh"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The main weakness of the paper lies in its presentation, which significantly reduces clarity regarding the approach and design. These issues can be summarized as follows:\n\n**1)** **Mesanet's online learning objective** is significantly under-explained, starting from Equation 4:\n\n$$\n\\mathcal{L}(\\Phi) = \\frac{1}{2} \\sum^t_{\\tau=1} |v_\\tau - \\Phi k_\\tau|^2 + \\frac{\\text{Tr} (\\Phi^\\top \\Lambda \\Phi)}{2}\n$$\n\nBy taking the gradient and setting it to zero, we arrive at:\n\n$$\n\\sum^t_{\\tau=1} (v_\\tau - \\Phi k_\\tau)k^\\top_\\tau + \\Lambda \\Phi = 0, \\quad\n\\sum^t_{\\tau=1} v_\\tau k^\\top_\\tau - \\Phi \\sum^t_{\\tau=1} k_\\tau k^\\top_\\tau + \\Lambda \\Phi = 0 \\quad \\text{(Eq. 1)}\n$$\n\nThe above result corresponds to the optimal $\\Phi$. Since in an autoregressive regime the summations above can be expressed recursively, one can write the following recurrences:\n\n$$\nG_\\tau = G_{\\tau-1} + v_\\tau k^\\top_\\tau, \\quad\nH_\\tau = H_{\\tau-1} + k_\\tau k^\\top_\\tau\n$$\n\nFrom here, one can substitute the states $G_t$ and $H_t$ into Equation (1) to obtain the optimal Mesa update rule:\n$\\Phi = G_t(H_t + \\Lambda_t)^{-1}$.\n\nMoreover, by including forget gates in the recurrences above, the online learning objective will also change, which is inconsistent with the paper’s notation. I suggest reformulating the online learning loss (Equation 4 in the paper) to explicitly include these forget gates, as also described in *Gated DeltaNet’s Table 1* [1]. Moreover, is $\\Lambda_t$ time-dependet or not since in equation 4 of paper is suggested as not but exactly on paragraph bellow it is depending on forget gates.\n\n----\n\n**2) The full recurrence of MesaNet is not shown and not compared with other baselines.**\nI recommend moving the recurrence equation of MesaNet (currently on page 21) into the main body of the paper, along with Table 2 for clarity and comparison with other methods.\n\n---\n\n**3) Motivation behind the Mesa layer.**\nSince the main motivation is the connection to test-time training (TTT), I wonder why one should use MesaNet instead of DeltaNet or Gated DeltaNet. My main question is: given that DeltaNet and Gated DeltaNet leverage TTT to improve recall abilities, and indeed show significantly better recall, does MesaNet outperform them in this aspect? MesaNet introduces additional complexity in both architecture and training. As Table 14 suggests, if the motivation for this design is to enhance recall ability, this should be clearly stated and justified in the paper.\n\n---\n\n**4) Hawk-Mesa underperforms significantly compared to many baselines and even MesaNet itself in recall tasks (Table 14).**\nIf the main motivation behind building a model based on the Mesa layer is to introduce an optimal way to structure memory (as suggested by the TTT framework), why does Hawk-Mesa perform substantially worse than MesaNet in Table 14? Why should one construct such a model if it does not improve recall? I ask this because I believe recall ability is one of the primary motivations behind the design of MesaNet and the TTT framework in general.\n\n---\n\n**5) Tables and plots are not clearly visible.**\nMany numerical results presented as bar plots are difficult to read. I highly recommend replacing them with tables, particularly Figure 5, to improve clarity.\n\n---\n\n**6) Missing baselines: Atlas and Titans.**\nI believe the baselines **Atlas** and **Titans** are missing from the paper’s comparisons. These models address a very similar problem and propose closely related solutions, especially Titans, which should be included for a fair evaluation.\n\n\n---\n\n### References\n\n[1] Gated Delta Networks: Improving Mamba2 with Delta Rule: Songlin Yang, Jan Kautz, Ali Hatamizadeh"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764358047837,"tcdate":1761593194916,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18821/Reviewer_mKy1"],"signatures":["ICLR.cc/2026/Conference/Submission18821/Reviewer_mKy1"],"forum":"xa3OnTb6c3","number":1,"license":"CC BY 4.0","cdate":1761593194916,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission18821/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764358047837,"domain":"ICLR.cc/2026/Conference","replyto":"xa3OnTb6c3","id":"OfXhSUUzfa","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Sequence modeling","test-time training","RNN transformer alternatives"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compute linearly during inference. A recent stream of work linearized the softmax operation, resulting in powerful recurrent neural network (RNN) models with constant memory and compute costs such as DeltaNet, Mamba or xLSTM. These models can be unified by noting that their recurrent layer dynamics can all be derived from an in-context regression objective, approximately optimized through an online learning rule. Here, we join this line of work and introduce a numerically stable, chunkwise parallelizable version of the recently proposed Mesa layer (von Oswald et al., 2024), which could only run sequentially in time and was therefore not scalable. This layer again stems from an in-context loss, but which is now minimized to optimality at every time point using a fast conjugate gradient solver. Through an extensive suite of experiments study up to the billion-parameter scale, we show that optimal test-time training enables reaching lower language modeling perplexity and higher downstream benchmark performance than previous RNNs, especially on tasks requiring long context understanding. This performance gain comes at the cost of additional flops spent during inference time. Our results are therefore intriguingly related to recent trends of increasing test-time compute to improve performance -- here by spending compute to solve sequential optimization problems within the neural network itself."},"_bibtex":{"value":"@inproceedings{\noswald2026mesanet,\ntitle={MesaNet: Sequence Modeling by Locally Optimal Test-Time Training},\nauthor={Johannes von Oswald and Nino Scherrer and Seijin Kobayashi and Luca Versari and Songlin Yang and Maximilian Schlegel and Kaitlin Maile and Yanick Schimpf and Oliver Sieberling and Alexander Meulemans and Guillaume Lajoie and Rif A. Saurous and Charlotte Frenkel and Razvan Pascanu and Blaise Aguera y Arcas and Joao Sacramento},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=xa3OnTb6c3}\n}"},"title":{"value":"MesaNet: Sequence Modeling by Locally Optimal Test-Time Training"},"pdf":{"value":"/pdf/55189e529b3da4e2e90fd1988db5f2bb30a909ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"oswald|mesanet_sequence_modeling_by_locally_optimal_testtime_training"},"authorids":{"value":["~Johannes_von_Oswald2","~Nino_Scherrer1","~Seijin_Kobayashi1","~Luca_Versari1","~Songlin_Yang1","~Maximilian_Schlegel1","~Kaitlin_Maile1","~Yanick_Schimpf1","~Oliver_Sieberling1","~Alexander_Meulemans1","~Guillaume_Lajoie1","~Rif_A._Saurous1","~Charlotte_Frenkel1","~Razvan_Pascanu1","~Blaise_Aguera_y_Arcas1","~Joao_Sacramento1"]},"authors":{"value":["Johannes von Oswald","Nino Scherrer","Seijin Kobayashi","Luca Versari","Songlin Yang","Maximilian Schlegel","Kaitlin Maile","Yanick Schimpf","Oliver Sieberling","Alexander Meulemans","Guillaume Lajoie","Rif A. Saurous","Charlotte Frenkel","Razvan Pascanu","Blaise Aguera y Arcas","Joao Sacramento"]}},"version":2},{"content":{"comment":{"value":"> **W1: The term \"information classification\" is confusing and does not clearly reflect the intended mechanism.**\n\nThank you for the insightful suggestion. As you pointed out, the mechanism can indeed be regarded as a row-selection process. We will therefore reduce the emphasis on the term \"classification\" and refine the terminology to improve clarity for readers.\n\nIn Section 3, we introduce the *row-sparse state update* framework, where the current input is assigned to a subset of state rows and only the selected rows are updated. Our original motivation for using the term *information classification* was to convey that different rows can be interpreted as latent categories, emphasizing the role of $f(\\mathbf{x}_t, \\mathbf{W}_k)$ as a classification function. This naturally motivates the *Softmax-top-$k$ hard classification* mechanism introduced in Section 3.2.\n\nIn the revised version, we will preserve the conceptual intuition where needed, but describe the operation primarily as row selection to ensure clarity.\n\n---\n\n> **W2: The paper misses important baselines such as Mamba/Mamba2.**\n\nBased on your suggestions, we have included several additional baselines to provide a more comprehensive comparison. The results show that **SSE continues to deliver the strongest performance**, particularly on recall-intensive tasks such as FDA.\n\n1. At the 600M-parameter & 15B-token scale, we trained Mamba [1], Mamba2 [2] , and KDA [3] (a recent linear attention method from Moonshot). We also evaluated the combination of SSE with GDN, referred to as SSE-GDN (see our response to Reviewer Jg4D Weakness 1 for additional details).\n\n| Model            | Wiki↓ | PIQA  | Hella. | Wino. | ARC-e | ARC-c | SIQA  | **CSR-Avg.** | -  | FDA   | SWDE  | SQuAD | **Recall-Avg.**  |\n|------------------|-------|-------|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| Transformer      | 33.47 | 61.92 | 31.60  | 51.38 | 48.02 | 23.12 | 37.31 | 42.22 | -  | 74.50 | 60.67 | 32.67 | 55.95 |\n| GLA              | 43.97 | 61.21 | 29.61  | 51.46 | 47.01 | 23.29 | 36.59 | 41.53 | -  | 9.44  | 23.40 | 23.06 | 18.63 |\n| Mamba            | 36.62 | 63.82 | 32.20  | 50.43 | 49.62 | 23.38 | 38.59 | 43.01 | -  | 6.35  | 21.60 | 24.93 | 17.63 |\n| Mamba2           | 34.89 | 62.89 | 32.84  | 48.93 | 49.20 | 23.81 | 37.92 | 42.60 | -  | 20.51 | 30.42 | 27.21 | 26.05 |\n| KDA              | 34.76 | 62.19 | 31.81  | 51.22 | 49.96 | 24.83 | 37.41 | 42.90 | -  | 24.50 | 35.55 | 26.71 | 28.92 |\n| GDN              | 35.26 | 62.79 | 32.16  | 51.85 | 50.46 | 24.23 | 36.80 | **43.05** | -  | 14.79 | 33.48 | 26.24 | 24.84 |\n| **SSE-n4k1** (Ours)     | 35.52 | 61.59 | 31.71  | 51.62 | 49.45 | 24.74 | 38.33 | 42.91 | -  | 33.67 | 33.30 | 26.51 | **31.16** |\n| **SSE-n4k2** (Ours)     | 35.33 | 62.95 | 31.64  | 50.43 | 49.66 | 25.34 | 38.28 | **43.05** | -  | 30.85 | 36.90 | 27.82 | **31.86** |\n| **SSE-GDN-n4k1** (Ours)   | **34.15** | 63.11| 32.22| 50.36| 50.59| 23.98| 37.46| 42.95 | -  | 41.56 | 41.13 | 30.83 | **37.84** |\n\n\n2. At the 2B-parameter & 100B-token scale, we further conducted experiments for Mamba2. \n\n| Model            | Wiki↓ | PIQA  | Hella. | Wino. | ARC-e | ARC-c | SIQA  | CSR-Avg. | -  | FDA   | SWDE  | SQuAD | Recall-Avg.  |\n|------------------|-------|-------|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| Transformer  | 16.46 | 71.82 | 52.70 | 58.80 | 63.76 | 31.31 | 42.89 | 53.55 | -  | 86.48 | 82.99 | 49.53 | 73.00 |\n| GLA          | 21.30 | 68.44 | 42.82 | 54.38 | 59.43 | 28.24 | 41.45 | 49.13 | -  | 51.27 | 59.86 | 36.73 | 49.29 |\n| Mamba2       | 17.27 | 71.87 | 53.34 | 57.06 | 64.52 | 32.59 | 42.17 | 53.59 | -  | 56.62 | 63.28 | 43.06 | 54.32 |\n| GDN          | 17.08 | 72.25 | 53.20 | 57.70 | 65.61 | 32.68 | 42.89 | 54.05 | -  | 54.63 | 66.88 | 39.88 | 53.79 |\n| **SSE-n4k1** (Ours) | **16.92**  | 71.98 | 53.62 | 60.14 | 66.67 | 32.94 | 42.07 | **54.57** | -  | 71.23 | 69.94 | 43.20  | **61.46** |\n\n\n*NOTE:* It is important to note that, to ensure a fair comparison, all of our models vary only in the attention mixer and **do not** include convolutional layers—except for the Mamba family, which inherently uses 1D convolutions and unified blocks. \n\n[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. https://arxiv.org/abs/2312.00752   \n[2] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. https://arxiv.org/abs/2405.21060   \n[3] Kimi Linear: An Expressive, Efficient Attention Architecture. https://arxiv.org/abs/2510.26692  \n\n---"},"title":{"value":"[1/2] Response to Reviewer dHEC (W1~W2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763523262739,"tcdate":1763523262739,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8934/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8934/Authors"],"forum":"R6DrJ4tnGV","number":2,"license":"CC BY 4.0","cdate":1763523262739,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8934/-/Official_Comment"],"mdate":1763523262739,"domain":"ICLR.cc/2026/Conference","replyto":"ZyCXTzc2XM","id":"REczUU6wMY","forumContent":{"TLDR":{"value":"SSE is a partitioned state expansion method under a row-sparse update framework for linear attention, improving retrieval and reasoning."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Linear Attention","Language Model"]},"supplementary_material":{"value":"/attachment/6e3475d7248015327464c0fe2192bc1fb8d0142d.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The Transformer architecture, despite its widespread success, struggles with long-context scenarios due to quadratic computation and linear memory growth. While various linear attention variants mitigate these efficiency constraints by compressing context into fixed-size states, they often degrade performance in tasks such as in-context retrieval and reasoning. To address this limitation and achieve more effective context compression, we propose two key innovations. First, we introduce a row-sparse update formulation for linear attention by conceptualizing state updating as information categorization. This enables sparse state updates via softmax-based top-$k$ row selection, thereby extending receptive fields and reducing information interference. Second, we present Sparse State Expansion (SSE) within the sparse framework, which expands the contextual state into multiple partitions, effectively decoupling parameter size from state capacity while maintaining the sparse row-selection paradigm. Supported by efficient parallelized implementations, our design achieves highly discriminative state representations. We extensively validate SSE in both pure linear and hybrid (SSE-H) architectures across language modeling, in-context retrieval, and mathematical reasoning benchmarks. SSE demonstrates strong retrieval performance and scales favorably with state size. Moreover, after reinforcement learning (RL) training, our 2B SSE-H model achieves state-of-the-art mathematical reasoning performance among small reasoning models, scoring 64.5 on AIME24 and 50.2 on AIME25, significantly outperforming similarly sized open-source Transformers. These results highlight SSE as a promising and efficient architecture for long-context modeling."},"_bibtex":{"value":"@inproceedings{\npan2026scaling,\ntitle={Scaling Linear Attention Capacity with Sparse State Expansion},\nauthor={Yuqi Pan and Yongqi An and Zheng Li and Yuhong Chou and Rui-Jie Zhu and Xiaohui Wang and Mingxuan Wang and Jinqiao Wang and Guoqi Li},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=R6DrJ4tnGV}\n}"},"title":{"value":"Scaling Linear Attention Capacity with Sparse State Expansion"},"pdf":{"value":"/pdf/73d9d62767b69cd0649b8caaa9d65c613f858782.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"pan|scaling_linear_attention_capacity_with_sparse_state_expansion"},"authorids":{"value":["~Yuqi_Pan2","~Yongqi_An1","~Zheng_Li48","~Yuhong_Chou1","~Rui-Jie_Zhu2","~Xiaohui_Wang4","~Mingxuan_Wang1","~Jinqiao_Wang1","~Guoqi_Li1"]},"authors":{"value":["Yuqi Pan","Yongqi An","Zheng Li","Yuhong Chou","Rui-Jie Zhu","Xiaohui Wang","Mingxuan Wang","Jinqiao Wang","Guoqi Li"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Bellman Diffusion, a novel generative model designed to approximate both the gradient and scalar field of the data’s probability distribution. Bellman Diffusion is distinguished by its linear modeling property, meaning that if we aim to model a combined distribution f+g, we can simply sum the fields of f and g. This property is particularly advantageous for distributional reinforcement learning (RL). By first modeling the distribution of returns in terminal states, we can subsequently perform Bellman-like updates on the approximated fields, ultimately deriving the distribution for all states in the Markov Decision Process (MDP). In essence, Bellman Diffusion’s linearity makes it well-suited for distribution modeling tasks where data exhibits a linear structure. The authors validate Bellman Diffusion’s effectiveness through evaluations on low-dimensional toy tasks, high-dimensional benchmarks, and applications in distributional RL."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"One concurrent work [1] also seems to optimize the diffusion model in a temporal difference manner, and they used existing diffusion models, rather than proposing a new SDE with linear operators as in Bellman Diffusion. Although I understand that [1] deals with the successor measure instead of value distribution, could the authors briefly discuss about the difference or relationship between [1] and Bellman Diffusion? \n\n[1] Liam Schramm and Abdeslam Boularias. Bellman Diffusion Models."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The motivation behind this paper is both clear and compelling. In reinforcement learning, many distributions exhibit inherent structures that can be effectively described through Bellman iteration. Beyond the return or value distribution, other examples include the successor state distribution, which describes the probability of transitioning to a state s' at any time after taking an action a in a state s. Learning these distributions—even with powerful generative models—can be challenging if the data’s inherent structure isn’t properly respected. In this regard, Bellman Diffusion offers a practical solution, enabling the modeling of such structured distributions in a way that aligns with their underlying characteristics."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The presentation of the paper is poor. Since the authors motivate their approach through its application in distributional reinforcement learning, it would be more effective to explain how Bellman Diffusion can be integrated into and facilitate distributional RL (currently in Appendix D) directly within the main method section.\n\nThe evaluation of the proposed method is quite limited in both comparisons with existing methods and also the explanations about the performance. \n+ In Section 6.1, the authors primarily demonstrate that Bellman Diffusion can model synthetic data distributions (though there are apparent modeling inconsistencies). Including comparisons with established baseline methods, such as DDPM, would provide a clearer picture of the advantages Bellman Diffusion offers.\n+ In Section 6.2, given that the tasks lack internal structures, it raises the question of why Bellman Diffusion outperforms DDPM in certain tasks. Further explanation here would clarify the observed performance benefits and also certain claims made in line 431. Regarding Section 6.3, the benchmark environments (FrozenLake and CartPole) are too simple to truly assess the method’s effectiveness. These environments have relatively small state and action spaces, leading to straightforward return distributions. In contrast, existing methods like C51 are validated on more complex tasks, such as those in Atari environments, which exhibit greater structural complexity. In light of this, I think it would be necessary to extend the evaluations to more complex environments to truly reflect the effectiveness of the proposed method."}},"nonreaders":[],"tmdate":1731428232580,"tcdate":1730706495887,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6909/Reviewer_57hc"],"signatures":["ICLR.cc/2025/Conference/Submission6909/Reviewer_57hc"],"forum":"kIqA447T5c","number":2,"license":"CC BY 4.0","cdate":1730706495887,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6909/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428232580,"domain":"ICLR.cc/2025/Conference","replyto":"kIqA447T5c","id":"4GQEOrZErG","forumContent":{"TLDR":{"value":"The first Deep Generative Model for Markov Decision Processes, such as Planning and Distributional Reinforcement Learning"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Deep Generative Models; Markov Decision Processes; Foundation of Distributional Reinforcement Learning"]},"supplementary_material":{"value":"/attachment/2cccd23943410f30efaae6f541c6030b1952483c.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Deep Generative Models (DGMs), including Score-based Generative Models, have made significant progress in approximating complex continuous distributions. However, their application to Markov Decision Processes (MDPs), particularly in distributional Reinforcement Learning (RL), is underexplored. The field remains dominated by classical histogram-based methods, which suffer from discretization errors, leading to instability and slower convergence. This work highlights that this gap stems from the nonlinear operators used in modern DGM's modelings, which map neural network functions to the target distribution. These nonlinearities conflict with the linearity required by the Bellman equation, which relates the return distribution of a state to a linear combination of future states' return distributions. To address this, we introduce Bellman Diffusion, a new DGM that preserves the necessary linearity by modeling both the gradient and scalar fields. We propose a novel divergence-based training technique to optimize neural network proxies and introduce a new stochastic differential equation for sampling. With these innovations, Bellman Diffusion is guaranteed to converge to the target distribution. Our experiments show that Bellman Diffusion not only achieves accurate field estimations and serves as an effective image generator, but also converges $1.5\\times$ faster than traditional histogram-based baselines in distributional RL tasks. This work paves the way for the effective integration of DGMs into MDP applications, enabling more advanced decision-making frameworks."},"_bibtex":{"value":"@misc{\nli2025bellman,\ntitle={Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space},\nauthor={Yangming Li and Chieh-Hsin Lai and Carola-Bibiane Sch{\\\"o}nlieb and Yuki Mitsufuji and Stefano Ermon},\nyear={2025},\nurl={https://openreview.net/forum?id=kIqA447T5c}\n}"},"title":{"value":"Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space"},"pdf":{"value":"/pdf/488f28556d28dd33ed5549306a3b9d8e040a9134.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"li|bellman_diffusion_generative_modeling_as_learning_a_linear_operator_in_the_distribution_space"},"authorids":{"value":["~Yangming_Li1","~Chieh-Hsin_Lai2","~Carola-Bibiane_Schönlieb1","~Yuki_Mitsufuji1","~Stefano_Ermon1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yangming Li","Chieh-Hsin Lai","Carola-Bibiane Schönlieb","Yuki Mitsufuji","Stefano Ermon"]}},"version":2},{"content":{"comment":{"value":"We are grateful for Reviewer LroV's efforts and time on reviewing our work. We have taken your valuable suggestions and carefully revise our manuscript to incoporate some initial solutions for mitigating the stated problems. Please see our responses to your questions:\n\n**1. Discussion on possible methods for mitigation.**\n\nWe thank Reviewer LroV for this thoughtful suggestion. We added the following dicussion on the potential solutions based on our theoretical results.\n\nAs our theory suggests, the maximal and minimal values in $A_t$ directly governs the locality and over-smoothing issues in SSMs, respectively. Encouraging the diversity of values in $A_t$ is thus crucial. While $A_{max} \\approx 1$ and $A_{min} \\approx 0$ could could theoretically occur simultaneously, our new experiments in Fig. 6 reveal that this does not happen in practice, as the values in $A_t$ are often concentrated within a narrow range. A simple yet promising solution to address this issue is to fix one component of $A_t$ as a constant 1, another as a constant 0, and others freely learnable. This approach ensures that one dimension in the state memory consistently focuses on the current token, counteracting over-smoothing by preventing mixing with previous tokens. Simultaneously, another dimension exclusively retains information from past tokens, avoiding locality issues by preserving the complete history. We are currently in the process of empirically validating this simple solution and will notify the reviewer and update the draft as soon as we have the results.\n\n**2. Discussion comparing recency and vanishing gradients.**\n\nWe appreciate Reviewer LroV's comments on connecting our theory with vanishing gradients. Vanishing gradients refer to a challenge in RNNs, where backpropagation-based learning is impeded due to gradient magnitudes decaying exponentially over time. The diminishing dependencies among distant tokens are a fundamental cause of this issue [1]. SSMs were initially proposed to address this limitation by explicitly modeling long-range dependencies, as highlighted in [2, 3]. Subsequent work, such as Mamba, extends this approach by adopting their initialization alongside newly proposed selection mechanisms, which are widely believed to enhance these capabilities. **Our Theorem 3.1 (formerly Theorem 3.2) lies in theoretically challenging this claim.** We demonstrate that modern SSMs still suffer from the recency bias, which not only undermines their ability to capture long-term dependencies but also potentially exacerbates the vanishing gradient problem. We have included this discussion to Appendix B of our revision.\n\n[1] Bengio et al., Learning long-term dependencies with gradient descent is difficult\n\n[2] Gu et al., Hippo: Recurrent memory with optimal polynomial projections\n\n[3] Gu et al., Combining recurrent, convolutional, and continuous-time models with linear state space layers.\n\n**3. The experimental details for Section 4.1 are lacking.**\n\nOur scaling experiments are aligned with the setting adopted in Mamba (see Appendix E.2.1 in [1]). Specifically, in each curve of Fig. 4, we vary the size of models from 140M to 550M by changing the hidden dimension. Then we train these varying-sized Mamba with a certain amount of tokens determined by Chinchilla laws. We have included a detailed version in our Appendix D.3 of the revised draft. \n\n[1] Gu & Dao, Mamba: Linear-time sequence modeling with selective state spaces\n\n\n**4. Performance downgradation caused by oversmoothing does not appear in Section 4.1.**\n\nOver-smoothing does not always result in performance degradation. In many prior studies [1,2,3], over-smoothing has been observed to manifest as performance saturation, where the model's performance plateaus as its depth increases.\n\nIn our new experiments, we further increased the depth of Mamba, following the setup described in Sec. 4.1. We observed a noticeable performance drop when the depth exceeded 48 layers. As shown in Fig. 4 (of the revision), the validation perplexity begins to rise for both models with sequence lengths of 2048 and 8192, indicating a decline in performance (e.g., in the 64-layer and 72-layer models).\n\n\n[1] Zhou et al., Deepvit: Towards deeper vision transformer\n\n[2] Wang et al., Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice\n\n[3] Chen et al., Bag of tricks for training deeper graph neural networks: A comprehensive benchmark study\n\n**5. Typos and writing errors.**\n\nWe thank Reviewer LroV for pointing out writing errors. We have resolved them in our revision. We have moved Lemma 3.1 (currently Lemma C.1) to Appendix C.1 to save some space for other more meaningful discussions."}},"tmdate":1732272694616,"tcdate":1732272694616,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1195/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission1195/Authors"],"forum":"pymXpl4qvi","number":4,"license":"CC BY 4.0","cdate":1732272694616,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1195/-/Official_Comment"],"mdate":1732272694616,"domain":"ICLR.cc/2025/Conference","replyto":"qPOc1Dmog4","id":"ARgFVjABdN","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"We theoretically and empirically reveal and tackle locality and over-smoothing bottlenecks for state space models."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["State Space Models","Long-context LLMs","Over-smoothing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Structured State Space Models (SSMs) have emerged as alternatives to transformers.\nWhile SSMs are often regarded as effective in capturing long-sequence dependencies, we rigorously demonstrate that they are inherently limited by strong recency bias.\nOur empirical studies also reveal that this bias impairs the models' ability to recall distant information and introduces robustness issues. Our scaling experiments then discovered that deeper structures in SSMs can facilitate the learning of long contexts.\nHowever, subsequent theoretical analysis reveals that as SSMs increase in depth, they exhibit another inevitable tendency toward over-smoothing, e.g., token representations becoming increasingly indistinguishable.\nThis *fundamental dilemma* between recency and over-smoothing hinders the scalability of existing SSMs. \nInspired by our theoretical findings, we propose to *polarize* two channels of the state transition matrices in SSMs, setting them to zero and one, respectively, simultaneously addressing recency bias and over-smoothing.\nExperiments demonstrate that our polarization technique consistently enhances the associative recall accuracy of long-range tokens and unlocks SSMs to benefit further from deeper architectures.\nAll source codes are released at https://github.com/VITA-Group/SSM-Bottleneck."},"_bibtex":{"value":"@inproceedings{\nwang2025understanding,\ntitle={Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing},\nauthor={Peihao Wang and Ruisi Cai and Yuehao Wang and Jiajun Zhu and Pragya Srivastava and Zhangyang Wang and Pan Li},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=pymXpl4qvi}\n}"},"title":{"value":"Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing"},"pdf":{"value":"/pdf/982acd7979eff95835985d1c18fb057152350bf1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wang|understanding_and_mitigating_bottlenecks_of_state_space_models_through_the_lens_of_recency_and_oversmoothing"},"authorids":{"value":["~Peihao_Wang1","~Ruisi_Cai1","~Yuehao_Wang1","~Jiajun_Zhu1","~Pragya_Srivastava1","~Zhangyang_Wang1","~Pan_Li2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Peihao Wang","Ruisi Cai","Yuehao Wang","Jiajun Zhu","Pragya Srivastava","Zhangyang Wang","Pan Li"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a novel framework for modeling Temporal Point Processes (TPPs) through continuous-time editing operations. \n\nThe paper presents the EDITPP framework, which integrates random set interpolation methods for TPPs with discrete sequence editing flow methods. This combination allows for a more effective modeling of event sequences.\nThe framework parameterizes instantaneous rates of insertion, deletion, and substitution operations within a Continuous-Time Markov Chain (CTMC) framework. This approach significantly reduces the number of editing operations required during the generation process.\nThe experimental results demonstrate that EDITPP achieves state-of-the-art performance on various real-world and synthetic datasets for both unconditional and conditional tasks.\nBy introducing an auxiliary alignment space, the framework can be flexibly applied to tasks of varying complexity, enhancing its versatility in different scenarios.\nOverall, the paper showcases the effectiveness of EDITPP in improving the modeling and generation of event sequences in TPPs."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Could you provide more detailed guidelines or best practices for hyperparameter tuning? Given that the performance of EDITPP may heavily rely on this aspect, clearer instructions could help practitioners achieve optimal results more efficiently.\n\n2. How does EDITPP perform when applied to datasets that differ significantly from those used in training? Are there any specific limitations observed in terms of generalization?\n\n3. What measures have been taken to ensure the computational efficiency of EDITPP during both training and inference? Are there specific scenarios where the model may struggle with efficiency?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. EDITPP effectively combines random set interpolation methods for Temporal Point Processes (TPPs) with discrete sequence editing flow methods, allowing for a more comprehensive modeling approach.\n2. By parameterizing the instantaneous rates of insertion, deletion, and substitution within a Continuous-Time Markov Chain (CTMC) framework, EDITPP significantly reduces the number of editing operations required during the generation process, enhancing efficiency.\n3. The framework demonstrates state-of-the-art performance on various real-world and synthetic datasets for both unconditional and conditional tasks, showcasing its effectiveness and reliability in practical applications.\n4. The introduction of an auxiliary alignment space allows EDITPP to be flexibly applied to tasks of varying complexity, making it versatile for different modeling scenarios and enhancing its applicability across diverse domains."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. While the framework shows strong performance on specific datasets, its generalization capabilities to unseen data or different domains may be limited, necessitating further validation across a broader range of applications.\n2. Although EDITPP reduces the number of editing operations, the overall computational efficiency during training and inference may still be a concern, especially when dealing with large-scale datasets or real-time applications."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931052446,"tcdate":1761904249950,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19001/Reviewer_BAAf"],"signatures":["ICLR.cc/2026/Conference/Submission19001/Reviewer_BAAf"],"forum":"FNf9IV1P2L","number":3,"license":"CC BY 4.0","cdate":1761904249950,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19001/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931052446,"domain":"ICLR.cc/2026/Conference","replyto":"FNf9IV1P2L","id":"HQeaROvIxS","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Generative Modelling","Forecasting","Events","Sequences","Sets","Continuous Time","CTMC"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Temporal point processes (TPPs) are a fundamental tool for modeling event sequences in continuous time, but most existing approaches rely on autoregressive parameterizations that are limited by their sequential sampling. Recent non-autoregressive, diffusion-style models mitigate these issues by jointly interpolating between noise and data through event insertions and deletions in a discrete Markov chain. In this work, we generalize this perspective and introduce an Edit Flow process for TPPs that transports noise to data via insert, delete, and substitute edit operations. By learning the instantaneous edit rates within a continuous-time Markov chain framework, we attain a flexible and efficient model that effectively reduces the total number of necessary edit operations during generation. Empirical results demonstrate the generative flexibility of our unconditionally trained model in a wide range of unconditional and conditional generation tasks on benchmark TPPs."},"_bibtex":{"value":"@inproceedings{\nludke2026editbased,\ntitle={Edit-Based Flow Matching for Temporal Point Processes},\nauthor={David L{\\\"u}dke and Marten Lienen and Marcel Kollovieh and Stephan G{\\\"u}nnemann},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=FNf9IV1P2L}\n}"},"title":{"value":"Edit-Based Flow Matching for Temporal Point Processes"},"pdf":{"value":"/pdf/fa10f2b770ff5a86ae8cac879cf976a7a8c843ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lüdke|editbased_flow_matching_for_temporal_point_processes"},"authorids":{"value":["~David_Lüdke1","~Marten_Lienen1","~Marcel_Kollovieh1","~Stephan_Günnemann1"]},"authors":{"value":["David Lüdke","Marten Lienen","Marcel Kollovieh","Stephan Günnemann"]}},"version":2},{"content":{"summary":{"value":"The authors generalise RoPE to a mechanism allowing it to choose angles in a way that is input dependent. The authors perform analysis mainly on linear-attention models and show that selective rope seems to improve performance over baselines such as RoPE or NoPE."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please see questions in the weaknesses"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The connection between SSMs, linear attention, and RoPE is interesting. I particularly liked the presentation in Table 1."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"My main area of research is in Transformers and not linear attention although I have some experience with linear attention and RFF.s \n\nI do not think I quite understand the point of \"Softmax attention implicitly applies a selective rotation, to encode relative positional information between tokens.\" Are you arguing that the rotations come from the relationship between the softmax kernel and RFF? So you can view the softmax kernel as applying RoPE but where the angles are sampled IID from a Gaussian. This however would really only be true if your angle samples tend to infinity of course. Is this how you are connecting RoPE with a \"NoPE\" softmax?\n\nI found the notation slightly hard to follow especially as someone not coming from SMMs. I mainly found confusing that RoPE is really a method used in Transformers, but the paper seems to only implemented the selective mechanism for linear attention and was not implemented for normal quadratic Transformers? Is there something stopping you from implementing for a normal Transformer? \n\nMinor\nTypo in abstract: rotation in all angels -> rotation in all angles"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762941772689,"tcdate":1761741659188,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21436/Reviewer_Rgdu"],"signatures":["ICLR.cc/2026/Conference/Submission21436/Reviewer_Rgdu"],"forum":"AQo1SEElNb","number":1,"license":"CC BY 4.0","cdate":1761741659188,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission21436/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762941772689,"domain":"ICLR.cc/2026/Conference","replyto":"AQo1SEElNb","id":"ZDa85mPL54","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce Selective RoPE, an input-dependent rotary embedding that enhances gated linear transformers."},"keywords":{"value":["RoPE","Linear Transformer","Attention","State Space Models","Forget Gate"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Positional information is essential for language modeling. Softmax Transformers with Rotary Position Embeddings (RoPE) encode it with fixed-angle rotations, while linear Transformers rely on input-dependent gates that only decay past key-value norms. We provide a theoretical argument for the necessity of a rotation and decay component in well-performing sequence models, and observe that the missing ingredient in linear models is precisely the rotation that softmax attention performs implicitly. We introduce Selective Rotary Position Embedding (*Selective RoPE*), an input-dependent, learnable rotary embedding that generalizes RoPE to arbitrary angles and composes seamlessly with decay gates. Equipping gated linear attention with *Selective RoPE* yields a complex-valued recurrent layer that can be implemented efficiently with the “RoPE trick”. On synthetic benchmarks (MQAR, copying, state tracking) and 370M-parameter language-model pre-training, the method improves recall, downstream accuracy, and expressivity while adding minimal architectural overhead. We open-source our implementation [here](https://github.com/timurcarstensen/selective-rope)."},"_bibtex":{"value":"@inproceedings{\nmovahedi2026selective,\ntitle={Selective Rotary Position Embedding},\nauthor={Sajad Movahedi and Timur Carstensen and Arshia Afzal and Frank Hutter and Antonio Orvieto and Volkan Cevher},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=AQo1SEElNb}\n}"},"title":{"value":"Selective Rotary Position Embedding"},"pdf":{"value":"/pdf/943324e41b7e6a418a5aab0f6ad1261b8e48ad8e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"movahedi|selective_rotary_position_embedding"},"authorids":{"value":["~Sajad_Movahedi1","~Timur_Carstensen1","~Arshia_Afzal1","~Frank_Hutter1","~Antonio_Orvieto3","~Volkan_Cevher1"]},"authors":{"value":["Sajad Movahedi","Timur Carstensen","Arshia Afzal","Frank Hutter","Antonio Orvieto","Volkan Cevher"]}},"version":2},{"content":{"TLDR":{"value":"Delta-SSM introduces a parallelizable spiking Mamba-2 variant that simplifies training to activation quantization while reducing effective FLOPs by about 55% with minimal performance loss."},"venue":{"value":"AXIOM Poster"},"pdf":{"value":"/pdf/3e5a9fa5de2bb9be77af7ecac3163fca530eec31.pdf"},"keywords":{"value":["State Space Models","Spiking Neural Networks","Event-Driven Computing","Sparse Computation","Quantization","Sequence Modeling"]},"venueid":{"value":"NeurIPS.cc/2026/Workshop/AXIOM"},"paperhash":{"value":"seghairi|deltassm_exact_parallelization_of_spiking_state_space_models_for_efficient_sequence_modeling"},"authorids":{"value":["~Abderraouf_Seghairi1","~Emre_Neftci1","~Younes_Bouhadjar2"]},"abstract":{"value":"State Space Models (SSMs) have emerged as an efficient alternative for sequence modeling. However, their deployment on edge hardware remains limited by dense matrix multiplications, which make it difficult to satisfy strict computational constraints. Spiking Neural Networks (SNNs) address this issue through event-driven computation, leveraging sparse spiking activity to reduce computational cost. However, existing spiking SSM variants inherit the sequential dynamics of SNNs, hindering parallelization and thereby limiting training efficiency and scalability. To address these limitations, we propose Delta-SSM, a spiking SSM variant based on the Mamba-2 architecture. By reducing redundant operations across time, Delta-SSM lowers computational cost and enables event-based computation. This is achieved through a custom integer spiking neuron whose sequential dynamics admit an exact time-parallel formulation that reduces to activation quantization. This enables fully parallel training without approximating the underlying spiking model. We evaluate Delta-SSM on a range of audio and language tasks, showing that it achieves approximately 45% of the original computational cost with limited performance degradation."},"_bibtex":{"value":"@inproceedings{\nseghairi2026deltassm,\ntitle={Delta-{SSM}: Exact parallelization of spiking State Space Models for Efficient Sequence Modeling},\nauthor={Abderraouf Seghairi and Emre Neftci and Younes Bouhadjar},\nbooktitle={AXIOM: Foundations of Efficient Deep Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=0b75H3NcHp}\n}"},"track":{"value":"Short Paper track (4 pages, excluding references and appendix)"},"title":{"value":"Delta-SSM: Exact parallelization of spiking State Space Models for Efficient Sequence Modeling"},"authors":{"value":["Abderraouf Seghairi","Emre Neftci","Younes Bouhadjar"]}},"tmdate":1791113214744,"pdate":1790710352987,"tcdate":1788045922710,"writers":["NeurIPS.cc/2026/Workshop/AXIOM","NeurIPS.cc/2026/Workshop/AXIOM/Submission189/Authors"],"signatures":["NeurIPS.cc/2026/Workshop/AXIOM/Submission189/Authors"],"forum":"0b75H3NcHp","license":"CC BY 4.0","number":189,"cdate":1788045922710,"readers":["everyone"],"invitations":["NeurIPS.cc/2026/Workshop/AXIOM/-/Submission","NeurIPS.cc/2026/Workshop/AXIOM/-/Post_Submission","NeurIPS.cc/2026/Workshop/AXIOM/-/Edit","NeurIPS.cc/2026/Workshop/AXIOM/-/PC_Revision"],"mdate":1791113214744,"odate":1790710352987,"domain":"NeurIPS.cc/2026/Workshop/AXIOM","id":"0b75H3NcHp","version":2},{"content":{"venue":{"value":"AISTATS 2026 Poster"},"abstract":{"value":"Transformers play an important role in molecular representation learning, enabling unsupervised learning from large scale unlabeled molecule datasets. However, existing Transformer based methods suffer from heavy training computation and slow inference. To accelerate the computation and relieve the burdensome pre-training, we propose a Mamba-based framework that leverages selective state space models to learn molecular representations more efficiently. Unlike conventional methods, our model, GeoMamba-SE(3), offers streamlined computation with linear-time complexity. However, naively applying Mamba to molecules struggles with SE(3) symmetry, representations can drift under rotations/translations—leading to chemically inconsistent features. To address this, we introduce a geometry and statistics aware design: (i) complete local frames at atoms by converting geometric vectors into scalar channels suitable for SSMs; (ii) multi-stream Mamba blocks are modulated by SE(3)-invariant scalars to preserve geometric stability; and (iii) we impose statistical symmetry constraints via orbit-kernel losses and invariant risk minimization, treating SE(3) actions and conformers as environments. This yields practical SE(3) stability without heavy high-order tensor representations. Experiments show that our method achieves new state-of-the-art performance benchmarks on the MoleculeNet datasets, while using only one-sixth of the training computation and 57\\% less computation for inference."},"_bibtex":{"value":"@inproceedings{\nqin2026from,\ntitle={From Transformers to State Spaces: GeoMamba-{SE}(3) for Fast and Accurate Molecular Learning},\nauthor={Jiayu Qin and Zhengquan Luo and Jian Chen and Xuhui Li and Jiayi Chen and zhiqiang xu},\nbooktitle={The 29th International Conference on Artificial Intelligence and Statistics},\nyear={2026},\nurl={https://openreview.net/forum?id=CG2BZY2HIe}\n}"},"title":{"value":"From Transformers to State Spaces: GeoMamba-SE(3) for Fast and Accurate Molecular Learning"},"pdf":{"value":"/pdf/f6a2987ac157cf6d0957cb16c86ae103791dc288.pdf"},"Signed_copyright_form":{"value":"/attachment/f3b0fbd31ba706f3dbe6bec18be57144a2b8cab3.pdf"},"code_dataset_url":{"value":"https://github.com/jiayujack/GeoMamba-SE3"},"venueid":{"value":"aistats.org/AISTATS/2026/Conference"},"paperhash":{"value":"qin|from_transformers_to_state_spaces_geomambase3_for_fast_and_accurate_molecular_learning"},"authorids":{"value":["~Jiayu_Qin1","~Zhengquan_Luo1","~Jian_Chen9","~Xuhui_Li3","~Jiayi_Chen20","~zhiqiang_xu1"]},"format_confirmation":{"value":"I agree that I have read and followed the formatting instructions for the camera ready version."},"code_dataset_promise":{"value":"Yes"},"authors":{"value":["Jiayu Qin","Zhengquan Luo","Jian Chen","Xuhui Li","Jiayi Chen","zhiqiang xu"]}},"tmdate":1784233601735,"pdate":1770123857853,"tcdate":1758776965446,"writers":["aistats.org/AISTATS/2026/Conference","aistats.org/AISTATS/2026/Conference/Submission1089/Authors"],"signatures":["aistats.org/AISTATS/2026/Conference/Submission1089/Authors"],"forum":"CG2BZY2HIe","license":"CC BY 4.0","number":1089,"cdate":1758776965446,"readers":["everyone"],"invitations":["aistats.org/AISTATS/2026/Conference/-/Submission","aistats.org/AISTATS/2026/Conference/-/Post_Submission","aistats.org/AISTATS/2026/Conference/Submission1089/-/Full_Submission","aistats.org/AISTATS/2026/Conference/-/PC_Revision","aistats.org/AISTATS/2026/Conference/-/Edit","aistats.org/AISTATS/2026/Conference/Submission1089/-/Camera_Ready_Revision"],"mdate":1784233601735,"odate":1770123857853,"domain":"aistats.org/AISTATS/2026/Conference","id":"CG2BZY2HIe","version":2},{"content":{"summary":{"value":"This paper decouples variations of different scales in multi-variate long-term time series based on their periodicity. It then leverages the respective modeling strengths of CNNs and Transformer models to represent these scale-distinct variations. Extensive experiments on multiple long-term time series forecasting datasets demonstrate that the proposed Periodic Decoupling Framework (PDF) method outperforms the latest state-of-the-art approaches across various forecasting time intervals."},"presentation":{"value":"3 good"},"contribution":{"value":"4 excellent"},"soundness":{"value":"3 good"},"strengths":{"value":"Strengths:\n1) The proposed periodicity decoupling framework is a novel and efficient solution to capture 2D temporal variation modeling for long-term series forecasting. Besides, modeling long-term and short-term variations also offers a new perspective for time series forecasting. \n2) The fusion of CNNs and Transformers is prevalent in CV and NLP. This paper bridges time series studies with other advanced domains, introducing the PDF framework by leveraging the strengths of both architectures. The PDF demonstrates outstanding performance and efficiency in time series tasks, as evidenced by thorough experiments.\n3) The authors analyze the computational burden with other state-of-the-art methods and demonstrate the efficiency of the proposed method.\n4) The overall paper is well-organized and easy to follow."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1) The channel-independent strategy focuses on the dependencies of the time dimension adopted in this work and is inspired by PatchTST [1]. However, channel dependency is also helpful. In other works, such as Crossformer [2], modeling channels have also been beneficial to some extent in forecasting results. A more robust justification for this choice is needed.\n2) The innovativeness of the patching method in this work needs to be articulated more specifically. What distinguishes this patching approach from other recent patch-based works (such as PatchTST [1] and PETformer [3])?\n\nOther minor suggestions：\nIn the first sentence of Section 4.4, one of the occurrences of \"complexity\" should be removed.\n\n[1] Nie, Yuqi, et al. \"A time series is worth 64 words: Long-term forecasting with transformers.\", 2022, https://arxiv.org/pdf/2211.14730.\n[2] Zhang, Yunhao, and Junchi Yan. \"Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.\", 2022, https://openreview.net/pdf?id=vSVLM2j9eie.\n[3] Lin, Shengsheng, et al. \"PETformer: Long-term time series forecasting via placeholder-enhanced transformer.\", 2023, https://arxiv.org/pdf/2308.04791."},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"1) What are the benefits of long-term variation and short-term variation extractors?\n2) The periodicity obtained is adaptive to the time series of input?\n3) It is reasonable to model 2D temporal short-term and long-term variations. Is it possible to model 3D variations in time series?"},"rating":{"value":"8: accept, good paper"},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636162728,"tcdate":1698765699501,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission2303/Reviewer_fZ8f"],"signatures":["ICLR.cc/2024/Conference/Submission2303/Reviewer_fZ8f"],"forum":"dp27P5HBBt","number":4,"license":"CC BY 4.0","cdate":1698765699501,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission2303/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636162728,"domain":"ICLR.cc/2024/Conference","replyto":"dp27P5HBBt","id":"DeVcZ1fgxM","forumContent":{"TLDR":{"value":"We propose a novel Periodicity Decoupling Framework (PDF) for long-term series forecasting by capturing 2D temporal variation modeling."},"venue":{"value":"ICLR 2024 poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Long-term time series forecasting","Transformer","CNN"]},"supplementary_material":{"value":"/attachment/af75d7b944def30bd8774f9a8338b05efd04f540.zip"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Convolutional neural network (CNN)-based and Transformer-based methods have recently made significant strides in time series forecasting, which excel at modeling local temporal variations or capturing long-term dependencies. However, real-world time series usually contain intricate temporal patterns, thus making it challenging for existing methods that mainly focus on temporal variations modeling from the 1D time series directly. Based on the intrinsic periodicity of time series, we propose a novel Periodicity Decoupling Framework (PDF) to capture 2D temporal variations of decoupled series for long-term series forecasting. Our PDF mainly consists of three components: multi-periodic decoupling block (MDB), dual variations modeling block (DVMB), and variations aggregation block (VAB). Unlike the previous methods that model 1D temporal variations, our PDF mainly models 2D temporal variations, decoupled from 1D time series by MDB. After that, DVMB attempts to further capture short-term and long-term variations, followed by VAB to make final predictions. Extensive experimental results across seven real-world long-term time series datasets demonstrate the superiority of our method over other state-of-the-art methods, in terms of both forecasting performance and computational efficiency. Code is available at https://github.com/Hank0626/PDF."},"_bibtex":{"value":"@inproceedings{\ndai2024periodicity,\ntitle={Periodicity Decoupling Framework for Long-term Series Forecasting},\nauthor={Tao Dai and Beiliang Wu and Peiyuan Liu and Naiqi Li and Jigang Bao and Yong Jiang and Shu-Tao Xia},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=dp27P5HBBt}\n}"},"title":{"value":"Periodicity Decoupling Framework for Long-term Series Forecasting"},"pdf":{"value":"/pdf/930c90425bd54115335d9bda13d4a63c60a432f7.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference"},"paperhash":{"value":"dai|periodicity_decoupling_framework_for_longterm_series_forecasting"},"authorids":{"value":["~Tao_Dai3","~Beiliang_Wu1","~Peiyuan_Liu1","~Naiqi_Li1","~Jigang_Bao1","~Yong_Jiang3","~Shu-Tao_Xia1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tao Dai","Beiliang Wu","Peiyuan Liu","Naiqi Li","Jigang Bao","Yong Jiang","Shu-Tao Xia"]}},"version":2},{"content":{"summary":{"value":"The paper presents SymmetricDiffusers, a novel discrete diffusion model for learning distributions over permutations within symmetric groups. This model addresses the complexity of directly modeling the vast and discrete state space of permutations by decomposing the task into simpler, more manageable transitions.\n\nKey contributions include:\n1) Forward Diffusion Process Using Card Shuffling Methods: Symmetric diffusers introduce noise to permutations using classical shuffling methods (riffle shuffles, random transpositions, and random insertions). These methods facilitate a gradual transformation toward a known noise distribution, simplifying the learning process.\n2) Generalized Plackett-Luce (PL) Distribution for Reverse Diffusion: To return the noisy state to its original distribution, the model leverages a neural network-based generalized PL distribution, enhancing expressiveness and effectively reconstructing complex dependencies within permutations.\n3) Theoretically Grounded Denoising Schedule: An optimized denoising schedule merges reverse steps to boost sampling efficiency and learning performance, reducing computational requirements without sacrificing accuracy.\n\nThe model demonstrates state-of-the-art or comparable results in tasks such as sorting, jigsaw puzzle assembly, and solving traveling salesman problems, validating its effectiveness in permutation-based applications."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Comparative Analysis with Other Discrete Diffusion Models:\nQuestion: Could you provide additional comparisons between symmetric diffusers and other discrete diffusion models, such as discrete denoising diffusion probabilistic models (D3PMs), particularly regarding tasks that involve smaller permutation spaces?\nSuggestion: Including benchmarks with other models for more straightforward tasks could help clarify SymmetricDiffusers' unique benefits in terms of accuracy, expressiveness, and computational efficiency. If it outperforms or scales better than current discrete diffusion models, it would provide valuable evidence of its novelty.\n\nScalability for Larger :\nQuestion: How do symmetric diffusers handle scalability as increases, and what strategies do you envision for managing the factorial growth in the state space for large permutations?\nSuggestion: An empirical analysis or theoretical discussion on the scalability limits and potential optimizations (e.g., sparse transition matrices or modular architectures) would provide greater insight into the model’s practicality for large-scale tasks.\n\nDenoising Schedule Efficiency:\nQuestion: How significant is the denoising schedule's impact on computational efficiency and model performance? Have you considered alternative denoising schedules that might further improve efficiency?\nSuggestion: An ablation study on the denoising schedule’s impact on performance and efficiency would be beneficial. Exploring adaptive denoising strategies or alternative merging techniques might also enhance the model’s flexibility and efficiency.\n\nForward and Reverse Diffusion Process Clarifications:\nQuestion: Could you clarify certain technical details of the forward and reverse diffusion processes, especially for readers unfamiliar with symmetric groups? Specifically, could you break down the mathematical formulations for each shuffling method?\nSuggestion: To make these sections more accessible to readers, illustrative examples for the shuffling and reverse diffusion steps or a more detailed explanation of terms such as “stationary distribution” and “mixing time” could be added.\n\nBroader Applicability and Adaptation to Other Discrete Structures:\nQuestion: Can symmetric diffusers be adapted for other structured discrete data beyond permutations, such as specific types of graph structures or other combinatorial tasks?\nSuggestion: A brief discussion of the model’s applicability to other structured discrete data or potential adaptations would help clarify its versatility. For instance, could SymmetricDiffusers’ approach be generalized to data with hierarchical or relational structures?\n\nGeneralized Plackett-Luce Distribution Benefits:\nQuestion: Could you provide more insights or experiments to demonstrate the practical benefits of using the generalized Plackett-Luce (PL) distribution over the standard PL model? In which cases does the generalized PL significantly enhance performance?\nSuggestion: Examples or comparisons that highlight the generalized PL distribution’s expressiveness—especially in complex permutation tasks—would illustrate its value over standard PL models."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"Originality: This paper demonstrates notable originality by advancing discrete diffusion models to tackle the problem of learning distributions over permutations in symmetric groups. The task of modeling permutations is inherently challenging due to the factorial growth of the state space, and SymmetricDiffusers introduces an innovative solution by utilizing card shuffling techniques (riffle shuffle, random transpositions, and random insertions) as part of a structured forward diffusion process. Combining classic combinatorial methods with modern neural-based diffusion modeling is an inspired choice well-suited to the discrete and combinatorial nature of permutations. Compared to related works on discrete diffusion—such as Discrete Denoising Diffusion Probabilistic Models (D3PMs), which focus on multinomial categories—SymmetricDiffusers addresses a unique domain with a permutation-focused framework that current D3PMs do not target.\n\nFurthermore, introducing a generalized Plackett-Luce (PL) distribution for the reverse process sets this work apart from other discrete diffusion models, such as score-based continuous-time discrete-state models, which operate on categorical data but lack this flexibility. The generalized PL distribution is well suited for structured dependencies in permutations, enabling greater expressiveness and more accurate learning over complex permutations.\n\nQuality: The paper demonstrates high methodological rigor. Each component of SymmetricDiffusers—the forward process using card shuffling, the generalized PL distribution for reverse diffusion, and the denoising schedule—is technically well developed and grounded in established theories of random walks and Markov chains on finite groups. This adds a layer of mathematical credibility to the proposed model, particularly in the careful treatment of transition probabilities and mixing times associated with the shuffling methods.\n\nAdditionally, the experiments are comprehensive and well-chosen to validate the model across diverse tasks, such as sorting, jigsaw puzzle completion, and traveling salesman problems (TSPs). This diversity of functions effectively showcases the robustness and generalizability of SymmetricDiffusers, achieving state-of-the-art or comparable results in each case. Unlike other discrete diffusion models focused on simpler domains (e.g., categorical image data), SymmetricDiffusers’ application to more complex combinatorial problems uniquely highlights the model’s practical value and robustness.\n\nClarity: The paper is generally well-organized and easy to follow, with a clear introduction to the challenges and requirements of permutation modeling. Complex ideas are supported by notations, figures, and a well-structured flow that gradually builds the reader’s understanding from the background on symmetric groups to the technical construction of SymmetricDiffusers. Visualizing the forward and reverse processes in figure form is particularly helpful, illustrating the model’s approach to diffusion over permutations.\nHowever, compared to some other papers on discrete diffusion (e.g., D3PMs), certain sections—particularly those explaining the forward and reverse processes in detailed mathematical terms—might benefit from additional simplification for readers less familiar with random walks on finite groups. Nevertheless, the overall clarity and structure make the contributions accessible and understandable.\n\nSignificance: SymmetricDiffusers addresses an important area in machine learning by advancing the state of permutation modeling within discrete diffusion frameworks. Distributions over permutations are critical in domains like ranking, combinatorial optimization, and sequence alignment, and current models struggle with the computational complexity posed by large discrete spaces. This work opens up new possibilities for generative modeling within these areas, especially with its robust performance in tasks requiring complex combinatorial reasoning (e.g., TSP).\n\nCompared to recent discrete diffusion works on more general categorical or sequential data, this paper contributes uniquely to discrete generative modeling by directly addressing permutation structures. This is significant because it establishes a pathway for diffusion models to utilize high-complexity tasks beyond traditional applications effectively. By providing an effective means of modeling permutations, SymmetricDiffusers could substantially impact practical applications and inspire future research in discrete-state diffusion for structured data.\n\nThe paper’s strengths lie in its original approach, which combines structured combinatorial methods with diffusion modeling. It also has high-quality methodological rigor, clearly presents complex concepts, and significantly contributes to modeling distributions over large permutation spaces, positioning it as a valuable advancement in discrete diffusion research."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Limited Comparative Analysis with Other Discrete Diffusion Models: While SymmetricDiffusers clearly advances permutation modeling, the paper could benefit from a more detailed comparison with other discrete diffusion models, particularly those handling high-dimensional categorical or sequential data, such as Discrete Denoising Diffusion Probabilistic Models (D3PMs). The current related work section mentions other models briefly but does not delve into how SymmetricDiffusers directly compares in terms of handling large discrete spaces. Including more detailed experiments or ablation studies to demonstrate SymmetricDiffusers’ advantages over such models regarding efficiency or performance on simpler permutation tasks could clarify its relative strengths and limitations.\n\nLack of Exploration on Scalability to Larger: The factorial growth of permutations means that scaling to larger sizes can become computationally intensive. The paper currently demonstrates its model on small-to-moderate values (e.g., tasks like 4-digit MNIST sorting). However, it does not provide clear insights into the scalability limits or potential strategies for scaling SymmetricDiffusers to larger values, where the computational load may become a bottleneck. Future work should consider including benchmarks on larger values or discussing ways to optimize the model’s performance, such as using sparse transition matrices or adopting modular architectures.\n\nClarity in the Forward and Reverse Processes: While the paper is generally well-structured, some sections—particularly those detailing the forward and reverse diffusion processes—are highly technical and may be challenging for readers less familiar with symmetric groups and permutation modeling. Additional clarifications or simplified explanations could improve accessibility. Specifically, breaking down the mathematical formulations for each shuffling method and reverse diffusion process with more illustrative examples would make these sections easier to understand. Clearer definitions of key terms, such as “stationary distribution” and “mixing time” in the context of random walks, could also make the content more accessible to a broader audience.\n\nEfficiency of the Denoising Schedule: The paper introduces a theoretically grounded denoising schedule to merge reverse steps and improve efficiency, but it lacks concrete benchmarks or ablation studies to assess its impact. Comparing SymmetricDiffusers with and without the denoising schedule in terms of computational time and performance would provide readers with a clearer understanding of its practical benefits. Additionally, exploring alternative denoising schedules or adaptive strategies that adjust based on task complexity could further optimize the model’s performance.\n\nBroader Applicability and Practical Implications: Although SymmetricDiffusers demonstrates promising results in permutation-specific tasks (sorting, TSP, jigsaw puzzles), the paper could better communicate its broader applicability and potential limitations. For example, could the model be applied to non-permutation-based tasks with discrete structures, such as certain types of graph-based tasks? A brief discussion of the boundaries of SymmetricDiffusers’ applicability and how it might adapt to related yet distinct discrete structures would clarify its versatility and limitations.\n\nSuggestions for Improvement:\n1) Enhance comparative experiments by including Symmetric Diffusers alongside other discrete diffusion models (e.g., D3PMs) on more straightforward permutation tasks for more explicit benchmarks.\n2) Expand on the model’s scalability for larger spaces, possibly with benchmarks on larger permutation spaces or theoretical discussions on extending to high-dimensional settings.\n3) Provide additional clarifications and illustrative examples for complex forward and reverse diffusion processes sections.\nInclude an ablation study on the denoising schedule to quantify its impact on performance and efficiency.\n4) Discuss the model's broader applicability to other discrete structures beyond permutations, providing readers with insight into its potential versatility."}},"nonreaders":[],"tmdate":1731428471447,"tcdate":1731105168011,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5686/Reviewer_H6zv"],"signatures":["ICLR.cc/2025/Conference/Submission5686/Reviewer_H6zv"],"forum":"EO8xpnW7aX","number":5,"license":"CC BY 4.0","cdate":1731105168011,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5686/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428471447,"domain":"ICLR.cc/2025/Conference","replyto":"EO8xpnW7aX","id":"N7oxQ3jTCS","forumContent":{"venue":{"value":"ICLR 2025 Oral"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Finite Symmetric Groups","Discrete Diffusion","Permutations","Riffle Shuffles","Plackett-Luce Distribution","Sorting","Jigsaw Puzzle"]},"supplementary_material":{"value":"/attachment/7c39a098c0ca040daeee21af5f493423863b3e95.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The group of permutations $S_n$, also known as the finite symmetric groups, are essential in fields such as combinatorics, physics, and chemistry. However, learning a probability distribution over $S_n$ poses significant challenges due to its intractable size and discrete nature. In this paper, we introduce *SymmetricDiffusers*, a novel discrete diffusion model that simplifies the task of learning a complicated distribution over $S_n$ by decomposing it into learning simpler transitions of the reverse diffusion using deep neural networks. We identify the riffle shuffle as an effective forward transition and provide empirical guidelines for selecting the diffusion length based on the theory of random walks on finite groups. Additionally, we propose a generalized Plackett-Luce (PL) distribution for the reverse transition, which is provably more expressive than the PL distribution. We further introduce a theoretically grounded \"denoising schedule\" to improve sampling and learning efficiency. Extensive experiments show that our model achieves state-of-the-art or comparable performance on solving tasks including sorting 4-digit MNIST images, jigsaw puzzles, and traveling salesman problems. Our code is released at <https://github.com/DSL-Lab/SymmetricDiffusers>."},"_bibtex":{"value":"@inproceedings{\nzhang2025symmetricdiffusers,\ntitle={SymmetricDiffusers: Learning Discrete Diffusion Models over Finite Symmetric Groups},\nauthor={Yongxing Zhang and Donglin Yang and Renjie Liao},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=EO8xpnW7aX}\n}"},"title":{"value":"SymmetricDiffusers: Learning Discrete Diffusion on Finite Symmetric Groups"},"pdf":{"value":"/pdf/62dd307a745e636daae0693aef02e033303bd251.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"zhang|symmetricdiffusers_learning_discrete_diffusion_on_finite_symmetric_groups"},"authorids":{"value":["~Yongxing_Zhang1","~Donglin_Yang1","~Renjie_Liao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yongxing Zhang","Donglin Yang","Renjie Liao"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2507.12803v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"zhang|fldmamba_integrating_fourier_and_laplace_transform_decomposition_with_mamba_for_enhanced_time_series_prediction"},"authorids":{"value":["~Qianru_Zhang2","https://dblp.org/search/pid/api?q=author:Chenglei_Yu:","https://dblp.org/search/pid/api?q=author:Haixin_Wang_0003:","https://dblp.org/search/pid/api?q=author:Yudong_Yan:","https://dblp.org/search/pid/api?q=author:Yuansheng_Cao:","https://dblp.org/search/pid/api?q=author:Siu-Ming_Yiu:","https://dblp.org/search/pid/api?q=author:Tailin_Wu:","~Hongzhi_Yin2"]},"html":{"value":"https://doi.org/10.48550/arXiv.2507.12803"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2507-12803,\n  publtype={informal},\n  author={Qianru Zhang and Chenglei Yu and Haixin Wang and Yudong Yan and Yuansheng Cao and Siu-Ming Yiu and Tailin Wu and Hongzhi Yin},\n  title={FLDmamba: Integrating Fourier and Laplace Transform Decomposition with Mamba for Enhanced Time Series Prediction},\n  year={2025},\n  month={July},\n  cdate={1751328000000},\n  journal={CoRR},\n  volume={abs/2507.12803},\n  url={https://doi.org/10.48550/arXiv.2507.12803}\n}\n"},"abstract":{"value":"Time series prediction, a crucial task across various domains, faces significant challenges due to the inherent complexities of time series data, including non-stationarity, multi-scale periodicity, and transient dynamics, particularly when tackling long-term predictions. While Transformer-based architectures have shown promise, their quadratic complexity with sequence length hinders their efficiency for long-term predictions. Recent advancements in State-Space Models, such as Mamba, offer a more efficient alternative for long-term modeling, but they cannot capture multi-scale periodicity and transient dynamics effectively. Meanwhile, they are susceptible to data noise issues in time series. This paper proposes a novel framework, FLDmamba (Fourier and Laplace Transform Decomposition Mamba), addressing these limitations. FLDmamba leverages the strengths of both Fourier and Laplace transforms to effectively capture both multi-scale periodicity, transient dynamics within time series data, and improve the robustness of the model to the data noise issue. Our extensive experiments demonstrate that FLDmamba achieves superior performance on time series prediction benchmarks, outperforming both Transformer-based and other Mamba-based architectures. To promote the reproducibility of our method, we have made both the code and data accessible via the following URL:{\\href{https://github.com/AI4Science-WestlakeU/FLDmamba}{https://github.com/AI4Science-WestlakeU/\\model}."},"title":{"value":"FLDmamba: Integrating Fourier and Laplace Transform Decomposition with Mamba for Enhanced Time Series Prediction"},"authors":{"value":["Qianru Zhang","Chenglei Yu","Haixin Wang","Yudong Yan","Yuansheng Cao","Siu-Ming Yiu","Tailin Wu","Hongzhi Yin"]}},"tmdate":1771387711929,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2507-12803"],"tcdate":1768178465840,"writers":["~"],"signatures":["~Hongzhi_Yin2"],"forum":"ZA7HIDdJK9","license":"CC BY-SA 4.0","number":743383,"cdate":1751328000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1771387711929,"domain":"DBLP.org","id":"ZA7HIDdJK9","version":2},{"content":{"summary":{"value":"This paper addresses the contextual nonstationary bandit problem, presenting NeuralPES, a deep neural network-based algorithm as a solution. The algorithm is a combination of ensemble sampling and future reward prediction, achieved through sequence modeling.\n\nIn essence, NeuralPES can be interpreted as a neural network implemented version of Linear Predictive Sampling (LinPS), a model that holds regret guarantees under various nonstationary environments.\n\nThe effectiveness of this approach in dealing with nonstationarity is corroborated by experiments conducted with (i) synthetic data, (ii) a real-world dataset featuring short-term nonstationarity (one week), and (iii) a real-world dataset with long-term nonstationarity (two months)."},"presentation":{"value":"3 good"},"contribution":{"value":"2 fair"},"soundness":{"value":"3 good"},"strengths":{"value":"This paper, to my knowledge, is the first to address nonstationarity by considering the rate at which information disappears in the future, taking into account the applicability to real-world data.\n\nIn real-world data, there exist high-dimensional, non-linear, and diverse features (or contexts). This paper proposes a neural network-based method and architecture that allows efficient handling of these features while effectively addressing non-stationarity (via prediction). The usefulness of this approach has been verified across various applications.\n\nAs an algorithm, it enables NN-based sequence reward modeling, which predicts future rewards based on the weight sequence of past models, and ensemble sampling, which can be applied even when it is difficult to calculate the posterior distribution, to be combined.\n\nThe authors have been able to experimentally demonstrate superior performance compared to other neural-based bandit algorithms, as well as their sliding window versions."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"I have many concerns regarding this paper, all of which I believe to be addressed by the authors.\n\n1. The paper lacks a clear comparative analysis between PS and NeurPS. It would be beneficial for the readers if the authors could elucidate the apparent differences between the two.\n\n2. The authors mention that PS \"suffers from their scalability\" and \"it does not efficiently scale\", however, the specifics of these issues are not clearly outlined. I would recommend providing concrete examples or explanations to support these statements.\n\n3. It seems that PS (Liu et al. 2023) operates on an infinite reward sequence for decision-making, yet this element is not present in the current paper. It would be interesting to know how this differs or if it is similar. Either way, a discussion on this point seems necessary.\n\n4. The theoretical analysis may appear almost identical to that of (Liu et al. 2023). Are there any differences that the authors could highlight?\n\n5. How much execution time is required for NeuPS's learning and inference (or decision-making)? Considering the time-intensive nature of neural network training and inference, and even more so for ensemble models (depending on the value of M), it would be important to discuss the scalability implications of the execution time.\n\n6. Given that the authors are using an \"A100 40GB GPU\", if the learning time is long, this could significantly increase the cost, which is a critical factor for practical applications.\n\n7. The authors state that \"the regret of LinPS is zero\" and \"LinPS achieves 0 regret\". Is this regret referring to long-run average regret?\n\n8. The paper mentions that the algorithm \"is such optimal\". It would be helpful if the authors could clarify what they mean by \"optimal\", as many algorithms can achieve a long-run average regret of 0.\n\n9. How is the ensemble size M set? Is it adjusted for each experiment? Knowing the effect of its size on reward changes and computation time seems necessary for demonstrating scalability.\n\n10. For the long-term experiments, how is the data processed? The authors mention grouping every 12 hours of recommendation into a contextual bandit format. Does this mean that the order of recommendations presented within the 12-hour period is ignored? If so, could this introduce bias?\n\n11. Despite the major contributions of the paper appearing to lie in the experimental section, the experiments are not written in a reproducible manner. The code is not shared, and it seems that there are parameters, such as the value of M, that are not mentioned in the paper."},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"I hope you can answer me about the comments I wrote in Weaknesses."},"rating":{"value":"3: reject, not good enough"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699637067841,"tcdate":1698558693709,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission8538/Reviewer_Coy3"],"signatures":["ICLR.cc/2024/Conference/Submission8538/Reviewer_Coy3"],"forum":"qVILwUxjLG","number":2,"license":"CC BY 4.0","cdate":1698558693709,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission8538/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699637067841,"domain":"ICLR.cc/2024/Conference","replyto":"qVILwUxjLG","id":"xtMDTZa7FA","forumContent":{"TLDR":{"value":"We introduce a pioneering non-stationary neural contextual bandit algorithm that is not only scalable with deep neural nets but also prioritizes acquiring pertinent information that remains relevant for a long period of time under nonstationarity."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Nonstationary Contextual Bandit","Neural Bandit Learning","Continual Learning","Exploration vs Exploitation"]},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Real-world applications of contextual bandits often exhibit non-stationarity due to seasonality, serendipity, and evolving social trends. \nWhile a number of non-stationary contextual bandit learning algorithms have been proposed in the literature, they excessively explore due to a lack of prioritization for information of enduring value, or are designed in ways that do not scale in modern applications with high-dimensional user-specific features and large action set, or both. In this paper, we introduce a novel non-stationary contextual bandit algorithm that addresses these concerns. It combines a scalable, deep-neural-network-based architecture with a carefully designed exploration mechanism that strategically prioritizes collecting information with the most lasting value in a non-stationary environment. Through empirical evaluations on two real-world recommendation datasets, which exhibit pronounced non-stationarity, we demonstrate that our approach significantly outperforms the state-of-the-art baselines."},"_bibtex":{"value":"@misc{\nzhu2024nonstationary,\ntitle={Non-stationary Contextual Bandit Learning via Neural Predictive Ensemble Sampling},\nauthor={Zheqing Zhu and Yueyang Liu and Kuang Xu and Benjamin Van Roy},\nyear={2024},\nurl={https://openreview.net/forum?id=qVILwUxjLG}\n}"},"title":{"value":"Non-stationary Contextual Bandit Learning via Neural Predictive Ensemble Sampling"},"pdf":{"value":"/pdf/47a7201f864944c003943cc5dfb3d9ee3280d4b4.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"zhu|nonstationary_contextual_bandit_learning_via_neural_predictive_ensemble_sampling"},"authorids":{"value":["~Zheqing_Zhu1","~Yueyang_Liu1","~Kuang_Xu1","~Benjamin_Van_Roy1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zheqing Zhu","Yueyang Liu","Kuang Xu","Benjamin Van Roy"]}},"version":2},{"content":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["3D Object Detection","Mamba","Foreground Representation"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Linear modeling methods like Mamba have been merged as the effective backbone for the 3D object detection task. However, previous Mamba-based methods utilize the bidirectional encoding for the whole non-empty voxel sequence, which contains abundant useless background information in the scenes. Though directly encoding foreground voxels appears to be a plausible solution, it tends to degrade detection performance. We attribute this to the response attenuation and restricted context representation in the linear modeling for fore-only sequences. To address this problem, we propose a novel backbone, termed Fore-Mamba3D, to focus on the foreground enhancement by modifying Mamba-based encoder. The foreground voxels are first sampled according to the predicted scores. Considering the response attenuation existing in the interaction of foreground voxels across different instances, we design a regional-to-global slide window (RGSW) to propagate the information from regional split to the entire sequence. Furthermore, a semantic-assisted and state spatial fusion module (SASFMamba) is proposed to enrich contextual representation by enhancing semantic and geometric awareness within the Mamba model. Our method emphasizes foreground-only encoding and alleviates the distance-based and causal dependencies in the linear autoregression model. The superior performance across various benchmarks demonstrates the effectiveness of Fore-Mamba3D in the 3D object detection task."},"_bibtex":{"value":"@inproceedings{\nning2026foremambad,\ntitle={Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection},\nauthor={Zhiwei Ning and Xuanang Gao and Jiaxi Cao and Runze Yang and Huiying Xu and Xinzhong Zhu and JIE YANG and Wei Liu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=e4t1775UJ1}\n}"},"title":{"value":"Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection"},"pdf":{"value":"/pdf/c88d1a4b5a3d89c3710599dac8b4ca8c7c9c1d6e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"ning|foremamba3d_mambabased_foregroundenhanced_encoding_for_3d_object_detection"},"authorids":{"value":["~Zhiwei_Ning2","~Xuanang_Gao1","~Jiaxi_Cao1","~Runze_Yang2","~Huiying_Xu1","~Xinzhong_Zhu1","~JIE_YANG18","~Wei_Liu38"]},"authors":{"value":["Zhiwei Ning","Xuanang Gao","Jiaxi Cao","Runze Yang","Huiying Xu","Xinzhong Zhu","JIE YANG","Wei Liu"]}},"tmdate":1775877064244,"pdate":1769436020666,"tcdate":1758165256315,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10258/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission10258/Authors"],"forum":"e4t1775UJ1","license":"CC BY 4.0","number":10258,"cdate":1758165256315,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission10258/-/Full_Submission","ICLR.cc/2026/Conference/Submission10258/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission10258/-/Camera_Ready_Revision"],"mdate":1775877064244,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"e4t1775UJ1","version":2},{"content":{"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"kapse|fast_vision_mamba_pooling_spatial_dimensions_for_accelerated_processing"},"authorids":{"value":["~Saarthak_Kapse1"]},"html":{"value":"https://arxiv.org/pdf/2502.00594"},"abstract":{"value":"State Space Models (SSMs) with selective scan (Mamba) have been adapted into efficient vision models. Mamba, unlike Vision Transformers, achieves linear complexity for token interactions through a recurrent hidden state process. This sequential processing is enhanced by a parallel scan algorithm, which reduces the computational time of recurrent steps from L sequential steps to log(L) parallel steps with respect to the number of input tokens (L). In this work, we propose Fast Vision Mamba (FastVim), that further reduces the computational time of the SSM block by reducing the number of recurrent steps in Vision Mamba models while still retaining model performance. By alternately pooling tokens along image dimensions across Mamba blocks, we obtain a 2× reduction in the number of parallel steps in SSM block. Our model offers up to 72.5% speedup in inference speed compared to baseline Vision Mamba models on high resolution (2048×2048) images. Our experiments demonstrate state-of-the-art performance with dramatically improved throughput in a range of tasks such as image classification, cell perturbation prediction, segmentation, and object detection. Code is made available at github.com/insitro/FastVim"},"title":{"value":"Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing"},"authors":{"value":["Saarthak Kapse"]}},"tmdate":1739405787047,"pdate":1738386000000,"tcdate":1739405787047,"writers":["~Saarthak_Kapse1"],"signatures":["~Saarthak_Kapse1"],"forum":"zCK8Kb1Qu4","license":"CC BY 4.0","number":32779,"cdate":1739405787047,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1739405787047,"domain":"OpenReview.net/Archive","id":"zCK8Kb1Qu4","version":2},{"content":{"comment":{"value":"---\n## Question Group 2\n- [W2] How MambaSL captures intrinsic temporal properties/patterns? Especially for classification?\n- [W3] Extensive Mamba background makes readers more confuse about the W2. At the same time, readers may still find it unclear how MambaSL effectively learns task-relevant features for classification.\n---\n\nAll four hypotheses (H1–H4) we propose in the paper correspond to minimal structural modifications designed to **help Mamba better learn from time series classification datasets**. Unlike classical time series classification models, which often explicitly focus on capturing temporal \"patterns,\" **MambaSL is primarily aimed at helping Mamba’s SSM retain meaningful state information over long horizons**. In that sense, we believe the question \"How does MambaSL capture temporal properties/patterns?\" is essentially equivalent to \"How does Mamba capture temporal properties/patterns?\".\n\nThus, the overarching motivation of this work is that **naively adopting the original Mamba architecture (or typical time series classification pipeline) can easily lead to overfitting in time series classification**. As mentioned at the end of `Section 6 (Conclusion)`, H1 and H2 are designed with the \"time series\" domain in mind, whereas H3 and H4 are tailored to the \"classification\" task:\n- **H1**: If the receptive field used to build input embeddings is too small, Mamba’s *gating mechanism may only capture very local patterns*. We therefore scale the receptive field with sequence length or sampling rate so that it can observe a longer context.  \n- **H2**: When data are collected from a time-invariant system, training with the original selective SSM may actually *prevent the Mamba from capturing time-invariant properties*. We thus make time (in)variance a tunable hyperparameter.\n- **H3**: For classification, each label summarizes the entire sequence rather than a single position. *Overly strong skip connections and large model capacity can interfere with capturing global properties*, so we remove the skip connection and reduce capacity.  \n- **H4**: In classification, the label may correspond to *either a global summary of the sequence or a specific temporal event*. Instead of pure global average pooling, we introduce an adaptive pooling mechanism that allows the model to choose pooling behavior (average or max) that better fits each domain. Since Mamba is a recurrent model, cherry-picking the most relevant time step in the classifier still allows the model to consider the entire sequence history.\n\n\n&nbsp;\n\nWe first apologize for any confusion caused by the lengthy explanation of Mamba. We will try to clarify the motivation behind `Section 3.2 (Time Variant SSM in Mamba)` in this response.\n\n`Section 3.2.1 (LTI SSM)` introduces SSMs for readers who may not be familiar with Mamba or SSMs, so that they can still follow the paper. It also emphasizes that when applying an SSM to multivariate sequences, all parameters except $\\boldsymbol{A}$ are broadcasted, making the dynamics **channel-independent**.\n\nThe reason we emphasize this is that **Mamba’s selective SSM performs channel mixing** unlike earlier SSM-based models such as S4 (as described in `Section 3.2.2 (Selective SSM)`). Because this behavior is not explicitly described in the Mamba literature, Mamba is sometimes (incorrectly) assumed to be channel-independent like traditional SSMs. For instance, in [DeltaNet](https://openreview.net/forum?id=UvTo3tVBk2), the authors state that \"this allows DeltaNet to perform simultaneous token- and channel-mixing, while Mamba can only perform token-mixing,\" which reflects such a misunderstanding.\n\nTo our knowledge, very few papers have **explicitly formulated the selective SSM for multivariate inputs**. Thus, we **carefully specify the tensor dimensions of $\\boldsymbol{B}$, $\\boldsymbol{C}$, and $\\boldsymbol{\\Delta}$** in `Section 3.2.2 (Selective SSM)` to clearly articulate Mamba’s channel-mixing behavior. Building on this formulation, `Section 3.2.3 (On Time Variance of B,C,Δ)` then explains the **role of each parameter** and how Mamba captures temporal characteristics of time series. In that sense, `Section 3.2.2` and `Section 3.2.3` go beyond basic background and directly motivate our hypothesis H2.  \n\nThus, we view **the entirety of `Section 3.2` as a conceptual and mathematical setup specifically for H2**, rather than general background that might be safely omitted.\nWe admit that this full explanation may have been too dense for readers unfamiliar with Mamba or SSMs, and we will strive to clarify the presentation in the final version.\n\n&nbsp;"},"title":{"value":"Official Comment by Authors (part 3/4)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763911925187,"tcdate":1763911925187,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11231/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission11231/Authors"],"forum":"YDl4vqQqGP","number":8,"license":"CC BY 4.0","cdate":1763911925187,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11231/-/Official_Comment"],"mdate":1763911925187,"domain":"ICLR.cc/2026/Conference","replyto":"pog7d1pEPA","id":"dVtloPK6PR","forumContent":{"TLDR":{"value":"We introduce MambaSL, a minimally redesigned single-layer Mamba that achieves state-of-the-art accuracy on the UEA30 benchmark, with reproducible evaluation covering all baselines."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["time series classification","single-layer Mamba","modular selective SSM","multi-head adaptive pooling","skip connection"]},"supplementary_material":{"value":"/attachment/dfc093493a30882d173d5eedf1107e0c42d4d9fc.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Despite recent advances in state space models (SSMs) such as Mamba across various sequence domains, research on their standalone capacity for time series classification (TSC) has remained limited. \nWe propose MambaSL, a framework that minimally redesigns the selective SSM and projection layers of a single-layer Mamba, guided by four TSC-specific hypotheses.\nTo address benchmarking limitations—restricted configurations, partial University of East Anglia (UEA) dataset coverage, and insufficiently reproducible setups—we re-evaluate 20 strong baselines across all 30 UEA datasets under a unified protocol. \nAs a result, MambaSL achieves state-of-the-art performance with statistically significant average improvements, while ensuring reproducibility via public checkpoints for all evaluated models.\nTogether with visualizations, these results demonstrate the potential of Mamba-based architectures as a TSC backbone."},"_bibtex":{"value":"@inproceedings{\njung2026mambasl,\ntitle={Mamba{SL}: Exploring Single-Layer Mamba for Time Series Classification},\nauthor={Yoo-Min Jung and Leekyung Kim},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=YDl4vqQqGP}\n}"},"title":{"value":"MambaSL: Exploring Single-Layer Mamba for Time Series Classification"},"pdf":{"value":"/pdf/36a22e78ae778d70b99d6fc145e8d5a9be7e0c1d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jung|mambasl_exploring_singlelayer_mamba_for_time_series_classification"},"authorids":{"value":["~Yoo-Min_Jung1","~Leekyung_Kim1"]},"authors":{"value":["Yoo-Min Jung","Leekyung Kim"]}},"version":2},{"content":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce Selective RoPE, an input-dependent rotary embedding that enhances gated linear transformers."},"keywords":{"value":["RoPE","Linear Transformer","Attention","State Space Models","Forget Gate"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Positional information is essential for language modeling. Softmax Transformers with Rotary Position Embeddings (RoPE) encode it with fixed-angle rotations, while linear Transformers rely on input-dependent gates that only decay past key-value norms. We provide a theoretical argument for the necessity of a rotation and decay component in well-performing sequence models, and observe that the missing ingredient in linear models is precisely the rotation that softmax attention performs implicitly. We introduce Selective Rotary Position Embedding (*Selective RoPE*), an input-dependent, learnable rotary embedding that generalizes RoPE to arbitrary angles and composes seamlessly with decay gates. Equipping gated linear attention with *Selective RoPE* yields a complex-valued recurrent layer that can be implemented efficiently with the “RoPE trick”. On synthetic benchmarks (MQAR, copying, state tracking) and 370M-parameter language-model pre-training, the method improves recall, downstream accuracy, and expressivity while adding minimal architectural overhead. We open-source our implementation [here](https://github.com/timurcarstensen/selective-rope)."},"_bibtex":{"value":"@inproceedings{\nmovahedi2026selective,\ntitle={Selective Rotary Position Embedding},\nauthor={Sajad Movahedi and Timur Carstensen and Arshia Afzal and Frank Hutter and Antonio Orvieto and Volkan Cevher},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=AQo1SEElNb}\n}"},"title":{"value":"Selective Rotary Position Embedding"},"pdf":{"value":"/pdf/943324e41b7e6a418a5aab0f6ad1261b8e48ad8e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"movahedi|selective_rotary_position_embedding"},"authorids":{"value":["~Sajad_Movahedi1","~Timur_Carstensen1","~Arshia_Afzal1","~Frank_Hutter1","~Antonio_Orvieto3","~Volkan_Cevher1"]},"authors":{"value":["Sajad Movahedi","Timur Carstensen","Arshia Afzal","Frank Hutter","Antonio Orvieto","Volkan Cevher"]}},"tmdate":1778677944617,"pdate":1769436552095,"tcdate":1758317562764,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21436/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission21436/Authors"],"forum":"AQo1SEElNb","license":"CC BY 4.0","number":21436,"cdate":1758317562764,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission21436/-/Full_Submission","ICLR.cc/2026/Conference/Submission21436/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission21436/-/Camera_Ready_Revision"],"mdate":1778677944617,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"AQo1SEElNb","version":2},{"content":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Visual Mamba","Visual Recognition","Vision Backbone"]},"supplementary_material":{"value":"/attachment/daeea7fc669370d7e0181fde8442c4c77113891a.zip"},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Recent advancements in State Space Models, notably Mamba, have demonstrated superior performance over the dominant Transformer models, particularly in reducing the computational complexity from quadratic to linear. Yet, difficulties in adapting Mamba from language to vision tasks arise due to the distinct characteristics of visual data, such as the spatial locality and adjacency within images and large variations in information granularity across visual tokens. Existing vision Mamba approaches either flatten tokens into sequences in a raster scan fashion, which breaks the local adjacency of images, or manually partition tokens into windows, which limits their long-range modeling and generalization capabilities. To address these limitations, we present a new vision Mamba model, coined QuadMamba, that effectively captures local dependencies of varying granularities via quadtree-based image partition and scan. Concretely, our lightweight quadtree-based scan module learns to preserve the 2D locality of spatial regions within learned window quadrants. The module estimates the locality score of each token from their features, before adaptively partitioning tokens into window quadrants. An omnidirectional window shifting scheme is also introduced to capture more intact and informative features across different local regions. To make the discretized quadtree partition end-to-end trainable, we further devise a sequence masking strategy based on Gumbel-Softmax and its straight-through gradient estimator. Extensive experiments demonstrate that QuadMamba achieves state-of-the-art performance in various vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Our code and models will be released."},"_bibtex":{"value":"@inproceedings{\nxie2024quadmamba,\ntitle={QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model},\nauthor={Fei Xie and Weijia Zhang and Zhongdao Wang and Chao Ma},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=TtcwVuBZu1}\n}"},"title":{"value":"QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model"},"pdf":{"value":"/pdf/9d8f78fdd61ffd61d29a20c3000ebe33c0a5e8bf.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"xie|quadmamba_learning_quadtreebased_selective_scan_for_visual_state_space_model"},"authorids":{"value":["~Fei_Xie3","~Weijia_Zhang7","~Zhongdao_Wang2","~Chao_Ma3"]},"authors":{"value":["Fei Xie","Weijia Zhang","Zhongdao Wang","Chao Ma"]}},"tmdate":1730873850264,"pdate":1727287665756,"tcdate":1714711204267,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission1601/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission1601/Authors"],"forum":"TtcwVuBZu1","license":"CC BY 4.0","number":1601,"cdate":1714711204267,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/-/Submission","NeurIPS.cc/2024/Conference/-/Edit","NeurIPS.cc/2024/Conference/-/Post_Submission","NeurIPS.cc/2024/Conference/Submission1601/-/Revision","NeurIPS.cc/2024/Conference/Submission1601/-/Camera_Ready_Revision"],"mdate":1730873850264,"odate":1730873850191,"domain":"NeurIPS.cc/2024/Conference","id":"TtcwVuBZu1","version":2},{"content":{"summary":{"value":"The paper introduces MambaMatch, a Mamba-based correspondence learning method with a focus on effective data selection. Specifically, the Channel-Aware Mamba Filter (CAMF), parametrizes SSM parameters to distinguish inliers and outliers in high-dimensional latent space, especially in outlier-heavy scenarios. To address Mamba’s limitations, the authors enhance channel-awareness and add a Local Context Enhancement Module (LCEM) for improved local context capture. Contributions include pioneering Mamba for sparse feature matching, developing a custom outlier filter and LCEM, and validating MambaMatch’s effectiveness on real-world datasets."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Clarify how to upscale C to F in the section of problem formation as well as what value is for d dimensional. \n2. Authors declared selective state space models can ensure the preciseness and efficiency of exploitintg pertinent information. Complexity of the methods should be analyzed to illustrate the greatness of the proposed methods in computational overhead. \n3. What is AUC@5 at line 485 in the caption of Table 5.\n4. The Mamba does not need to be verbosely introduced in this paper since it is a well-known method that is capable of handling both continuous and discretized problems. \n5. Not all SSMs (e.g., the structural SMM) have selectivity and thus Section 2.2 should give more statements for selective SSMs rather than rough SSMs."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper is well-written and easy to follow, making the methodology and results accessible to the reader.\n2. This work inaugurally explore well-known Mamba which key component is SSM to solve key-point correspindence of paired-view images.   \n3. The proposed method, MambaMatch, demonstrates superior performance over other methods on two benchmark datasets, underscoring its effectiveness in real-world scenarios."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The motivation for the proposed approach is unclear. Since correspondence filtering inherently aims to retain correct matches and discard incorrect ones, the novelty of MambaMatch should be clarified. Specifically, it would help to explain how MambaMatch’s selective capabilities address limitations in previous methods that lack an effective mechanism for prioritizing true correspondences over false ones.\n2. The claim that the Mamba-based approach focuses on correct matches, discards false ones, and prevents attention leakage is interesting. However, the paper would benefit from a clearer explanation of how the Mamba structure achieves these capabilities.\n3. The comparative experiments are insufficient to fully validate the effectiveness of the proposed approach. For example, the authors claim the use of a specific loss function; however, there is a lack of comparative experiments to demonstrate its effectiveness."}},"nonreaders":[],"tmdate":1731428644246,"tcdate":1730686448543,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6894/Reviewer_Lypq"],"signatures":["ICLR.cc/2025/Conference/Submission6894/Reviewer_Lypq"],"forum":"YuJdtpPV4n","number":2,"license":"CC BY 4.0","cdate":1730686448543,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6894/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428644246,"domain":"ICLR.cc/2025/Conference","replyto":"YuJdtpPV4n","id":"17mneAK2uz","forumContent":{"TLDR":{"value":"Correspondences Learning with Selective State Spaces"},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Correspondence learning","image matching","feature matching"]},"supplementary_material":{"value":"/attachment/2d09b8e2610b05f0cb43cedcc37982b32e359fa9.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Two-view correspondence learning aims to discern true and false correspondences between image pairs by recognizing their underlying different information. Previous methods either treat the information equally or fail to discard the superfluous information of false correspondences, tending to be invalid in practical scenarios. Therefore, inspired by Mamba's inherent competence of selectivity, we propose MambaMatch as a Mamba-based correspondence filter to selectively mine information from true correspondences and to dispose of the potentially interfering information of false correspondences. Specifically, the selection is achieved by adaptively adjusting model parameters in a high-dimensional latent space, which also avoids attention leakage and implements context compression, ensuring the precise and efficient exploitation of pertinent information. Meanwhile, channel awareness is tailored to serve as a complementary aspect of comprehensive information acquisition. Moreover, we design a novel local-context enhancement module to capture reasonable local context that is crucial for correspondence pruning. Extensive experiments demonstrate that our approach outperforms existing state-of-the-art methods on several visual tasks while saving time and space costs."},"_bibtex":{"value":"@misc{\nfang2024mambamatch,\ntitle={MambaMatch: Learning Two-View Correspondences with Selective State Spaces},\nauthor={Xiang Fang and Shihua Zhang and Jiayi Ma},\nyear={2024},\nurl={https://openreview.net/forum?id=YuJdtpPV4n}\n}"},"title":{"value":"MambaMatch: Learning Two-View Correspondences with Selective State Spaces"},"pdf":{"value":"/pdf/63484318bfc95ae154ff8a779ed5675dcfd84a4f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"fang|mambamatch_learning_twoview_correspondences_with_selective_state_spaces"},"authorids":{"value":["~Xiang_Fang2","~Shihua_Zhang2","~Jiayi_Ma2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xiang Fang","Shihua Zhang","Jiayi Ma"]}},"version":2},{"content":{"summary":{"value":"This paper introduces the generator-matching (GM) framework for modeling arbitrary Markov processes in both continuous (i.e., $\\mathbb{R}^d$) and discrete (i.e., $\\\\{1,...,N\\\\}^D$) domains. GM generalizes the (continuous and discrete) flow-matching (FM) approach, extending its concept of transforming the complex task of learning *marginal* distributions/scores/flows/generators to a simpler task of specifying *data-conditional* distributions/scores/flows/generators, and approximating the marginal ones through a loss based on conditional expectations. The framework not only unifies a variety of existing methods across continuous and discrete generative modeling, but also holds promise for multimodal applications, where the state space comprises the product of multiple different state spaces."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. In appendix D.1, line 1319 suggests that the solution $\\lambda_ t(x)$ is not unique. The derivation selects the form that achieves equality on line 1319 (see line 1334). Are there alternative choices of $\\lambda_ t(x)$ that also satisfy the constraint? If so, are all choices of $\\lambda_ t(x)$ equivalent in some sense, or do they have different characteristics in practical uses (given that they all correspond to the same conditional path $p_ {t|1}(x|z)$)?\n\n2. I don't fully understand the argument in appendix D.2, starting from line 1473. Specifically, can you have a detailed elaboratation on the reason why the prior has compact support poses a potential issue (i.e., lines 1473-1474)? What implications do the non-zero boundaries of $\\tilde\\sigma_ t^2(\\cdot|z)$ have? Moreover, I hope that the author(s) could have a review of the reflected SDE (the process ${\\rm d}L_ t$) so that the readers can better understand the intuition behind how this pure diffusion solution to mixture path works.\n\n3. Minor comments:\n\n- I personally don't quite appreciate the unconventional notation $p_ t\\odot f$ and $p_ {t+h|t}\\odot f$ in this paper. Typically, $p_ t\\odot f$ would be denoted as $\\mathbb{E}_ {p_ t}f$; in standard textbooks of Markov semigroup (e.g., Bakry et al., 2014), $p_ {t+h|t}\\odot f$ in this paper would appear as $P_ {t,t+h}f$, where the semigroup $(P_ {t,s})_ {s\\in[t,\\infty)}$ is defined as $P_ {t,s}f=\\mathbb{E}(f(X_ s)|X_ t=\\cdot)$. However, I recognize these non-standard notations may be acceptable within the machine learning community.\n- Line 274, *no* diffusion coefficient? Besides, the $x_ 1$ in equation (9) seems to be a typo.\n- Line 667: $\\mathbb{E}_ {x\\sim\\mu_ 1}[f(x)]=\\mathbb{E}_ {x\\sim\\mu_ 2}[f(x)]$.\n- Line 669: $(X_ t)_ {0\\le t\\le1}$.\n- Line 725: $p_ {t+h|t}(A_ {t+h}|x)$.\n- Line 755: the limit operation is missing.\n- Line 863: should mention the definition of matrix inner product $A\\cdot B=\\operatorname{tr}(AB)$ for positive definite $A,B$.\n- Lines 901 and 968-978: should be \"with probability $...+o(h)$ when $h\\to0$\".\n- Lines 917, 1001-1017, 1050, 1060, 1067, 1072: the parentheses are missing: should be $\\int_ {y\\ne x}(f(y)-f(x))Q_ t({\\rm d}y;x)$. \n- Lines 1112-1125: $B(F_ t^\\theta(x))$ is missing.\n- Line 1155: the first generator is $\\bar{\\cal L}_ t$, not ${\\cal L}'_ t$.\n- Appendix C.6, proof for conditional generator: I suggest deriving the equation by directly applying derivative by parts to line 1178, which simplifies the proof and reduces its length.\n- Line 1264, should mention that $\\lambda_ t(x)$, $Q_ t(y;x)$ and $J_ t(y;x)$ may depend on $z$, and here this dependence is omitted for conciseness. Similarly, in line 1426, should mention that $a_ t,b_ t$ may depend on $z$ but are independent of $x$.\n- Appendix D.1, time-derivative $\\frac{\\partial}{\\partial t}p_ t$: I suggest rewriting the proof by directly calculating $\\partial_ t\\log{\\cal N}(x|tz,(1-t)^2)$, which would simplify the presentation.\n- Appendix D.2, I suggest replacing all $p_ {\\rm simple}$ with $p_ 0$.\n- Line 1421: the definitions of $G_ 0(x)$ and $G_ {z,\\sigma_ {\\min}}(x)$ have typos.\n- Line 1476: \"Let's *consider* a ...\"\n\nI will be happy to raise the score if the author(s) could address my concern.\n\n**References**\n\n- Bakry et al. Analysis and Geometry of Markov Diffusion Operators. Springer 2014.\n- Bertazzi et al. Piecewise deterministic generative models. Arxiv 2407.19448, 2024.\n- Bose et al. SE(3)-Stochastic Flow Matching for Protein Backbone Generation. ArXiv 2310.02391, 2023.\n- Campbell et al. Trans-Dimensional Generative Modeling via Jump Diffusion Models. NeurIPS 2023.\n- Campbell et al. Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design. ICML 2024.\n- Del Moral & Penev. Stochastic Processes: From Applications to Theory. Chapman & Hall 2014.\n- Gat et al. Discrete Flow Matching. NeurIPS 2024.\n- Huguet et al. Sequence-Augmented SE (3)-Flow Matching For Conditional Protein Backbone Generation. ArXiv 2405.20313, 2024.\n- Lou et al. Scaling Riemannian Diffusion Models. NeurIPS 2023."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"As discussed above, The GM framework extends the FM approach, building on the well-established relationships between marginal distributions/scores/flows and their data-conditional counterparts. From my persepective, this framework presents several notable innovations:\n\n1. The introduction of jump, a type of stochastic process in the continuous domain with non-continuous trajectories, which largely unexplored within the generative AI community. Particularly compelling is the result showing that the widely-used conditional path $p_ {t|1}(x|z)={\\cal N}(x|tz,(1-t)^2)$ can also be achieved through jumps, highlighting its potential utility for generative tasks.\n\n2. An elegant theorem establishing that the generators of interest in $\\mathbb{R}^d$ for applications can be decomposed to three basic types: flow (ODE), diffusion (scaled Brownian motion), and jump. This decomposition underscores the advantages of using the generator perspective in modeling.\n\nThe paper also introduces methodologies for: (a) combining models that use different generators, and (b) modeling multimodal data distributions. While not entirely novel, these contributions effectively systematize existing techniques through the GM perspective. Prior works have addressed these topics, for example: the equivalence of backward SDE and backward probability-flow ODE, the detailed balance condition for CTMC (Campbell et al., 2024), and the predictor-corrector for CTMC (Gat et al., 2024) are all examples of model combination; similarly, multimodal flows introduced in Campbell et al. (2024) and earlier models such as FrameFlow and FrameDiff have laid groundwork in multimodal data modeling, potentially inspiring aspects of this work.\n\nFinally, the paper is well-written, featuring clear notations, a strong motivation, and a rigorous mathematical foundation."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The writing style in some part of the paper, especially section 6, feels overly general and abstract. While the GM framework is indeed comprehensive, it primarily applies to only 4 specific examples: flow, diffusion, jump in $\\mathbb{R}^d$, and CTMC in finite state spaces. Although the theoretical concepts like target-affine (TA) loss and Bregman divergences are elegant, they lack practical utility in these contexts. This abstraction, without sufficient illustrative examples, limits the reader's ability to grasp the practical impact of the framework. I suggest exploring these theoretical constructs further to enrich the paper. For instance, theoretically or numerically comparing the performance of the TA-loss and discussing generator choice in more general state spaces would provide valuable insights. Additionally, a discussion on how TA-loss may facilitate the development of better training losses for generative tasks would be appreciated.\n\n2. The literature review for jump-based models is insufficient. Jump is related to piecewise deterministic process (flow + jump) and piecewise diffusion process (flow + diffusion + jump) in the stochastic process literature (see, e.g., Del Moral & Penev, 2014), and there have been several works leveraging such process for generative modeling, e.g., Campbell et al. (2023) and Bertazzi et al. (2024), which the author(s) have not mentioned. Including these citations would better position GM within the broader literature and highlight its contributions.\n\n3. The experiments are not convincing enough and lack the design details necessary for fair comparisons.\n\n- For $\\mathbb{R}^d$, the paper introduces two constructions of the conditional probability path (mixtures and geometric averages), each with realizations based on flow, diffusion, jump, or a mixture of these (Markov superposition). The toy example in figure 2 is worth further explorations: (1) Which conditional probability path is recommended for better empirical performance? Comparing the performance on more complex target distributions, such as $6\\times6$ checkerboard patterns (Lou et al., 2023), would be informative. (2) Although these processes all generate the same marginal probability distribution in theory, practical implementations face approximation and discretization errors. Which realization provides the best sampling quality? (3) One potential advantage of jump over flow and diffusion is the ability of exploring complicated landscape of target distribution due to its non-continuous nature, but this benefit isn’t evident in the experiments. Therefore, it remains unclear whether the new algorithms offer practical advantages over existing methods, such as FM.\n\n- The details of image generation experiment are lacking, such as the model size, neural network structure, and sampling schedule. The baselines (DDPM, VPSDE, and EDM) are outdated, and the datasets (CIFAR10 and ImageNet32) are less challenging than those commonly used today, such as ImageNet256.\n\n- Despite emphasizing the GM framework’s potential for multimodal data, there are no numerical results on joint multimodal generation. For example, instead of considering protein backbone generation or pure image generation (which are unimodal tasks where existing algorithms already achieved amazing performance), evaluating GM on protein structure-sequence co-generation or image-text joint generation (which has been mentioned in section 7.2) would better showcase its advantage in multimodal settings.\n\n- For the protein experiments, critical experimental details are absent, such as sampling NFE, training time, and model size. The target task is restricted to protein with length less than $128$, a relatively unchallenging setting. Designability, as an important metric for evaluating protein backbone, is not presented. Additionally, more up-to-date work like FoldFlow (Bose et al., 2023) and FoldFlow-2 (Huguet et al., 2024) are missing from the benchmarks. The performance gain achieved by the proposed approach seems to be minimal at the cost of training a separate network with more computational resources. Thus, its efficacy for protein generation remains unclear, especially in comparison to existing methods. It would certainly be more compelling to apply GM to multimodal tasks in protein design by incorporating additional sequence information."}},"nonreaders":[],"tmdate":1732813652047,"tcdate":1730689606042,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3162/Reviewer_rGQU"],"signatures":["ICLR.cc/2025/Conference/Submission3162/Reviewer_rGQU"],"forum":"RuP17cJtZo","number":3,"license":"CC BY 4.0","cdate":1730689606042,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3162/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732813652047,"domain":"ICLR.cc/2025/Conference","replyto":"RuP17cJtZo","id":"0GXsGx1U95","forumContent":{"venue":{"value":"ICLR 2025 Oral"},"TLDR":{"value":"The core principles of flow matching can be vastly generalized to practically all continuous-time Markov processes using Markov generators, unifying all previous methods and opening the door to new generative models agnostic to data modality."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Flow matching","Markov process","Diffusion model","Generative Modeling"]},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We introduce Generator Matching, a modality-agnostic framework for generative modeling using arbitrary Markov processes. Generators characterize the infinitesimal evolution of a Markov process, which we leverage for generative modeling in a similar vein to flow matching: we construct conditional generators which generate single data points, then learn to approximate the marginal generator which generates the full data distribution. We show that Generator Matching unifies various generative modeling methods, including diffusion models, flow matching and discrete diffusion models. Furthermore, it expands the design space to new and unexplored Markov processes such as jump processes. Finally, Generator Matching enables the construction of superpositions of Markov generative models and enables the construction of multimodal models in a rigorous manner. We empirically validate our method on image and multimodal generation, e.g. showing that superposition with a jump process improves performance."},"_bibtex":{"value":"@inproceedings{\nholderrieth2025generator,\ntitle={Generator Matching: Generative modeling with arbitrary Markov processes},\nauthor={Peter Holderrieth and Marton Havasi and Jason Yim and Neta Shaul and Itai Gat and Tommi Jaakkola and Brian Karrer and Ricky T. Q. Chen and Yaron Lipman},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=RuP17cJtZo}\n}"},"title":{"value":"Generator Matching: Generative modeling with arbitrary Markov processes"},"pdf":{"value":"/pdf/100734ddc124d7050b323387376a4e539099e69c.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"holderrieth|generator_matching_generative_modeling_with_arbitrary_markov_processes"},"authorids":{"value":["~Peter_Holderrieth2","~Marton_Havasi1","~Jason_Yim1","~Neta_Shaul1","~Itai_Gat1","~Tommi_S._Jaakkola1","~Brian_Karrer1","~Ricky_T._Q._Chen1","~Yaron_Lipman1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Peter Holderrieth","Marton Havasi","Jason Yim","Neta Shaul","Itai Gat","Tommi Jaakkola","Brian Karrer","Ricky T. Q. Chen","Yaron Lipman"]}},"version":2},{"content":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Rejected by TMLR"},"abstract":{"value":"Multivariate time series classification (TSC) is critical for various applications in fields such as healthcare and finance. While various approaches for TSC have been explored, important properties of time series, such as shift equivariance and inversion invariance, are largely underexplored by existing works. To fill this gap, we propose a novel multi-view approach to capture patterns with properties like shift equivariance. Our method integrates diverse features, including spectral, temporal, local, and global features, to obtain rich, complementary contexts for TSC. We use continuous wavelet transform to capture time-frequency features that remain consistent even when the input is shifted in time. These features are fused with temporal convolutional or multilayer perceptron features to provide complex local and global contextual information. We utilize the Mamba state space model for efficient and scalable sequence modeling and capturing long-range dependencies in time series. Moreover, we introduce a new scanning scheme for Mamba, called tango scanning, to effectively model sequence relationships and leverage inversion invariance, thereby enhancing our model's generalization and robustness. Experiments on two sets of benchmark datasets (10 datasets each) demonstrate our approach's effectiveness, achieving average accuracy improvements of 4.01-6.45% and 8.77% respectively over leading TSC models such as TimesNet and TSLANet."},"_bibtex":{"value":"@article{\nanonymous2024tscmamba,\ntitle={{TSCM}amba: Mamba Meets Multi-View Learning for Time Series Classification},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2024},\nurl={https://openreview.net/forum?id=cpHGwrkbbb},\nnote={Rejected}\n}"},"title":{"value":"TSCMamba: Mamba Meets Multi-View Learning for Time Series Classification"},"changes_since_last_submission":{"value":"We use texts in green for new additions and red strikes for deletions. Also, we have thoroughly rewritten the introduction part. Since this part is almost completely rewritten, it is noted that we did not highlight the changes in this part (to avoid busy changes of text colors)."},"pdf":{"value":"/pdf/7e97b63ef143bde3bfcf00a049a5f7efb435466e.pdf"},"venueid":{"value":"TMLR/Rejected"},"assigned_action_editor":{"value":"~Wei_Liu3"}},"tmdate":1729283645511,"tcdate":1717698458679,"writers":["TMLR","TMLR/Paper2814/Authors"],"signatures":["TMLR/Paper2814/Authors"],"forum":"cpHGwrkbbb","license":"CC BY 4.0","number":2814,"cdate":1717698458679,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/Paper2814/-/Revision","TMLR/-/Rejected"],"mdate":1729283645511,"odate":1719196429192,"domain":"TMLR","id":"cpHGwrkbbb","version":2},{"content":{"summary":{"value":"This paper introduces a novel video frame interpolation (VFI) method called VFIMamba. VFIMamba is the first method that combines the State-Space Model Mamba with VFI architectures and therefore, it has the advantage of a linearly growing complexity w.r.t. the resolution while maintaining the ability to utilize global receptive fields similar to vision transformers. In order to apply the idea to VFI, architectural modifications have been proposed to handle 2 frames as input. Further, a novel curriculum learning strategy is used to increase the models performance across a large range of motions. The method has been evaluated on multiple datasets and w.r.t. to various other methods, achieving state-of-the-art PSNR values, especially improving the performance for high-resolution frames (2K and 4K)."},"soundness":{"value":4},"confidence":{"value":5},"questions":{"value":"- Table 3: There is a discrepancy between many of the reported number in their respective original papers and in this paper on X-TEST for 2k and 4k, e.g.,   EMA-VFI-S reported 30.89dB instead of 29.91dB and BiFormer reported 31.32dB instead 31.18. Based on the experience of the reviewer it is possible to reproduce these numbers using the correct evaluation protocol.\n\n- How is 8x interpolation performed?\n\n- Unclear why there is no FLOPs measurement and no evaluation of SNU-FILM for SoftSplat given that the source code + trained models are publicly available (https://github.com/sniklaus/softmax-splatting)"},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- Combination of an emerging alternative to transformers (MAMBA) with VFI methods that allow for higher resolution frame interpolation due to linear complexity growth. The paper is interesting to read.\n - Discussion and evaluation of different sequence arrangements.\n- Introduction of a relatively simple curriculum learning strategy for VFI with experiments confirming that this strategy is beneficial for VFI methods in general when having to deal with large motions.\n- Reaches new state-of-the-art performance\n- Exhaustive ablation studies proving the effectiveness for each of their introduced modules\n- Supplement contains a video with qualitative comparisons, although only a few short sequences and given the coarse time steps, it is difficult to clearly judge the temporal consistency."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- It is unclear how frames at arbitrary time steps are computed to perform 8x interpolation in table 3. It should also be directly clear from the caption of table 3, that 8x interpolation is evaluated (these details are only in the appendix).\n- In general, a lot of important information is in the appendix. It is helpful to add at least a reference from the main paper to the appendix that there is more information, e.g., such as the evaluation and the experiment on generlalization of curriculum learning.\n- Unclear how the FLOPs requirement scales with resolution. A plot/table showing the FLOPs compared to the input resolution for VFIMamba and other methods would be interesting to get a better feeling for the scaling w.r.t. frame resolution and a comparison of memory footprints might be interesting.\n- Ideally, it should be mentioned in table 2 and 3 on which dataset the other methods have been trained on, especially for methods where the original paper proposed multiple versions such as XVFI.\n- Some recent methods especially for high-resolution data are missing in Table 3 such as [A] and [B]. Especially [A] has only been included in Tab. 2 although their focus is also on high-resolution datasets and code for X-Test is directly available: https://github.com/feinanshan/M2M_VFI/blob/main/Test/bench_xtest.py\n\n\n[A] Hu et al. Many-to-many Splatting for Efficient Video Frame Interpolation, 2022\n\n[B] Nottebaum et al., Efficient Feature Extraction for High-resolution Video Frame Interpolation, 2022.\nIt would be nice to have longer sequences, and playing them a bit more fluently to get a feeling for the temporal and visual consistency.\n\nMinor: \n- Some colors difficult to see in overlay in Figure 1.\n- L. 126, Check sentence “where contains”"},"limitations":{"value":"Discussion of limitations is only in the appendix. It would be better to discuss them already in the main paper. \n\nAdditionaly, has the method similar limitations as MAMBA (the base model for this entire work). Therefore, the compute requirements are still relatively high even though they do not have to compute an attention matrix anymore."}},"nonreaders":[],"tmdate":1730879360640,"tcdate":1720720929962,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission10081/Reviewer_9Hn5"],"signatures":["NeurIPS.cc/2024/Conference/Submission10081/Reviewer_9Hn5"],"forum":"4s5UsBUsUS","number":2,"license":"CC BY 4.0","cdate":1720720929962,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission10081/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879360640,"domain":"NeurIPS.cc/2024/Conference","replyto":"4s5UsBUsUS","id":"3YYKr0H4gU","forumContent":{"TLDR":{"value":"A novel video frame interpolation method that leverages the S6 model for efficient and dynamic inter-frame modeling"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Video Frame Interpolation","State Space Models","Low Level Vision"]},"supplementary_material":{"value":"/attachment/4e30c4c3e65ec2f279d5012fbc32cd6e3f5d90e8.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Inter-frame modeling is pivotal in generating intermediate frames for video frame interpolation (VFI). Current approaches predominantly rely on convolution or attention-based models, which often either lack sufficient receptive fields or entail significant computational overheads. Recently, Selective State Space Models (S6) have emerged, tailored specifically for long sequence modeling, offering both linear complexity and data-dependent modeling capabilities. In this paper, we propose VFIMamba, a novel frame interpolation method for efficient and dynamic inter-frame modeling by harnessing the S6 model. Our approach introduces the Mixed-SSM Block (MSB), which initially rearranges tokens from adjacent frames in an interleaved fashion and subsequently applies multi-directional S6 modeling. This design facilitates the efficient transmission of information across frames while upholding linear complexity. Furthermore, we introduce a novel curriculum learning strategy that progressively cultivates proficiency in modeling inter-frame dynamics across varying motion magnitudes, fully unleashing the potential of the S6 model. Experimental findings showcase that our method attains state-of-the-art performance across diverse benchmarks, particularly excelling in high-resolution scenarios. In particular, on the X-TEST dataset, VFIMamba demonstrates a noteworthy improvement of 0.80 dB for 4K frames and 0.96 dB for 2K frames."},"_bibtex":{"value":"@inproceedings{\nzhang2024vfimamba,\ntitle={{VFIM}amba: Video Frame Interpolation with State Space Models},\nauthor={Guozhen Zhang and Chunxu Liu and Yutao Cui and Xiaotong Zhao and Kai Ma and Limin Wang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=4s5UsBUsUS}\n}"},"title":{"value":"VFIMamba: Video Frame Interpolation with State Space Models"},"pdf":{"value":"/pdf/de833adbf41a6cd01e783ddcc38df5b08822bc0f.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"zhang|vfimamba_video_frame_interpolation_with_state_space_models"},"authorids":{"value":["~Guozhen_Zhang2","~Chunxu_Liu1","~Yutao_Cui1","~Xiaotong_Zhao1","~Kai_Ma2","~Limin_Wang1"]},"authors":{"value":["Guozhen Zhang","Chunxu Liu","Yutao Cui","Xiaotong Zhao","Kai Ma","Limin Wang"]}},"version":2},{"content":{"summary":{"value":"The paper introduces Rotor-Invariant Shift Estimation (RISE), a geometric framework for understanding and controlling semantic transformations in multilingual embedding spaces. It treats meaning changes as rotational operations on the curved manifold where sentence embeddings lie, rather than as linear shifts in Euclidean space. Evaluated across multiple models and seven morphologically diverse languages, the method shows that certain discourse-level semantic transformations exhibit consistent geometric structure across both languages and model architectures."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. **Universality:** \nIf cross-model experiments reveal a consistent English-centric bias, how can the authors justify the claim of universal geometric operations across languages?\n\n2. **Semantic vs. Grammatical Structure:** \nGiven that RISE achieves near-perfect results on syntactic (BLiMP) tasks but only moderate performance on semantic (SICK) tasks, how do the authors support the claim that RISE captures semantic rather than primarily grammatical transformations?\n\n3. **Synthetic Data Validity:**\nSince all data were generated by GPT-4.5 without human or native-speaker validation, how do the authors ensure that RISE learns genuine linguistic geometric structures rather than artifacts of the data-generation model’s English-centric bias?\n\n4. **Experimental Scope and Generalization:**\nWith experiments limited to three linguistic phenomena (negation, conditionality, politeness), how can the authors substantiate broader conclusions about general geometric principles in embedding spaces?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper provides a principled geometric extension of the Linear Representation Hypothesis to non-Euclidean (Riemannian) spaces, moving beyond heuristic linear probing.\n\n2. The study’s inclusion of seven languages and three embedding architectures (OpenAI, BGE-M3, mBERT) demonstrates broad applicability and robustness."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. **Limited empirical grounding for broad claims.**\nThe paper’s experiments are not sufficiently strong or comprehensive to substantiate its broader theoretical claims. For instance, it asserts “theoretical contributions to the usefulness of steering vectors for embedding models’ controllability and interpretation,” yet the evidence presented does not convincingly bridge this conceptual gap. The “steering vector” literature predominantly addresses interventions in the activations of generative or autoregressive models, whereas this work focuses on embedding models, a fundamentally different setting. Without stronger empirical validation or comparative studies, the claimed connection to the steering-vector paradigm remains tenuous.\n\n2. **Mismatch between ambition and experimental scope.**\nThis limitation reflects a broader issue: the paper makes overgeneralized theoretical claims based on a narrow experimental design. By restricting its evaluation to a small set of highly specific discourse-level transformations, the work lacks the empirical breadth needed to justify its sweeping conclusions about universal geometric principles or semantic control in embedding spaces. The authors acknowledge this limitation, but it significantly constrains the strength of their theoretical arguments.\n\n3. **Incremental methodological novelty.**\nDespite its mathematical rigor, the proposed shift from linear to rotational/geodesic steering may appear incremental rather than groundbreaking. The core contribution lies primarily in applying standard Riemannian geometry tools (e.g., exponential and logarithmic maps on the hypersphere) to embedding analysis, rather than introducing fundamentally new geometric or algorithmic mechanisms. As a result, the perceived innovation may fall short of the paper’s ambitious framing."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762920037536,"tcdate":1761843421022,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8045/Reviewer_i6Zc"],"signatures":["ICLR.cc/2026/Conference/Submission8045/Reviewer_i6Zc"],"forum":"qEmKKvYr07","number":3,"license":"CC BY 4.0","cdate":1761843421022,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8045/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762920037536,"domain":"ICLR.cc/2026/Conference","replyto":"qEmKKvYr07","id":"XhKmcmDxYI","forumContent":{"TLDR":{"value":"Rotor-Invariant Shift Estimation (RISE), a geometric framework that enables cross-lingual and cross-model semantic transfer with improved performance on transformer-based embeddings."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Embedding Models","Steering Vectors","High Dimensional Geometry","LLMs","Unit Hypersphere"]},"primary_area":{"value":"interpretability and explainable AI"},"abstract":{"value":"Understanding how language and embedding models  encode semantic relationships is fundamental to model interpretability. \nWhile early word embeddings exhibited intuitive vector arithmetic  (''king'' - ''man'' + ''woman'' = ''queen''), modern high-dimensional text representations lack straightforward interpretable geometric properties. \nWe introduce Rotor-Invariant Shift Estimation (RISE), a geometric approach that represents semantic-syntactic transformations as consistent rotational operations in embedding space, leveraging the manifold structure of modern language representations. \nRISE operations have the ability to operate across both languages and models without reducing performance, suggesting the existence of analogous cross-lingual geometric structure.\nWe compare and evaluate RISE using two baseline methods, three embedding models, three datasets, and seven morphologically diverse languages in five major language groups. \nOur results demonstrate that RISE consistently maps discourse-level semantic-syntactic transformations with distinct grammatical features (e.g., negation and conditionality) across languages and models.\nThis work provides the first demonstration that discourse-level semantic-syntactic transformations correspond to consistent geometric operations in multilingual embedding spaces, empirically supporting the linear representation hypothesis at the sentence level."},"_bibtex":{"value":"@inproceedings{\nfreenor2026mapping,\ntitle={Mapping Semantic \\& Syntactic Relationships with Geometric Rotation},\nauthor={Michael Freenor and Lauren Alvarez},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=qEmKKvYr07}\n}"},"title":{"value":"Mapping Semantic & Syntactic Relationships with Geometric Rotation"},"pdf":{"value":"/pdf/4e0b40dbacc1e4691cf951d4eb41f381d0acb5ec.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"freenor|mapping_semantic_syntactic_relationships_with_geometric_rotation"},"authorids":{"value":["~Michael_Freenor1","~Lauren_Alvarez1"]},"authors":{"value":["Michael Freenor","Lauren Alvarez"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2509.08753v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"zeghidour|streaming_sequencetosequence_learning_with_delayed_streams_modeling"},"html":{"value":"https://doi.org/10.48550/arXiv.2509.08753"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2509-08753,\n  publtype={informal},\n  author={Neil Zeghidour and Eugene Kharitonov and Manu Orsini and Václav Volhejn and Gabriel de Marmiesse and Edouard Grave and Patrick Pérez and Laurent Mazaré and Alexandre Défossez},\n  title={Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling},\n  year={2025},\n  month={September},\n  cdate={1756684800000},\n  journal={CoRR},\n  volume={abs/2509.08753},\n  url={https://doi.org/10.48550/arXiv.2509.08753}\n}\n"},"abstract":{"value":"We introduce Delayed Streams Modeling (DSM), a flexible formulation for streaming, multimodal sequence-to-sequence learning. Sequence-to-sequence generation is often cast in an offline manner, where the model consumes the complete input sequence before generating the first output timestep. Alternatively, streaming sequence-to-sequence rely on learning a policy for choosing when to advance on the input stream, or write to the output stream. DSM instead models already time-aligned streams with a decoder-only language model. By moving the alignment to a pre-processing step,and introducing appropriate delays between streams, DSM provides streaming inference of arbitrary output sequences, from any input combination, making it applicable to many sequence-to-sequence problems. In particular, given text and audio streams, automatic speech recognition (ASR) corresponds to the text stream being delayed, while the opposite gives a text-to-speech (TTS) model. We perform extensive experiments for these two major sequence-to-sequence tasks, showing that DSM provides state-of-the-art performance and latency while supporting arbitrary long sequences, being even competitive with offline baselines. Code, samples and demos are available at https://github.com/kyutai-labs/delayed-streams-modeling"},"title":{"value":"Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling"},"authors":{"value":[{"fullname":"Neil Zeghidour","username":""},{"fullname":"Eugene Kharitonov","username":""},{"fullname":"Manu Orsini","username":""},{"fullname":"Václav Volhejn","username":""},{"fullname":"Gabriel de Marmiesse","username":""},{"fullname":"Edouard Grave","username":""},{"fullname":"Patrick Pérez","username":""},{"fullname":"Laurent Mazaré","username":""},{"fullname":"Alexandre Défossez","username":"~Alexandre_Défossez1"}]}},"tmdate":1779801568934,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2509-08753"],"tcdate":1779801565149,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Alexandre_Défossez1"],"forum":"gdi4LHpOoZ","license":"CC BY-SA 4.0","number":29900,"cdate":1756684800000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1779801568934,"domain":"OpenReview.net/Public_Article","id":"gdi4LHpOoZ","version":2},{"content":{"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["reinforcement learning","continual learning"]},"primary_area":{"value":"reinforcement_learning"},"abstract":{"value":"Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based continual learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space. Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 17 baselines, our method reaches the SOTA performance."},"_bibtex":{"value":"@inproceedings{\nhu2025tackling,\ntitle={Tackling Continual Offline {RL} through Selective Weights Activation on Aligned Spaces},\nauthor={Jifeng Hu and Sili Huang and Li Shen and Zhejian Yang and Shengchao Hu and Shisong Tang and Hechang Chen and Lichao Sun and Yi Chang and Dacheng Tao},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=KfRfTAJpjh}\n}"},"title":{"value":"Tackling Continual Offline RL through Selective Weights Activation on Aligned Spaces"},"pdf":{"value":"/pdf/66d488dcd1fe0c41ab29f7b6c286eb16f546f443.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"hu|tackling_continual_offline_rl_through_selective_weights_activation_on_aligned_spaces"},"authorids":{"value":["~Jifeng_Hu1","~Sili_Huang1","~Li_Shen1","~Zhejian_Yang1","~Shengchao_Hu1","~Shisong_Tang1","~Hechang_Chen2","~Lichao_Sun1","~Yi_Chang4","~Dacheng_Tao1"]},"authors":{"value":["Jifeng Hu","Sili Huang","Li Shen","Zhejian Yang","Shengchao Hu","Shisong Tang","Hechang Chen","Lichao Sun","Yi Chang","Dacheng Tao"]}},"tmdate":1783627430734,"pdate":1758216677571,"tcdate":1746263766896,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission6316/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission6316/Authors"],"forum":"KfRfTAJpjh","license":"CC BY 4.0","number":6316,"cdate":1746263766896,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission6316/-/Camera_Ready_Revision"],"mdate":1783627430734,"odate":1761704789071,"domain":"NeurIPS.cc/2025/Conference","id":"KfRfTAJpjh","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.15698v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"hu|solving_continual_offline_rl_through_selective_weights_activation_on_aligned_spaces"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jifeng_Hu:","https://dblp.org/search/pid/api?q=author:Sili_Huang:","https://dblp.org/search/pid/api?q=author:Li_Shen_0008:","https://dblp.org/search/pid/api?q=author:Zhejian_Yang:","https://dblp.org/search/pid/api?q=author:Shengchao_Hu:","","~Hechang_Chen2","~Yi_Chang4","https://dblp.org/search/pid/api?q=author:Dacheng_Tao:","https://dblp.org/search/pid/api?q=author:Lichao_Sun_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.15698"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-15698,\n  publtype={informal},\n  author={Jifeng Hu and Sili Huang and Li Shen and Zhejian Yang and Shengchao Hu and Shisong Tang and Hechang Chen and Yi Chang and Dacheng Tao and Lichao Sun},\n  title={Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.15698},\n  url={https://doi.org/10.48550/arXiv.2410.15698}\n}\n"},"abstract":{"value":"Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based lifelong learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space. Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 16 baselines, our method reaches the SOTA performance."},"title":{"value":"Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces"},"authors":{"value":["Jifeng Hu","Sili Huang","Li Shen","Zhejian Yang","Shengchao Hu","Shisong Tang","Hechang Chen","Yi Chang","Dacheng Tao","Lichao Sun"]}},"tmdate":1749847627904,"pdate":1704067200000,"tcdate":1736016706345,"writers":["~"],"signatures":["~Shisong_Tang2"],"forum":"sXXzN9Iq2J","license":"CC BY-SA 4.0","number":260158,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1749847627904,"domain":"DBLP.org","id":"sXXzN9Iq2J","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We propose to solve continual offline RL with various state and action spaces by adopting selective weights activation on the quantized alignment spaces."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["continual offline reinforcement learning"]},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based continual learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space. Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 16 baselines, our method reaches the SOTA performance."},"_bibtex":{"value":"@misc{\nhu2025solving,\ntitle={Solving Continual Offline {RL} through Selective Weights Activation on Aligned Spaces},\nauthor={Jifeng Hu and Sili Huang and Li Shen and Zhejian Yang and Shengchao Hu and Shisong Tang and Hechang Chen and Yi Chang and Dacheng Tao and Lichao Sun},\nyear={2025},\nurl={https://openreview.net/forum?id=Hr3TBaZl4S}\n}"},"title":{"value":"Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces"},"pdf":{"value":"/pdf/19ade0bc8e9c98b3410f06b22f7f2a308a2ed7fe.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"hu|solving_continual_offline_rl_through_selective_weights_activation_on_aligned_spaces"},"authorids":{"value":["~Jifeng_Hu1","~Sili_Huang1","~Li_Shen1","~Zhejian_Yang1","~Shengchao_Hu1","~Shisong_Tang1","~Hechang_Chen2","~Yi_Chang4","~Dacheng_Tao1","~Lichao_Sun1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jifeng Hu","Sili Huang","Li Shen","Zhejian Yang","Shengchao Hu","Shisong Tang","Hechang Chen","Yi Chang","Dacheng Tao","Lichao Sun"]}},"tmdate":1738735713985,"tcdate":1727317064550,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5556/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission5556/Authors"],"forum":"Hr3TBaZl4S","license":"CC BY 4.0","number":5556,"cdate":1727317064550,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission5556/-/Full_Submission","ICLR.cc/2025/Conference/Submission5556/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735713985,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"Hr3TBaZl4S","version":2},{"content":{"summary":{"value":"This paper proposes SPTD (Selective Prediction based on neural network Training Dynamics), a new approach to selective prediction problem. In this approach, SPTD captures the final model along with many intermediate models learned during the SGD style training method. SPTD runs these intermediate models and find the disagreement between the final model prediction and the intermediate model predictions. With some weighting scheme, a threshold based gating function is introduced to estimate the selection region. Given that this method introduces no architectural changes, it has no train-time impact (the intermediate checkpoint storage is an additional overhead). This also means that such an approach can be utilized for not only classification tasks, but other tasks such as regression."},"presentation":{"value":"3 good"},"contribution":{"value":"2 fair"},"soundness":{"value":"2 fair"},"strengths":{"value":"- No architectural changes implies no training time changes \n- Applicable for not only selective classification problems but other tasks such as regression and time-series forecasting"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Added storage overhead for the intermediate models\n- Added inference cost for the prediction using the intermediate models compared to other selective prediction models that only require one forward pass through the architecture and the gating mechanism.\n- It is unclear which of the many intermediate checkpoints should be used for the inference stage."},"confidence":{"value":"5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully."},"questions":{"value":"-  Have you plotted other baselines for the Figure 2 to see what impact these baselines have compared to SPTD?\n-  How do you select which of the intermediate checkpoints should be used for the inference stage? It is possible to design clever selection strategies during training to reduce the storage and inference cost rather than storing intermediate points at fixed checkpoint intervals.\n-  Have you tried other weighting schemes than (t/T)^k? \n-  Have you compared the inference cost of STPD with other methods (which only require one forward pass through the network and some gating mechanism)?"},"rating":{"value":"5: marginally below the acceptance threshold"},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636365215,"tcdate":1698764564981,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission4024/Reviewer_2hdK"],"signatures":["ICLR.cc/2024/Conference/Submission4024/Reviewer_2hdK"],"forum":"flg9EB6ikY","number":1,"license":"CC BY 4.0","cdate":1698764564981,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission4024/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636365215,"domain":"ICLR.cc/2024/Conference","replyto":"flg9EB6ikY","id":"ADv3hCHdHg","forumContent":{"venue":{"value":"Submitted to ICLR 2024"},"TLDR":{"value":"We propose a novel approach for selective prediction based on the prediction disagreement evolution of intermediate models with the final model's prediction."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["selective prediction","training dynamics","example difficulty","forging","reject option","uncertainty quantification","reliability"]},"supplementary_material":{"value":"/attachment/ae1f8afaa163b43fbee1e2c11b645219a584a233.zip"},"primary_area":{"value":"societal considerations including fairness, safety, privacy"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Selective prediction is the task of rejecting inputs a model would predict incorrectly on through a trade-off between input space coverage and model utility. Current methods for selective prediction typically impose constraints on either the model architecture or the loss function; this inhibits their usage in practice. In contrast to prior work, we show that state-of-the-art selective prediction performance can be attained solely from studying the (discretized) training dynamics of a model. We propose a general framework that, given a test input, monitors metrics capturing the instability of predictions from intermediate models obtained during training w.r.t. the final model's prediction. In particular, we reject data points exhibiting too much disagreement with the final prediction at late stages in training. The proposed scoring mechanism is domain-agnostic (i.e., it works for both discrete and real-valued prediction) and can be flexibly combined with existing selective prediction approaches as it does not require any train-time modifications. Our experimental evaluation on image classification, regression, and time series forecasting problems shows that our method beats past state-of-the-art accuracy/utility trade-offs on typical selective prediction benchmarks."},"_bibtex":{"value":"@misc{\nrabanser2024selective,\ntitle={Selective Prediction via Training Dynamics},\nauthor={Stephan Rabanser and Anvith Thudi and Kimia Hamidieh and Adam Dziedzic and Israfil Bahceci and Akram Bin Sediq and HAMZA SOKUN and Nicolas Papernot},\nyear={2024},\nurl={https://openreview.net/forum?id=flg9EB6ikY}\n}"},"title":{"value":"Selective Prediction via Training Dynamics"},"pdf":{"value":"/pdf/423d746e5f704f6a5de8d91a782b606befaa5cc1.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"rabanser|selective_prediction_via_training_dynamics"},"authorids":{"value":["~Stephan_Rabanser1","~Anvith_Thudi1","~Kimia_Hamidieh1","~Adam_Dziedzic1","~Israfil_Bahceci1","~Akram_Bin_Sediq1","~HAMZA_SOKUN1","~Nicolas_Papernot1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Stephan Rabanser","Anvith Thudi","Kimia Hamidieh","Adam Dziedzic","Israfil Bahceci","Akram Bin Sediq","HAMZA SOKUN","Nicolas Papernot"]}},"version":2},{"content":{"summary":{"value":"The paper presents a method to correctly apply LRP to Mamba models. Through careful analysis, the authors demonstrate that applying LRP directly results in poor performance and propose modifications to recover the propagation rules. The obtained method outperforms the alternatives, grounded by theory, and the authors show several applications of their method, including identifying gender biases and measuring the long-range abilities of S6."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1.\tMamba LRP uses \"Detach\" on the SSM recurrent matrices. Similarly, [4] uses an attention matrix determined by A, B, and C parameters.  It seems that both methods ignore the influence of the input on the system matrices (which is the core of the selection mechanism of Mamba). Am I right? Could addressing this issue provide a way to improve both methods? I would be glad to hear what the authors think about it.\n2.\tMinor: Traditionally in the SSM domain, Delta denotes the step size. I suggest the authors replace \\Delta with \\delta when discussing the differences between the two scores.\n3.\tIn the Needle-in-a-haystack experiment (Figure 7), is there a reason not to increase the context length? It seems that the most interesting part of the figure is missing (which could show if there are edge cases, for example, regimes where the model succeeds in finding the needle, but Mamba-LRP fails).\n4.\tMinor: Perhaps a relevant work that is missing is “Does Transformer Interpretability Transfer to RNNs?” by Paulo et al.\n5.\tThere is a standard trend in the SSM literature to omit D (and treat it as a parameter-based skip connection). Is it used in Mamba LRP, or is it ignored (like other biases, which are typically non-attributable)? It would be better if it were written explicitly in the paper.\n6.\tI wonder if the authors can explore the potential limitations or failure cases of the proposed Mamba-LRP. Such information can help the community improve the method in the future. Are there cases where [4] or naive LRP might be better than Mamba-LRP?\n7.\tHalf propagation: While I'm not an expert in LRP, I suspect there are more effective methods to manage the gating mechanism. For instance, instead of normalizing the scores by averaging (0.5(x + y)), it may be better to use a weighted approach such as $$(1 - a) \\cdot x + a\\cdot y$$\nwhere the value of a is determined by the actual norms of x and y, with different a values for each channel. Is there something I'm missing? Can this method improve the conservation properties?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1) **Impact:** Mamba is an emerging architecture (~600+ citations, 10K+ stars on GitHub). Thus, providing an attribution method for these models is crucial. In this essence, correctly applying LRP is an important direction that facilitates the community in improving and understanding these models.\n\n2) **Simplicity:** The method and modifications are very simple, with few hyper-parameters. While this might be seen as a lack of novelty, I view it as an advantage, making the method easy to use and adaptable to various applications, such as other variants of Mamba (Mamba2) and other domains (DNA, speech and more). \n\n3) **Informative Ablation studies** and justification of decision choices are insightful. For example, Figure 3 and Tables 1, 2, 6, 9 provide comparisons with naive LRP and also present ablation studies that allow the reader to measure the contribution of each modification to the method.\n\n4) Section 6 (use cases) shows that the method is **applicable** and allows the authors to explore the gender bias and long-range abilities of Mamba models and provide insightful analyses about Mamba models."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1) **The comparison with previous work should be improved:**\n\n- 1.1) Metrics and Benchmarking: Can the authors highlight which results are reproduced by them and which results are taken from previous work? Moreover, as far as I understand, the method in [4] is the only method that developed an interpretability method for Mamba before (perhaps in parallel) to this work. If I understand correctly, although the two methods share and use the same pre-trained models, there is no overlap in the metrics. Am I correct? If so, is there a reason for this discrepancy? Can the authors compare results directly with previous work using previously proposed metrics to ensure the gap doesn’t arise from employing [4] incorrectly?\n\n- 1.2)  Informative Comparison: Additionally, I think the comparison with [4] can be more accurate. The method in [4] applies its method only to S6 layers (without gating and convs), while Mamba LRP is an end-to-end method (which is a strength of Mamba-LRP). However, it is still important to make some apple-to-apple comparisons. Can the authors check both methods on a model without conv and gating layers to determine which approach is better for providing an explanation to S6 layers? Alternatively, it seems that the issue with [4] is fixed in [Uni]. Would the authors be able to compare their method to [Uni]? Providing this comparison would be highly valuable to the community. (I understand that this is a very new paper, so I will not decrease my score if the results are less favorable than those of [Uni]).\n\n2) **Insufficient empirical analysis:**\n\n- 2.1) For the long-range experiments (Figure 6), it would be very insightful to compare the behaviour of Mamba to Transformers (Pythia pre-trained models can serve as the transformer baselines since they are trained on the same data). Additionally, analyzing the trend with larger models could yield valuable information on how increased model size enhances long-context capabilities in practice, perhaps the 7B models from [Sca] can be used (which exist in Hugging Face).\n\n- 2.2)  \n>  “Residual lack of conservation is due to the presence of biases in linear and convolution layers, which are typically non-attributable” \n\n Can the authors empirically justify this claim? This can be easily validated by taking a pre-trained model, omitting the biases, fine-tuning it for several epochs, and then checking the conservation again. \n\n3) **Novelty:** The novelty of the method is somewhat limited. One could argue that it merely involves a few applications of detach and the simple half-propagation rule (in addition to previously proposed contributions), which can be easily summarized in a few lines of code (as detailed in Algorithm 2). However, I believe this is not a significant drawback, particularly given the demand for such simple tools in the community. Additionally, the thorough evaluation, including insightful ablation studies, novel test case evaluations, and theoretical justification, is sufficiently robust.\n\n[Uni] A Unified Implicit Attention Formulation for Gated-Linear Recurrent Sequence. Models. Zimerman et al.\n\n[Sca] An Empirical Study of Mamba-based Language Models. Waleffe et al."},"limitations":{"value":"From my perspective, the authors address most of the limitations, except those pointed out in the weaknesses section and question 6 (failure cases)."}},"nonreaders":[],"tmdate":1730879313038,"tcdate":1720386595925,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission9423/Reviewer_dNt6"],"signatures":["NeurIPS.cc/2024/Conference/Submission9423/Reviewer_dNt6"],"forum":"2n1Ysn1EDl","number":1,"license":"CC BY 4.0","cdate":1720386595925,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission9423/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879313038,"domain":"NeurIPS.cc/2024/Conference","replyto":"2n1Ysn1EDl","id":"PQqGFxIXiq","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Explainable AI","explainability","interpretability","state space models","Mamba","long-range dependency"]},"primary_area":{"value":"interpretability_and_explainability"},"abstract":{"value":"Recent sequence modeling approaches using selective state space sequence models, referred to as Mamba models, have seen a surge of interest. These models allow efficient processing of long sequences in linear time and are rapidly being adopted in a wide range of applications such as language modeling, demonstrating promising performance. To foster their reliable use in real-world scenarios, it is crucial to augment their transparency. Our work bridges this critical gap by bringing explainability, particularly Layer-wise Relevance Propagation (LRP), to the Mamba architecture. Guided by the axiom of relevance conservation, we identify specific components in the Mamba architecture, which cause unfaithful explanations. To remedy this issue, we propose MambaLRP, a novel algorithm within the LRP framework, which ensures a more stable and reliable relevance propagation through these components. Our proposed method is theoretically sound and excels in achieving state-of-the-art explanation performance across a diverse range of models and datasets. Moreover, MambaLRP facilitates a deeper inspection of Mamba architectures, uncovering various biases and evaluating their significance. It also enables the analysis of previous speculations regarding the long-range capabilities of Mamba models."},"_bibtex":{"value":"@inproceedings{\njafari2024mambalrp,\ntitle={Mamba{LRP}: Explaining Selective State Space Sequence Models},\nauthor={Farnoush Rezaei Jafari and Gr{\\'e}goire Montavon and Klaus Robert Muller and Oliver Eberle},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=2n1Ysn1EDl}\n}"},"title":{"value":"MambaLRP: Explaining Selective State Space Sequence Models"},"pdf":{"value":"/pdf/d253dfe5b75f15bb96b1033361699ed2984da98a.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"jafari|mambalrp_explaining_selective_state_space_sequence_models"},"authorids":{"value":["~Farnoush_Rezaei_Jafari1","~Grégoire_Montavon1","~Klaus_Robert_Muller1","~Oliver_Eberle1"]},"authors":{"value":["Farnoush Rezaei Jafari","Grégoire Montavon","Klaus Robert Muller","Oliver Eberle"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.09564v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"gou|mambadriven_multiperspective_structural_understanding_for_molecular_groundstate_conformation_prediction"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Yuxin_Gou:","https://dblp.org/search/pid/api?q=author:Aming_Wu:","~Richang_Hong1","~Meng_Wang2"]},"html":{"value":"https://doi.org/10.48550/arXiv.2511.09564"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-09564,\n  publtype={informal},\n  author={Yuxin Gou and Aming Wu and Richang Hong and Meng Wang},\n  title={Mamba-driven multi-perspective structural understanding for molecular ground-state conformation prediction},\n  year={2025},\n  month={November},\n  cdate={1761955200000},\n  journal={CoRR},\n  volume={abs/2511.09564},\n  url={https://doi.org/10.48550/arXiv.2511.09564}\n}\n"},"abstract":{"value":"A comprehensive understanding of molecular structures is important for the prediction of molecular ground-state conformation involving property information. Meanwhile, state space model (e.g., Mamba) has recently emerged as a promising mechanism for long sequence modeling and has achieved remarkable results in various language and vision tasks. However, towards molecular ground-state conformation prediction, exploiting Mamba to understand molecular structure is underexplored. To this end, we strive to design a generic and efficient framework with Mamba to capture critical components. In general, molecular structure could be considered to consist of three elements, i.e., atom types, atom positions, and connections between atoms. Thus, considering the three elements, an approach of Mamba-driven multi-perspective structural understanding (MPSU-Mamba) is proposed to localize molecular ground-state conformation. Particularly, for complex and diverse molecules, three different kinds of dedicated scanning strategies are explored to construct a comprehensive perception of corresponding molecular structures. And a bright-channel guided mechanism is defined to discriminate the critical conformation-related atom information. Experimental results on QM9 and Molecule3D datasets indicate that MPSU-Mamba significantly outperforms existing methods. Furthermore, we observe that for the case of few training samples, MPSU-Mamba still achieves superior performance, demonstrating that our method is indeed beneficial for understanding molecular structures."},"title":{"value":"Mamba-driven multi-perspective structural understanding for molecular ground-state conformation prediction"},"authors":{"value":["Yuxin Gou","Aming Wu","Richang Hong","Meng Wang"]}},"tmdate":1769474202766,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2511-09564"],"tcdate":1768795930669,"writers":["~"],"signatures":["~Richang_Hong1"],"forum":"oL5fgg6fWM","license":"CC BY-SA 4.0","number":770843,"cdate":1761955200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769474202766,"domain":"DBLP.org","id":"oL5fgg6fWM","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2512.06657v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"zhao|textmamba_scene_text_detector_with_mamba"},"html":{"value":"https://doi.org/10.48550/arXiv.2512.06657"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2512-06657,\n  publtype={informal},\n  author={Qiyan Zhao and Yue Yan and Da-Han Wang},\n  title={TextMamba: Scene Text Detector with Mamba},\n  year={2025},\n  month={December},\n  cdate={1764547200000},\n  journal={CoRR},\n  volume={abs/2512.06657},\n  url={https://doi.org/10.48550/arXiv.2512.06657}\n}\n"},"abstract":{"value":"In scene text detection, Transformer-based methods have addressed the global feature extraction limitations inherent in traditional convolution neural network-based methods. However, most directly rely on native Transformer attention layers as encoders without evaluating their cross-domain limitations and inherent shortcomings: forgetting important information or focusing on irrelevant representations when modeling long-range dependencies for text detection. The recently proposed state space model Mamba has demonstrated better long-range dependencies modeling through a linear complexity selection mechanism. Therefore, we propose a novel scene text detector based on Mamba that integrates the selection mechanism with attention layers, enhancing the encoder's ability to extract relevant information from long sequences. We adopt the Top\\_k algorithm to explicitly select key information and reduce the interference of irrelevant information in Mamba modeling. Additionally, we design a dual-scale feed-forward network and an embedding pyramid enhancement module to facilitate high-dimensional hidden state interactions and multi-scale feature fusion. Our method achieves state-of-the-art or competitive performance on various benchmarks, with F-measures of 89.7\\%, 89.2\\%, and 78.5\\% on CTW1500, TotalText, and ICDAR19ArT, respectively. Codes will be available."},"title":{"value":"TextMamba: Scene Text Detector with Mamba"},"authors":{"value":[{"fullname":"Qiyan Zhao","username":"~Qiyan_Zhao1"},{"fullname":"Yue Yan","username":""},{"fullname":"Da-Han Wang","username":""}]}},"tmdate":1784725925496,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2512-06657"],"tcdate":1784725919425,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Qiyan_Zhao1"],"forum":"npXAn3q96b","license":"CC BY-SA 4.0","number":91560,"cdate":1764547200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784725925496,"domain":"OpenReview.net/Public_Article","id":"npXAn3q96b","version":2},{"content":{"TLDR":{"value":"We introduce TiledFlashLinearAttention a faster kernel algorithm for Linear RNNs and mLSTMs by improved Sequence Parallelism."},"venue":{"value":"ICLR 2025 FM-Wild Workshop"},"pdf":{"value":"/pdf/5fd4361c26bb5a4031976b03ed77ce45a4bf1f62.pdf"},"keywords":{"value":["xLSTM","FlashLinearAttention","FlashAttention","Linear Attention","Linear RNN","RNN","mLSTM","Attention","Self-Attention","Transformers","LLM","Kernels","Systems","Efficiency"]},"venueid":{"value":"ICLR.cc/2025/Workshop/FM-Wild"},"paperhash":{"value":"beck|tiled_flash_linear_attention_more_efficient_linear_rnn_and_xlstm_kernels"},"authorids":{"value":["~Maximilian_Beck1","~Korbinian_Pöppel1","~Phillip_Lippe1","~Sepp_Hochreiter1"]},"abstract":{"value":"Linear RNNs with gating recently demonstrated competitive performance compared to Transformers in language modeling.\nAlthough their linear compute scaling in sequence length offers theoretical runtime advantages over Transformers, realizing these benefits in practice requires optimized custom kernels, as Transformers rely on the highly efficient FlashAttention kernels (Dao, 2024).\nLeveraging the chunkwise-parallel formulation of linear RNNs, FlashLinearAttention (FLA) (Yang & Zhang, 2024) shows that linear RNN kernels are faster than FlashAttention, by parallelizing over chunks of the input sequence.\nHowever, since the chunk size of FLA is limited, many intermediate states must be materialized in GPU memory. \nThis causes high memory consumption and IO cost, especially for long-context pre-training.  \nIn this work, we present Tiled Flash Linear Attention (TFLA), a novel kernel algorithm for linear RNNs, that enables arbitrary large chunk sizes by introducing an additional level of sequence parallelization within each chunk.\nFirst, we apply TFLA to the xLSTM with matrix memory, the mLSTM (Beck et al., 2024). \nSecond, we propose an mLSTM variant with sigmoid input gate and reduced computation for even faster kernel runtimes at equal language modeling performance.\nIn our speed benchmarks, we show that our new mLSTM kernels based on TFLA outperform highly optimized FlashAttention, Linear Attention and Mamba kernels, setting a new state of the art for efficient long-context sequence modeling primitives."},"_bibtex":{"value":"@inproceedings{\nbeck2025tiled,\ntitle={Tiled Flash Linear Attention: More Efficient Linear {RNN} and x{LSTM} Kernels},\nauthor={Maximilian Beck and Korbinian P{\\\"o}ppel and Phillip Lippe and Sepp Hochreiter},\nbooktitle={ICLR 2025 Workshop on Foundation Models in the Wild},\nyear={2025},\nurl={https://openreview.net/forum?id=P6CPFdexbk}\n}"},"title":{"value":"Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels"},"authors":{"value":["Maximilian Beck","Korbinian Pöppel","Phillip Lippe","Sepp Hochreiter"]}},"tmdate":1742397093340,"pdate":1741257916196,"tcdate":1738676311217,"writers":["ICLR.cc/2025/Workshop/FM-Wild","ICLR.cc/2025/Workshop/FM-Wild/Submission30/Authors"],"signatures":["ICLR.cc/2025/Workshop/FM-Wild/Submission30/Authors"],"forum":"P6CPFdexbk","license":"CC BY 4.0","number":30,"cdate":1738676311217,"readers":["everyone"],"invitations":["ICLR.cc/2025/Workshop/FM-Wild/-/Submission","ICLR.cc/2025/Workshop/FM-Wild/-/Post_Submission","ICLR.cc/2025/Workshop/FM-Wild/-/Edit","ICLR.cc/2025/Workshop/FM-Wild/Submission30/-/Camera-Ready_Version"],"mdate":1742397093340,"odate":1741257916196,"domain":"ICLR.cc/2025/Workshop/FM-Wild","id":"P6CPFdexbk","version":2},{"content":{"summary":{"value":"This paper introduces the selective state space model, renowned for its long-range modeling within linear complexity, into the image deblurring task, further developing an aggregate local and global information block. Experimental results demonstrate that the proposed algorithm outperforms state-of-the-art approaches on various benchmarks."},"suitability":{"value":2},"strengths":{"value":"1. The proposed approach conduct the complementary global and local branch, implemented by a selective state space model and channel attention. \n\n2. The proposed ALGNet achieves the superior performance while preserving efficiency."},"confidence":{"value":3},"rating":{"value":3},"limitations":{"value":"1. MambaIR designs a Residue State Space Block by cascading VSSM and channel attention. It seem that the ALGNet reformulates this cascading structure into a parallel one. The authors are encouraged to discuss the specific improvements brought by this reformulation, excluding the performance gains. \n\n2. The authors claim that this combination of global and local brancher not only addresses issues like local pixel forgetting and channel redundancy but also empowers the network to capture more enriched and precise features. However, Figure 3 only illustrates the channel redundancy of the global branch. It does not adequately demonstrate how introducing the local branch alleviates this problem. Additional evidence is needed to support this claim.\n\n3. The reviewer is curious whether cascading global and local branches can achieve comparable performance and diverse channel responses. This alternative structure should be explored and compared with the proposed parallel design.\n\n4. The compared baselines in Tables 1, 2, and 4 are inconsistent. The authors are suggested to include the missing baselines in Tables 2 and 4. Specifically, Table 4 lacks comparisons with efficient algorithms like FFTformer [1]. Including such comparisons would provide a more comprehensive evaluation of the proposed method’s performance and efficiency. \n\n[1] Kong L, Dong J, Ge J, et al. Efficient frequency domain-based transformers for high-quality image deblurring[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 5886-5895."}},"nonreaders":[],"tmdate":1721657179151,"tcdate":1715935693239,"writers":["acmmm.org/ACMMM/2024/Conference","acmmm.org/ACMMM/2024/Conference/Submission1277/Reviewer_K25P"],"signatures":["acmmm.org/ACMMM/2024/Conference/Submission1277/Reviewer_K25P"],"forum":"yoofksFKk9","number":1,"license":"CC BY 4.0","cdate":1715935693239,"readers":["everyone"],"invitations":["acmmm.org/ACMMM/2024/Conference/Submission1277/-/Official_Review","acmmm.org/ACMMM/2024/Conference/-/Edit","acmmm.org/ACMMM/2024/Conference/Submission1277/Official_Review1/-/Final_Rating"],"mdate":1721657179151,"domain":"acmmm.org/ACMMM/2024/Conference","replyto":"yoofksFKk9","id":"VGHntoENhb","forumContent":{"venue":{"value":"MM2024 Poster"},"supplementary_material":{"value":"/attachment/e6771c423b45e04e75a97a0afcf72c23724da4f0.zip"},"abstract":{"value":"Image deblurring aims to restore a high-quality image from its corresponding blurred. The emergence of CNNs and Transformers has enabled significant progress. However, these methods often face the dilemma between eliminating long-range degradation perturbations and maintaining computational efficiency. While the selective state space model (SSM) shows promise in modeling long-range dependencies with linear complexity, it also encounters challenges such as local pixel forgetting and channel redundancy. To address this issue, we propose an efficient image deblurring network that leverages selective state spaces model to aggregate enriched and accurate features. Specifically, we introduce an aggregate local and global information block (ALGBlock) designed to effectively capture and integrate both local invariant properties and non-local information. The ALGBlock comprises two primary modules: a module for capturing local and global features (CLGF), and  a feature aggregation module (FA).  The CLGF module is composed of two branches: the global branch captures long-range dependency features via a selective state spaces model, while the local branch employs simplified channel attention to model local connectivity, thereby reducing local pixel forgetting and channel redundancy. In addition, we design a FA module to accentuate the local part by recalibrating the weight during the aggregation of the two branches for restoration. Experimental results demonstrate that the proposed method outperforms state-of-the-art approaches on widely used benchmarks."},"relevance_to_conference":{"value":"In this paper, we propose an efficiently aggregates local and global features network (ALGNet) that leverages selective state spaces model for image deblurring. Extensive experiments demonstrate that our ALGNet achieves favorably performance against state-of-the-art methods."},"_bibtex":{"value":"@inproceedings{\ngao2024learning,\ntitle={Learning Enriched Features via Selective State Spaces Model for Efficient Image Deblurring},\nauthor={Hu Gao and Bowen Ma and Ying Zhang and Jingfan Yang and Jing Yang and Depeng Dang},\nbooktitle={ACM Multimedia 2024},\nyear={2024},\nurl={https://openreview.net/forum?id=yoofksFKk9}\n}"},"title":{"value":"Learning Enriched Features via Selective State Spaces Model for Efficient Image Deblurring"},"secondary_subject_area":{"value":["[Experience] Multimedia Applications"]},"pdf":{"value":"/pdf/edef71b7f4e8b881c4803643a7655e9bd3b3fda3.pdf"},"venueid":{"value":"acmmm.org/ACMMM/2024/Conference"},"paperhash":{"value":"gao|learning_enriched_features_via_selective_state_spaces_model_for_efficient_image_deblurring"},"primary_subject_area":{"value":"[Content] Vision and Language"},"authorids":{"value":["~Hu_Gao1","~Bowen_Ma3","~Ying_Zhang27","~Jingfan_Yang1","~Jing_Yang18","~Depeng_Dang1"]},"authors":{"value":["Hu Gao","Bowen Ma","Ying Zhang","Jingfan Yang","Jing Yang","Depeng Dang"]}},"version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"venue":{"value":"HiLD at ICML 2026 Poster"},"TLDR":{"value":"We show that the many-token limit of Mamba is a causal Volterra equation with an explicit memory kernel, unlike the mean-field attention limits of transformers."},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["selective state space models","Mamba","Volterra equations","continuum limits","interacting particle systems"]},"abstract":{"value":"Transformers admit continuum descriptions based on mean-field interactions, but selective state space models fall into a different class.\nWe show that the many-token limit of an input-conditioned, single-head SISO Mamba-3 block is a \\emph{causal Volterra equation} on the sphere with an explicit exponential memory kernel. The key reason is that Mamba's causal mask is chain-dependent: the influence of one token on another is transmitted through the full sequence of intermediate gates rather than a single pairwise weight.\nUnder a constant-horizon scaling, we prove convergence to this limit, validate it numerically, and show that the same framework covers the SISO Mamba-3 rotation, with Mamba-2 as a special case. We also characterize the resulting memory-horizon interpolation and relate the kernel scale to pretrained Mamba-2 heads."},"_bibtex":{"value":"@inproceedings{\nchang2026causal,\ntitle={Causal Volterra Dynamics of Mamba},\nauthor={Ming-Ching Chang and Ting Yu Tsai and Davis Wertheimer and Felix X.-F. Ye},\nbooktitle={High-dimensional Learning Dynamics 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=aIbMv2GhHp}\n}"},"title":{"value":"Causal Volterra Dynamics of Mamba"},"pdf":{"value":"/pdf/4e6f8108f8a654799cab30b96e8225c2083d3e48.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/HiLD"},"paperhash":{"value":"chang|causal_volterra_dynamics_of_mamba"},"authorids":{"value":["~Ming-Ching_Chang1","~Ting_Yu_Tsai1","~Davis_Wertheimer1","~Felix_X.-F._Ye1"]},"authors":{"value":["Ming-Ching Chang","Ting Yu Tsai","Davis Wertheimer","Felix X.-F. Ye"]}},"tmdate":1780074341620,"pdate":1780074340584,"tcdate":1777431378016,"writers":["ICML.cc/2026/Workshop/HiLD","ICML.cc/2026/Workshop/HiLD/Submission9/Authors"],"signatures":["ICML.cc/2026/Workshop/HiLD/Submission9/Authors"],"forum":"aIbMv2GhHp","license":"CC BY 4.0","number":9,"cdate":1777431378016,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/HiLD/-/Submission","ICML.cc/2026/Workshop/HiLD/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/HiLD/-/Submission_Change_Before_Bidding","ICML.cc/2026/Workshop/HiLD/Submission9/-/Camera_Ready_Revision","ICML.cc/2026/Workshop/HiLD/-/Submission_Release"],"mdate":1780074341620,"odate":1780074340584,"domain":"ICML.cc/2026/Workshop/HiLD","id":"aIbMv2GhHp","version":2},{"content":{"review":{"value":"The proposal presents a small-scale Mamba architecture-based protein sequence generation model, aiming to efficiently handle long sequences and simulate protein dynamics through selective state-space updates. The study emphasizes achieving competitive performance while reducing computational resource requirements, making advanced protein sequence modeling more accessible to the academic research community. With a multi-dimensional evaluation framework and experimental workflow, the study is expected to provide valuable references for the protein engineering field, especially in computationally constrained academic settings.\n\n### Strengths:\n\n①The proposed Mamba architecture offers linear computational complexity, efficient handling of long-range sequence dependencies, and natural alignment with protein dynamics.\n②The model design considers parameter optimization to balance expressiveness and efficiency, adaptable hidden state dimensions based on sequence complexity, and efficient batch processing for training acceleration.\n③The comprehensive evaluation framework covers multiple aspects of protein sequence analysis and generation, including sequence modeling capability, computational efficiency analysis, and sequence quality assessment.\n④The experimental design is rigorous, using the UniRef50 dataset to ensure comprehensive coverage while maintaining computational feasibility.\n### Weaknesses:\n\n①Although the advantages of the Mamba architecture are proposed, the proposal lacks specific details on how to handle complex biochemical characteristics within protein sequences.\n②While the experimental part is well-designed, there is a lack of assessment of the model's generalization ability, especially its performance on different types of protein sequences.\n③The proposal is not clear enough on the model's scalability and the direction of future work."},"confidence":{"value":4},"rating":{"value":9},"title":{"value":"Brief review"}},"nonreaders":[],"tmdate":1731406854400,"tcdate":1731406854400,"writers":["tsinghua.edu.cn/THU/2024/Fall/AML","~ChenJian1"],"signatures":["~ChenJian1"],"forum":"I2FbCxyYVS","number":8,"license":"CC BY 4.0","cdate":1731406854400,"readers":["everyone"],"invitations":["tsinghua.edu.cn/THU/2024/Fall/AML/Submission52/-/Official_Review"],"mdate":1731406854400,"domain":"tsinghua.edu.cn/THU/2024/Fall/AML","replyto":"I2FbCxyYVS","id":"KSv1WStVlA","forumContent":{"TLDR":{"value":"This proposal explores implementing a Mamba architecture for efficient protein sequence generation, aiming to balance performance with reduced computational resource demands and making the architecture more suited to biological dynamics."},"venue":{"value":"THU 2024 Fall AML Submission"},"pdf":{"value":"/pdf/8d5ee7449a5695662b51c83289c76642879f2be4.pdf"},"keywords":{"value":["protein sequence generation","Mamba architecture","efficient modeling","protein language models","biological dynamics"]},"venueid":{"value":"tsinghua.edu.cn/THU/2024/Fall/AML/Submission"},"paperhash":{"value":"jiang|[proposalml]protein_sequence_generation_model"},"authorids":{"value":["~pan_jiang1","zeeshanzulfiqar79@outlook.com","ivanyazykov.mipt@gmail.com"]},"abstract":{"value":"This proposal outlines a novel approach for protein sequence generation, a key area in computational biology focused on generating sequences with specific functional and structural characteristics. Despite recent advances with deep learning models, challenges remain in handling long sequences and capturing biological dynamics. We propose implementing a Mamba architecture that leverages selective state-space updates to achieve efficient protein sequence generation. With linear computational complexity and effective handling of long-range dependencies, this architecture is designed to be more suited to biological dynamics while reducing resource requirements. A comprehensive evaluation on the UniRef50 dataset will demonstrate its potential to deliver competitive performance, providing a viable solution for research environments with limited computational resources."},"_bibtex":{"value":"@inproceedings{\njiang2024proposalmlprotein,\ntitle={[Proposal-{ML}]Protein Sequence Generation Model},\nauthor={pan jiang and Zeeshan Zulfiqar and Ivan Iazykov},\nbooktitle={Submitted to Tsinghua University Course: Advanced Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=I2FbCxyYVS},\nnote={under review}\n}"},"title":{"value":"[Proposal-ML]Protein Sequence Generation Model"},"authors":{"value":["pan jiang","Zeeshan Zulfiqar","Ivan Iazykov"]}},"version":2},{"content":{"summary":{"value":"The paper investigates the limits of prompt tuning in transformer-based models by focusing on the simplest architectures with a single self-attention layer and head. The authors prove that such simple transformers can universally approximate any Lipschitz continuous sequence-to-sequence function through prompt tuning. They also demonstrate that these transformers can memorize any dataset via prompt tuning by establishing an exponential lower bound on the required number of soft-prompt tokens based on data dimension, sequence length, and desired error tolerance. Additionally, they identify a computational efficiency phase transition determined by the norms of the soft-prompt-induced keys and queries. They establish an efficiency threshold beyond which no sub-quadratic algorithm exists under the SETH, but within this threshold, they prove the existence of almost linear-time inference algorithms for prompt tuning."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. The paper \"Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?\" stated that they easily extended their analysis of contextual mapping to masked self-attention, which is what's actually used in practice. Since such an extension might be beneficial for completeness, could the authors elaborate on why they haven't addressed it? Was it more challenging in this case?\n\n2. The identified efficiency criterion (Theorem 3.1) requires controlling the norms of the soft-prompt-induced keys and queries to be $O(\\log(L_p + L))$. In practical applications, particularly with large-scale models and datasets, how feasible is it to meet this criterion?\n\n**Minor Remarks**\n\n1. In order to make them multiplication-compatible, $W_O^i$ should be $d \\times s$, and this doesn’t propagate to later sections.\n\n2. The infinity norm is already defined in matrices, which gives the maximum row sum, and perhaps a better notation would be $||M||_{max}$ for clarity."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"1. The paper advances the theory of prompt tuning by proving that even the simplest transformer architectures—with a single attention head and a single self-attention layer—can universally approximate any Lipschitz continuous sequence-to-sequence function.\n \n2. The paper notably discovers a phase transition in computational efficiency based on the norms of the soft-prompt-induced keys and queries. It provides a criterion under which prompt tuning can be performed in sub-quadratic time and demonstrates the existence of almost linear time algorithms under certain conditions.\n\n3. The paper introduces new applications, such as the chained reduction of piece-wise constant approximations, to prove universality results. It also extends the contextual mapping property to any-rank weight matrices in single-layer attention mechanisms, improving the rank-1 limitation of previous work."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper contains numerous grammatical errors, and some sentences are poorly constructed. A thorough proofreading is required to enhance the readability of the text.\n\n2. While the paper brings new applications of previous concepts to function approximation, the technical novelty appears limited, with many proofs in Section 2 overlapping significantly with those used in \"Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?\" and Universality and Limitations of Prompt Tuning\" papers.\n\n3. The paper does not include experiments to showcase any of the theoretical results. The authors could demonstrate the approximation capabilities of single-head, single-layer transformers for sequence-to-sequence functions in Section 2. The authors can even use synthetic datasets designed to have known Lipschitz properties. Deeper transformers with more heads and layers might be used for comparison.\n \n4. Another empirical comparison might be examining the relationship between the number of soft-prompt tokens and the model's ability to memorize datasets, as suggested by the exponential lower bound. Again, one might utilize synthetic datasets where the complexity (e.g., vocabulary size, sequence length) can be controlled.\n\n5. I think the authors should perhaps discuss and more on the results and analysis of the “Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?” and compare the results as they prove that transformers with one self-attention layer are universal approximators for permutation equivariant continuous functions. Also, explaining how the results in the current paper relate to or differ from theirs might provide insights."}},"nonreaders":[],"tmdate":1731428260304,"tcdate":1730684103962,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4923/Reviewer_CLcT"],"signatures":["ICLR.cc/2025/Conference/Submission4923/Reviewer_CLcT"],"forum":"jDpdQPMosW","number":4,"license":"CC BY 4.0","cdate":1730684103962,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4923/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428260304,"domain":"ICLR.cc/2025/Conference","replyto":"jDpdQPMosW","id":"iEE4kS2Mne","forumContent":{"TLDR":{"value":"We study the statistical and computational limits of prompt tuning in single-layer, single-head transformer, showing it is universal seq2seq approximator and, supports almost linear efficient inference."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Foundation Model","Prompt Tuning","Transformer","Universal Approximation","Memory Capacity","Computational Efficiency","Fine-Grained Complexity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We investigate the statistical and computational limits of prompt tuning for transformer-based foundation models.  Our key contributions are that prompt tuning on *single-head* transformers with only a *single* self-attention layer:  (i) is universal, and  (ii) supports efficient (even almost-linear time) algorithms under the Strong Exponential Time Hypothesis (SETH).  Statistically,  we prove that prompt tuning on such the simplest possible transformers are universal approximators for sequence-to-sequence Lipschitz functions.  In addition, we provide an exponential-in-$dL$ and -in-$(1/\\epsilon)$ lower bound on the required soft-prompt tokens for prompt tuning to memorize any dataset with 1-layer, 1-head transformers.  Computationally, we identify a phase transition in the efficiency of prompt tuning, determined by the norm of the *soft-prompt-induced* keys and queries, and provide an upper bound criterion.  Beyond this criterion, no sub-quadratic (efficient) algorithm for prompt tuning exists under SETH.  Within this criterion,  we showcase our theory by proving the existence of almost-linear time prompt tuning inference algorithms.  These fundamental limits provide important necessary conditions for designing expressive and efficient prompt tuning methods for practitioners."},"_bibtex":{"value":"@inproceedings{\nhu2025fundamental,\ntitle={Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency},\nauthor={Jerry Yao-Chieh Hu and Wei-Po Wang and Ammar Gilani and Chenyang Li and Zhao Song and Han Liu},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=jDpdQPMosW}\n}"},"title":{"value":"Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency"},"pdf":{"value":"/pdf/1007439db913bf6449631a810f98cc17ef76fc84.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"hu|fundamental_limits_of_prompt_tuning_transformers_universality_capacity_and_efficiency"},"authorids":{"value":["~Jerry_Yao-Chieh_Hu1","~Wei-Po_Wang1","~Ammar_Gilani1","~Chenyang_Li6","~Zhao_Song3","~Han_Liu4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jerry Yao-Chieh Hu","Wei-Po Wang","Ammar Gilani","Chenyang Li","Zhao Song","Han Liu"]}},"version":2},{"content":{"summary":{"value":"[AI Review] This proposal titled \"Automated Exoplanet Transit Detection from Stellar Light Curves\" aims to use a Mamba + Attention architecture for exoplanet transit detection but fails to deliver substantive technical content. The paper reads as a literature review with project aspirations rather than a concrete technical proposal, oscillating between search and vetting tasks without committing to either. 50% of the content is textbook background that should be compressed, while core components like architecture definition, experimental design, and methodology remain unspecified. With significant revision including architecture diagrams, task definition, and experiment plans, the paper could reach a score of approximately 7/10."},"strengths":{"value":"1. Addresses a relevant and scientifically meaningful problem in automated exoplanet detection from stellar light curves.\n2. Identifies the potential of modern sequence modeling approaches (Mamba + Attention) for time-domain astronomy.\n3. Acknowledges the closest prior work (Salinas et al. 2025), showing awareness of recent literature.\n4. Considers cross-domain applicability between Kepler and TESS missions, showing ambition for generalizability.\n5. Proposes data augmentation via false label injection, indicating some thought about training challenges."},"weaknesses":{"value":"1. No architecture defined despite mentioning \"Mamba + Attention\" — no diagram, layer specifications, or input/output mappings provided (severity 10).\n2. Fundamental task ambiguity: oscillates between search (finding candidates) and vetting (validating candidates) without clearly defining which problem is being solved (severity 10).\n3. Zero experimental design — no metrics, baselines, dataset splits, or success criteria specified (severity 9).\n4. Undefined cross-domain training approach: proposes Kepler↔TESS mixing without acknowledging or addressing the significant domain gap between instruments (severity 9).\n5. Vague data augmentation: \"injecting false labels\" uses undefined terminology and lacks methodological detail (severity 9).\n6. Only 6 references, missing seminal works including Shallue & Vanderburg 2018 and Kovács et al. 2002 (BLS).\n7. 50% of paper is textbook background (transit photometry, BLS) that should be compressed to 25%.\n8. No differentiation from closest prior work (Salinas et al. 2025) despite citing it.\n9. Timeline allocates only 2 weeks for core algorithm development, which is unrealistic.\n10. Non-periodic detection claim in introduction is unsupported by any proposed method.\n11. Incomplete proofreading: text contains \"figure...\" with ellipsis indicating unfinished sections."},"confidence":{"value":"4: Confident"},"questions":{"value":"1. Can you provide a concrete architecture diagram with layer specifications, dimensions, and input/output mappings for the proposed Mamba + Attention model?\n2. Is this project addressing the search problem (identifying transit signals in raw light curves) or the vetting problem (classifying candidates as true planets vs false positives)? Please commit to one clearly.\n3. What specific metrics (precision, recall, F1, AUC-ROC, etc.) will you use to evaluate performance, and what are your success criteria?\n4. How do you plan to handle the domain gap between Kepler and TESS data (different cadences, bandpasses, noise properties)? Will you train separate models, use domain adaptation, or fine-tune?\n5. What exactly does \"injecting false labels\" mean for data augmentation? Are you synthesizing transit signals, injecting false positives, or something else?\n6. How does your proposed approach differ from or improve upon Salinas et al. 2025, which you cite as the closest prior work?\n7. Given the 2-week timeline for core algorithm development, what is the minimum viable experiment you plan to demonstrate?"},"rating":{"value":"5: Average"}},"nonreaders":[],"tmdate":1779868067987,"tcdate":1779109627985,"writers":["tsinghua.edu.cn/THU/2026/Spring/ANM","tsinghua.edu.cn/THU/2026/Spring/ANM/Submission8/Reviewers"],"signatures":["tsinghua.edu.cn/THU/2026/Spring/ANM/Submission8/Reviewer_FbqQ"],"forum":"1Per9rP2px","number":1,"license":"CC BY 4.0","cdate":1779109627985,"readers":["everyone"],"invitations":["tsinghua.edu.cn/THU/2026/Spring/ANM/Submission8/-/Official_Review"],"mdate":1779868067987,"domain":"tsinghua.edu.cn/THU/2026/Spring/ANM","replyto":"1Per9rP2px","id":"wzL7vpKBhs","forumContent":{"venue":{"value":"THU 2026 Spring ANM Submission"},"pdf":{"value":"/pdf/c01c4b37fd02f5360173780690808a2cb6a50a62.pdf"},"keywords":{"value":["Time Series Intelligence","Transit Photometry"]},"venueid":{"value":"tsinghua.edu.cn/THU/2026/Spring/ANM/Submission"},"paperhash":{"value":"dallamico|automated_exoplanet_transit_detection_from_stellar_light_curves"},"authorids":{"value":["~Tommaso_Dall'Amico1","~Mikhail_Kuvakin1","~Lamat_Foschini_Antichi1"]},"abstract":{"value":"The discovery of exoplanets is a crucial frontier in modern astrophysics. It expands our understanding\nof planetary formation, contextualizes the uniqueness of our own solar system, and drives the search\nfor potentially habitable worlds. Just as ancient explorers like Herodotus gathered knowledge about\nislands of habitability outside the known world, modern astronomers seek out habitable zones beyond\nour local cosmic neighborhood. Until recent years, the field relied heavily on classical approaches\nand manual interpretation of potential exoplanet candidates. These methods are labor-intensive,\nsubject to human error, and often struggle to isolate weak signals hidden in noise. The recent\nintroduction of Machine Learning has driven significant progress, leading to the automated discovery\nof many new potential exoplanets. In this course project, we aim to contribute to this intersection of\nPlanetary Science and ML by bringing state-of-the-art Time-Series Intelligence techniques into the\nexisting framework of exoplanet discovery. Specifically, we plan to focus on the automated search for\nsmaller exoplanets with weak or non-periodic transit signals, which are currently difficult for existing\narchitectures to recognize."},"_bibtex":{"value":"@inproceedings{\nkuvakin2026automated,\ntitle={Automated Exoplanet Transit Detection from Stellar Light Curves},\nauthor={Mikhail Kuvakin},\nbooktitle={Submitted to Advanced Networking Management (Time Series Intelligence) - Spring 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=1Per9rP2px},\nnote={under review}\n}"},"title":{"value":"Automated Exoplanet Transit Detection from Stellar Light Curves"},"authors":{"value":["Tommaso Dall'Amico","Mikhail Kuvakin","Lamat Foschini Antichi"]}},"version":2},{"content":{"TLDR":{"value":"A mamba-transformer tracking framework is proposed for resolution-scalable visual tracking."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Single Object Tracking"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Current one-stream Transformer-based trackers are quality but unfriendly to memory consumption of large resolution and long sequence, both of which are crucial keys to tracking tasks. Recently structured state space model (SSM) demonstrates promising performance and efficiency in sequence modeling but struggles to retrieve due to the limited hidden state number. To solve the computation challenge and explore the potential of Mamba, we propose TrackMamba, a Mamba-Transformer tracker containing TrackMamba Blocks and Attention Blocks. In order to better harness the scanning in TrackMamba Blocks for inter- and intra-frame modeling, we introduce various scan patterns for rearrangement and flipping. Furthermore, we propose Target Enhancement, including Temporal Token for target aggregation and search enhancement, and Temporal Mamba for target information cross-frame propagation. Extensive experiments show TrackMamba performs better than the first-generation one-stream Transformer-based tracker at same resolution and mitigates consumption growth when enlarging resolution, exhibiting the potential of Mamba-based model for large-resolution tracking."},"_bibtex":{"value":"@misc{\nzhang2024trackmamba,\ntitle={TrackMamba: Mamba-Transformer Tracking},\nauthor={Jiaming Zhang and Cheng Liang and Yutao Cui and Xiangbo Shu and Gangshan Wu and Limin Wang},\nyear={2024},\nurl={https://openreview.net/forum?id=V7QRVEZ0le}\n}"},"title":{"value":"TrackMamba: Mamba-Transformer Tracking"},"pdf":{"value":"/pdf/b6fd4058e862b558436284a5d3bc1c3711911485.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zhang|trackmamba_mambatransformer_tracking"},"authorids":{"value":["~Jiaming_Zhang6","~Cheng_Liang1","~Yutao_Cui1","~Xiangbo_Shu1","~Gangshan_Wu1","~Limin_Wang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jiaming Zhang","Cheng Liang","Yutao Cui","Xiangbo Shu","Gangshan Wu","Limin Wang"]}},"tmdate":1731589710154,"tcdate":1727275170539,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4593/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission4593/Authors"],"forum":"V7QRVEZ0le","license":"CC BY 4.0","number":4593,"cdate":1727275170539,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission4593/-/Full_Submission","ICLR.cc/2025/Conference/-/Withdrawn_Submission"],"mdate":1731589710154,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"V7QRVEZ0le","version":2},{"content":{"summary":{"value":"This manuscript proposes a novel visual backbone network called Vision Filter (ViF). Its innovation lies in the introduction of a core component, the Fourier Neural Filter (FNF). Building on the Fourier Neural Operator (FNO), FNF introduces an input-dependent adaptive integral kernel. Through selective activation and adaptive modulation, it establishes a dynamic information flow between the frequency and time domains, addressing the bandwidth bottleneck and over-smoothing issues inherent in FNO. ViF outperforms current mainstream Transformer, Mamba, and Fourier-based models on ImageNet-1K image classification, COCO object detection, and ADE20K semantic segmentation tasks, while maintaining high computational efficiency."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"As mentioned above."},"rating":{"value":4},"details_of_ethics_concerns":{"value":"None"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1.The manuscript 's motivation is sound, and it clearly identifies the limitations of FNO, such as bandwidth bottlenecks and over-smoothing.\n2.This manuscript introduces an input-dependent integral kernel into the Fourier operator, constructing a unified time-frequency representation space, which is quite innovative.\n3.An obvious advantage of this method is computational efficiency. Compared with Transformer-type models, it has lower computational complexity and is more advantageous than Mamba-type models in spatial structure modeling.\n4.This manuscript achieves state-of-the-art or competitive results on multiple mainstream vision tasks, particularly on ImageNet-1K, significantly outperforming similar Fourier methods (such as GFNet)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.Although FNF solves some of the shortcomings of FNO, its core idea is still based on the traditional framework of \"frequency domain processing + time domain supplementation\". It does not propose a new visual representation learning paradigm, which weakens the innovation of this article.\n2.Although ViF performs well on ImageNet-1K, its improvement on COCO and ADE20K compared to models such as ViM is relatively small (e.g., mAP increases by 0.2-0.4), failing to significantly widen the gap.\n3.ViF only tested three small and medium-sized variants, ViF-T/S/B, and did not evaluate the performance of larger models, such as those of ViT-L/16, on larger datasets.\n4.Definition 7 mentions that adaptive modulation is achieved through the amplitude-sensitive weighting function \\(\\|z\\|\\) and learnable parameters \\(\\alpha\\) and \\(\\beta\\), but does not give the range of \\(\\alpha\\) and \\(\\beta\\) and whether they are adjusted with training iterations."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918428777,"tcdate":1761807376861,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6047/Reviewer_3JLT"],"signatures":["ICLR.cc/2026/Conference/Submission6047/Reviewer_3JLT"],"forum":"Q5SSA6IonA","number":2,"license":"CC BY 4.0","cdate":1761807376861,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6047/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918428777,"domain":"ICLR.cc/2026/Conference","replyto":"Q5SSA6IonA","id":"pWyathJVpa","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["representation learning"]},"supplementary_material":{"value":"/attachment/46b601e6cab658387a955095350002cd51924698.pdf"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"Effective information extraction has long been a central challenge in Computer Vision (CV). Transformer- and Mamba-based backbones have significantly advanced this field by providing powerful long-range modeling capability, even though they are initially developed for Natural Language Processing (NLP). Recent progress has highlighted the potential of Fourier Neural Operator (FNO), which, with its favorable quasi-linear complexity and strong global modeling capacity, offers a promising alternative for visual representation learning. However, FNO exhibits a fundamental limitation in capturing local high-frequency patterns due to the over-smoothing effect and bandwidth bottleneck. To address this limitation, we propose Vision Filter (ViF), as a generic backbone for CV, consisting of two complementary components: adaptive modulation for enhancing sensitivity to high-frequency component in the frequency domain, and selective activation for balancing local time-domain and global frequency-domain information flow. Extensive experiments reveal that ViF consistently outperforms prominent variants of Transformer- and Mamba-based backbones across diverse visual tasks, including image classification, object detection, and semantic segmentation. ViF demonstrates lower computational complexity than Transformer-based models and better structural modeling than Mamba-based models, which suffer from spatial disruption due to their directional scanning mechanism. The joint time- and frequency-domain mechanism introduced in ViF may establish a promising paradigm for designing effective visual representation learning, bridging local high-frequency information with global low-frequency information."},"_bibtex":{"value":"@misc{\nxu2026fourier,\ntitle={Fourier Neural Filter as Generic Vision Backbone},\nauthor={Chenheng Xu and Yuanhao Ban and Dan Wu and Cho-Jui Hsieh and Ying Nian Wu and Yixin Zhu},\nyear={2026},\nurl={https://openreview.net/forum?id=Q5SSA6IonA}\n}"},"title":{"value":"Fourier Neural Filter as Generic Vision Backbone"},"pdf":{"value":"/pdf/357d80af090cafb82d30a7f6570ca1a0576217e3.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xu|fourier_neural_filter_as_generic_vision_backbone"},"authorids":{"value":["~Chenheng_Xu1","~Yuanhao_Ban1","~Dan_Wu7","~Cho-Jui_Hsieh1","~Ying_Nian_Wu1","~Yixin_Zhu1"]},"authors":{"value":["Chenheng Xu","Yuanhao Ban","Dan Wu","Cho-Jui Hsieh","Ying Nian Wu","Yixin Zhu"]}},"version":2},{"content":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Sequence modeling","test-time training","RNN transformer alternatives"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compute linearly during inference. A recent stream of work linearized the softmax operation, resulting in powerful recurrent neural network (RNN) models with constant memory and compute costs such as DeltaNet, Mamba or xLSTM. These models can be unified by noting that their recurrent layer dynamics can all be derived from an in-context regression objective, approximately optimized through an online learning rule. Here, we join this line of work and introduce a numerically stable, chunkwise parallelizable version of the recently proposed Mesa layer (von Oswald et al., 2024), which could only run sequentially in time and was therefore not scalable. This layer again stems from an in-context loss, but which is now minimized to optimality at every time point using a fast conjugate gradient solver. Through an extensive suite of experiments study up to the billion-parameter scale, we show that optimal test-time training enables reaching lower language modeling perplexity and higher downstream benchmark performance than previous RNNs, especially on tasks requiring long context understanding. This performance gain comes at the cost of additional flops spent during inference time. Our results are therefore intriguingly related to recent trends of increasing test-time compute to improve performance -- here by spending compute to solve sequential optimization problems within the neural network itself."},"_bibtex":{"value":"@inproceedings{\noswald2026mesanet,\ntitle={MesaNet: Sequence Modeling by Locally Optimal Test-Time Training},\nauthor={Johannes von Oswald and Nino Scherrer and Seijin Kobayashi and Luca Versari and Songlin Yang and Maximilian Schlegel and Kaitlin Maile and Yanick Schimpf and Oliver Sieberling and Alexander Meulemans and Guillaume Lajoie and Rif A. Saurous and Charlotte Frenkel and Razvan Pascanu and Blaise Aguera y Arcas and Joao Sacramento},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=xa3OnTb6c3}\n}"},"title":{"value":"MesaNet: Sequence Modeling by Locally Optimal Test-Time Training"},"pdf":{"value":"/pdf/55189e529b3da4e2e90fd1988db5f2bb30a909ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"oswald|mesanet_sequence_modeling_by_locally_optimal_testtime_training"},"authorids":{"value":["~Johannes_von_Oswald2","~Nino_Scherrer1","~Seijin_Kobayashi1","~Luca_Versari1","~Songlin_Yang1","~Maximilian_Schlegel1","~Kaitlin_Maile1","~Yanick_Schimpf1","~Oliver_Sieberling1","~Alexander_Meulemans1","~Guillaume_Lajoie1","~Rif_A._Saurous1","~Charlotte_Frenkel1","~Razvan_Pascanu1","~Blaise_Aguera_y_Arcas1","~Joao_Sacramento1"]},"authors":{"value":["Johannes von Oswald","Nino Scherrer","Seijin Kobayashi","Luca Versari","Songlin Yang","Maximilian Schlegel","Kaitlin Maile","Yanick Schimpf","Oliver Sieberling","Alexander Meulemans","Guillaume Lajoie","Rif A. Saurous","Charlotte Frenkel","Razvan Pascanu","Blaise Aguera y Arcas","Joao Sacramento"]}},"tmdate":1775877279046,"pdate":1769436410587,"tcdate":1758291137989,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18821/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission18821/Authors"],"forum":"xa3OnTb6c3","license":"CC BY 4.0","number":18821,"cdate":1758291137989,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission18821/-/Full_Submission","ICLR.cc/2026/Conference/Submission18821/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission18821/-/Camera_Ready_Revision"],"mdate":1775877279046,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"xa3OnTb6c3","version":2},{"content":{"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"huang|avmamba_crossmodality_selective_state_space_models_for_audiovisual_question_answering"},"authorids":{"value":["~Ziru_Huang1","~Jia_Li28","~Wenjie_Zhao2","~Yunhui_Guo2","~Yapeng_Tian1"]},"html":{"value":"https://sightsound.org/papers/2024/Huang_AV-Mamba_Cross-Modality_Selective_State_Space_Models_for_Audio-Visual_Question_Answering.pdf"},"abstract":{"value":"Audio-visual question answering (AVQA) aims to answer questions relevant to visual objects, sounds, and their relationships within videos [6, 10, 11]. This task delves into the complexities of multimodal scenes, which are both diverse and dynamic. The key challenges of AVQA lie in accurately identifying the audio and video segments that directly relate to the question and establishing whether the identified visual regions produce sounds relevant to the question. Prior research primarily leverages attention mechanisms to tackle these challenges. For instance, employing audio-guided visual attention helps localize sounding visual regions, while question-guided temporal attention aggregates relevant audio and visual segments [4–6]. Nonetheless, audio and visual segments don’t always correlate, and multimodal video segments can vary dynamically over time. Furthermore, when faced with lengthy sequences of audio-visual data accompanied by textual inputs, attention mechanisms may struggle to accurately discern the relationships across different modalities over extended durations. Consequently, the selection mechnism, driven by attention weights, may falter with long sequences, inadvertently aggregating irrelevant segments despite minimal weights. \nThe Mamba model [1] has proven its strength in modeling long sequences across diverse tasks. It dynamically adjusts the parameters of State Space Models (SSMs) [2] guided by the input, allowing for context-aware reasoning. Mamba’s unique capability to select and retain information indefinitely inspired us to expand its success story into the realm of multimodal video modeling. In this paper, we introduce CM-Mamba, a new extension that leverages a cross-modality selection mechanism within Mamba models. The model is designed to efficiently leverage information across audio, visual, and textual modalities, in which the parameters of SSMs will dynamically adjust in response to inputs from an alternative modality."},"title":{"value":"AV-Mamba: Cross-Modality Selective State Space Models for Audio-Visual Question Answering"},"authors":{"value":["Ziru Huang","Jia Li","Wenjie Zhao","Yunhui Guo","Yapeng Tian"]}},"tmdate":1727711387271,"pdate":1718546964430,"tcdate":1727711387271,"writers":["~Ziru_Huang1","~Jia_Li28","~Wenjie_Zhao2","~Yunhui_Guo2","~Yapeng_Tian1"],"signatures":["~Jia_Li28"],"forum":"7rft30bDiI","license":"CC BY 4.0","number":29455,"cdate":1727711387271,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1727711387271,"domain":"OpenReview.net/Archive","id":"7rft30bDiI","version":2},{"content":{"summary":{"value":"This paper proposes the Masked Autoencoder Cyclic Encoder (MACE), which innovatively takes the cyclic structure of time-series data as an inductive bias. It decomposes sequences into cyclic units and performs masked reconstruction at both the cycle and sequence levels to balance the computational efficiency and performance of long time-series modeling."},"strengths":{"value":"1. The paper integrates cyclic structure as an inductive bias into masked autoencoders. By splitting long time-series into cyclic units, it effectively reduces the quadratic complexity of Transformers in processing long sequences. Meanwhile, the dual-loss design ensures the accuracy of cycle-level and sequence-level modeling, with validity verified across multiple cyclic data scenarios.\n\n2.  On large cyclic datasets MACE not only outperforms mainstream baselines in performance but also significantly shortens fine-tuning time. This addresses the pain point of high computational costs for long time-series Transformer models, holding practical application value."},"weaknesses":{"value":"1. The paper does not explore the impact of key designs: fixed 75% sequence masking ratio, cycle segmentation methods  on performance, nor does it verify the necessity of the teacher-student architecture. Thus, the rationality of core designs lacks sufficient experimental support.\n\n2. On small datasets like Air quality, unoptimized computational overhead (from operations such as cycle splitting and padding) makes MACE’s fine-tuning efficiency significantly lower than that of MOMENT (SVM, 19 seconds) and TimeMAE (21 seconds), limiting its use in small-data scenarios."},"confidence":{"value":2},"rating":{"value":4},"justification":{"value":"Through clear architectural design (cycle decomposition for pre-training + cycle class token utilization for fine-tuning), experimental validation on multiple types of cyclic datasets, and dual improvements in efficiency and performance, the paper demonstrates the value of the \"cyclic inductive bias\" in time-series masked autoencoders. It responds to the field's demand for efficient long time-series modeling, holding both theoretical and practical significance."},"title":{"value":"Lack key ablations"}},"parentInvitations":"NLDL.org/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762462561928,"tcdate":1759112814936,"writers":["NLDL.org/2026/Conference","NLDL.org/2026/Conference/Submission58/Reviewer_FgbH"],"signatures":["NLDL.org/2026/Conference/Submission58/Reviewer_FgbH"],"forum":"qp5lal8QhP","number":2,"license":"CC BY 4.0","cdate":1759112814936,"readers":["everyone"],"invitations":["NLDL.org/2026/Conference/Submission58/-/Official_Review","NLDL.org/2026/Conference/-/Edit"],"mdate":1762462561928,"domain":"NLDL.org/2026/Conference","replyto":"qp5lal8QhP","id":"ylFkvbISKF","forumContent":{"TLDR":{"value":"Masked Autoencoder using a cycle inductive bias to reduce computational cost"},"venue":{"value":"NLDL 2026 Spotlight"},"keywords":{"value":["Masked autoencoder","self supervised learning","healthcare","masked learning","time series","."]},"presenter":{"value":["~Stuart_Gallina_Ottersen1"]},"permission":{"value":"/attachment/1858615223d666748d1e02127e322af830eb30a3.pdf"},"abstract":{"value":"Many time series exhibit cyclic structure — for example, in physiological signals such as ECG or EEG — yet most representation learning methods treat them as generic sequences. We propose a masked autoencoder (MAE) framework that explicitly leverages cycles as an inductive bias for more efficient and effective time-series modelling. Our method decomposes sequences into cycles and trains the model to reconstruct masked segments at both the cycle and sequence level. This cycle-based decomposition shortens the effective sequence length processed by the encoder by up to a factor of ten in our experiments, yielding substantial computational savings without loss in reconstruction quality. At the same time, the approach exposes the encoder to a greater diversity of temporal patterns, as each cycle forms an additional training instance, which enhances the ability to capture subtle intra-cycle variations. Empirically, our framework outperforms three competitive baselines across four cyclic datasets, while also reducing training time on larger datasets."},"_bibtex":{"value":"@inproceedings{\nottersen2026incorporating,\ntitle={Incorporating the Cycle Inductive Bias in Masked Autoencoders},\nauthor={Stuart Gallina Ottersen and Kerstin Bach},\nbooktitle={Northern Lights Deep Learning Conference 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=qp5lal8QhP}\n}"},"title":{"value":"Incorporating the Cycle Inductive Bias in Masked Autoencoders"},"pdf":{"value":"/pdf/492be03d864c4a5d0ed5214469ce2c4fc25e3505.pdf"},"venueid":{"value":"NLDL.org/2026/Conference"},"paperhash":{"value":"ottersen|incorporating_the_cycle_inductive_bias_in_masked_autoencoders"},"authorids":{"value":["~Stuart_Gallina_Ottersen1","~Kerstin_Bach1"]},"authors":{"value":["Stuart Gallina Ottersen","Kerstin Bach"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2411.02941v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"ma|a_mamba_foundation_model_for_time_series_forecasting"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Haoyu_Ma:","https://dblp.org/search/pid/api?q=author:Yushu_Chen:","~Wenlai_Zhao1","https://dblp.org/search/pid/api?q=author:Jinzhe_Yang:","https://dblp.org/search/pid/api?q=author:Yingsheng_Ji:","https://dblp.org/search/pid/api?q=author:Xinghua_Xu:","https://dblp.org/search/pid/api?q=author:Xiaozhu_Liu:","https://dblp.org/search/pid/api?q=author:Hao_Jing:","https://dblp.org/search/pid/api?q=author:Shengzhuo_Liu:","https://dblp.org/search/pid/api?q=author:Guangwen_Yang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2411.02941"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2411-02941,\n  publtype={informal},\n  author={Haoyu Ma and Yushu Chen and Wenlai Zhao and Jinzhe Yang and Yingsheng Ji and Xinghua Xu and Xiaozhu Liu and Hao Jing and Shengzhuo Liu and Guangwen Yang},\n  title={A Mamba Foundation Model for Time Series Forecasting},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2411.02941},\n  url={https://doi.org/10.48550/arXiv.2411.02941}\n}\n"},"abstract":{"value":"Time series foundation models have demonstrated strong performance in zero-shot learning, making them well-suited for predicting rapidly evolving patterns in real-world applications where relevant training data are scarce. However, most of these models rely on the Transformer architecture, which incurs quadratic complexity as input length increases. To address this, we introduce TSMamba, a linear-complexity foundation model for time series forecasting built on the Mamba architecture. The model captures temporal dependencies through both forward and backward Mamba encoders, achieving high prediction accuracy. To reduce reliance on large datasets and lower training costs, TSMamba employs a two-stage transfer learning process that leverages pretrained Mamba LLMs, allowing effective time series modeling with a moderate training set. In the first stage, the forward and backward backbones are optimized via patch-wise autoregressive prediction; in the second stage, the model trains a prediction head and refines other components for long-term forecasting. While the backbone assumes channel independence to manage varying channel numbers across datasets, a channel-wise compressed attention module is introduced to capture cross-channel dependencies during fine-tuning on specific multivariate datasets. Experiments show that TSMamba's zero-shot performance is comparable to state-of-the-art time series foundation models, despite using significantly less training data. It also achieves competitive or superior full-shot performance compared to task-specific prediction models. The code will be made publicly available."},"title":{"value":"A Mamba Foundation Model for Time Series Forecasting"},"authors":{"value":["Haoyu Ma","Yushu Chen","Wenlai Zhao","Jinzhe Yang","Yingsheng Ji","Xinghua Xu","Xiaozhu Liu","Hao Jing","Shengzhuo Liu","Guangwen Yang"]}},"tmdate":1769482834904,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2411-02941"],"tcdate":1769482822684,"writers":["~"],"signatures":["~Wenlai_Zhao1"],"forum":"em4U5xQLdL","license":"CC BY-SA 4.0","number":806006,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769482834904,"domain":"DBLP.org","id":"em4U5xQLdL","version":2},{"content":{"summary":{"value":"This paper introduces PointMamba, an interesting method for point cloud analysis that utilizes a linear complexity state space model (SSM) instead of traditional Transformer architectures. PointMamba employs space-filling curves for point tokenization and features a simple, non-hierarchical Mamba encoder. Additionally, the authors propose an effective order indicator and serialization-based mask modeling strategy. Comprehensive evaluations across multiple datasets demonstrate PointMamba's superior performance and significantly reduced computational costs compared to Transformer-based methods."},"soundness":{"value":4},"confidence":{"value":5},"questions":{"value":"See weakness."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":4},"strengths":{"value":"1. The paper reads very well. I appreciate the motivation of the paper and agree that we should make an effective method as simple as possible. As the first Mamba-based work for point cloud tasks, the proposed method is simple, elegant, and effective, establishing a solid Mamba-based baseline for point cloud analysis tasks.\n\n2. The paper considers the limitations of Mamba's unidirectional modeling and proposes practical solutions like serialization-based mask modeling strategies and order indicators.\n\n3. The theoretical analysis of Mamba used in point cloud is reasonable, and the paper is easy to reproduce.\n\n4. The experiments are convincing and support the main idea of the paper. The authors provide thorough evaluations, including comparisons with SOTA methods, ablation studies, and analyses of each component."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Some experiments are missing. For example, from Table 11, it would be beneficial to include more masking ratios, especially the performance when masking 90% point patches. Besides, from Table 12, could the authors provide a more detailed analysis of why average pooling performs better? Additionally, what about the performance of max pooling?\n\n2. Have the authors considered the potential benefits of integrating PointMamba with Transformer architectures? For example, a transformer layer can be used as the final prediction head. Such a combination can take advantage of both architectures and will not introduce many computational costs.\n\n3. The point clouds have complex structures, and this paper only considers two orders. In my view, introducing more orders (e.g., introducing three serialization methods and tripling the input length) can better capture the geometry information of the point clouds, which might be beneficial for learning. \n\n4. During the pre-training strategy, do the authors use different order indicators or the same indicator? Figure 4 is somewhat ambiguous.\n\nMinor: Using space-filling curves to scan point clouds is an interesting attempt. Although it is common knowledge for some readers, it still would be helpful if the authors provided a more detailed introduction to space-filling curves in the preliminaries part.\n\ntypos: PointMAE -> Point-MAE"},"limitations":{"value":"The authors have adequately discussed the limitations of their work."}},"nonreaders":[],"tmdate":1730878669019,"tcdate":1720768523304,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission940/Reviewer_QPV3"],"signatures":["NeurIPS.cc/2024/Conference/Submission940/Reviewer_QPV3"],"forum":"Kc37srXvan","number":3,"license":"CC BY 4.0","cdate":1720768523304,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission940/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878669019,"domain":"NeurIPS.cc/2024/Conference","replyto":"Kc37srXvan","id":"OkrO6NeqJE","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Point cloud analysis; State space model; Mamba; PointMamba"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Transformers have become one of the foundational architectures in point cloud analysis tasks due to their excellent global modeling ability. However, the attention mechanism has quadratic complexity, making the design of a linear complexity method with global modeling appealing. In this paper, we propose PointMamba, transferring the success of Mamba, a recent representative state space model (SSM), from NLP to point cloud analysis tasks. Unlike traditional Transformers, PointMamba employs a linear complexity algorithm, presenting global modeling capacity while significantly reducing computational costs. Specifically, our method leverages space-filling curves for effective point tokenization and adopts an extremely simple, non-hierarchical Mamba encoder as the backbone. Comprehensive evaluations demonstrate that PointMamba achieves superior performance across multiple datasets while significantly reducing GPU memory usage and FLOPs. This work underscores the potential of SSMs in 3D vision-related tasks and presents a simple yet effective Mamba-based baseline for future research. The code is available at https://github.com/LMD0311/PointMamba."},"_bibtex":{"value":"@inproceedings{\nliang2024pointmamba,\ntitle={PointMamba: A Simple State Space Model for Point Cloud Analysis},\nauthor={Dingkang Liang and Xin Zhou and Wei Xu and Xingkui Zhu and Zhikang Zou and Xiaoqing Ye and Xiao Tan and Xiang Bai},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=Kc37srXvan}\n}"},"title":{"value":"PointMamba: A Simple State Space Model for Point Cloud Analysis"},"pdf":{"value":"/pdf/799eef8a02512c60290e1c8180f95b4799a27ec9.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"liang|pointmamba_a_simple_state_space_model_for_point_cloud_analysis"},"authorids":{"value":["~Dingkang_Liang2","~Xin_Zhou17","~Wei_Xu15","~Xingkui_Zhu1","~Zhikang_Zou2","~Xiaoqing_Ye1","~Xiao_Tan3","~Xiang_Bai1"]},"authors":{"value":["Dingkang Liang","Xin Zhou","Wei Xu","Xingkui Zhu","Zhikang Zou","Xiaoqing Ye","Xiao Tan","Xiang Bai"]}},"version":2},{"content":{"comment":{"value":"We now proceed to address the remaining points one by one.\n\n&nbsp;\n\n---\n\n### > How MambaSL models inter-variable relationships?\n\nWe did not propose a separate structure that explicitly models inter-variable relationships. However, **the selective SSM in vanilla Mamba already inherently performs channel mixing** and thereby fulfills that role:\n- $\\boldsymbol{B}$ can capture relationships among variables when projecting inputs into the state, and  \n- $\\boldsymbol{C}$ can again reflect inter-variable relationships when mapping the state back to the output, while  \n- $\\boldsymbol{\\Delta}$ changes dynamically depending on the input, allowing time-varying structural relationships.\n\nBorrowing the terminology from the \\[survey paper] you cited,  \nMamba can be seen as inherently incorporating **embedding fusion** and modeling inter-variable relationships with **asymmetry** and **dynamism**.\n\nHowever, inter-variable relationships are **not always beneficial** for every dataset. On the contrary, we wanted to emphasize that for some time series datasets, **not reflecting inter-variable relationships can be more effective**. This is precisely the motivation behind our H2 hypothesis and the key structural design direction of MambaSL.\n\nWe **modularized $\\boldsymbol{B}$, $\\boldsymbol{C}$, and $\\boldsymbol{\\Delta}$** so that:\n- a selective SSM (time-variant, channel-mixing) and  \n- an LTI SSM (time-invariant, channel-independent)\n\ncan be chosen depending on dataset characteristics.\n\nAs a result, we observed that for certain datasets, time-invariant step sizes or channel-independent input embeddings actually yielded better performance (e.g., EthanolConcentration, Handwriting, etc.).  \nThis aligns with observations from the forecasting literature, where \\[PatchTST] demonstrated strong performance with a channel-independent model.\n\nIn summary,\n- Mamba inherently has a structure capable of modeling inter-variable relationships, and  \n- MambaSL (H2), conversely, provides a way to deactivate inter-variable relationships, enabling the selection of the most suitable inductive bias depending on the data.\n\n**> Reference**\n\n\\[survey paper] *A comprehensive survey of deep learning for multivariate time series forecasting: A channel strategy perspective*. arXiv, 2025.\n\n\\[PatchTST] *PatchTST: A Time Series is Worth 64 Words: Long-term Forecasting with Transformers*. ICLR, 2023.\n\n&nbsp;\n\n### > Further discussion on modeling temporal properties for classification\n\nWe fully recognize that feature engineering-based approaches (e.g., ROCKET) are powerful. However, in this work, we wanted to explore the possibility that **a well-learned latent space alone could be sufficient for effective classification**. As mentioned in the Introduction, we aimed to maximize the pure potential of Mamba itself, and to show that the structural advantages of MambaSL are effective across a broad range of time series classification datasets.\n\nThis is possible because, compared to traditional recurrent models or Transformers, *Mamba is particularly good at memorizing long-term properties*. Thus, it is indeed plausible for Mamba to store only the information necessary for classification in the state across the entire sequence.\n\nOne piece of evidence supporting this is the model ablation result where we use **only the latent vector at the last timestep** for classification (`Section 5.2` and `Appendix C.2`).\nIf the Mamba block could not memorize long-term characteristics for a given dataset, it would be almost impossible to perform reasonable classification using only the last-timestep latent vector. However, the results do not support this concern; in fact, for datasets such as JapaneseVowels and DuckDuckGeese, using only the last timestep even outperforms all other pooling strategies."},"title":{"value":"Reply to the Follow-up Questions (part 2/4)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764844588780,"tcdate":1764844507384,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11231/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission11231/Authors"],"forum":"YDl4vqQqGP","number":19,"license":"CC BY 4.0","cdate":1764844507384,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11231/-/Official_Comment"],"mdate":1764844588780,"domain":"ICLR.cc/2026/Conference","replyto":"pog7d1pEPA","id":"XvtWQcha93","forumContent":{"TLDR":{"value":"We introduce MambaSL, a minimally redesigned single-layer Mamba that achieves state-of-the-art accuracy on the UEA30 benchmark, with reproducible evaluation covering all baselines."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["time series classification","single-layer Mamba","modular selective SSM","multi-head adaptive pooling","skip connection"]},"supplementary_material":{"value":"/attachment/dfc093493a30882d173d5eedf1107e0c42d4d9fc.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"Despite recent advances in state space models (SSMs) such as Mamba across various sequence domains, research on their standalone capacity for time series classification (TSC) has remained limited. \nWe propose MambaSL, a framework that minimally redesigns the selective SSM and projection layers of a single-layer Mamba, guided by four TSC-specific hypotheses.\nTo address benchmarking limitations—restricted configurations, partial University of East Anglia (UEA) dataset coverage, and insufficiently reproducible setups—we re-evaluate 20 strong baselines across all 30 UEA datasets under a unified protocol. \nAs a result, MambaSL achieves state-of-the-art performance with statistically significant average improvements, while ensuring reproducibility via public checkpoints for all evaluated models.\nTogether with visualizations, these results demonstrate the potential of Mamba-based architectures as a TSC backbone."},"_bibtex":{"value":"@inproceedings{\njung2026mambasl,\ntitle={Mamba{SL}: Exploring Single-Layer Mamba for Time Series Classification},\nauthor={Yoo-Min Jung and Leekyung Kim},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=YDl4vqQqGP}\n}"},"title":{"value":"MambaSL: Exploring Single-Layer Mamba for Time Series Classification"},"pdf":{"value":"/pdf/36a22e78ae778d70b99d6fc145e8d5a9be7e0c1d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jung|mambasl_exploring_singlelayer_mamba_for_time_series_classification"},"authorids":{"value":["~Yoo-Min_Jung1","~Leekyung_Kim1"]},"authors":{"value":["Yoo-Min Jung","Leekyung Kim"]}},"version":2},{"content":{"summary":{"value":"The authors propose a sub-linear complexity speech summarization model by combining Q-former and pretrained Mamba. Segmented audios are processed by Whisper and compressed by Q-former with Mamba-processed query vectors, which are then fed to Mamba LLM to generate the summarization. A 3-stage training with different tasks is carried out, i.e. short-form ASR, long-form ASR, and summarization, accompanied by DPO. Empirical studies show better results and speed compared to cascaded models and a HuBERT+LLAMA E2E model."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Q-former-like mechanism should be able to compress input of any length into fixed length. Hence I'm a bit confused about the decision to compress every 0.33s of audio into 2 query vectors. This differs from previous works including arxiv:2309.13963 and arxiv:2407.02005, which considered 30~100 vectors for every 30s of audio. In this way, the contextual information further than 0.33s will not be captured by the Q-former. I guess that the reason is to avoid the high-cost quadratic cross-attention but it will be better for the author to discuss that explicitly. Also, compressing a short sequence (only dozens of vectors, as each Whisper frame takes 25ms) into merely 2 vectors is rather simple and I doubt if the complicated Q-former will really outperform a much simpler one, e.g. pooling or convolution, which may also process information within such a short context well. More ablation studies will be necessary to justify this decision, by comparing with pooling or CNN, and comparing with different context lengths.\n\nThere are many other approaches to compress speech signals into \"token-like\" embeddings to be processed by LMs, e.g. HuBERT units, speech tokens, and neural audio codec, while Q-former is somehow similar to a kind of VQ, but with continuous features. Can you elaborate on the reasons why you chose Q-former? Do you think there is any specific advantage?\n\nI am particularly concerned with the unidirectional Mamba used in Q-Mamba, and I fail to find the motivation to apply Mamba to the sequence of query vectors. Trainable query vectors should be already capable of introducing positional information. Ablation studies (e.g. by removing this Mamba layer) should be necessary to justify this choice.\n\nI also have some questions regarding the use of DPO. What is the experiment without DPO in Table 4? Using supervised fine-tuning only?\n\nIf the issue w/o DPO is that the summary will be too detailed, has the author considered any other more straightforward solution, e.g. length penalty during generation, downsampling the input sequence, or upweighting EOS during training?\n\nIt is commonly believed that instruction fine-tuning leads to better alignment, but at the cost of flexibility and adaptability to specific downstream tasks in fine-tuning, while the authors use a instruction fine-tuned version of Mamba-2.8B as the base LLM. Is there any specific reason to use Mamba-2.8B-Zephyr instead of the original Mamba-2.8B model?\n\nWhat is the LLM used in the Cascaded model? The original Mamba-2.8B or the instruction fine-tuned one? Is it further fine-tuned to summarization?\n\nCan you elaborate more on the speedup of the model compared to the cascaded one? With both of them using Whisper and Mamba (though the inputs to Mamba are different), I'm curious about the source of the extra overhead in the cascaded pipeline. Also, it can be helpful to report the average input sequence length to the final LLM model as a reference to the expected computational costs.\n\nUsing only synthesized datasets is a weak point of the empirical evaluation, particularly when the labels are also synthetic. It can be necessary to also report the results on real datasets, e.g. SLUE-SUMM, and include more examples and human evaluations.\n\nIt can be interesting to report the ASR performance of the model after either of the two Alignment stages.\n\nIt will be better to also include the original transcript in Appendix C.\n\nIn Figure 3, is Whisper frozen in the Fine-tuning Stage?\n\nMinor issues:\nL210: Figure 4.1?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"* The model enables efficient speech summarization by combining pretrained Mamba and Q-former.\n* Empirical results are strong."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* Ablation studies are not sufficient to demonstrate the advantages of the proposed methods and to identify the impact of each component. Some design choices are yet to be well-motivated.\n* The method is more like replacing transformers in existing methods (esp. arxiv:2407.02005) with Mamba, which leads to doubt on the technical novelty.\n* Only one synthetic dataset is used.\n\nSee questions below for details."}},"nonreaders":[],"tmdate":1731427688640,"tcdate":1730740710384,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9819/Reviewer_WB7G"],"signatures":["ICLR.cc/2025/Conference/Submission9819/Reviewer_WB7G"],"forum":"zOMa82W1HV","number":4,"license":"CC BY 4.0","cdate":1730740710384,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9819/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427688640,"domain":"ICLR.cc/2025/Conference","replyto":"zOMa82W1HV","id":"RqL0jIghgS","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Summarisation","Speech","Mamba"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Abstractive Speech Summarization (SSum) becomes increasingly difficult as the input speech length grows. To address this, we present SQuBa (Speech Querying Mamba Architecture), an end-to-end model designed explicitly for efficient speech summarization. SQuBa leverages a querying-attention Mamba projector to condense extended acoustic features into compact semantic tokens, which are subsequently summarized by the Mamba Large Language Model (LLM). The architecture’s computational complexity scales linearly with input length, enabling efficient handling of longer inputs. A two-stage training framework, complemented by bootstrapped Direct Preference Optimization (DPO) fine-tuning, empowers SQuBa to generate concise and coherent summaries. Experimental results demonstrate that SQuBa delivers competitive performance while significantly improving inference speed, making it ideal for real-world applications such as podcast and meeting transcriptions."},"_bibtex":{"value":"@misc{\neom2025squba,\ntitle={{SQ}uBa: Speech Mamba Language Model with Querying-Attention for Efficient Summarization},\nauthor={SooHwan Eom and Jay Shim and Eunseop Yoon and Hee Suk Yoon and Hyeonmok Ko and Mark A. Hasegawa-Johnson and Chang D. Yoo},\nyear={2025},\nurl={https://openreview.net/forum?id=zOMa82W1HV}\n}"},"title":{"value":"SQuBa: Speech Mamba Language Model with Querying-Attention for Efficient Summarization"},"pdf":{"value":"/pdf/5f8e99f39df6ded1708d0ef2eca02f6f98f7e983.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"eom|squba_speech_mamba_language_model_with_queryingattention_for_efficient_summarization"},"authorids":{"value":["~SooHwan_Eom1","~Jay_Shim1","~Eunseop_Yoon1","~Hee_Suk_Yoon1","~Hyeonmok_Ko1","~Mark_A._Hasegawa-Johnson1","~Chang_D._Yoo1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["SooHwan Eom","Jay Shim","Eunseop Yoon","Hee Suk Yoon","Hyeonmok Ko","Mark A. Hasegawa-Johnson","Chang D. Yoo"]}},"version":2},{"content":{"summary":{"value":"The paper proposes FLDmamba, a novel framework that integrates Fourier and Laplace Transform Decomposition with the Mamba State-Space Model (SSM) to enhance long-term time series prediction. The authors identify key challenges in existing models, particularly in capturing multi-scale periodicity, transient dynamics, and handling data noise. Extensive experiments on nine real-world datasets demonstrate that FLDmamba outperforms state-of-the-art Transformer-based and Mamba-based architectures"},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Can you provide more details on how the inverse Laplace transform is computed in practice within your framework? Given that inverse Laplace transforms can be numerically challenging, how do you ensure stability and efficiency in this component?\n2. Have you explored using alternative kernel functions beyond the RBF kernel for data smoothing? If so, how do they compare in terms of performance and computational cost?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper introduces a novel integration of Fourier and Laplace transforms into the Mamba framework, addressing the limitations of previous SSMs in capturing multi-scale periodicity and transient dynamics.\n2. The paper includes thorough experiments on nine diverse real-world datasets, covering various domains. The results consistently show that FLDmamba achieves superior performance compared to strong baselines.\n3. The model's robustness to data noise is evaluated, showing that FLDmamba maintains high performance even under increased noise levels, outperforming other methods like S-Mamba and iTransformer. Detailed ablation studies are conducted to isolate and demonstrate the contribution of each component in the FLDmamba framework."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. While the paper explains the intuition behind using the Laplace transform to capture transient dynamics, it lacks a deeper theoretical exploration of how exactly the inverse Laplace transform contributes to performance improvements in the context of the model.\n2. The experimental comparison focuses primarily on Transformer-based models and Mamba-based methods. Inclusion of more diverse SSM-based baselines, such as those based on S4 or other recent advances, would strengthen the evaluation."}},"nonreaders":[],"tmdate":1731428015814,"tcdate":1730252284359,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4022/Reviewer_6yv6"],"signatures":["ICLR.cc/2025/Conference/Submission4022/Reviewer_6yv6"],"forum":"9EiWIyJMNi","number":1,"license":"CC BY 4.0","cdate":1730252284359,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4022/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428015814,"domain":"ICLR.cc/2025/Conference","replyto":"9EiWIyJMNi","id":"42eZl9mqQu","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Time Series Prediction"]},"supplementary_material":{"value":"/attachment/1a7a0dbb359aaeb4b73874c96c7e6b288ffeb4cd.pdf"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Time series prediction, a crucial task across various domains, faces significant challenges due to the inherent complexities of time series data, including non-stationarity, multi-scale periodicity, and transient dynamics, particularly when tackling long-term predictions. While Transformer-based architectures have shown promise, their quadratic complexity with sequence length hinders their efficiency for long-term predictions. Recent advancements in State-Space Models, such as Mamba, offer a more efficient alternative for long-term modeling, but they lack the capability to capture multi-scale periodicity and transient dynamics effectively. Meanwhile, they are susceptible to the data noise issue in time series. This paper proposes a novel framework, FLDmamba (Fourier and Laplace Transform Decomposition Mamba), addressing these limitations. FLDmamba leverages the strengths of both Fourier and Laplace transforms to effectively capture both multi-scale periodicity, transient dynamics within time series data, and improve the robustness of the model to the data noise issue. By integrating Fourier analysis into Mamba, FLDmamba enhances its ability to capture global-scale properties, such as multi-scale periodicity patterns, in the frequency domain. Meanwhile, the Fourier Transform aids in isolating underlying patterns or trends from noise in time series data by emphasizing key frequency components, thereby enabling the model to mitigate noise effects. Additionally, incorporating Laplace analysis into Mamba improves its capacity to capture local correlations between neighboring data points, leading to a more accurate representation of transient dynamics. Our extensive experiments demonstrate that FLDmamba achieves superior performance on time series prediction benchmarks, outperforming both Transformer-based and other Mamba-based architectures. This work offers a computationally efficient and effective solution for long-term time series prediction, paving the way for its application in real-world scenarios. To promote the reproducibility of our method, we have made both the code and data accessible via the following URL: \\href{https://anonymous.4open.science/r/FLambas-AD7E/README.md}{https://anonymous.4open.science/r/FLDmamba}"},"_bibtex":{"value":"@misc{\nzhang2025fldmamba,\ntitle={{FLD}mamba:  Integrating Fourier and Laplace Transform Decomposition with Mamba for Enhanced Time Series Prediction},\nauthor={Qianru Zhang and Chenglei Yu and Haixin Wang and Yudong Yan and Yuansheng Cao and Hongzhi Yin and Siu Ming Yiu and Tailin Wu},\nyear={2025},\nurl={https://openreview.net/forum?id=9EiWIyJMNi}\n}"},"title":{"value":"FLDmamba:  Integrating Fourier and Laplace Transform Decomposition with Mamba for Enhanced Time Series Prediction"},"pdf":{"value":"/pdf/5dc7cf776ca2eac09c1caf0e8a4b4195707a1e3b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|fldmamba_integrating_fourier_and_laplace_transform_decomposition_with_mamba_for_enhanced_time_series_prediction"},"authorids":{"value":["~Qianru_Zhang2","~Chenglei_Yu1","~Haixin_Wang3","~Yudong_Yan1","~Yuansheng_Cao1","~Hongzhi_Yin2","~Siu_Ming_Yiu1","~Tailin_Wu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Qianru Zhang","Chenglei Yu","Haixin Wang","Yudong Yan","Yuansheng Cao","Hongzhi Yin","Siu Ming Yiu","Tailin Wu"]}},"version":2},{"content":{"summary":{"value":"The paper proposed a lightweight ECMNet for semantic segmentation, which effectively combines CNN and Mamba in a framework to leverage their complementary strengths in local detail capture and long-range dependency modeling. It introduces three key modules: an EDAB for lightweight feature refinement, a MSAU for spatial–channel aggregation, and a FFM for hierarchical feature integration. Experimental results on Cityscapes and CamVid show that ECMNet achieves 70.6% and 73.6% mIoU, respectively, with only 0.87M parameters and 8.27G FLOPs, demonstrating an balance between segmentation accuracy and computational efficiency."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1.The mathematical formulas (e.g., Equations 1–9) are densely stacked, which affects readability and clarity.\n2.The innovation is limited. The proposed network is essentially a combination of CNN and Mamba, where the EDAB module resembles the attention mechanism used in Transformers, and the MASU module is similar to CBAM, making the contribution incremental rather than fundamentally novel.\n3.The manuscript lacks a clear description of the loss function and evaluation metrics, which are necessary for reproducibility.\n4.The ablation study does not include qualitative comparisons, making it difficult to visually assess the impact of each component.\n5.The CamVid dataset lacks qualitative visual comparisons, and in Figure 7 (Cityscapes dataset), recent methods from the past two years are also missing.\n6.In Tables 2 and 3, the comparisons with recent methods are insufficient. Table 2 includes only two 2024 methods, and Table 3 includes only one 2024 method.\n7.The paper lacks any robustness analysis or validation, such as testing under noise, occlusion, or domain variation.\n8.In Table 4, the proposed method performs similarly to the 2019 method LEDNet; the reason for this comparable performance should be clearly explained."},"rating":{"value":4},"details_of_ethics_concerns":{"value":"1.The mathematical formulas (e.g., Equations 1–9) are densely stacked, which affects readability and clarity.\n2.The innovation is limited. The proposed network is essentially a combination of CNN and Mamba, where the EDAB module resembles the attention mechanism used in Transformers, and the MASU module is similar to CBAM, making the contribution incremental rather than fundamentally novel.\n3.The manuscript lacks a clear description of the loss function and evaluation metrics, which are necessary for reproducibility.\n4.The ablation study does not include qualitative comparisons, making it difficult to visually assess the impact of each component.\n5.The CamVid dataset lacks qualitative visual comparisons, and in Figure 7 (Cityscapes dataset), recent methods from the past two years are also missing.\n6.In Tables 2 and 3, the comparisons with recent methods are insufficient. Table 2 includes only two 2024 methods, and Table 3 includes only one 2024 method.\n7.The paper lacks any robustness analysis or validation, such as testing under noise, occlusion, or domain variation.\n8.In Table 4, the proposed method performs similarly to the 2019 method LEDNet; the reason for this comparable performance should be clearly explained."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"First, it effectively integrates CNN and Mamba to balance local detail capture with long-range dependency modeling, addressing the global context limitation common in semantic segmentation.\nSecond, it achieves an excellent accuracy–efficiency trade-off, reaching 70.6% mIoU on Cityscapes and 73.6% mIoU on CamVid with only 0.87M parameters and 8.27G FLOPs, demonstrating strong computational efficiency."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Incremental novelty: The core contribution is a module combination (CNN + Mamba + attention/fusion blocks) rather than a clear algorithmic breakthrough.\nInsufficient breadth of evaluation: Results are limited to Cityscapes and CamVid without robustness tests (domain shift, noise, resolution), or comparisons against the most recent lightweight/SSM-based segmenters."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919784152,"tcdate":1761575363886,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7728/Reviewer_iMoH"],"signatures":["ICLR.cc/2026/Conference/Submission7728/Reviewer_iMoH"],"forum":"0YV1iyqz16","number":1,"license":"CC BY 4.0","cdate":1761575363886,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7728/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919784152,"domain":"ICLR.cc/2026/Conference","replyto":"0YV1iyqz16","id":"WUYHkBP50L","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Semantic segmentation","Lightweight","Convolutional neural network","Mamba"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"In the past decade, Convolutional Neural Networks (CNNs) and Transformers have achieved wide application in semantic segmentation tasks. Although CNNs with Transformer models greatly improve performance, the global context modeling remains inadequate. Recently, Mamba achieved great potential in vision tasks, showing its advantages in modeling long-range dependency. In this paper, we propose a lightweight Efficient CNN-Mamba Network for semantic segmentation, dubbed as ECMNet.  ECMNet combines CNN with Mamba skillfully in a capsule-based framework to address their complementary weaknesses. Specifically, We design a Enhanced Dual-Attention Block (EDAB) for lightweight bottleneck. In order to improve the representations ability of feature, We devise a Multi-Scale Attention Unit (MSAU) to integrate multi-scale feature aggregation, spatial aggregation and channel aggregation. Moreover, a Mamba enhanced Feature Fusion Module (FFM) merges diverse level feature, significantly enhancing segmented accuracy. Extensive experiments on two representative datasets demonstrate that the proposed model excels in accuracy and efficiency balance, achieving 70.6% mIoU on Cityscapes and 73.6% mIoU on CamVid test datasets, with 0.87M parameters and 8.27G FLOPs on a single RTX 3090 GPU platform."},"_bibtex":{"value":"@misc{\ndu2025ecmnet,\ntitle={{ECMN}et: Lightweight Semantic Segmentation  with Efficient {CNN}-Mamba Network},\nauthor={Feixiang Du and Shengkun Wu and Xiang Wang and Aoxue Ding and Zhongliang Wang and Stephen Dou},\nyear={2025},\nurl={https://openreview.net/forum?id=0YV1iyqz16}\n}"},"title":{"value":"ECMNet: Lightweight Semantic Segmentation  with Efficient CNN-Mamba Network"},"pdf":{"value":"/pdf/44e22227c65b75759c71ad2a494e748f6d710a26.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"du|ecmnet_lightweight_semantic_segmentation_with_efficient_cnnmamba_network"},"authorids":{"value":["~Feixiang_Du1","~Shengkun_Wu1","~Xiang_Wang25","~Aoxue_Ding1","~Zhongliang_Wang2","~Stephen_Dou1"]},"authors":{"value":["Feixiang Du","Shengkun Wu","Xiang Wang","Aoxue Ding","Zhongliang Wang","Stephen Dou"]}},"version":2},{"content":{"summary":{"value":"The paper proposes Hierarchical Mamba (HiM), a model that integrates the Mamba2 state-space model with hyperbolic geometry (both Poincaré and Lorentz manifolds) for hierarchical reasoning tasks such as ontology and taxonomy inference. It introduces a SentenceMamba-16M encoder for sequence embedding and projects outputs into hyperbolic space via learnable curvature. A hyperbolic loss combining centripetal and clustering terms enforces parent–child relationships.\nExperiments on WordNet, DOID, FoodOn, and SNOMED-CT show improved F1 scores over Euclidean and hyperbolic transformer baselines, with claims of better scalability for long sequences."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1) Why does ontology reasoning require long-range sequence modeling? Are there sequential dependencies that justify using Mamba rather than a hyperbolic Transformer like HiT?\n\n2) How is HiM different from SHMamba (2024), HiT (2024), or Hyperbolic LoRA (2024) beyond substituting the backbone?\n\n3) Can you quantify how much of the performance gain arises from Mamba2 vs. the hyperbolic embedding itself?\n\n4) Have you compared HiM with strong non-hyperbolic ontology reasoning baselines like KG embeddings?\n\n5) What does “long-range dependency” mean in hierarchical reasoning? Can you formalize or visualize this relationship?\n\n6) How does the model handle non-tree or cyclic structures present in real ontologies like SNOMED-CT?\n\n7) Does learnable curvature consistently converge, or is it dataset-dependent?\n\n8) Why not test the approach on natural language tasks?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1) First work of intergrating Mamba2 and hyperbolic embeddings. \n\n2) The paper includes both Poincaré and Lorentz formulations, curvature regularization, and stability approximations.\n\n3) empirical study across four ontology datasets, including δ-hyperbolicity analysis and visualization of embedding hierarchies."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1) Questionable motivation for long-range dependency modeling. Ontology reasoning is structurally hierarchical but not sequential. It’s unclear why Mamba’s sequence modeling is advantageous here. The link between “long-range dependencies” and “multi-hop ontology reasoning” is weak and mismatched. \n\n2) Limited novelty. Integrating a known efficient sequence model (Mamba2) with hyperbolic embeddings is incremental, given prior works like SHMamba (2024) and Hyperbolic BERT / HiT (He et al 2024) / HypLoRA (Yang et al) already address hierarchy in both Euclidean and hyperbolic spaces.\n\n3) The “hyperbolic loss” largely reuses existing centripetal and triplet loss ideas without strong new formulation.\n\n4) The paper mixes terminology between hierarchical, long-range, and multi-hop reasoning without a clear theoretical bridge.\n\n5) Evaluation tasks are limited. Only ontology/knowledge graphs datasets are tested. There is no evidence of generalization to natural language or other structured tasks."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762921401865,"tcdate":1761757604806,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission9954/Reviewer_gJLi"],"signatures":["ICLR.cc/2026/Conference/Submission9954/Reviewer_gJLi"],"forum":"iR2hMzVu1k","number":1,"license":"CC BY 4.0","cdate":1761757604806,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission9954/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762921401865,"domain":"ICLR.cc/2026/Conference","replyto":"iR2hMzVu1k","id":"0j49AI8qod","forumContent":{"TLDR":{"value":"Hierarchical Mamba for Structure-Aware Language Embedding"},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Hierarchical language modeling","non-Euclidean Neural Networks","hyperbolic geometry","state-space models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Selective state-space models excel at long-sequence modeling, but their capacity for language representation -- in complex hierarchical reasoning -- remains underexplored. Most large language models rely on *flat* Euclidean embeddings, limiting their ability to capture latent hierarchies. To address this, we propose *Hierarchical Mamba (HiM)*, integrating efficient Mamba2 with hyperbolic geometry to learn hierarchy-aware language embeddings for deeper linguistic understanding. Mamba2-processed sequences are projected to the Poincar\\'e ball or Lorentzian manifold with \"learnable\" curvature, optimized with a hyperbolic loss. Our HiM model facilitates the capture of relational distances across varying hierarchical levels, enabling effective long-range reasoning for tasks like mixed-hop prediction and multi-hop inference in hierarchical classification. Experimental results show both HiM effectively capture hierarchical relationships across four linguistic and medical datasets, surpassing Euclidean baselines, with HiM-Poincar\\'e providing fine-grained distinctions with higher h-norms, while HiM-Lorentz offers more stable, compact, and hierarchy-preserving embeddings."},"_bibtex":{"value":"@misc{\npatil2025hierarchical,\ntitle={Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings},\nauthor={Sarang Rajendra Patil and Ashish Parmanand Pandey and Ioannis Koutis and Mengjia Xu},\nyear={2025},\nurl={https://openreview.net/forum?id=iR2hMzVu1k}\n}"},"title":{"value":"Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings"},"pdf":{"value":"/pdf/b82efbbaa2a39231a92a0bd359f74cc8bd82e0ab.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"patil|hierarchical_mamba_meets_hyperbolic_geometry_a_new_paradigm_for_structured_language_embeddings"},"authorids":{"value":["~Sarang_Rajendra_Patil1","~Ashish_Parmanand_Pandey1","~Ioannis_Koutis1","~Mengjia_Xu1"]},"authors":{"value":["Sarang Rajendra Patil","Ashish Parmanand Pandey","Ioannis Koutis","Mengjia Xu"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.07887v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"waleffe|an_empirical_study_of_mambabased_language_models"},"authorids":{"value":["~Roger_Waleffe1","~Wonmin_Byeon1","https://dblp.org/search/pid/api?q=author:Duncan_Riach:","https://dblp.org/search/pid/api?q=author:Brandon_Norick:","https://dblp.org/search/pid/api?q=author:Vijay_Korthikanti:","https://dblp.org/search/pid/api?q=author:Tri_Dao:","~Albert_Gu1","~Ali_Hatamizadeh1","https://dblp.org/search/pid/api?q=author:Sudhakar_Singh_0003:","https://dblp.org/search/pid/api?q=author:Deepak_Narayanan:","https://dblp.org/search/pid/api?q=author:Garvit_Kulshreshtha:","https://dblp.org/search/pid/api?q=author:Vartika_Singh:","https://dblp.org/search/pid/api?q=author:Jared_Casper:","~Jan_Kautz1","https://dblp.org/search/pid/api?q=author:Mohammad_Shoeybi:","~Bryan_Catanzaro1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.07887"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-07887,\n  publtype={informal},\n  author={Roger Waleffe and Wonmin Byeon and Duncan Riach and Brandon Norick and Vijay Korthikanti and Tri Dao and Albert Gu and Ali Hatamizadeh and Sudhakar Singh and Deepak Narayanan and Garvit Kulshreshtha and Vartika Singh and Jared Casper and Jan Kautz and Mohammad Shoeybi and Bryan Catanzaro},\n  title={An Empirical Study of Mamba-based Language Models},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.07887},\n  url={https://doi.org/10.48550/arXiv.2406.07887}\n}\n"},"abstract":{"value":"Selective state-space models (SSMs) like Mamba overcome some of the shortcomings of Transformers, such as quadratic computational complexity with sequence length and large inference-time memory requirements from the key-value cache. Moreover, recent studies have shown that SSMs can match or exceed the language modeling capabilities of Transformers, making them an attractive alternative. In a controlled setting (e.g., same data), however, studies so far have only presented small scale experiments comparing SSMs to Transformers. To understand the strengths and weaknesses of these architectures at larger scales, we present a direct comparison between 8B-parameter Mamba, Mamba-2, and Transformer models trained on the same datasets of up to 3.5T tokens. We also compare these models to a hybrid architecture consisting of 43% Mamba-2, 7% attention, and 50% MLP layers (Mamba-2-Hybrid). Using a diverse set of tasks, we answer the question of whether Mamba models can match Transformers at larger training budgets. Our results show that while pure SSMs match or exceed Transformers on many tasks, they lag behind Transformers on tasks which require strong copying or in-context learning abilities (e.g., 5-shot MMLU, Phonebook) or long-context reasoning. In contrast, we find that the 8B Mamba-2-Hybrid exceeds the 8B Transformer on all 12 standard tasks we evaluated (+2.65 points on average) and is predicted to be up to 8x faster when generating tokens at inference time. To validate long-context capabilities, we provide additional experiments evaluating variants of the Mamba-2-Hybrid and Transformer extended to support 16K, 32K, and 128K sequences. On an additional 23 long-context tasks, the hybrid model continues to closely match or exceed the Transformer on average. To enable further study, we release the checkpoints as well as the code used to train our models as part of NVIDIA's Megatron-LM project."},"title":{"value":"An Empirical Study of Mamba-based Language Models"},"authors":{"value":["Roger Waleffe","Wonmin Byeon","Duncan Riach","Brandon Norick","Vijay Korthikanti","Tri Dao","Albert Gu","Ali Hatamizadeh","Sudhakar Singh","Deepak Narayanan","Garvit Kulshreshtha","Vartika Singh","Jared Casper","Jan Kautz","Mohammad Shoeybi","Bryan Catanzaro"]}},"tmdate":1769714005782,"pdate":1704067200000,"tcdate":1730142570156,"writers":["~"],"signatures":["~Roger_Waleffe1"],"forum":"lutFMQeNIE","license":"CC BY-SA 4.0","number":162019,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769714005782,"domain":"DBLP.org","id":"lutFMQeNIE","version":2},{"content":{"venue":{"value":"IEEE Trans. Pattern Anal. Mach. Intell. 2024"},"pdf":{"value":"https://ieeexplore.ieee.org/iel7/34/10522060/10416705.pdf"},"venueid":{"value":"dblp.org/journals/PAMI/2024"},"paperhash":{"value":"xiao|selective_random_walk_for_transfer_learning_in_heterogeneous_label_spaces"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Qiao_Xiao:","~Yu_Zhang3","https://dblp.org/search/pid/api?q=author:Qiang_Yang_0001:"]},"html":{"value":"https://doi.org/10.1109/TPAMI.2024.3360118"},"_bibtex":{"value":"@article{DBLP:journals/pami/XiaoZY24,\n  author={Qiao Xiao and Yu Zhang and Qiang Yang},\n  title={Selective Random Walk for Transfer Learning in Heterogeneous Label Spaces},\n  year={2024},\n  month={June},\n  cdate={1717200000000},\n  journal={IEEE Trans. Pattern Anal. Mach. Intell.},\n  volume={46},\n  number={6},\n  pages={4476-4488},\n  url={https://doi.org/10.1109/TPAMI.2024.3360118}\n}\n"},"abstract":{"value":"Transfer learning has been widely used in different scenarios, especially in those lacking enough labeled data. However, most of the existing transfer learning methods are based on the assumption that the source and target domains should share the label space entirely or partially, which greatly limits their application scopes. In this article, a Selective Random Walk (SRW) method for transfer learning in heterogeneous label spaces is proposed to make full use of unlabeled auxiliary data, which acts as a bridge for knowledge transfer from the source domain to the target domain. The proposed SRW method can explicitly identify transfer sequences between source and target instances via auxiliary instances based on random walk techniques. Since not all of the transfer sequences generated by random walk are credible for the target task, the SRW method can learn to weight transfer sequences adaptively. Based on the weights of the transfer sequences, the SRW method leverages knowledge by forcing adjacent data points in the transfer sequence to be similar and making the target data point in the sequence represented by other data points in the same sequence. Experiments show that the SRW method outperforms state-of-the-art models in plenty of transfer learning tasks with heterogeneous label spaces constructed within and across several benchmark datasets."},"title":{"value":"Selective Random Walk for Transfer Learning in Heterogeneous Label Spaces"},"authors":{"value":["Qiao Xiao","Yu Zhang","Qiang Yang"]}},"tmdate":1747840703688,"pdate":1704067200000,"tcdate":1747840699840,"writers":["~"],"signatures":["~Yu_Zhang3"],"forum":"ajyMY20Vix","license":"CC BY-SA 4.0","number":547479,"cdate":1717200000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747840703688,"domain":"DBLP.org","id":"ajyMY20Vix","version":2},{"content":{"venue":{"value":"EMNLP 2023 Main"},"TLDR":{"value":"We investigate the impacts of states and natural language instructions on what knowledge Transformer-based sequence models can learn."},"Submission_Track_2":{"value":"Interpretability, Interactivity, and Analysis of Models for NLP"},"keywords":{"value":["Interpretability and Analysis; Decision Making via Sequence Modeling; Language Grounding to Vision and Beyond"]},"abstract":{"value":"Decision making via sequence modeling aims to mimic the success of language models, where actions taken by an embodied agent are modeled as tokens to predict. Despite their promising performance, it remains unclear if embodied sequence modeling leads to the emergence of internal representations that represent the environmental state information. A model that lacks abstract state representations would be liable to make decisions based on surface statistics which fail to generalize. We take the BabyAI environment, a grid world in which language-conditioned navigation tasks are performed, and build a sequence modeling Transformer, which takes a language instruction, a sequence of actions, and environmental observations as its inputs. In order to investigate the emergence of abstract state representations, we design a \"blindfolded\" navigation task, where only the initial environmental layout, the language instruction, and the action sequence to complete the task are available for training. Our probing results show that intermediate environmental layouts can be reasonably reconstructed from the internal activations of a trained model, and that language instructions play a role in the reconstruction accuracy. Our results suggest that many key features of state representations can emerge via embodied sequence modeling, supporting an optimistic outlook for applications of sequence modeling objectives to more complex embodied decision-making domains."},"_bibtex":{"value":"@inproceedings{\nyun2023emergence,\ntitle={Emergence of Abstract State Representations in Embodied Sequence Modeling},\nauthor={Tian Yun and Zilai Zeng and Kunal Handa and Ashish V Thapliyal and Bo Pang and Ellie Pavlick and Chen Sun},\nbooktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},\nyear={2023},\nurl={https://openreview.net/forum?id=FKNtgr0qQy}\n}"},"title":{"value":"Emergence of Abstract State Representations in Embodied Sequence Modeling"},"Submission_Type":{"value":"Regular Long Paper"},"pdf":{"value":"/attachment/bae11487c477e9f56d6de2e0869eef18bf16c31a.pdf"},"Submission_Track":{"value":"Language Grounding to Vision, Robotics and Beyond"},"venueid":{"value":"EMNLP/2023/Conference"},"paperhash":{"value":"yun|emergence_of_abstract_state_representations_in_embodied_sequence_modeling"},"authorids":{"value":["~Tian_Yun2","~Zilai_Zeng1","~Kunal_Handa1","~Ashish_V_Thapliyal1","~Bo_Pang3","~Ellie_Pavlick1","~Chen_Sun1"]},"authors":{"value":["Tian Yun","Zilai Zeng","Kunal Handa","Ashish V Thapliyal","Bo Pang","Ellie Pavlick","Chen Sun"]}},"tmdate":1701459769175,"pdate":1696709286971,"tcdate":1686955360697,"writers":["EMNLP/2023/Conference","EMNLP/2023/Conference/Submission4558/Authors"],"signatures":["EMNLP/2023/Conference/Submission4558/Authors"],"forum":"FKNtgr0qQy","number":4558,"cdate":1686955360697,"mdate":1701459769175,"readers":["everyone"],"invitations":["EMNLP/2023/Conference/-/Submission","EMNLP/2023/Conference/-/Post_Submission","EMNLP/2023/Conference/Submission4558/-/Revision","EMNLP/2023/Conference/-/Edit","EMNLP/2023/Conference/Submission4558/-/Camera_Ready_Revision"],"odate":1701459769162,"domain":"EMNLP/2023/Conference","id":"FKNtgr0qQy","version":2},{"content":{"summary":{"value":"This paper proposes MOHAWK, a three-stage distillation method to transfer knowledge from pretrained transformer models to subquadratic models such as Mamba-2. The key idea is to view both transformers and SSMs as applying different forms of mixing matrices over token sequences. Experiments demonstrate that Phi-Mamba, a variant of Mamba-2 architecture tailored for distillation, achieves strong performance on downstream tasks using less than 1% of the training data compared to training from scratch."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See weaknesses part."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The paper introduces an effective way to distill knowledge from pretrained transformer models to subquadratic models, enabling the linear attention community to obtain better models at a lower cost.\n2. Mamba-2 aligns well with self-attention by converting the forget gate of GLA into a data-dependent value, achieving linear attention-like structures by simply removing the softmax and adding a forget gate.\n3. The three-stage distillation process is well-designed and ablation studies demonstrate the importance of each stage, providing valuable insights for future research in this direction."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper does not provide a detailed comparison between the proposed distillation method and direct fine-tuning approaches like SUPRA (reference 21). While the three-stage distillation process achieves better performance with only 2.8B tokens, it is crucial to consider the additional computational overhead introduced by the need for both a teacher model and a student model during distillation. In contrast, direct fine-tuning methods, though requiring more tokens (e.g., +20B in SUPRA), may be simpler and more efficient in practice. The authors should discuss the trade-offs between these approaches and provide ablation studies to justify their choice of distillation over direct fine-tuning.\n\n2. There are several typos throughout the paper. For example, in line 159, \"Mamba-2 2\" should have a \"Figure\" mentioned before the second \"2\". The authors should carefully proofread the manuscript to address these issues.\n\n3. There is an inconsistency in the reported model size of Phi-1.5. The official documentation states that Phi-1.5 has 1.3B parameters, while this paper mentions a model size of 1.5B. The authors should clarify this discrepancy and ensure consistency throughout the paper.\n\n4. The paper does not report the performance of the Phi-Mamba model on the MMLU benchmark, which is a standard evaluation for the Phi-1.5 model. Given that Phi-1.5 (1.3B) has reported MMLU performance, it would be informative to compare the MMLU scores of the Phi-Mamba model to assess its performance on a wider range of tasks."},"limitations":{"value":"The authors have discussed the potential limitations in Section 6."}},"nonreaders":[],"tmdate":1730879576395,"tcdate":1718824084571,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission12748/Reviewer_Kdfj"],"signatures":["NeurIPS.cc/2024/Conference/Submission12748/Reviewer_Kdfj"],"forum":"FJlrSZBMCD","number":1,"license":"CC BY 4.0","cdate":1718824084571,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission12748/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879576395,"domain":"NeurIPS.cc/2024/Conference","replyto":"FJlrSZBMCD","id":"5i05JNToia","forumContent":{"TLDR":{"value":"We show that we can leverage pre-trained Transformers to create highly performant state space models using very little training data"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["distillation","mamba","sub-quadratic","ssm","state-space-models"]},"primary_area":{"value":"deep_learning_architectures"},"abstract":{"value":"Transformer architectures have become a dominant paradigm for domains like language modeling but suffer in many inference settings due to their quadratic-time self-attention. Recently proposed subquadratic architectures, such as Mamba, have shown promise, but have been pretrained with substantially less computational resources than the strongest Transformer models. In this work, we present a method that is able to distill a pretrained Transformer architecture into alternative architectures such as state space models (SSMs). The key idea to our approach is that we can view both Transformers and SSMs as applying different forms of mixing matrices over the token sequences. We can thus progressively distill the Transformer architecture by matching different degrees of granularity in the SSM: first matching the mixing matrices themselves, then the hidden units at each block, and finally the end-to-end predictions. Our method, called MOHAWK, is able to distill a Mamba-2 variant based on the Phi-1.5 architecture (Phi-Mamba) using only 3B tokens. Despite using less than 1% of the training data typically used to train models from scratch, Phi-Mamba boasts substantially stronger performance compared to all past open-source non-Transformer models. MOHAWK allows models like SSMs to leverage computational resources invested in training Transformer-based architectures, highlighting a new avenue for building such models."},"_bibtex":{"value":"@inproceedings{\nbick2024transformers,\ntitle={Transformers to {SSM}s: Distilling Quadratic Knowledge to Subquadratic Models},\nauthor={Aviv Bick and Kevin Li and Eric P. Xing and J Zico Kolter and Albert Gu},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=FJlrSZBMCD}\n}"},"title":{"value":"Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models"},"pdf":{"value":"/pdf/ddedaa9f0d6404305d1b4b3223cca34caab6ab83.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"bick|transformers_to_ssms_distilling_quadratic_knowledge_to_subquadratic_models"},"authorids":{"value":["~Aviv_Bick1","~Kevin_Li3","~Eric_Xing1","~J_Zico_Kolter1","~Albert_Gu1"]},"authors":{"value":["Aviv Bick","Kevin Li","Eric P. Xing","J Zico Kolter","Albert Gu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes to combine selective prediction and post-hoc calibration to achieve more reliable performance for classification tasks. The proposed optimization framework is based on another recent work which combines selective prediction and training-time calibration. In the proposed selective recalibration framework, a new loss function, Selective Top-Label Binary Cross Entropy (S-TLBCE) is proposed for training such a recalibrator and a selective model at the same time. The proposed approach is evaluated on real-world medical diagnosis datasets and image classification datasets, and the results show its effectiveness."},"presentation":{"value":"2 fair"},"contribution":{"value":"2 fair"},"soundness":{"value":"2 fair"},"strengths":{"value":"1. The idea of combining selective prediction and model calibration is very useful and realistic in many safety aware tasks. Also, previous work studied the combination of selective prediction and train-time calibration, the combination of selective prediction and recalibration seems more important as post-hoc calibration achieves better calibration performance generally.\n\n2. Except for the experiments on real-world datasets, the authors also give theoretic results based on a simple and intuitive data generation model."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The technical novelty and overall contribution is quite limited, primarily because the combination of selective prediction and model calibration has been studied in previous work and the proposed method is a straightforward combination of existing optimization framework and recalibration model. \n\n2. The writing and organization of this paper are not good enough. Many places involving notations are confusing, for example, some loss functions appear in the Methodology section but are not used in the following optimization framework, and some notations are not defined before their first appearance.\n\n3. The empirical evaluation is also limited, as a result, the effectiveness and soundness cannot be sufficiently shown. For example, some benchmark datasets for image classification, which are commonly used in the context of model calibration like SVHN/CIFAR-10, are not used in experiment section. Moreover, the experimental part does not provide sufficient information about the dataset used and the way it was divided (for training and validation), as well as the model structure."},"confidence":{"value":"2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked."},"questions":{"value":"Please refer to the weaknesses section."},"rating":{"value":"5: marginally below the acceptance threshold"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1700623070423,"tcdate":1698762857031,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6404/Reviewer_6mVa"],"signatures":["ICLR.cc/2024/Conference/Submission6404/Reviewer_6mVa"],"forum":"0kvrymILfy","number":2,"license":"CC BY 4.0","cdate":1698762857031,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6404/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1700623070423,"domain":"ICLR.cc/2024/Conference","replyto":"0kvrymILfy","id":"RTBsbRsty1","forumContent":{"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["calibration","statistical learning"]},"supplementary_material":{"value":"/attachment/87fbedc398ec82602df4467c794e39d1ddcc41fa.zip"},"primary_area":{"value":"general machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"A reliable deep learning system should be able to accurately express its confidence with respect to its predictions, a quality known as calibration.  One of the most effective ways to produce reliable confidence estimates with a pre-trained model is by applying a post-hoc recalibration method.  Popular recalibration methods like temperature scaling are typically fit on a small amount of data and work in the model's output space, as opposed to the more expressive feature embedding space, and thus usually have only one or a handful of parameters.  However, the target distribution to which they are applied is often complex and difficult to fit well with such a function.  To this end we propose selective recalibration, where a selection model learns to reject some user-chosen proportion of the data in order to allow the recalibrator to focus on regions of the input space that can be well-captured by such a model.  We provide theoretical analysis to motivate our algorithm, and test our method through comprehensive experiments on difficult medical imaging and zero-shot classification tasks.  Our results show that selective recalibration consistently leads to significantly lower calibration error than a wide range of selection and recalibration baselines."},"_bibtex":{"value":"@misc{\nzollo2024making,\ntitle={Making Predictors More Reliable with Selective Recalibration},\nauthor={Thomas P Zollo and Zhun Deng and Jake Snell and Toniann Pitassi and Richard Zemel},\nyear={2024},\nurl={https://openreview.net/forum?id=0kvrymILfy}\n}"},"title":{"value":"Making Predictors More Reliable with Selective Recalibration"},"pdf":{"value":"/pdf/beefb4832daf12115b8fb7eeca4d77f816c75a58.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"zollo|making_predictors_more_reliable_with_selective_recalibration"},"authorids":{"value":["~Thomas_P_Zollo1","~Zhun_Deng1","~Jake_Snell1","~Toniann_Pitassi3","~Richard_Zemel1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Thomas P Zollo","Zhun Deng","Jake Snell","Toniann Pitassi","Richard Zemel"]}},"version":2},{"content":{"summary":{"value":"This paper introduces an innovative time series forecasting model, DCTS (Discrete and Continuous Time Series Fusion), which combines discrete encoding and continuous encoding to enhance both pattern abstraction and numerical expression abilities.\nTraditional time series models typically focus solely on continuous features, ignoring the underlying discrete patterns in the data (such as workday/weekend or peak/off-peak characteristics). This can lead to overfitting to noise or neglecting key patterns. DCTS addresses this issue by introducing vector quantization (VQ) into continuous models, leveraging a multi-codebook structure to generate multi-dimensional discrete codes, significantly enhancing the feature space’s expressiveness. The model consists of Discrete Module and Continuous Module. Experiments on multiple real-world datasets, including ETT, Weather, Traffic, Electricity, and Solar, show that DCTS outperforms state-of-the-art models (e.g., SOFTS, PatchTST, iTransformer) in terms of both MSE and MAE. Ablation studies confirm the complementarity of the two modules, while contrastive experiments highlight the advantages of the multi-codebook mechanism for capturing complex time series patterns."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"See weakness"},"rating":{"value":2},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. DCTS is the first model to integrate discrete vector quantization with continuous time series modeling, utilizing multi-codebooks to expand the discrete encoding space. \n2.  DCTS improved MSE by 2.7% and MAE by 1.5% compared to state-of-the-art models, demonstrating its effectiveness and generalization ability\n3. The proposed dual-module structure (Discrete + Continuous) clearly separates the responsibilities: the discrete module captures global patterns, while the continuous module focuses on fine-grained changes."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Except for SOFT, all compared baselines are from 2023 or earlier, lacking comparisons with the most recent works.\n2. The performance improvement achieved by the model is minimal and remains lower than that of some more advanced baselines[1,2].\n3. There is no analysis of key hyperparameters. In addition, the input length is fixed at 96, but many of the compared baselines (such as PatchTST and DLinear) are generally not suitable for this setting. Overall, the experimental section is incomplete and insufficient.\n4. The core technology mainly consists of a combination of existing methods. Moreover, Vector Quantization has already been explored in prior studies [3,4]. Therefore, the technical innovation of this paper is limited.\n5. Since the authors emphasize the advantages of using a codebook, the following comparative experiments should be conducted: replace the codebook used in the proposed model with traditional patch or adaptive patch techniques [5], or alternatively, use simple temporal and variable encoding methods [6].\n6. There is a lack of visualization for the codebook, making it impossible for reviewers to determine whether issues such as dead codes exist.\n7. Since the authors emphasize that the proposed method can help the model recognize certain pattern characteristics of data across different time periods, and the example used is traffic data, corresponding experiments should be conducted to validate this motivation. As shown in paper [7], can the method accurately predict sudden events in datasets such as PEMS or METR-LA?\n8. In summary, I believe that the experimental results presented in the current version are insufficient. The proposed method is neither solid nor innovative. I recommend that the authors conduct a comprehensive revision, particularly by adding experiments that validate the motivation and by including comparisons with more advanced baselines.\n\n[1]  TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables\n\n[2] Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling\n\n[3] Sparse-vq transformer: An ffn-free framework with vector quantization for enhanced time series forecasting\n\n[4] Fusionsf: Fuse heterogeneous modalities in a vector quantized framework for robust solar power forecasting\n\n[5] Enhancing Time Series Forecasting through Selective Representation Spaces: A Patch Perspective\n\n[6] Spatial-temporal identity: A simple yet effective baseline for multivariate time series forecasting\n\n[7] Hutformer: Hierarchical u-net transformer for long-term traffic forecasting"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926481394,"tcdate":1761744063510,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16347/Reviewer_mLa9"],"signatures":["ICLR.cc/2026/Conference/Submission16347/Reviewer_mLa9"],"forum":"m4t7rpjUEs","number":2,"license":"CC BY 4.0","cdate":1761744063510,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16347/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926481394,"domain":"ICLR.cc/2026/Conference","replyto":"m4t7rpjUEs","id":"MARcEDcw3F","forumContent":{"TLDR":{"value":"Mixing discrete information to continuous model"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Deep Learning","Time Series Forecasting"]},"supplementary_material":{"value":"/attachment/dffc9ec6a642b46baea07828f6ce92d9f8a90f0d.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"In time series analysis, data is usually treated as a set of continuous values. Conventional methods do all the computations, from inputs to outputs, in continuous form. While this continuous representations are highly expressive, it can also pay too much attention to fine-grained details. This risks introducing noise and overlooking critical information. In contrast, discrete representations can assign a single code to each temporal pattern. In this way, key patterns underlain in the data are extracted more effectively, while some informative details are probably filtered out along with noises. In order to combine the advantages of both, we propose fusing discrete and continuous information for time series forecasting (DCTS), that incorporates both continuous and discrete approaches, it fuses the expressive power of continuous encoding and pattern-abstracting ability of discrete encoding. It uses a codebook learned via vector quantization to extract discrete encoding from the time series and then fuses it with the continuous encoding. In doing so, the model can benefit from the strengths of both continuous and discrete representations. Additionally, we use multiple codebooks to encode the time series. A single code can hardly cover the entire feature space of a time series. In contrast, multiple discrete values can be combined, exponentially expanding the encoding space and achieving much stronger expressive power. We evaluated our proposed method on multiple real-world datasets and achieved the best performance compared to the baseline methods."},"_bibtex":{"value":"@misc{\nliu2026dcts,\ntitle={{DCTS}: Fusing Discrete and Continuous Information for Time Series Forecasting},\nauthor={Shilin Liu and zipei fan and Xuan Song},\nyear={2026},\nurl={https://openreview.net/forum?id=m4t7rpjUEs}\n}"},"title":{"value":"DCTS: Fusing Discrete and Continuous Information for Time Series Forecasting"},"pdf":{"value":"/pdf/f52407a12fe041bd81bea6aefae222dfc1563a66.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"liu|dcts_fusing_discrete_and_continuous_information_for_time_series_forecasting"},"authorids":{"value":["~Shilin_Liu3","~zipei_fan1","~Xuan_Song5"]},"authors":{"value":["Shilin Liu","zipei fan","Xuan Song"]}},"version":2},{"content":{"comment":{"value":"We're willing to change the title to reflect the fact that our paper focuses on Mamba SSMs in particular if you think this is a significant improvement. But we'd like to clarify some potential misunderstandings first.\n\nThe paper is mostly about the copying task, which previous work has noted state space models struggle on in particular [0]. Consequently, we focus on SSMs which can actually perform on the copying task. This _excludes_ LRU and RetNet: they are *linear time invariant* and _cannot_ generalize on the copying task. *Selective* SSMs like Mamba 1, Mamba 2, and GLA implement an _input dependent_ operation that can actually generalize on copying tasks.\n\nOur usage of \"State Space Models\" is in line with previous work [0,1], assuming we're restricted to SSMs which _can_ copy and recall (beyond one fixed sequence length).\n\nWe concede that GLA may be a reasonable comparison to put in the paper. GLA is quite similar to the state space layer in Mamba 1 and 2, and the same technique from our paper applies: try to set underlying parameters such that the gating parameter is close to all-1s, making the layer resemble linear attention. The correlated C/B (equivalently, query/key) phenomenon also applies. However, in our experiments, we were not able to train networks using only GLA to copy. More experiment details below:\n\nIn particular, we replaced the SSD layer inside Mamba 2 with GLA with the same hidden dimension and number of heads. We attempted to set $\\alpha_t = \\sigma(x_t W_{\\alpha 1} W_{\\alpha 2} + b_\\alpha )$ so that $G_t = \\alpha_t^T 1 \\approx 1$. In the GLA implementation, $\\sigma(x) = \\exp( \\mathsf{LogSigmoid}(x)/16 )$, so we set $W_{\\alpha 1} W_{\\alpha 2} \\approx 0$ and $b_\\alpha \\approx 10$ to get $\\alpha_t \\approx 1$. We compare this to the default initialization, training on 50-length strings, using 4-layer models with hidden dim 1024.\n\n|Model | Acc. (Length 50) | Acc. (Length 200)|\n|--------|--------------------|-----------------------|\n|Baseline Mamba 2\t\t| 100%\t |\t39.1% |\n|Baseline Mamba 2 (init)\t        | 100%\t |\t88.2% |\n|All GLA               \t\t|  5.0%\t |\t4.2% |\n|All GLA (init)\t\t\t| 99.9%\t |\t8.10%      |\n|GLA Layer 2      \t\t| 100%\t |\t13.1% |\n|GLA Layer 2 (init)\t\t| 100%\t |\t86.1% |\n\nModels using only GLA were not able to learn to generalize on the copying task despite trying several initialization hyperparameters. Our initialization seems to help fit to the training length, at least. We also tried Mamba 2 models where we replaced SSD in only the second layer by GLA (with/without our init), and this performs comparably to Mamba 2 with our init. We note that GLA is actually closer to all-1s using its default init than Mamba 2 due to the particular sigmoid parameterization, so we might expect (at least this component) of our init to have less of an effect.\n\nWe're unsure why GLA on its own seems to be unable to learn to generalize on copying, but this makes us think that including it in the paper may be relatively low priority.\n\nRe: *\"SSM layers cannot mimic full Attention\"*. There's a misunderstanding here; selective SSMs such as Mamba 2 can in fact mimic (linear) attention, and this is in fact noted in the title of the Mamba 2 paper \"Transformers are SSMs\" [1]. We show this in our paper mathematically. Of course, due to the limited state size and lack of softmax, we can't mimic full (softmax) attention.\n\nWe can add some references, though we already cite [3] and the time invariant SSM and linear attention papers are not too relevant (as we use no specialized linear attention techniques). [5,7,8] are good references to add though. Let us know if you have further thoughts."},"title":{"value":"Response 1/2"}},"tmdate":1732135989520,"tcdate":1732135989520,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11529/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission11529/Authors"],"forum":"iVy7aRMb0K","number":4,"license":"CC BY 4.0","cdate":1732135989520,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11529/-/Official_Comment"],"mdate":1732135989520,"domain":"ICLR.cc/2025/Conference","replyto":"lbHCkjpQDC","id":"XXxAjC9wu3","forumContent":{"TLDR":{"value":"Mamba learns to recall more easily when we initialize it to be more like linear attention"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba","SSM","sequence models","language models","initialization","linear attention"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with respect to their input sequence length. But in practice, state space models have fairly large state sizes, and we conjecture that they should be able to perform much better at these tasks than previously reported. We investigate whether their poor copying and recall performance could be due in part to training difficulties rather than fundamental capacity constraints. Based on observations of their \"attention'' maps, we propose a structured initialization technique that allows state space layers to more readily mimic attention. Across a variety of architecture settings, our initialization makes it substantially easier for Mamba to learn to copy and do associative recall from scratch."},"_bibtex":{"value":"@misc{\ntrockman2025mimetic,\ntitle={Mimetic Initialization Helps State Space Models Learn to Recall},\nauthor={Asher Trockman and Hrayr Harutyunyan and J Zico Kolter and Sanjiv Kumar and Srinadh Bhojanapalli},\nyear={2025},\nurl={https://openreview.net/forum?id=iVy7aRMb0K}\n}"},"title":{"value":"Mimetic Initialization Helps State Space Models Learn to Recall"},"pdf":{"value":"/pdf/5b9624e012b1653fd84db25266eba2a30fcf0588.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"trockman|mimetic_initialization_helps_state_space_models_learn_to_recall"},"authorids":{"value":["~Asher_Trockman1","~Hrayr_Harutyunyan1","~J_Zico_Kolter1","~Sanjiv_Kumar1","~Srinadh_Bhojanapalli1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Asher Trockman","Hrayr Harutyunyan","J Zico Kolter","Sanjiv Kumar","Srinadh Bhojanapalli"]}},"version":2},{"content":{"review":{"value":"**Summary:**\n\nThis proposal presents a protein sequence generation model based on the Mamba architecture, aiming to improve efficiency and accessibility for protein modeling. By leveraging the Mamba model's selective state-space updates, which offer linear computational complexity and efficient handling of long-range dependencies, the authors seek to generate high-quality protein sequences with reduced computational resources. They plan to evaluate the model’s performance across multiple dimensions, including sequence generation quality, structural accuracy, and computational efficiency, with a focus on making protein modeling more accessible to researchers with limited resources.\n\n**Strengths:**\n1. **Efficient Model Choice:** The use of Mamba architecture, known for its computational efficiency, addresses accessibility issues and makes complex protein modeling more feasible in resource-limited settings.\n2. **Comprehensive Evaluation Framework:** The proposal outlines a multi-faceted evaluation framework, covering sequence quality, structural validation, and biological plausibility, which provides a thorough assessment of the model's capabilities.\n\n**Weaknesses:**\n1. **Evaluation of Long-Term Stability Missing:** There is no mention of testing the long-term stability or usability of generated sequences in practical protein engineering applications.\n\n**Questions:**\n1. How will the authors ensure that the generated sequences maintain functionality and stability in real biological contexts?\n2. Is there a plan to compare the model’s performance with existing transformer-based protein models, specifically in terms of computational cost and sequence quality?"},"confidence":{"value":3},"rating":{"value":9},"title":{"value":"Good proposal, well explained"}},"nonreaders":[],"tmdate":1731424267066,"tcdate":1731424236739,"writers":["tsinghua.edu.cn/THU/2024/Fall/AML","~Zhu_Zhang6"],"signatures":["~Zhu_Zhang6"],"forum":"I2FbCxyYVS","number":12,"license":"CC BY 4.0","cdate":1731424236739,"readers":["everyone"],"invitations":["tsinghua.edu.cn/THU/2024/Fall/AML/Submission52/-/Official_Review"],"mdate":1731424267066,"domain":"tsinghua.edu.cn/THU/2024/Fall/AML","replyto":"I2FbCxyYVS","id":"c4QkkGyfd4","forumContent":{"TLDR":{"value":"This proposal explores implementing a Mamba architecture for efficient protein sequence generation, aiming to balance performance with reduced computational resource demands and making the architecture more suited to biological dynamics."},"venue":{"value":"THU 2024 Fall AML Submission"},"pdf":{"value":"/pdf/8d5ee7449a5695662b51c83289c76642879f2be4.pdf"},"keywords":{"value":["protein sequence generation","Mamba architecture","efficient modeling","protein language models","biological dynamics"]},"venueid":{"value":"tsinghua.edu.cn/THU/2024/Fall/AML/Submission"},"paperhash":{"value":"jiang|[proposalml]protein_sequence_generation_model"},"authorids":{"value":["~pan_jiang1","zeeshanzulfiqar79@outlook.com","ivanyazykov.mipt@gmail.com"]},"abstract":{"value":"This proposal outlines a novel approach for protein sequence generation, a key area in computational biology focused on generating sequences with specific functional and structural characteristics. Despite recent advances with deep learning models, challenges remain in handling long sequences and capturing biological dynamics. We propose implementing a Mamba architecture that leverages selective state-space updates to achieve efficient protein sequence generation. With linear computational complexity and effective handling of long-range dependencies, this architecture is designed to be more suited to biological dynamics while reducing resource requirements. A comprehensive evaluation on the UniRef50 dataset will demonstrate its potential to deliver competitive performance, providing a viable solution for research environments with limited computational resources."},"_bibtex":{"value":"@inproceedings{\njiang2024proposalmlprotein,\ntitle={[Proposal-{ML}]Protein Sequence Generation Model},\nauthor={pan jiang and Zeeshan Zulfiqar and Ivan Iazykov},\nbooktitle={Submitted to Tsinghua University Course: Advanced Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=I2FbCxyYVS},\nnote={under review}\n}"},"title":{"value":"[Proposal-ML]Protein Sequence Generation Model"},"authors":{"value":["pan jiang","Zeeshan Zulfiqar","Ivan Iazykov"]}},"version":2},{"content":{"TLDR":{"value":"We uncover an interesting phenomenon where a single-layer Mamba represents the Bayes optimal Laplacian smoothing estimator when trained on Markov chains and we demonstrate it theoretically and empirically."},"venue":{"value":"ICLR 2026 Oral"},"keywords":{"value":["State-space models","Markov chains","In-context learning","Laplacian smoothing"]},"supplementary_material":{"value":"/attachment/7481132bbc16e969531b0db2b1a4e3d8dbafeefc.zip"},"primary_area":{"value":"interpretability and explainable AI"},"abstract":{"value":"While transformer-based language models have driven the AI revolution thus far, their computational complexity has spurred growing interest in viable alternatives, such as structured state space sequence models (SSMs) and Selective SSMs. Among these, Mamba (S6) and its variant Mamba-2 have shown remarkable inference speed-ups over transformers while achieving comparable or superior performance on complex language modeling tasks. However, despite these architectural innovations and empirical successes, the fundamental learning capabilities of Mamba remain poorly understood. In this paper, we address this gap by studying in-context learning (ICL) on Markov chains and uncovering an interesting phenomenon: even a single-layer Mamba efficiently learns the in-context Laplacian smoothing estimator, which is both Bayes and minimax optimal. To explain this, we theoretically characterize the representation capacity of Mamba and reveal the fundamental role of convolution in enabling it to represent the optimal Laplacian smoothing. These theoretical insights align strongly with empirical results and, to the best of our knowledge, represent the first formal connection between Mamba and optimal statistical estimators. Finally, we outline promising research directions inspired by these findings."},"_bibtex":{"value":"@inproceedings{\nbondaschi2026from,\ntitle={From Markov to Laplace: How Mamba In-Context Learns Markov Chains},\nauthor={Marco Bondaschi and Nived Rajaraman and Xiuying Wei and Razvan Pascanu and Caglar Gulcehre and Michael Gastpar and Ashok Vardhan Makkuva},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=kmK3WSCOCT}\n}"},"title":{"value":"From Markov to Laplace: How Mamba In-Context Learns Markov Chains"},"pdf":{"value":"/pdf/7c5974f3688d2ba6f4817747b376897b745b3197.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"bondaschi|from_markov_to_laplace_how_mamba_incontext_learns_markov_chains"},"authorids":{"value":["~Marco_Bondaschi1","~Nived_Rajaraman1","~Xiuying_Wei1","~Razvan_Pascanu1","~Caglar_Gulcehre1","~Michael_Gastpar1","~Ashok_Vardhan_Makkuva1"]},"authors":{"value":["Marco Bondaschi","Nived Rajaraman","Xiuying Wei","Razvan Pascanu","Caglar Gulcehre","Michael Gastpar","Ashok Vardhan Makkuva"]}},"tmdate":1775877247354,"pdate":1769436339032,"tcdate":1758274156934,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17277/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission17277/Authors"],"forum":"kmK3WSCOCT","license":"CC BY 4.0","number":17277,"cdate":1758274156934,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission17277/-/Full_Submission","ICLR.cc/2026/Conference/Submission17277/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission17277/-/Camera_Ready_Revision"],"mdate":1775877247354,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"kmK3WSCOCT","version":2},{"content":{"summary":{"value":"This paper proposes a novel time-adaptive RL method framework (TaCoS) for continuous-time systems with continuous state and action spaces. The framework shows that the settings of interactions having costs or a budget of interactions, can be formulated as extended MDPs, that can be solved with standard RL algorithms. The paper theoretically demonstrates this, and empirically verifies this. The paper also empirically demonstrates that TaCoS works across a range of interaction frequencies, and proposes OTaCoS a model-based RL approach."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"* Is the action applied from the policy kept constant until the next interaction? I presume the action is kept constant. Have you considered parameterizing the action and enabling non-constant actions or even continuous trajectories of actions between interaction times as a form of open-loop planning?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"* The paper is well written, and the intuitive explanations help the reader.\n* The problem of time-adaptive RL for continuous-time systems with continuous state and action spaces is of high significance to the community and is well-motived throughout.\n* The paper contributions are novel to the framework proposed by TaCoS and provide theoretical and empirical evidence.\n* The method works surprisingly well, with the surprise that TaCoS achieves better control performance even with a small budget of interactions."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* L154: “Intuitively, the more stochastic the environment, the more interactions we would require to stabilize the system.” the argument should either have a reference or be empirically verified.\n* Minor: Missing related work reference of (Nam et al. 2021).\n\nReferences:\n\n* Nam, HyunJi Alex, Scott Fleming, and Emma Brunskill. \"Reinforcement learning with state observation costs in action-contingent noiselessly observable Markov decision processes.\" Advances in Neural Information Processing Systems 34 (2021): 15650-15666."},"limitations":{"value":"Yes, they are discussed in Section 5."}},"nonreaders":[],"tmdate":1730879769104,"tcdate":1720836925509,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission15313/Reviewer_L1VF"],"signatures":["NeurIPS.cc/2024/Conference/Submission15313/Reviewer_L1VF"],"forum":"0JSKjdePGq","number":3,"license":"CC BY 4.0","cdate":1720836925509,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission15313/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879769104,"domain":"NeurIPS.cc/2024/Conference","replyto":"0JSKjdePGq","id":"tcXarkC7Fo","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["reinforcement learning","continuous-time"]},"supplementary_material":{"value":"/attachment/3d30eb4d166c74afa88948505ef8dccdd8e09d82.zip"},"primary_area":{"value":"reinforcement_learning"},"abstract":{"value":"Reinforcement learning (RL) excels in optimizing policies for discrete-time Markov decision processes (MDP). However, various systems are inherently continuous in time, making discrete-time MDPs an inexact modeling choice. \nIn many applications, such as greenhouse control or medical treatments, each interaction (measurement or switching of action) involves manual intervention and thus is inherently costly. Therefore, \nwe generally prefer a time-adaptive approach with fewer interactions with the system.\nIn this work, we formalize an RL framework, \n**T**ime-**a**daptive **Co**ntrol \\& **S**ensing (**TaCoS**), that tackles this challenge by optimizing over policies that besides control predict the duration of its application. Our formulation results in an extended MDP that any standard RL algorithm can solve.\nWe demonstrate that state-of-the-art RL algorithms trained on TaCoS drastically reduce the interaction amount over their discrete-time counterpart while retaining the same or improved performance, and exhibiting robustness over discretization frequency.\nFinally, we propose OTaCoS, an efficient model-based algorithm for our setting. We show that OTaCoS enjoys sublinear regret for systems with sufficiently smooth dynamics and empirically results in further sample-efficiency gains."},"_bibtex":{"value":"@inproceedings{\ntreven2024when,\ntitle={When to Sense and Control? A Time-adaptive Approach for Continuous-Time {RL}},\nauthor={Lenart Treven and Bhavya Sukhija and Yarden As and Florian Dorfler and Andreas Krause},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=0JSKjdePGq}\n}"},"title":{"value":"When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL"},"pdf":{"value":"/pdf/da0a3d70fa3dca0be69b763308ebef152bfbbc86.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"treven|when_to_sense_and_control_a_timeadaptive_approach_for_continuoustime_rl"},"authorids":{"value":["~Lenart_Treven1","~Bhavya_Sukhija1","~Yarden_As1","~Florian_Dorfler1","~Andreas_Krause1"]},"authors":{"value":["Lenart Treven","Bhavya Sukhija","Yarden As","Florian Dorfler","Andreas Krause"]}},"version":2},{"content":{"comment":{"value":"$\\textbf{Continued Reference}$:\n\n[31] Van Cutsem, Thierry. \"Voltage instability: phenomena, countermeasures, and analysis methods.\" Proceedings of the IEEE 88.2 (2000): 208-227.\n\n[32] Butcher, John Charles. Numerical methods for ordinary differential equations. John Wiley & Sons, 2016.\n\n[33] Hillebrecht, Birgit, and Benjamin Unger. \"Certified machine learning: A posteriori error estimation for physics-informed neural networks.\" 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022.\n\n[34] Hairer, Ernst, and Gerhard Wanner. \"Solving ordinary differential equations. II, Vol. 14 of.\" Springer Series in Computational Mathematics (Springer Berlin Heidelberg, Berlin, Heidelberg, 1996) 10 (1996): 978-3.\n\n[35] Habermann, Christian, and Fabian Kindermann. \"Multidimensional spline interpolation: Theory and applications.\" Computational Economics 30 (2007): 153-169.\n\n[36] Sacchi, Mauricio D., Tadeusz J. Ulrych, and Colin J. Walker. \"Interpolation and extrapolation using a high-resolution discrete Fourier transform.\" IEEE Transactions on Signal Processing 46.1 (1998): 31-38.\n\n[37] Donoho, David L. \"Compressed sensing.\" IEEE Transactions on information theory 52.4 (2006): 1289-1306.\n\n[38] Yi, Ming, et al. \"Bayesian High-Rank Hankel Matrix Completion for Nonlinear Synchrophasor Data Recovery.\" IEEE Transactions on Power Systems 39.1 (2023): 2198-2208.\n\n[39] Fukami, Kai, Koji Fukagata, and Kunihiko Taira. \"Machine-learning-based spatio-temporal super resolution reconstruction of turbulent flows.\" Journal of Fluid Mechanics 909 (2021): A9.\n\n[40] Yin, Yuan, et al. \"Continuous pde dynamics forecasting with implicit neural representations.\" arXiv preprint arXiv:2209.14855 (2022).\n\n[41] Vaswani, A. \"Attention is all you need.\" Advances in Neural Information Processing Systems (2017).\n\n[42] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n\n[43] Mahabadi, Rabeeh Karimi, et al. \"Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks.\" arXiv preprint arXiv:2106.04489 (2021).\n\n[44] Schug, Simon, et al. \"Attention as a Hypernetwork.\" arXiv preprint arXiv:2406.05816 (2024)."}},"tmdate":1732264924240,"tcdate":1732264924240,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11781/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission11781/Authors"],"forum":"CgXAophhEb","number":17,"license":"CC BY 4.0","cdate":1732264924240,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11781/-/Official_Comment"],"mdate":1732264924240,"domain":"ICLR.cc/2025/Conference","replyto":"pPaoRw0tgD","id":"pwS08OduAb","forumContent":{"TLDR":{"value":"This paper proposes a unified geometric perspective to model adaptive physical dynamics and align data qualities with provable guarantees."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["adaptive physical systems","dynamical modeling","data quality alignment","geometric representation learning","geometric optimization"]},"supplementary_material":{"value":"/attachment/e0ee0affe36167b0d0e3157f9f7b88d000a51bab.zip"},"primary_area":{"value":"applications to physical sciences (physics, chemistry, biology, etc.)"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The Neural ODE family has shown promise in modeling complex systems but often assumes consistent data quality, making them less effective in real-world applications with irregularly sampled, incomplete, or multi-resolution data. Current methods, such as ODE-RNN, aim to address these issues but lack formal performance guarantees and can struggle with highly evolving dynamical systems. To tackle this, we propose a novel approach that leverages parameter manifolds to improve robustness in system dynamical modeling. Our method utilizes the orthogonal group as the underlying structure for the parameter manifold, facilitating both quality alignment and dynamical learning in a unified framework. Unlike previous methods, which primarily focus on empirical performance, our approach offers stronger theoretical guarantees of error convergence thanks to the novel architecture and well-posed optimization with orthogonality. Numerical experiments demonstrate significant improvements in interpolation and prediction tasks, particularly in scenarios involving high- and low-resolution data, irregular sampling intervals, etc. Our framework provides a step toward more reliable dynamics learning in changing environments where data quality cannot be assumed."},"_bibtex":{"value":"@misc{\nli2025galignnet,\ntitle={G-AlignNet: Geometry-Driven Quality Alignment for Robust Dynamical Systems Modeling},\nauthor={Haoran Li and CHENHAN XIAO and Muhao Guo and Yang Weng},\nyear={2025},\nurl={https://openreview.net/forum?id=CgXAophhEb}\n}"},"title":{"value":"G-AlignNet: Geometry-Driven Quality Alignment for Robust Dynamical Systems Modeling"},"pdf":{"value":"/pdf/3af322c76c9fcf83036d7bcedd45805e857bd191.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"li|galignnet_geometrydriven_quality_alignment_for_robust_dynamical_systems_modeling"},"authorids":{"value":["~Haoran_Li6","~CHENHAN_XIAO1","~Muhao_Guo1","~Yang_Weng1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Haoran Li","CHENHAN XIAO","Muhao Guo","Yang Weng"]}},"version":2},{"content":{"summary":{"value":"This paper introduces the notion of effective state size (ESS), and demonstrate that it correlates with downstream performance on recall-based tasks better than theoretical state size (TSS).  The effective state size (ESS) is the smallest dimension required to represent the causal sequence operation as a linear recurrence.  They test their method for a variety of training runs, and task complexities."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Line 142: I believe $C \\in \\mathbb{R}^{d \\times n_i}$ (dimensions should be swapped)\n\nHow expensive is it to compute the ESS?\n\n“which are comprised of two sequence mixing and two channel mixing layers) detailed above”. Do you do any ablations to consider effect of greater depth?\n\nWhat is (total ESS) vs ESS Figure 2?\n\nWhy such low correlation for WLA method with KV = 2^7 but high at 2^6?\n\nFigure 4b is a bit confusing to read since ESS and Accuracy are on the same axis.  It would also to help explain the experimental process, as it seems there are multiple configurations and you take the mean accuracy across those."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"* The effective state size metric is intuitive and of conceptual value.\n* The experiments show better correlation with accuracy on recall tasks than theoretical state size\n* The method may be useful to compare the effective memory capabilities of different architectures, in contrast with their theoretical expressivity."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The paper seems to be lacking ablations on network depth, at least in the main body.  Showing how the results change with the number of layers could make the paper stronger.\n* There is little discussion about the computational costs of the method.  The state sizes of modern SSMs like Mamba are very large, thus it would help to have some reassurance that the method is scalable to modern architectures.\n* The authors briefly mention hybrid architectures in the main body.  However, since Hybrid architectures are becoming more prominent, some further discussion might be warranted.  Since Hybrid architectures have attention layers and skip connections, we might expect that they have full recall capabilities within their context window, which could permit the SSM layers memorize less in their state."}},"nonreaders":[],"tmdate":1732226965410,"tcdate":1731113190289,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9386/Reviewer_3frp"],"signatures":["ICLR.cc/2025/Conference/Submission9386/Reviewer_3frp"],"forum":"DHVjLvSps6","number":3,"license":"CC BY 4.0","cdate":1731113190289,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9386/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732226965410,"domain":"ICLR.cc/2025/Conference","replyto":"DHVjLvSps6","id":"YS1BaivF14","forumContent":{"TLDR":{"value":"We propose the Effective State-Size (ESS) metric to quantitatively analyze how sequence models utilize memory and context, offering insights into model in-context recall, initialization strategies, efficiency, and architecture design."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["model analysis","interpretability","linear systems","attention","state-space models","sequence models","memory utilization","context utilization"]},"primary_area":{"value":"interpretability and explainable AI"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"As the space of causal sequence modeling architectures continues to grow, the need to develop a general framework for their analysis becomes increasingly important. With this aim, we draw insights from classical signal processing and control theory, to develop a quantitative measure of *memory utilization*: the internal mechanisms through which a model stores past information to produce future outputs. This metric, which we call **effective state-size** (ESS), is tailored to the fundamental class of systems with *input-invariant* and *input-varying linear operators*, encompassing a variety of computational units such as variants of attention, convolutions, and recurrences. Unlike prior work on memory utilization, which either relies on raw operator visualizations (e.g. attention maps), or simply the total *memory capacity* (i.e. cache size) of a model, our metrics provide highly interpretable and actionable measurements. In particular, we show how ESS can be leveraged to improve initialization strategies, inform novel regularizers and advance the performance-efficiency frontier through model distillation. Furthermore, we demonstrate that the effect of context delimiters (such as end-of-speech tokens) on ESS highlights cross-architectural differences in how large language models utilize their available memory to recall information. Overall, we find that ESS provides valuable insights into the dynamics that dictate memory utilization, enabling the design of more efficient and effective sequence models."},"_bibtex":{"value":"@misc{\nparnichkun2025quantifying,\ntitle={Quantifying Memory Utilization with Effective State-Size},\nauthor={Rom Parnichkun and Neehal Tumma and Armin W Thomas and Alessandro Moro and Qi An and Taiji Suzuki and Atsushi Yamashita and Michael Poli and Stefano Massaroli},\nyear={2025},\nurl={https://openreview.net/forum?id=DHVjLvSps6}\n}"},"title":{"value":"Quantifying Memory Utilization with Effective State-Size"},"pdf":{"value":"/pdf/9de503dde3c70436d5a15c5372fb7d21e789bc5a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"parnichkun|quantifying_memory_utilization_with_effective_statesize"},"authorids":{"value":["~Rom_Parnichkun1","~Neehal_Tumma1","~Armin_W_Thomas1","~Alessandro_Moro1","~Qi_An6","~Taiji_Suzuki1","~Atsushi_Yamashita1","~Michael_Poli1","~Stefano_Massaroli1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Rom Parnichkun","Neehal Tumma","Armin W Thomas","Alessandro Moro","Qi An","Taiji Suzuki","Atsushi Yamashita","Michael Poli","Stefano Massaroli"]}},"version":2},{"content":{"venue":{"value":"ICPR (12) 2026"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-032-31920-3_9.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"li|lsmamba_gated_spatiotemporal_modulation_and_bidirectional_state_space_modeling_for_eeg_emotion_recognition"},"html":{"value":"https://doi.org/10.1007/978-3-032-31920-3_9"},"_bibtex":{"value":"@inproceedings{DBLP:conf/icpr/LiGHX26,\n  author={Zitao Li and Jiayue Gu and Ziyi Huang and Ze Xiong},\n  title={LS-Mamba: Gated Spatio-Temporal Modulation and Bidirectional State Space Modeling for EEG Emotion Recognition},\n  year={2026},\n  cdate={1767225600000},\n  pages={126-139},\n  url={https://doi.org/10.1007/978-3-032-31920-3_9},\n  booktitle={ICPR (12)},\n  crossref={conf/icpr/2026-12}\n}\n"},"abstract":{"value":"Decoding fine-grained emotional states from electroencephalography (EEG) signals presents a significant challenge due to their inherent non-stationarity and low signal-to-noise ratio. While deep learning has advanced this field, designing an architecture that effectively balances local feature extraction with long-range temporal modeling remains a difficult trade-off. Current Convolutional Neural Networks (CNNs) are computationally efficient but often constrained by limited receptive fields, whereas Transformer-based models capture global context but typically incur high parameter costs and computational overhead. To address this challenge, we propose LS-Mamba, a lightweight hybrid framework. The architecture incorporates a Large-Small Gating (LSG) module to dynamically modulate features, aiming to separate local variations from global trends via dual-branch convolutions. Subsequently, a bidirectional Mamba encoder is employed to model the continuous temporal evolution of emotional states, capturing long-range dependencies with linear computational complexity. The proposed framework achieves efficient modeling with a compact parameter footprint of only 0.06M. Experiments on the SEED and SEED-IV benchmarks demonstrate that LS-Mamba achieves competitive accuracies of 89.96% and 76.48%, respectively. Notably, the model outperforms representative baselines on the complex four-class task and maintains consistent performance stability across different subjects. The source code is publicly available at https://github.com/xtt-lzt/LS-Mamba."},"title":{"value":"LS-Mamba: Gated Spatio-Temporal Modulation and Bidirectional State Space Modeling for EEG Emotion Recognition"},"authors":{"value":[{"fullname":"Zitao Li","username":"~Zitao_Li1"},{"fullname":"Jiayue Gu","username":""},{"fullname":"Ziyi Huang","username":""},{"fullname":"Ze Xiong","username":""}]}},"tmdate":1788049515655,"pdate":1798675200000,"externalIds":["dblp:conf/icpr/LiGHX26"],"tcdate":1788049507914,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Zitao_Li1"],"forum":"onJeqjfyEu","license":"CC BY-SA 4.0","number":144846,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1788049515655,"domain":"OpenReview.net/Public_Article","id":"onJeqjfyEu","version":2},{"content":{"venue":{"value":"ICML 2025 spotlightposter"},"keywords":{"value":["test-time training"]},"primary_area":{"value":"deep_learning->foundation_models"},"abstract":{"value":"Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden states. We present a practical framework for instantiating sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Similar to Transformer, TTT-Linear and TTT-MLP can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research."},"_bibtex":{"value":"@inproceedings{\nsun2025learning,\ntitle={Learning to (Learn at Test Time): {RNN}s with Expressive Hidden States},\nauthor={Yu Sun and Xinhao Li and Karan Dalal and Jiarui Xu and Arjun Vikram and Genghan Zhang and Yann Dubois and Xinlei Chen and Xiaolong Wang and Sanmi Koyejo and Tatsunori Hashimoto and Carlos Guestrin},\nbooktitle={Forty-second International Conference on Machine Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=wXfuOj9C7L}\n}"},"title":{"value":"Learning to (Learn at Test Time): RNNs with Expressive Hidden States"},"link_to_code":{"value":"https://github.com/test-time-training/ttt-lm-jax"},"pdf":{"value":"/pdf/d1e0fbc0e618156674fa947fc622a2ce3a1a8387.pdf"},"lay_summary":{"value":"Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden states. We present a practical framework for instantiating sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Similar to Transformer, TTT-Linear and TTT-MLP can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research."},"venueid":{"value":"ICML.cc/2025/Conference"},"paperhash":{"value":"sun|learning_to_learn_at_test_time_rnns_with_expressive_hidden_states"},"authorids":{"value":["~Yu_Sun1","~Xinhao_Li3","~Karan_Dalal1","~Jiarui_Xu1","~Arjun_Vikram1","~Genghan_Zhang1","~Yann_Dubois1","~Xinlei_Chen1","~Xiaolong_Wang3","~Sanmi_Koyejo1","~Tatsunori_Hashimoto1","~Carlos_Guestrin1"]},"authors":{"value":["Yu Sun","Xinhao Li","Karan Dalal","Jiarui Xu","Arjun Vikram","Genghan Zhang","Yann Dubois","Xinlei Chen","Xiaolong Wang","Sanmi Koyejo","Tatsunori Hashimoto","Carlos Guestrin"]}},"tmdate":1753293003052,"pdate":1746105465283,"tcdate":1737611623291,"writers":["ICML.cc/2025/Conference","ICML.cc/2025/Conference/Submission9249/Authors"],"signatures":["ICML.cc/2025/Conference/Submission9249/Authors"],"forum":"wXfuOj9C7L","license":"CC BY 4.0","number":9249,"cdate":1737611623291,"readers":["everyone"],"invitations":["ICML.cc/2025/Conference/-/Submission","ICML.cc/2025/Conference/-/Post_Submission","ICML.cc/2025/Conference/Submission9249/-/Full_Submission","ICML.cc/2025/Conference/-/Edit","ICML.cc/2025/Conference/Submission9249/-/Camera_Ready_Revision"],"mdate":1753293003052,"odate":1750231289675,"domain":"ICML.cc/2025/Conference","id":"wXfuOj9C7L","version":2},{"content":{"summary":{"value":"The paper develops a pre-training technique for Mamba family of models and applies this to solve time-series forecasting task. Empirical results show lift resulting from the proposed pretraining techniques when applied to Mamba models."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"- Can you include the recent baselines from https://arxiv.org/pdf/2405.14616 and https://arxiv.org/pdf/2403.09898 for comparison?\n- Can you provide empirical evidence that freezing pretrained matrix A indeed improves Mamba selectivity? Please define selectivity and measure the improvement. Are there qualitative results demonstrating improved selectivity?\n- Can you provide ablation of Intra-sequence contrast and Repetitive Contrastive Learning. What happens when only one of them is used? Which one is more powerful? Do they exhibit synergistic effects (more than additive gain)\n- Can you provide more details on the experimental setup? For example, how exactly are the metrics in Table 1 computed? Why is the presentation format different than in the other works relying on the same datasets?\n- Please extend related work on time series contrastive pretraining works and explain how your work is different and novel. \n- Can you include existing time-series pretraining techniques I identified as baselines?\n- In related work, the following MLP-based architectures are missing, please include in the discussion and as baselines: https://arxiv.org/pdf/1905.10437, https://arxiv.org/pdf/2201.12886\n- The proposed contrastive learning techniques, are they applicable only to Mamba model, or they are more general? If they are general, why are not you exploring their effects on other model families? Is it viable to obtain evidence of accuracy lifts on other model families?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- Improved results compared to mamba models without pretraining"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Experiment tables lack comparison to recent baselines and reporting format is different than the accepted in the literature. It is not clear where the proposed techniques stand against current state of the art.\n- Comparison against univariate baselines is missing. Does you approach beat the univariate baselines on these datasets?\n- The title claims that the pretraining improves Mamba's selectivity. The word selectivity appears in two places in the paper other than the title: \"we freeze the matrix A that controls the mamba’s selectivity\" and \"Freezing matrix A can retain the enhanced selectivity obtained during pre-training.\" While A controls mamba’s selectivity and authors hypothesize that pretraining results in a matrix A that improves the selectivity, there is no empirical result in the paper that would help to support the hypothesis. The claim is basically unsupported.\n- There are now ablation studies of the two proposed pretraining techniques: Intra-sequence contrast and Repetitive Contrastive Learning.\n- Experiments are not clearly described\n- The idea of contrastive pretraining does not seem to be entirely novel, even in the time-series literature. The related work section is misleading as it puts a lot of emphasis on contrastive learning in image/textual domains. However, the following works exist in the time-series domain: https://arxiv.org/pdf/2303.11911, https://arxiv.org/pdf/2312.16424, https://arxiv.org/pdf/2206.08496."}},"nonreaders":[],"tmdate":1731427908292,"tcdate":1730928054087,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9847/Reviewer_Fxmp"],"signatures":["ICLR.cc/2025/Conference/Submission9847/Reviewer_Fxmp"],"forum":"tIURLNBTPx","number":4,"license":"CC BY 4.0","cdate":1730928054087,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9847/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427908292,"domain":"ICLR.cc/2025/Conference","replyto":"tIURLNBTPx","id":"V9wqdjzUhp","forumContent":{"TLDR":{"value":"We propose a Repetitive Contrastive Learning method to enhance the sequence selection capability of Mamba SSM, which can bring improvements to multiple Mamba-based models in time series prediction problems."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Mamba; Self-supervised learning; Time Series Prediction"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The prediction of long sequences has always been a challenge in time series forecasting tasks. Due to Mamba's sequence selection capability, many Mamba-based models have been proposed, achieving state-of-the-art results in long sequence prediction problems. However, much research has focused on integrating mamba-ssm into specific model structures for better performance, while the core of mamba-ssm, its sequence selection capability, has not been deeply explored. We believe there is significant potential in Mamba's sequence selection capability and propose a Repetitive Contrastive Learning (RCL) method to enhance it. Specifically, we utilize Repeating Sequence Augmentation to expand the sequence while introducing Gaussian noise, thereby enhancing the Mamba block's sequence selection capability through both inter-sequence and intra-sequence contrastive techniques. Then our methods transfer parameters directly from a single pretrained Mamba block to a variety of Mamba-based models. This approach provides superior initialization for forecasting tasks. Our experiments consistently demonstrate that this technique improves the forecasting performance of many Mamba-based models, without imposing additional memory requirements."},"_bibtex":{"value":"@misc{\nyan2025repetitive,\ntitle={Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction},\nauthor={WenBo Yan and Hanzhong Cao and Ying Tan},\nyear={2025},\nurl={https://openreview.net/forum?id=tIURLNBTPx}\n}"},"title":{"value":"Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction"},"pdf":{"value":"/pdf/25ba192abf3d52ee17ddd05bad92e932a9f5e0c0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"yan|repetitive_contrastive_learning_enhances_mambas_selectivity_in_time_series_prediction"},"authorids":{"value":["~WenBo_Yan1","~Hanzhong_Cao1","~Ying_Tan1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["WenBo Yan","Hanzhong Cao","Ying Tan"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a new algorithm for time-series imputation and forecasting via using implicit neural representations. The proposed method particularly leverages an idea of latent modulation, extending a previous approach by making the latent vectors evolve over time. During the inference, for both imputation and forecasting, the method assumes there exist a few samples and fine-tunes the INR via auto-decoding. The paper tests the proposed methods on well-known time-series benchmark datasets and compares the result with several time-series modeling methods that can be considered as the current state-of-the-art."},"presentation":{"value":"3 good"},"contribution":{"value":"3 good"},"soundness":{"value":"3 good"},"strengths":{"value":"- The paper is written clearly, elaborating the architecture design, and training/test algorithms. \n\n- Time-series modeling has not been investigated much in the INR literature and this paper provides some insights that modeling time-series data in a continuous neural function can be beneficial.\n\n- The paper compares the proposed method with several important baselines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Although the domain of the application (i.e., time-series modeling) is new and the proposed design brings an idea of the latent state evolution (in forecasting), the novelty seems to be limited. The overall architectural design follows the FFN architecture (Tancik, et al, 2020) without any consideration on how to handle multivariate time-series. Also, the idea of the latent modulation and the meta-learning-based training algorithm largely follow the previous approach (Dupont, et al, 2022). Finally, a similar idea of employing temporally-refined latent variables has been explored in (Yin, et al, 2023). \n\n- Regarding auto-decoding process:\n  \n  - For imputation, it is natural to assume that there is a given set of measurements for a new time-series, and to set up the goal to fill-in unseen data via imputation.  For forecasting, however, the assumption of having a separate training period and a look-back window raises some concerns. Having a separate look-back window suggests that the method needs to wait until the new observations are collected to make forecasting. Some of the datasets that are considered in the paper have hourly sampling rate and this time gap might provide enough time to fine-tune other baseline models (with many model parameters, e.g., Transformers). If the ultimate goal is to achieve accurate prediction, with the given time period (an hour), fine-tuning those baselines with a new observation may provide better prediction results.  \n\n  - Similarly, another concern is fairness on the comparisons. Although it is just 3 gradient steps, auto-decoding is considered as solving an optimization problem to fine-tune the model for the new observations. What happens if the small portion of the other baselines (e.g., the last layer) is fine-tuned during the inference? For example, in forecasting, the model can be fine-tuned after making predictions on the current sliding window and then make predictions on the next sliding window with the updated models. \n \n  - Although the method seems to provide accurate predictions both in imputation and forecasting, the method seems to struggle in predicting peaks accurately. In many applications, predicting peaks accurately would have more importance than simply minimizing MSEs (e.g., to properly prepare the electricity supply or properly set up the cost during the peak time period). Based on the eye-ball examination (Figure 5 for example), the model does not seem to provide accurate predictions in peak values."},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"- As mentioned in the weakness section, could the author provide more justifications for performing auto-decoding while other baselines are used only for inference? Also, could the authors mention more about the fairness of the comparisons? \n\n- Appendix section D, Tables 13 and 14 emphasize the performance degradation in Transformers models as the testing window is far apart from the training period. Would there be realistic cases where we have only the old history for training, measurements are stopped for a while, and collecting results regularly again afterwards?\n\n- In Appendix, the paper provides experimental results for varying dimensionalities on the latent vectors and the number of gradient steps in auto-decoding during the inference. However, these experiments are performed in a limited experiment setting. Could the authors provide more insight on the effect of these hyper-parameters? For example, Table 6 provides the results with a specific dataset (Electricity) with a horizon length 720 and a look-back window length 512 and essentially says there will be no improvement after 10 or 50 gradient steps. Would this observation be valid for other datasets, for other sizes of windows? Also, what happens if the dimensionality of the latent vector is changed? Are these results also obtained by the multiple number of runs? \n\n- Could the authors also provide justifications on not to include other time-continuous models for their baselines? Such as neural ODEs and their variants for irregular time-series modeling?"},"rating":{"value":"5: marginally below the acceptance threshold"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636864245,"tcdate":1698769940578,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission7252/Reviewer_QdLC"],"signatures":["ICLR.cc/2024/Conference/Submission7252/Reviewer_QdLC"],"forum":"w8JizpeY4y","number":2,"license":"CC BY 4.0","cdate":1698769940578,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission7252/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636864245,"domain":"ICLR.cc/2024/Conference","replyto":"w8JizpeY4y","id":"dVJbGR2g2t","forumContent":{"TLDR":{"value":"We propose a deep continuous model for time series imputation and forecasting which can handle irregular sampled time series and new samples."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time series","Continuous modeling","Forecasting","Imputation","Implicit Neural Representation"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"We introduce a novel modeling approach for time series imputation and forecasting, tailored to address the challenges often encountered in real-world data, such as irregular samples, missing data, or unaligned measurements from multiple sensors. Our method relies on a continuous-time-dependent model of the series' evolution dynamics. It leverages adaptations of conditional, implicit neural representations for sequential data. A modulation mechanism, driven by a meta-learning algorithm, allows adaptation to unseen samples and extrapolation beyond observed time-windows for long-term predictions. The model provides a highly flexible and unified framework for imputation and forecasting tasks across a wide range of challenging scenarios. It achieves state-of-the-art performance on classical benchmarks and outperforms alternative time-continuous models."},"_bibtex":{"value":"@misc{\nnaour2024time,\ntitle={Time Series Continuous Modeling for Imputation and Forecasting with Implicit Neural Representations},\nauthor={Etienne Le Naour and Louis Serrano and L{\\'e}on Migus and Yuan Yin and Ghislain Agoua and Nicolas Baskiotis and patrick gallinari and Vincent Guigue},\nyear={2024},\nurl={https://openreview.net/forum?id=w8JizpeY4y}\n}"},"title":{"value":"Time Series Continuous Modeling for Imputation and Forecasting with Implicit Neural Representations"},"pdf":{"value":"/pdf/8e1fe1b1fffa3eb03aba32b5b707f562779ce308.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"naour|time_series_continuous_modeling_for_imputation_and_forecasting_with_implicit_neural_representations"},"authorids":{"value":["~Etienne_Le_Naour1","~Louis_Serrano1","~Léon_Migus1","~Yuan_Yin1","~Ghislain_Agoua1","~Nicolas_Baskiotis2","~patrick_gallinari1","~Vincent_Guigue1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Etienne Le Naour","Louis Serrano","Léon Migus","Yuan Yin","Ghislain Agoua","Nicolas Baskiotis","patrick gallinari","Vincent Guigue"]}},"version":2},{"content":{"summary":{"value":"This work focuses on improving the performance of Mamba architecture in image super-resolution task. It proposes to enhance long-range modeling capability using Hierarchical Mabma Block. Meanwhile, mamba blocks with various scanning direction are employed to further enrich the spatial information. Three models with different size are introduced, with extensive experimental results to verify its superiority compared to previous Mamba- and Transformer-based methods under comparable or less computational costs."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. Have you experimented replacing the Avgpool in Region SSM with other downsampling methods, e.g., strided conv.\n2. In what aspects do you think the mamba-based methods perform better compared to the Transformer-based methods. As far as I know, several recent Transformer-based HAT, ATD, IPG have achieved much outstanding results, could you please make a comparison with them.\n3. Have you compared the LAM visualization of Hi-Mamba with GRL-B and SRFormer, since they have an enlarged window-size compared to SwinIR."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. Various techniques are proposed to improve the modeling ability of Mamba-based SR model, with comprehensive ablation studies validating the effectiveness.\n2. The Hi-Mamba-L strikes a better balance between performance and computational cost compared to the previous mamba-based method MambaIR."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The results of SRFormer are presented in Table 3, however, its model complexity and GPU Latency results are not included in Table 4 and Figure 5. \n2. Most citations that be in parenthesis are not in parenthesis, which impacts the readibility. \n3. The novelty seems limited."}},"nonreaders":[],"tmdate":1731427887519,"tcdate":1730622379520,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3583/Reviewer_Nx39"],"signatures":["ICLR.cc/2025/Conference/Submission3583/Reviewer_Nx39"],"forum":"iPYwddLhXR","number":4,"license":"CC BY 4.0","cdate":1730622379520,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3583/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427887519,"domain":"ICLR.cc/2025/Conference","replyto":"iPYwddLhXR","id":"qtTV1dNAui","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Image Super-Resolution","Mamba","Structured State Space Models"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSM), such as Mamba, have shown strong representation ability in modeling long-range dependency with linear complexity, achieving successful applications from high-level to low-level vision tasks. However, SSM's sequential nature necessitates multiple scans in different directions to compensate for the loss of spatial dependency when unfolding the image into a 1D sequence. This multi-direction scanning strategy significantly increases the computation overhead and is unbearable for high-resolution image processing. To address this problem, we propose a novel Hierarchical Mamba network, namely, Hi-Mamba, for image super-resolution (SR).  Hi-Mamba consists of two key designs: (1) The Hierarchical Mamba Block (HMB) assembled by a Local SSM (L-SSM) and a Region SSM (R-SSM) both with the single-direction scanning, aggregates multi-scale representations to enhance the context modeling ability. (2) The Direction Alternation  Hierarchical Mamba Group (DA-HMG) allocates the isomeric single-direction scanning into cascading HMBs to enrich the spatial relationship modeling.\nExtensive experiments demonstrate the superiority of Hi-Mamba across five benchmark datasets for efficient SR. For example, Hi-Mamba achieves a significant PSNR improvement of 0.29 dB on Manga109 for $\\times3$ SR, compared to the strong lightweight MambaIR."},"_bibtex":{"value":"@misc{\nqiao2024himamba,\ntitle={Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution},\nauthor={Junbo Qiao and Jincheng Liao and Wei Li and Yulun Zhang and Yong Guo and Yi Wen and Zhangxizi Qiu and Jiao Xie and Jie Hu and Shaohui Lin},\nyear={2024},\nurl={https://openreview.net/forum?id=iPYwddLhXR}\n}"},"title":{"value":"Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution"},"pdf":{"value":"/pdf/e75e168c76e93b40ac8e29dd9015c5e7d72edc8b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"qiao|himamba_hierarchical_mamba_for_efficient_image_superresolution"},"authorids":{"value":["~Junbo_Qiao1","~Jincheng_Liao1","~Wei_Li60","~Yulun_Zhang1","~Yong_Guo1","~Yi_Wen3","~Zhangxizi_Qiu2","~Jiao_Xie1","~Jie_Hu8","~Shaohui_Lin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Junbo Qiao","Jincheng Liao","Wei Li","Yulun Zhang","Yong Guo","Yi Wen","Zhangxizi Qiu","Jiao Xie","Jie Hu","Shaohui Lin"]}},"version":2},{"content":{"summary":{"value":"This paper introduces the first Mamba-based method for dynamic NLOS imaging, featuring three key modules: the spatial-temporal Mamba, the cross ST-Mamba, and the phasor field loss. The spatial-temporal Mamba extracts and integrates both causal and non-causal transient components, while the cross ST-Mamba combines deep features from adjacent frames. The phasor field loss ensures the consistency of transients in the phasor field. Evaluations on synthetic and self-captured real-world transients demonstrate that the proposed method outperforms existing approaches. Ablation studies further validate the effectiveness of the proposed components."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please refer to Weakness."},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"(1)\tThe proposed method of leveraging adjacent frame information and cross Mamba to enhance single-frame reconstruction is outstanding. As hidden objects move, fixed few-shot scanning points can capture information that other frames miss, which is crucial for recovering lost signals.\n(2)\tThis paper introduces deep learning into dynamic NLOS imaging for the first time. Its capability for single-pass rapid inference helps advance dynamic NLOS imaging towards real-time imaging.\n(3)\tThe captured real-world dataset will be a valuable resource for future research in dynamic NLOS imaging."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"(1)\tUsing cross-mamba to extract inter-frame causality in transients is understandable. However, the values in single-frame transients  are based on the distance from the hidden object to the wall. This seems to be the spatial correlation rather than time-resolving causality. Please provide further explanation to justify the motivation for using ST-Mamba to process single-frame transients.\n(2)\tLines 180-184 are unclear. Restoring deep features of transients to the original resolution seems to be a common practice. Can transient recovered be done directly at the original resolution? The temporal information in low spatial-resolution transients might be compromised by down-sampling and up-sampling operations.\n(3)\tThe framework is similar to USM, consisting of a transient super-resolution network and a volume reconstruction network. The proposed components are mainly applied to the transient super-resolution network. Please compare with USM's recovered high spatial-resolution transients to highlight the proposed method’s contributions."},"limitations":{"value":"Please refer to Weakness."}},"nonreaders":[],"tmdate":1730878940027,"tcdate":1720838528693,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission4624/Reviewer_CVND"],"signatures":["NeurIPS.cc/2024/Conference/Submission4624/Reviewer_CVND"],"forum":"QiCJomIW3l","number":3,"license":"CC BY 4.0","cdate":1720838528693,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission4624/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730878940027,"domain":"NeurIPS.cc/2024/Conference","replyto":"QiCJomIW3l","id":"XNdOeW1kYe","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"TLDR":{"value":"We propose a novel  spatial-temporal Mamba based approach for dynamic NLOS reconstruction, as well as  a new dataset for training and testing."},"keywords":{"value":["dynamic","non-line-of-sight imaging","spatial-temporal Mamba"]},"supplementary_material":{"value":"/attachment/af9bfb4f504be74ca2c3a0f0a41c613d33f41b6f.zip"},"primary_area":{"value":"machine_learning_for_physical_sciences"},"abstract":{"value":"Dynamic reconstruction in confocal non-line-of-sight imaging encounters great challenges since the dense raster-scanning manner limits the practical frame rate. A fewer pioneer works reconstruct high-resolution volumes from the under-scanning transient measurements but overlook temporal consistency among transient frames. To fully exploit multi-frame information, we propose the first spatial-temporal Mamba (ST-Mamba) based method tailored for dynamic reconstruction of transient videos. Our method capitalizes on neighbouring transient frames to aggregate the target 3D hidden volume. Specifically, the interleaved features extracted from the input transient frames are fed to the proposed ST-Mamba blocks, which leverage the time-resolving causality in transient measurement. The cross ST-Mamba blocks are then devised to integrate the adjacent transient features. The target high-resolution transient frame is subsequently recovered by the transient spreading module. After transient fusion and recovery, a physical-based network is employed to reconstruct the hidden volume. To tackle the substantial noise inherent in transient videos, we propose a wave-based loss function to impose constraints within the phasor field. Besides, we introduce a new dataset, comprising synthetic videos for training and real-world videos for evaluation. Extensive experiments showcase the superior performance of our method on both synthetic data and real world data captured by different imaging setups. The code and data are available at https://github.com/Depth2World/Dynamic_NLOS."},"_bibtex":{"value":"@inproceedings{\nli2024toward,\ntitle={Toward Dynamic Non-Line-of-Sight Imaging with Mamba Enforced Temporal Consistency},\nauthor={Yue Li and Yi Sun and Shida Sun and Juntian Ye and Yueyi Zhang and Feihu Xu and Zhiwei Xiong},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=QiCJomIW3l}\n}"},"title":{"value":"Toward Dynamic Non-Line-of-Sight Imaging with Mamba Enforced Temporal Consistency"},"pdf":{"value":"/pdf/8ceb8c4575a29d6db10203bc67a5f763ebcf0ef7.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"li|toward_dynamic_nonlineofsight_imaging_with_mamba_enforced_temporal_consistency"},"authorids":{"value":["~Yue_Li11","~Yi_Sun20","~Shida_Sun1","~Juntian_Ye1","~Yueyi_Zhang2","~Feihu_Xu1","~Zhiwei_Xiong1"]},"authors":{"value":["Yue Li","Yi Sun","Shida Sun","Juntian Ye","Yueyi Zhang","Feihu Xu","Zhiwei Xiong"]}},"version":2},{"content":{"comment":{"value":"> The experimental evaluation, while broad, has several limitations. The benefits over baselines are modest in most tasks, with the exception of the selective copying task, which seems carefully chosen to showcase the method's strengths. \n\nWe would like to clarify that the primary goal of our work is not to achieve the best performance on individual tasks, but to demonstrate that simplifying traditional, decades-old RNN architectures can yield results comparable to modern, state-of-the-art sequence models.\n\nWe selected the Selective Copying task because it was used in the Mamba paper (Gu & Dao, 2023) as one of the primary benchmarks to highlight how much their state-space model, S6, and the Mamba module improved over prior state-of-the-art models such as S4 and Hyena. Mamba is a widely recognized paper that builds on earlier work in state-space models and has garnered over 1300 citations in less than a year.\n\nIn our work, we show that simplified versions of decades-old RNNs can achieve comparable performance to Mamba on this challenging task. This result is both surprising and valuable, as it challenges the prevailing assumption in the sequence modelling community that increasing architectural and algorithmic complexity is necessary to achieve state-of-the-art performance."},"title":{"value":"Response to Reviewer jHLY (2/4)"}},"tmdate":1732169664448,"tcdate":1732169664448,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8241/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission8241/Authors"],"forum":"GrmFFxGnOR","number":8,"license":"CC BY 4.0","cdate":1732169664448,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8241/-/Official_Comment"],"mdate":1732169664448,"domain":"ICLR.cc/2025/Conference","replyto":"co3S9pN8U7","id":"gAVpJOXs3p","forumContent":{"TLDR":{"value":"Revisiting decade-old RNNs (LSTMs and GRUs), we introduce minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters, (2) are parallelizable during training, and (3) match the performance of recent sequence models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Recurrent Neural Networks","RNNs","Sequence Modelling","Efficiency","LSTMs","GRUs","Parallel Scan"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers—particularly with respect to sequence length—have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively.\nIn this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers."},"_bibtex":{"value":"@misc{\nfeng2025were,\ntitle={Were {RNN}s All We Needed?},\nauthor={Leo Feng and Frederick Tung and Mohamed Osama Ahmed and Yoshua Bengio and Hossein Hajimirsadeghi},\nyear={2025},\nurl={https://openreview.net/forum?id=GrmFFxGnOR}\n}"},"title":{"value":"Were RNNs All We Needed?"},"pdf":{"value":"/pdf/a56137c0e88e5c5b4003b438b48e61ee149bd102.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"feng|were_rnns_all_we_needed"},"authorids":{"value":["~Leo_Feng1","~Frederick_Tung1","~Mohamed_Osama_Ahmed2","~Yoshua_Bengio1","~Hossein_Hajimirsadeghi1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Leo Feng","Frederick Tung","Mohamed Osama Ahmed","Yoshua Bengio","Hossein Hajimirsadeghi"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2506.13183v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"liu|mtpcr_a_hybrid_mambatransformer_with_spatial_serialization_for_hierarchical_point_cloud_registration"},"authorids":{"value":["~Bingxi_Liu1","https://dblp.org/search/pid/api?q=author:An_Liu:","https://dblp.org/search/pid/api?q=author:Hao_Chen:","https://dblp.org/search/pid/api?q=author:Jinqiang_Cui:","https://dblp.org/search/pid/api?q=author:Yiqun_Wang:","https://dblp.org/search/pid/api?q=author:Hong_Zhang_0013:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2506.13183"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2506-13183,\n  publtype={informal},\n  author={Bingxi Liu and An Liu and Hao Chen and Jinqiang Cui and Yiqun Wang and Hong Zhang},\n  title={MT-PCR: A Hybrid Mamba-Transformer with Spatial Serialization for Hierarchical Point Cloud Registration},\n  year={2025},\n  month={June},\n  cdate={1748736000000},\n  journal={CoRR},\n  volume={abs/2506.13183},\n  url={https://doi.org/10.48550/arXiv.2506.13183}\n}\n"},"abstract":{"value":"Point cloud registration (PCR) is a fundamental task in 3D computer vision and robotics. Most existing learning-based PCR methods rely on Transformers, which suffer from quadratic computational complexity. This limitation restricts the resolution of point clouds that can be processed, inevitably leading to information loss. In contrast, Mamba-a recently proposed model based on state space models (SSMs)-achieves linear computational complexity while maintaining strong long-range contextual modeling capabilities. However, directly applying Mamba to PCR tasks yields suboptimal performance due to the unordered and irregular nature of point cloud data. To address this challenge, we propose MT-PCR, the first point cloud registration framework that integrates both Mamba and Transformer modules. Specifically, we serialize point cloud features using Z-order space-filling curves to enforce spatial locality, enabling Mamba to better model the geometric structure of the input. Additionally, we remove the order indicator module commonly used in Mamba-based sequence modeling, leads to improved performance in our setting. The serialized features are then processed by an optimized Mamba encoder, followed by a Transformer refinement stage. Extensive experiments on multiple benchmarks demonstrate that MT-PCR outperforms Transformer-based and concurrent state-of-the-art methods in both accuracy and efficiency, significantly reducing while GPU memory usage and FLOPs."},"title":{"value":"MT-PCR: A Hybrid Mamba-Transformer with Spatial Serialization for Hierarchical Point Cloud Registration"},"authors":{"value":["Bingxi Liu","An Liu","Hao Chen","Jinqiang Cui","Yiqun Wang","Hong Zhang"]}},"tmdate":1769151458469,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2506-13183"],"tcdate":1769151455854,"writers":["~"],"signatures":["~Bingxi_Liu1"],"forum":"ocsrYNUtf4","license":"CC BY-SA 4.0","number":790862,"cdate":1748736000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769151458469,"domain":"DBLP.org","id":"ocsrYNUtf4","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2410.23356v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"lee|sequential_orderrobust_mamba_for_time_series_forecasting"},"html":{"value":"https://doi.org/10.48550/arXiv.2410.23356"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-23356,\n  publtype={informal},\n  author={Seunghan Lee and Juri Hong and Kibok Lee and Taeyoung Park},\n  title={Sequential Order-Robust Mamba for Time Series Forecasting},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2410.23356},\n  url={https://doi.org/10.48550/arXiv.2410.23356}\n}\n"},"abstract":{"value":"Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data. However, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD. Extensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios. Code is available at https://github.com/seunghan96/SOR-Mamba."},"title":{"value":"Sequential Order-Robust Mamba for Time Series Forecasting"},"authors":{"value":[{"fullname":"Seunghan Lee","username":"~Seunghan_Lee1"},{"fullname":"Juri Hong","username":""},{"fullname":"Kibok Lee","username":""},{"fullname":"Taeyoung Park","username":""}]}},"tmdate":1783946655488,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2410-23356"],"tcdate":1783946643902,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Seunghan_Lee1"],"forum":"tuBi5lWbBu","license":"CC BY-SA 4.0","number":52651,"cdate":1704067200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1783946655488,"domain":"OpenReview.net/Public_Article","id":"tuBi5lWbBu","version":2},{"content":{"summary":{"value":"This paper proposes a Transformer Masked Language Model (MLM) variant of concept-bottleneck generative models (CBGM) with three goals in mind: controllable generation, interpretability, and debuggability. This paper adapts the training objectives of CBGM to the masked language model architecture, and proposes a control method specifically for their CB MLM. Evaluation was conducted on the domain of protein sequence modeling, for controllability, interpretability and debuggability. The proposed model leads to improvements over non concept-bottlenecked baselines on controllability. The proposed model demonstrates interpretability and debuggability via inspection of the weights of the linear decoder head. \n\nComment: I have experience with language modeling but not with protein domain, so my review will focus more on the method part of the paper (as recommended by AC). Please take this into account when considering my review and recommendation."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"Q1 How does compute / training vary across C-pLM, CC-pLM, and CB-pLM. Is total compute/wallclock time controlled (e.g. 24 hr, as the cited cramming paper) or perhaps early stopping on validation is used?\n\nQ2 It is interesting that there is negative scaling happening in figures 10 and 11, with sometimes large decreases in performance as the model becomes larger. Out of curiosity, do authors have any speculations about why this happens?\n\nQ3 In lines 142-144, “This enables us to answer [… q1…], or [q2] counterfactual questions like if we want to decrease hydrophobicity, which amino acid should we replace “E” with?”. I understand q1 is an interpretability question and linear decoder does help with that. For q2 however, I am not sure I fully understood. Is it accurate to rephrase it as “if we intervened on the concept strength of hydrophobicity, which amino acid would the decoder output?” My confusion is on the verb “enables” – I believe the linear layer gives you a more interpretable answer to q2 (say compared to a nonlinear decoder), but you would still be able to answer q2 with a nonlinear decoder,  by intervening on the concept value $\\hat{c}_i$ and then running a forward pass of the decoder.\n\nQ4 What’s the spread along the x-axis on figure 4b? Is it naturalness? It might be helpful to also plot the reference lines for no-change (for both naturalness and intervened property) as you do in figure 4a."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"* Adapting concept-bottleneck generative models to masked language models is an interesting and novel idea.\n* Using integrated gradients to find tokens to resample to is an interesting and novel idea.\n* Writing is generally easy to follow.\n* This work focuses on improving controllability, interpretability, and debuggability of language models (and protein sequence in particular as an example) which is an important topic with large potential impact."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Method\n\n(considering a broader application of the proposed method to general language/sequence modeling, as authors mentioned in the abstract) \n\nM.1 The model proposed is only a conditional generative model given a partial sequence rather than a full generative model of an entire sequence. Classical masked language models (like BERT) do not model the joint distribution over sequences [1]. There are ways to derive joints from MLMs by making their conditionals consistent with one another (see, e.g. [1] and the related works cited there), or by modifying the training objective so that the conditionals are encouraged to be consistent during training (e.g. [2] [3]). Other protein models based on MLM (e.g. [3], which the authors cited) have done the latter kind of training modifications to enable sampling full sequences. I'm not sure how this limitation matters for protein modeling, could authors give more discussions about this? \n\nM.2 Naturalness of the controllably generated sequence is not considered during intervention, but only used as a measurement afterwards. See point below for consequence.\n\nM.3 The proposed controllability technique only proposes changes local to a particular known sequence x, by changing n% (5 in experiments) of x at a time. Again, I cannot comment on the significance of this limitation for protein modeling. For other modalities like images and language, it can be difficult to change sequence-level concepts via chaining a sequence of such local changes. Even if it is possible, since naturalness is not considered as a factor during intervention but only afterwards, applying a sequence of interventions could lead to going outside of the training distribution (e.g. the interventions produced in Figure 20 goes outside of the color-MNIST training distribution, where all digits are uniformly colored). Empirically though, (Fig 4b 15 16) the proposed method seems to outperform baselines so maybe this is less of an issue in practice, for the concepts in this paper and/or for the protein domain.\n\nExperiments\n\nE.1 The model is trained with 25% masking, but is used to generate with 5% masking in the experiments. This could make the model worse because it’s a different input distribution. Is there a particular justification for masking 5%?\n\nE.2 When evaluating controllability against baselines (and more specifically CC-pLM, since it's closer to CB-pLM), it’s not clear whether the gain comes from intervening at better positions (due to applying integrated gradients to a different architecture), better sensitivity of the model to interventions (due to the proposed concept bottleneck), or a combination, making it hard to evaluate the limitations / benefits of the architecture versus the intervention technique.\n\n[1] Hennigen and Kim, 2023. Deriving Language Models from Masked Language Models.\n\n[2] Germain et al., 2015. MADE: Masked Autoencoder for Distribution Estimation.\n\n[3] Hayes et al., 2024. Simulating 500 million years of evolution with a language model."}},"nonreaders":[],"tmdate":1733087090725,"tcdate":1731183638827,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5756/Reviewer_QrUQ"],"signatures":["ICLR.cc/2025/Conference/Submission5756/Reviewer_QrUQ"],"forum":"Yt9CFhOOFe","number":4,"license":"CC BY 4.0","cdate":1731183638827,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5756/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733087090725,"domain":"ICLR.cc/2025/Conference","replyto":"Yt9CFhOOFe","id":"irRpH7WBvX","forumContent":{"TLDR":{"value":"We insert a concept bottleneck layer into language models."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLMs","protein design","concept bottleneck"]},"primary_area":{"value":"interpretability and explainable AI"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We introduce Concept Bottleneck Protein Language Models (CB-pLM), a generative masked language model with a layer where each neuron corresponds to an interpretable concept. Our architecture offers three key benefits: i) Control: We can intervene on concept values to precisely control the properties of generated proteins, achieving a 3$\\times$ larger change in desired concept values compared to baselines. ii) Interpretability: A linear mapping between concept values and predicted tokens allows transparent analysis of the model's decision-making process. iii) Debugging: This transparency facilitates easy debugging of trained models. Our models achieve pre-training perplexity and downstream task performance comparable to traditional masked protein language models, demonstrating that interpretability does not compromise performance. While adaptable to any language model, we focus on masked protein language models due to their importance in drug discovery and the ability to validate our model's capabilities through real-world experiments and expert knowledge. We scale our CB-pLM from 24 million to 3 billion parameters, making them the largest Concept Bottleneck Models trained and the first capable of generative language modeling."},"_bibtex":{"value":"@inproceedings{\nismail2025concept,\ntitle={Concept Bottleneck Language Models For Protein Design},\nauthor={Aya Abdelsalam Ismail and Tuomas Oikarinen and Amy Wang and Julius Adebayo and Samuel Don Stanton and Hector Corrada Bravo and Kyunghyun Cho and Nathan C. Frey},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=Yt9CFhOOFe}\n}"},"title":{"value":"Concept Bottleneck Language Models For Protein Design"},"pdf":{"value":"/pdf/60097e91c1f58e314e11f6555e1a84ae2649fc83.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"ismail|concept_bottleneck_language_models_for_protein_design"},"authorids":{"value":["~Aya_Abdelsalam_Ismail1","~Tuomas_Oikarinen1","~Amy_Wang2","~Julius_Adebayo1","~Samuel_Don_Stanton1","~Hector_Corrada_Bravo1","~Kyunghyun_Cho1","~Nathan_C._Frey1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Aya Abdelsalam Ismail","Tuomas Oikarinen","Amy Wang","Julius Adebayo","Samuel Don Stanton","Hector Corrada Bravo","Kyunghyun Cho","Nathan C. Frey"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a novel Neural Stochastic Differential Equation (Neural SDE) framework for continuous-domain sequence modeling. It treats time-series data as samples from an underlying continuous dynamical system and models its evolution using an SDE with neural networks for both the drift (deterministic trend) and diffusion (stochastic fluctuation) terms. Key contributions include a simulation-free maximum likelihood training objective and a decoupled two-stage optimizer, which eliminate the need for costly forward simulations during training."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. In Equation 7, $𝑋_t$ is constructed using stochastic interpolation (Equation 5). However, since stochastic interpolation lacks physical realism, how can the consistency between its gradients and the vector field be guaranteed?\n\n2. The paper proposes two simplifying assumptions in Section 5. Could the authors comment on the framework's potential for future extensions regarding these two aspects?\n\n3. When the Δt in the first term of Equation 11 varies, the effect of the loss function is expected to differ. For instance, when Δt is relatively large, Δx/Δt may not adequately approximate the gradient, potentially leading to a significant bias in the first term.\n\n4. See weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The proposed Neural SDE framework introduces a simulation-free maximum likelihood approach, enabling more efficient modeling of continuous-time sequences.\n\n2. Its effectiveness has been validated across multiple experiments, including imitation learning and video prediction tasks.\n\n3. The manuscript is well-written, with clear and concise presentation, ensuring strong readability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Generative models do not focus on intermediate processes; consequently, their efficacy in sequence modeling tasks remains somewhat limited.\n\n2. To better verify the effectiveness of the proposed Neural SDE approach, it is necessary to design ablative experiments concerning its innovations."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925706974,"tcdate":1761480196142,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15431/Reviewer_CvJZ"],"signatures":["ICLR.cc/2026/Conference/Submission15431/Reviewer_CvJZ"],"forum":"0QTiRsosTz","number":1,"license":"CC BY 4.0","cdate":1761480196142,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15431/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925706974,"domain":"ICLR.cc/2026/Conference","replyto":"0QTiRsosTz","id":"D5tjeFscOI","forumContent":{"TLDR":{"value":"We proposed a new sequence modeling approach tailored to continuous domain, as a unified framework for embodied and generative AI tasks."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Neural Stochastic Differential Equations","Flow Matching","Diffusion Models","Continuous-Time Sequence Modeling"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Inspired by the ubiquitous use of differential equations to model continuous dynamics across diverse scientific and engineering domains, we propose a novel and intuitive approach to continuous sequence modeling. Our method interprets timeseries data as discrete samples from an underlying continuous dynamical system, and models its time evolution using Neural Stochastic Differential Equation (Neural SDE), where both the flow (drift) and diffusion terms are parameterized by neural networks. We derive a principled maximum likelihood objective and a simulationfree scheme for efficient training of our Neural SDE model. We demonstrate the versatility of our approach through experiments on sequence modeling tasks across both embodied and generative AI. Notably, to the best of our knowledge, this is the first work to show that SDEbased continuous-time modeling also excels in such complex scenarios, and we hope that our work opens up new avenues for research of SDE models in high-dimensional and temporally intricate domains."},"_bibtex":{"value":"@misc{\ncheng2026neural,\ntitle={Neural {SDE}s as a Unified Approach to Continuous-Domain Sequence Modeling},\nauthor={Chen Cheng and Macheng Shen},\nyear={2026},\nurl={https://openreview.net/forum?id=0QTiRsosTz}\n}"},"title":{"value":"Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling"},"pdf":{"value":"/pdf/08fde7c3e4c62de396a588a83fcba00f3aa2309d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cheng|neural_sdes_as_a_unified_approach_to_continuousdomain_sequence_modeling"},"authorids":{"value":["~Chen_Cheng1","~Macheng_Shen2"]},"authors":{"value":["Chen Cheng","Macheng Shen"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2507.12760v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"zhang|unified_medical_image_segmentation_with_state_space_modeling_snake"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Ruicheng_Zhang:","https://dblp.org/search/pid/api?q=author:Haowei_Guo:","https://dblp.org/search/pid/api?q=author:Kanghui_Tian:","~Jun_Zhou14","https://dblp.org/search/pid/api?q=author:Mingliang_Yan:","https://dblp.org/search/pid/api?q=author:Zeyu_Zhang:","https://dblp.org/search/pid/api?q=author:Shen_Zhao:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2507.12760"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2507-12760,\n  publtype={informal},\n  author={Ruicheng Zhang and Haowei Guo and Kanghui Tian and Jun Zhou and Mingliang Yan and Zeyu Zhang and Shen Zhao},\n  title={Unified Medical Image Segmentation with State Space Modeling Snake},\n  year={2025},\n  month={July},\n  cdate={1751328000000},\n  journal={CoRR},\n  volume={abs/2507.12760},\n  url={https://doi.org/10.48550/arXiv.2507.12760}\n}\n"},"abstract":{"value":"Unified Medical Image Segmentation (UMIS) is critical for comprehensive anatomical assessment but faces challenges due to multi-scale structural heterogeneity. Conventional pixel-based approaches, lacking object-level anatomical insight and inter-organ relational modeling, struggle with morphological complexity and feature conflicts, limiting their efficacy in UMIS. We propose Mamba Snake, a novel deep snake framework enhanced by state space modeling for UMIS. Mamba Snake frames multi-contour evolution as a hierarchical state space atlas, effectively modeling macroscopic inter-organ topological relationships and microscopic contour refinements. We introduce a snake-specific vision state space module, the Mamba Evolution Block (MEB), which leverages effective spatiotemporal information aggregation for adaptive refinement of complex morphologies. Energy map shape priors further ensure robust long-range contour evolution in heterogeneous data. Additionally, a dual-classification synergy mechanism is incorporated to concurrently optimize detection and segmentation, mitigating under-segmentation of microstructures in UMIS. Extensive evaluations across five clinical datasets reveal Mamba Snake's superior performance, with an average Dice improvement of 3\\% over state-of-the-art methods."},"title":{"value":"Unified Medical Image Segmentation with State Space Modeling Snake"},"authors":{"value":["Ruicheng Zhang","Haowei Guo","Kanghui Tian","Jun Zhou","Mingliang Yan","Zeyu Zhang","Shen Zhao"]}},"tmdate":1767886780355,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2507-12760"],"tcdate":1767886776388,"writers":["~"],"signatures":["~Jun_Zhou14"],"forum":"PHfjSUtdVF","license":"CC BY-SA 4.0","number":736938,"cdate":1751328000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1767886780355,"domain":"DBLP.org","id":"PHfjSUtdVF","version":2},{"content":{"venue":{"value":"ICONIP (4) 2024"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-981-96-6585-3_28.pdf"},"venueid":{"value":"dblp.org/conf/ICONIP/2024"},"paperhash":{"value":"pan|mambafuse_fusing_multiscale_mamba_and_cnn_features_for_seizure_prediction"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Ding_Pan:","~Guibo_Luo2","https://dblp.org/search/pid/api?q=author:Yuesheng_Zhu:"]},"html":{"value":"https://doi.org/10.1007/978-981-96-6585-3_28"},"_bibtex":{"value":"@inproceedings{DBLP:conf/iconip/PanLZ24,\n  author={Ding Pan and Guibo Luo and Yuesheng Zhu},\n  title={MambaFuse: Fusing Multi-scale Mamba and CNN Features for Seizure Prediction},\n  year={2024},\n  cdate={1704067200000},\n  pages={399-413},\n  url={https://doi.org/10.1007/978-981-96-6585-3_28},\n  booktitle={ICONIP (4)},\n  crossref={conf/iconip/2024-4}\n}\n"},"abstract":{"value":"Recently, significant progress has been made in seizure prediction, attributed to the advancement of basic models. Specifically, the structures combining convolutional networks with transformers can successfully extract both local and global features. However, the disadvantage of the transformer is also inherited, namely, the quadratic computational complexity with respect to the length of the sequence. Inspired by the recent model with linear complexity for long-distance modeling, named Mamba, we explore the potential of this state space model for the seizure prediction task in terms of effectiveness and efficiency in the paper. Nevertheless, directly applying Mamba yields poor performance since multi-scale global and local features are not taken into consideration. To address this issue, we propose a novel design called MambaFuse. Specifically, a parallel Mamba and CNN block is initially developed to extract multi-scale global and local features. Firstly, the multi-scale Mamba module is proposed to leverage the global modeling of Mamba at linear complexity. Secondly, the scale-adapt CNN module is proposed to extract local features through residual learning. Finally, the enhanced cross-attention fusion module is proposed to dynamically combine global Mamba features and local convolution features for a multi-scale global-local representation. Experimental results on the dataset of CHB-MIT have demonstrated that MambaFuse outperforms state-of-the-art methods, providing a promising approach for seizure prediction."},"title":{"value":"MambaFuse: Fusing Multi-scale Mamba and CNN Features for Seizure Prediction"},"authors":{"value":["Ding Pan","Guibo Luo","Yuesheng Zhu"]}},"tmdate":1752377738373,"pdate":1704067200000,"tcdate":1752377727937,"writers":["~"],"signatures":["~Guibo_Luo2"],"forum":"tGhrIOi1Fa","license":"CC BY-SA 4.0","number":567118,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1752377738373,"domain":"DBLP.org","id":"tGhrIOi1Fa","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2403.09977v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"pei|efficientvmamba_atrous_selective_scan_for_light_weight_visual_mamba"},"authorids":{"value":["~Xiaohuan_Pei1","https://dblp.org/search/pid/api?q=author:Tao_Huang_0020:","https://dblp.org/search/pid/api?q=author:Chang_Xu_0002:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2403.09977"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2403-09977,\n  publtype={informal},\n  author={Xiaohuan Pei and Tao Huang and Chang Xu},\n  title={EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2403.09977},\n  url={https://doi.org/10.48550/arXiv.2403.09977}\n}\n"},"abstract":{"value":"Prior efforts in light-weight model development mainly centered on CNN and Transformer-based designs yet faced persistent challenges. CNNs adept at local feature extraction compromise resolution while Transformers offer global reach but escalate computational demands $\\mathcal{O}(N^2)$. This ongoing trade-off between accuracy and efficiency remains a significant hurdle. Recently, state space models (SSMs), such as Mamba, have shown outstanding performance and competitiveness in various tasks such as language modeling and computer vision, while reducing the time complexity of global information extraction to $\\mathcal{O}(N)$. Inspired by this, this work proposes to explore the potential of visual state space models in light-weight model design and introduce a novel efficient model variant dubbed EfficientVMamba. Concretely, our EfficientVMamba integrates a atrous-based selective scan approach by efficient skip sampling, constituting building blocks designed to harness both global and local representational features. Additionally, we investigate the integration between SSM blocks and convolutions, and introduce an efficient visual state space block combined with an additional convolution branch, which further elevate the model performance. Experimental results show that, EfficientVMamba scales down the computational complexity while yields competitive results across a variety of vision tasks. For example, our EfficientVMamba-S with $1.3$G FLOPs improves Vim-Ti with $1.5$G FLOPs by a large margin of $5.6\\%$ accuracy on ImageNet. Code is available at: \\url{https://github.com/TerryPei/EfficientVMamba}."},"title":{"value":"EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba"},"authors":{"value":["Xiaohuan Pei","Tao Huang","Chang Xu"]}},"tmdate":1731465237265,"pdate":1704067200000,"tcdate":1731464183230,"writers":["~"],"signatures":["~Xiaohuan_Pei1"],"forum":"gdE87A5Bvz","license":"CC BY-SA 4.0","number":189697,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1731465237265,"domain":"DBLP.org","id":"gdE87A5Bvz","version":2},{"content":{"summary":{"value":"The paper introduces a novel perspective on output target color space for object-centric representation learning. The authors provide an in-depth discussion of various color spaces as prediction targets, including RGB, HSV, and CIE-LAB. In particular, using Slot Attention as the backbone, the authors conduct experiments with different target color spaces, encompassing both individual and composite color spaces."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1) The authors only discuss grayscale for the single-channel approach. Considering the high correlation between RGB channels, what would the impact for OCRL be if using just one of the RGB channels as the target?\n\n2)  Why does introducing composite color spaces improve object-centric representation? Could you provide a more detailed explanation from a feature perspective?\n\n3) How does the model behave for different values of the number of slots? Is there an ablation showing the effect of that parameter?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1) As most previous works focused on the different color space input, the exploration and discussion of using different color space as the target is beneficial for the community.\n2) This paper finds that predicting composite color spaces offers a slight improvement over traditional RGB targets in unsupervised object-centric representation learning (OCRL). This approach not only inspires advancements in OCRL but also provides a direction for image modeling-based unsupervised representation learning.\n3) The discussion of different color spaces is also beneficial for the computer vision community."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**1) Although the authors present an interesting approach to enhancing model understanding of foreground objects in a scene by predicting different color spaces, the paper primarily discusses the impact of various color spaces from a results-oriented perspective without providing a more in-depth analysis. For example, influence on model parameters and feature characteristics are lacking. I hope the authors can offer further insights into how different target color spaces influence the model.**\n\n**2) Although the authors state that their work focuses on color spaces, the essence of their approach lies in enhancing the model's perceptual abilities by adjusting target objectives. Therefore, using only an RGB-RGB baseline may not be reliable. It would be more robust to include comparisons with methods that use different representations as targets, such as (Seitzer et al., 2023).**\n\n**3) While the authors' idea is interesting, they do not provide a reliable analysis from a representation perspective on the inclusion of different color space targets. For instance, in Figure 2 of the main paper, there is a noticeable similarity between saturation and the mask in highlighting foreground objects; however, the mask is discrete and contains categorical information, while saturation focuses solely on object shape. There is insufficient technical analysis explaining the impact of using saturation.**\n\nM. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow, T. Xiao, C.-J. Simon-Gabriel, T. He, Z. Zhang, B. Sch¨olkopf, T. Brox, and F. Locatello. Bridging the Gap to Real-World Object-Centric Learning, March 2023. URL http://arxiv.org/abs/2209.14860. arXiv:2209.14860 [cs]."}},"nonreaders":[],"tmdate":1733845318786,"tcdate":1730751031118,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11349/Reviewer_puBu"],"signatures":["ICLR.cc/2025/Conference/Submission11349/Reviewer_puBu"],"forum":"gRrjx9GzRP","number":2,"license":"CC BY 4.0","cdate":1730751031118,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11349/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733845318786,"domain":"ICLR.cc/2025/Conference","replyto":"gRrjx9GzRP","id":"bCzQAAvAkf","forumContent":{"TLDR":{"value":"Enhancing the common RGB color space with additional uncorrelated color channels significantly increases object detection capabilities of unsupervised object-centric representation learning."},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["object detection","visual reasoning","segmentation","unsupervised learning","object centric representation learning","computer vision"]},"supplementary_material":{"value":"/attachment/347730634b1dc96f836eb787209393475af8e944.zip"},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Object-centric architectures can learn to extract distinct object representations from visual scenes, enabling downstream applications on the object level. \nSimilarly to autoencoder-based image models, object-centric approaches have been trained on the unsupervised reconstruction loss of images encoded by RGB color spaces. \nIn our work, we challenge the common assumption that RGB images are the optimal target for unsupervised learning in computer vision.\nWe discuss conceptually and empirically that other color spaces, such as HSV, bear essential characteristics for object-centric representation learning, like robustness to lighting conditions. We further show that models improve when requiring them to predict additional color channels.\nSpecifically, we propose the RGB-S space, which extends RGB with HSV's saturation component and leads to markedly better reconstruction and disentanglement for five common evaluation datasets.\nThe use of composite color spaces can be implemented with basically no computational overhead, is agnostic of the models' architecture, and is universally applicable across a wide range of visual computing tasks and training types. \nThe findings of our approach encourage additional investigations in computer vision tasks beyond object-centric learning."},"_bibtex":{"value":"@misc{\njackl2024leveraging,\ntitle={Leveraging Color Channel Independence for Improved Unsupervised Object Detection},\nauthor={Bastian J{\\\"a}ckl and Yannick Metz and Udo Schlegel and Daniel A. Keim and Maximilian T. Fischer},\nyear={2024},\nurl={https://openreview.net/forum?id=gRrjx9GzRP}\n}"},"title":{"value":"Leveraging Color Channel Independence for Improved Unsupervised Object Detection"},"pdf":{"value":"/pdf/ffed47473d32ce8b6d3d3fd63d3c001e4a1b67e5.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"jäckl|leveraging_color_channel_independence_for_improved_unsupervised_object_detection"},"authorids":{"value":["~Bastian_Jäckl1","~Yannick_Metz1","~Udo_Schlegel1","~Daniel_A._Keim1","~Maximilian_T._Fischer1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Bastian Jäckl","Yannick Metz","Udo Schlegel","Daniel A. Keim","Maximilian T. Fischer"]}},"version":2},{"content":{"comment":{"value":"Thank you for your valuable feedback. We want to address your comments and questions as below:\n\n> Lack of Real-Time Runtime Analysis\n\nMeasuring the end-to-end latency is challenging in our case, since it heavily depends on upstream tasks. Our goal is to propose a flexible approach that can be adapted to any streaming conditions with the best possible latency. We provide some scenarios with latency analysis as below:\n\n- Voice Conversion: we consider text-based voice conversion (which can also be challenging tasks such as accent conversion, whisper to speech conversion) as the upstream task, where the model outputs word by word. With chunk length of 1 and lookahead of 1 (extreme cases need furthur finetuning as reported in Table 13&14), our latency is 2 words\n- Live Translation: we consider an upstream model that can output text chunks instead of full sentences (e.g. SeamlessM4T). If each chunk has an average of 4 words and there is one chunk lookahead, our latency is 8 words. Our approach also benefits from speech generated to keep pace with the text stream head, which is non measurable. For example, starting from the time 0, if the first chunk arrives at the time $t_1$, but needs $t'_1>t_1$ to play, the playback of the second chunk will be delayed by the audio-text duration gap \n$|t'_1-t_1|$. We remove this latency by making sure that $t'_1=t_1$ (audio-text stream synchronization)\n- Voice Assistant: adding voice to an LLM with our model is trickier since if LLM generates text fast enough, there will be sufficient lookaheads most of the time. We consider a scenario when LLM generates words at the same pace with the speech (e.g., large model running on device), in which our model aims to follow the latest words from LLM. Similar to voice conversion, our model can have as low as 2 word lookahead.\n\nIn these examples, we ignore the time-to-first-frame, which should be as fast as a single step, since the model can stream out audio frames as soon as they are generated, and adding text chunks introduces zero overhead (only embedded, not encoded).\n\n> Baseline Comparison with Transformer Decoder: Incorporating a baseline using a transformer-based decoder could further validate the performance and efficiency advantages of the Mamba architecture. There are well-known techniques to improve the inference speed and local attention mechanism. This addition would provide a stronger basis for the choice of Mamba over more conventional architectures.\n\nWe rely on the original papers [1] and [2] to justify our choice for Mamba in terms of inference speed and computational complexity, which is critical to address in on-device streaming settings. Our hypothesis is that a full access to long context length may not be required for the TTS task, since the content and the speaker characteristic are accessible via cross attention and the model only needs local context to ensure continuous generation. With that hypothesis, a transformer with local attention may work as well as Mamba in terms of speed and performance. We only demonstrate the performance in our experiments, where the main observation is that Mamba can produce results as good as competitive transformer baselines.\n\n[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces [2] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality\n\nPlease let us know if these have addressed your concerns or if you have other concerns. We will provide the updated manuscript at the end of the rebuttal period."},"title":{"value":"Thank you for your response!"}},"tmdate":1732633862882,"tcdate":1732633815290,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1359/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission1359/Authors"],"forum":"RK3Gj9J5my","number":7,"license":"CC BY 4.0","cdate":1732633815290,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1359/-/Official_Comment"],"mdate":1732633862882,"domain":"ICLR.cc/2025/Conference","replyto":"zcmO6dVluJ","id":"i9seR67h8G","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"We introduce a zero-shot text-to-speech model that is capable of generating speech from a continuous text stream, with support for on-device streaming."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["zero-shot text-to-speech","streaming","state-space models"]},"supplementary_material":{"value":"/attachment/eb3ee0f6f9fc423a83e128f51cb138fde2d018cc.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Existing zero-shot text-to-speech (TTS) systems are typically designed to process complete sentences and are constrained by the maximum duration for which they have been trained. However, in many streaming applications, texts arrive continuously in short chunks, necessitating instant responses from the system. We identify the essential capabilities required for chunk-level streaming and introduce L3Speech, a stream-aware model that supports non-stop speech generation, text-audio stream synchronization, and seamless transitions between short speech chunks. To achieve these, we propose (1) adopting Mamba, a class of sequence modeling distinguished by linear-time decoding, which is augmented by cross-attention mechanisms for conditioning, (2) utilizing rotary positional embeddings in the computation of cross-attention, enabling the model to process a non-stop text stream by sliding a window, and (3) decoding with semantic guidance, a technique that aligns speech with the transcript during inference with minimal overhead. Experimental results demonstrate that our models are competitive with state-of-the-art language model-based zero-shot TTS models, while also providing flexibility to support a wide range of streaming scenarios."},"_bibtex":{"value":"@misc{\ndang2025zeroshot,\ntitle={Zero-Shot Text-to-Speech from Continuous Text Streams},\nauthor={Trung Dang and David Aponte and Dung Tran and Tianyi Chen and Kazuhito Koishida},\nyear={2025},\nurl={https://openreview.net/forum?id=RK3Gj9J5my}\n}"},"title":{"value":"Zero-Shot Text-to-Speech from Continuous Text Streams"},"pdf":{"value":"/pdf/ae6d24286703bb7fbfcd629543b4f9e8d221ff52.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"dang|zeroshot_texttospeech_from_continuous_text_streams"},"authorids":{"value":["~Trung_Dang1","~David_Aponte1","~Dung_Tran2","~Tianyi_Chen3","~Kazuhito_Koishida1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Trung Dang","David Aponte","Dung Tran","Tianyi Chen","Kazuhito Koishida"]}},"version":2},{"content":{"TLDR":{"value":"A Mamba–Transformer hybrid error correcting codes decoder"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Error-Correcting Codes","Mamba"]},"supplementary_material":{"value":"/attachment/79782159142db6be93456ee89de2c7c8ab69d7fd.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"We introduce a novel deep learning method for decoding error correction codes based on the Mamba architecture, enhanced with Transformer layers. Our approach proposes a hybrid decoder that leverages Mamba’s efficient sequential modeling while maintaining the global context capabilities of Transformers. To further improve performance, we design a novel layer-wise masking strategy applied to each Mamba layer, allowing selective attention to relevant code features at different depths. Additionally, we introduce a progressive layer-wise loss, supervising the network at intermediate stages and promoting robust feature extraction throughout the decoding process. Comprehensive experiments across a range of linear codes demonstrate that our method significantly outperforms Transformer-only decoders and standard Mamba models."},"_bibtex":{"value":"@misc{\ncohen2026hybrid,\ntitle={Hybrid Mamba{\\textendash}Transformer Decoder for Error-Correcting Codes},\nauthor={Shy-El Cohen and Yoni Choukroun and Eliya Nachmani},\nyear={2026},\nurl={https://openreview.net/forum?id=BlKB91pkCd}\n}"},"title":{"value":"Hybrid Mamba–Transformer Decoder for Error-Correcting Codes"},"pdf":{"value":"/pdf/e4d1c6cde1b5afdf339765037e957bdd3c51da3f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cohen|hybrid_mambatransformer_decoder_for_errorcorrecting_codes"},"authorids":{"value":["~Shy-El_Cohen1","~Yoni_Choukroun1","~Eliya_Nachmani1"]},"authors":{"value":["Shy-El Cohen","Yoni Choukroun","Eliya Nachmani"]}},"tmdate":1770804815641,"tcdate":1756973850338,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1978/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission1978/Authors"],"forum":"BlKB91pkCd","license":"CC BY 4.0","number":1978,"cdate":1756973850338,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission1978/-/Full_Submission","ICLR.cc/2026/Conference/Submission1978/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804815641,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"BlKB91pkCd","version":2},{"content":{"comment":{"value":"2. We further run extended experiments studying the effect of batch size on the step-wise speedup comparing Muon and AdamW.\n\nOur research focuses on token-wise comparison, and therefore considers descent along the batch-size dimension as well. However, as reviewer xiBh points out, batch size is not always a tunable factor. It is thus informative to examine the speedup under fixed batch sizes. In response, we conducted additional experiments analyzing how batch size affects the speedup between Muon and AdamW, representing matrix-level and scalar-level optimizers, respectively. The results are now provided in the new **Appendix J**. Across model sizes (130M–1.2B), we find that Muon’s performance degrades more slowly as batch size increases, compared to AdamW. Consequently, the relative speedup of Muon over AdamW is smaller when the batch size is low. We also verify that the batch sizes used in the main paper are generally below or equal to the critical batch size. \n\n\n[1] (ICML 2025) MARS: Unleashing the Power of Variance Reduction for Training Large Models\n\n[2] (ICLR 2025 Spotlight) u-μP: The Unit-Scaled Maximal Update Parametrization\n\n[3] (ICLR 2025) How Does Critical Batch Size Scale in Pre-training?\n\n[4] (ICML 2024) Scaling Exponents Across Parameterizations and Optimizers\n\n[5] (Kimi Technical Report) Muon is Scalable for LLM Training\n\n[6] Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization\n\n[7] Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes \n\n[8] Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training \n\n[9] Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why gradient Accumulation Is Wasteful\n\n[10] Practical Efficiency of Muon for Pretraining \n\n[11] Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining\n\n[12] (COLM 2024) Mamba: Linear-Time Sequence Modeling with Selective State Spaces\n\n[13] (ICLR 2025)  Gated Delta Networks: Improving Mamba2 with Delta Rule\n\n[14] (NeurIPS 2025) The Road Less Scheduled\n\n[15] (Kimi Technical Report)  Kimi Linear: An Expressive, Efficient Attention Architecture"},"title":{"value":"General Response (continued)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763795450845,"tcdate":1763795450845,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission9895/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission9895/Authors"],"forum":"2J51qUZ0iG","number":3,"license":"CC BY 4.0","cdate":1763795450845,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission9895/-/Official_Comment"],"mdate":1763795450845,"domain":"ICLR.cc/2026/Conference","replyto":"KejPAoFGPt","id":"vBeMoAwStB","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"Fair comparisons reveal that new optimizers only offer modest speedups over AdamW; matrix level optimizers' advantage shrinks with scale—from 1.4× at 0.1B to just 1.1× at 1.2B parameters—once proper hyperparameter tuning and evaluation are applied."},"keywords":{"value":["optimizer","benchmarking","pretrain"]},"supplementary_material":{"value":"/attachment/16d8aeabd609c47808be8933467ac1e7a351ddfd.zip"},"primary_area":{"value":"optimization"},"abstract":{"value":"AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2$\\times$ speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups.\nTo address these two issues, we conduct a systematic study of ten deep learning optimizers across four model scales (0.1B–1.2B parameters) and data-to-model ratios (1--8$\\times$ the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperparameter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training.\nFirst, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. \nSecond, the actual speedup of many proposed optimizers over well-tuned baselines is lower than claimed and decreases with model size to only 1.1$\\times$ for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers can flip during training due to learning rate decay.\nThrough our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices as preconditioners --- multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4$\\times$ over AdamW for 0.1B parameter models to merely 1.1$\\times$ for 1.2B parameter models."},"_bibtex":{"value":"@inproceedings{\nwen2026fantastic,\ntitle={Fantastic Pretraining Optimizers and Where to Find Them},\nauthor={Kaiyue Wen and David Leo Wright Hall and Tengyu Ma and Percy Liang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=2J51qUZ0iG}\n}"},"title":{"value":"Fantastic Pretraining Optimizers and Where to Find Them"},"pdf":{"value":"/pdf/c2fc9a9f06c380b7631bd48bca832db7dea63a0b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"wen|fantastic_pretraining_optimizers_and_where_to_find_them"},"authorids":{"value":["~Kaiyue_Wen1","~David_Leo_Wright_Hall1","~Tengyu_Ma1","~Percy_Liang1"]},"authors":{"value":["Kaiyue Wen","David Leo Wright Hall","Tengyu Ma","Percy Liang"]}},"version":2},{"content":{"summary":{"value":"This paper presents a new self-supervised learning-based denoising method for diffusion MRI (dMRI). The proposed method leveraged the diffusion modeling concept, but instead of training a diffusion model with “clean images” as x_0 and noise as x_T, it utilized two diffusion weighted images (DWIs) with different diffusion encodings at both ends of a “diffusion-like” process. A denoising network was trained by predicting one DWI using a linear combination of two DWIs and an added noise term. The linear combination coefficients are time-dependent and determined via a scheduling strategy similar to training a diffusion model. The network was then used for a conditional sampling step for generating the final denoised images. The idea to utilize images acquired with different diffusion encodings to denoise one of them is interesting and the training strategy is an interesting approach to leverage the diffusion modeling concept, especially with training only latter diffusion steps to reduce hallucinations. However, several key assumptions made are questionable and the overall methodology and presentation lacks clarity. Evaluation using only dMRI signal model goodness of fit is limited and can be biased. There are a few overstatements that can mislead the readers. Detailed comments can be found below."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"In Eq. (5) on Line 155, the authors highlighted a specific term as the ”major difference” between xt−1 and x^bar_t−1. Could the authors clarify why this particular term is considered the primary source of difference? Furthermore, can the authors elaborate on the underlying reason(s) for the “drift” in the model and how it emerges during the reverse diffusion process?\n\nAccording to the definition of the Fusion process in Eq. (6)  and the “forward process” in Eq. (7), it appears that the starting point for the forward process changes based on t, as x_t* is dependent on t. This dependence implies that the Fusion process dynamically adjusts the starting point of the forward process at each step, which is unconventional compared to typical diffusion models. Could the authors clarify the rationale behind this design?\n\nOther more recent self-supervised denoising methods should be compared, if not for all, e.g., Noise2Score and Recorrupted2Recorrupted etc."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"A diffusion-like modeling that learns the relationship between two DWI volumes with different diffusion encodings to denoise one or each other.\n\nTraining only later step diffusion to avoid hallucination\n\nA fusion strategy that exploits linear combination of two DWIs with different contrasts with time-dependent coefficients and iterative refinement.\n\nExtensive evaluations using both simulations that exactly followed the assumptions for the proposed methodology and practical magnitude DWI data."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"There are statements that can be misleading in the context of MR physics (aka domain knowledge). For example, \"the noise predominantly originates from physical interferences (Fadnavis et al., 2020a)\". This statement about physical interferences is  both vague and inaccurate. This work is dealing with thermal noise or noise resulting from thermal noise in the measurements, which is not really physical interferences depending on how ones interpret them. Another example, \"Different clinical applications require varying numbers of diffusion vectors and acquisition strategies, which makes modeling the noise distribution and further implementing denoising techniques challenging\". Acquiring DWIs with varying numbers of diffusion vectors had nothing to do with the difficulty of  modeling noise distribution.\n\nMany key assumptions for the proposed method was built on do not hold which made the theoretical/mathematical foundations questionable, e.g.,\na) It seems that the authors assumed DWIs acquired with different diffusion encodings had the same underlying “clean” image and were corrupted by independent noise. This is inaccurate. In fact, two DWIs can have rather different contrasts due to the diffusion encoding effects, e.g., different diffusion encoding directions. More specifically, x and x’ cannot be simply modeled as the same y plus different noise. What are the implications of this assumption not met?\n\nb) Line 111: The authors claimed that that the proposed method does not require an explicit noise model. This is an overstatement. The J-invariance assumption, which formed the basis of the training objective in Eq. (9) implicitly requires that the noise distribution be zero-means and conditionally independent. Furthermore, additive noise model was assumed, x = y + n1 (Line 200). In dMRI, the magnitude images with higher b-values (stronger diffusion weightings) can have lower SNR for which additive noise may not hold. These need to be clarified.\n\n-  Overall, the presentation lacks clarity and there seem to be some concerning inaccuracies.\na) The linear combination relationship claimed in Section 3.1 does not seem accurate. I checked the derivation. Eq. 31 is correct which is known (so this is not a contribution of the authors), but I'm not sure about going from Eq. 31 to 32 as F_theta predicts x_0, but they are not equal, and there is also an additional term of sigma_t^2*z. Therefore, I don't think it's a correct statement to say x_(t-1) is a linear interpolation between x_out and x_t. But is this really needed for the proposed method? I really don’t see a connection between what’s argued theoretically and what’s actually being implemented.\n\nb) There are a few other inaccurate mathematical statements and notations which are confusing. For example, Eq. 7, the left side has q(x1:T |xt*) which is a joint distribution for x1 to xT, and the right side is a Gaussian distribution for xt. \nOn Line 160: {xt}1:T was described as”obtained from the reverse process.” However, in\nFigure 1 and on the right side of Equation (7) on Line 186, it appears that xt is a corrupted version of xt*.  This interpretation, along with the notation in the Fig. 1, implies that {xt}1:T would represent a forward process. It appears to this reviewer the authors had not been using a consistent definition of forward and reverse diffusion which made the overall description rather confusing. These are just examples of inconsistencies found.\n\nc) According to the J-invariance property, the noise should ideally have zero mean\nand be conditionally independent of the target output. This requirement is necessary to ensure that the expected loss for self-supervised training asymptotical approaching the supervised loss. However, the input to F(.) in Eq. (9) includes xt*, which is a linear combination of x and x’ (Eq. (6)). Given that x serves as the supervision signal for the loss, this implies a correlation between the input x∗t and the target x, which would violate the conditional independence requirement for J-invariance."}},"nonreaders":[],"tmdate":1731427772072,"tcdate":1730677502627,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13426/Reviewer_Dn7R"],"signatures":["ICLR.cc/2025/Conference/Submission13426/Reviewer_Dn7R"],"forum":"wxPnuFp8fZ","number":3,"license":"CC BY 4.0","cdate":1730677502627,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13426/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427772072,"domain":"ICLR.cc/2025/Conference","replyto":"wxPnuFp8fZ","id":"5HIUhft58i","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Diffusion based models","Self-supervised MRI denoising"]},"supplementary_material":{"value":"/attachment/ede8b02e96523f02c3ef761521a3601807463623.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Magnetic Resonance Imaging (MRI), including diffusion MRI (dMRI), serves as a ``microscope'' for anatomical structures and routinely mitigates the influence of low signal-to-noise ratio scans by compromising temporal or spatial resolution. However, these compromises fail to meet clinical demands for both efficiency and precision. Consequently, denoising is a vital preprocessing step, particularly for dMRI, where clean data is unavailable. In this paper, we introduce Di-Fusion, a fully self-supervised denoising method that leverages the latter diffusion steps and an adaptive sampling process. Unlike previous approaches, our single-stage framework achieves efficient and stable training without extra noise model training and offers adaptive and controllable results in the sampling process. Our thorough experiments on real and simulated data demonstrate that Di-Fusion achieves state-of-the-art performance in microstructure modeling, tractography tracking, and other downstream tasks. Code is available at https://github.com/FouierL/Di-Fusion."},"_bibtex":{"value":"@inproceedings{\nwu2025selfsupervised,\ntitle={Self-Supervised Diffusion {MRI} Denoising via Iterative and Stable Refinement},\nauthor={Chenxu Wu and Qingpeng Kong and Zihang Jiang and S Kevin Zhou},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=wxPnuFp8fZ}\n}"},"title":{"value":"Self-Supervised Diffusion MRI Denoising via Iterative and Stable Refinement"},"pdf":{"value":"/pdf/6bd7c307edad6542cdf2945234d9af92905ebf8f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wu|selfsupervised_diffusion_mri_denoising_via_iterative_and_stable_refinement"},"authorids":{"value":["~Chenxu_Wu1","~Qingpeng_Kong1","~Zihang_Jiang1","~S_Kevin_Zhou1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chenxu Wu","Qingpeng Kong","Zihang Jiang","S Kevin Zhou"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2504.07654v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"karadag|msmamba_multiscale_mamba_for_timeseries_forecasting"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Yusuf_Meric_Karadag:","~Sinan_Kalkan1","https://dblp.org/search/pid/api?q=author:Ipek_Gursel_Dino:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2504.07654"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2504-07654,\n  publtype={informal},\n  author={Yusuf Meric Karadag and Sinan Kalkan and Ipek Gursel Dino},\n  title={ms-Mamba: Multi-scale Mamba for Time-Series Forecasting},\n  year={2025},\n  month={April},\n  cdate={1743465600000},\n  journal={CoRR},\n  volume={abs/2504.07654},\n  url={https://doi.org/10.48550/arXiv.2504.07654}\n}\n"},"abstract":{"value":"The problem of Time-series Forecasting is generally addressed by recurrent, Transformer-based and the recently proposed Mamba-based architectures. However, existing architectures generally process their input at a single temporal scale, which may be sub-optimal for many tasks where information changes over multiple time scales. In this paper, we introduce a novel architecture called Multi-scale Mamba (ms-Mamba) to address this gap. ms-Mamba incorporates multiple temporal scales by using multiple Mamba blocks with different sampling rates ($\\Delta$s). Our experiments on many benchmarks demonstrate that ms-Mamba outperforms state-of-the-art approaches, including the recently proposed Transformer-based and Mamba-based models."},"title":{"value":"ms-Mamba: Multi-scale Mamba for Time-Series Forecasting"},"authors":{"value":["Yusuf Meric Karadag","Sinan Kalkan","Ipek Gursel Dino"]}},"tmdate":1762249797251,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2504-07654"],"tcdate":1762249793302,"writers":["~"],"signatures":["~Sinan_Kalkan1"],"forum":"fe8CUgOe1J","license":"CC BY-SA 4.0","number":652567,"cdate":1743465600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1762249797251,"domain":"DBLP.org","id":"fe8CUgOe1J","version":2},{"content":{"TLDR":{"value":"We introduce TiledFlashLinearAttention a faster kernel algorithm for Linear RNNs and mLSTMs by improved Sequence Parallelism."},"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["xLSTM","FlashLinearAttention","FlashAttention","Linear Attention","Linear RNN","RNN","mLSTM","Attention","Self-Attention","Transformers","LLM","Kernels","Systems","Efficiency"]},"supplementary_material":{"value":"/attachment/27f52d1c05b14f8d6ed5b85f0d659f139da368a5.zip"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Linear RNNs with gating recently demonstrated competitive performance compared to Transformers in language modeling. Although their linear compute scaling in sequence length offers theoretical runtime advantages over Transformers, realizing these benefits in practice requires optimized custom kernels, as Transformers rely on the highly efficient Flash Attention kernels (Dao, 2024). Leveraging the chunkwise-parallel formulation of linear RNNs, Flash Linear Attention (FLA) (Yang & Zhang, 2024) shows that linear RNN kernels are faster than Flash Attention, by parallelizing over chunks of the input sequence. However, since the chunk size of FLA is limited, many intermediate states must be materialized in GPU memory. This leads to low arithmetic intensity and causes high memory consumption and IO cost, especially for long-context pre-training. In this work, we present Tiled Flash Linear Attention (TFLA), a novel kernel algorithm for linear RNNs, that enables arbitrary large chunk sizes and high arithmetic intensity by introducing an additional level of sequence parallelization within each chunk. First, we apply TFLA to the xLSTM with matrix memory, the mLSTM (Beck et al., 2024). Second, we propose an mLSTM variant with sigmoid input gate and reduced computation for even faster kernel runtimes at equal language modeling performance. In our speed benchmarks, we show that our new mLSTM kernels based on TFLA outperform highly optimized Flash Attention, Linear Attention and Mamba kernels, setting a new state of the art for efficient long-context sequence modeling primitives. Our code is available at: https://github.com/NX-AI/mlstm_kernels"},"_bibtex":{"value":"@inproceedings{\nbeck2025tiled,\ntitle={Tiled Flash Linear Attention: More Efficient Linear {RNN} and x{LSTM} Kernels},\nauthor={Maximilian Beck and Korbinian P{\\\"o}ppel and Phillip Lippe and Sepp Hochreiter},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=b6H64u6TqI}\n}"},"title":{"value":"Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels"},"pdf":{"value":"/pdf/2bdf00cbea0d464718aa6d15927ba3e546bd5025.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"beck|tiled_flash_linear_attention_more_efficient_linear_rnn_and_xlstm_kernels"},"authorids":{"value":["~Maximilian_Beck1","~Korbinian_Pöppel1","~Phillip_Lippe1","~Sepp_Hochreiter1"]},"authors":{"value":["Maximilian Beck","Korbinian Pöppel","Phillip Lippe","Sepp Hochreiter"]}},"tmdate":1783627396972,"pdate":1758216852042,"tcdate":1746735892863,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission11030/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission11030/Authors"],"forum":"b6H64u6TqI","license":"CC BY 4.0","number":11030,"cdate":1746735892863,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission11030/-/Full_Submission","NeurIPS.cc/2025/Conference/Submission11030/-/Supplementary_Material","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission11030/-/Camera_Ready_Revision"],"mdate":1783627396972,"odate":1761704854861,"domain":"NeurIPS.cc/2025/Conference","id":"b6H64u6TqI","version":2},{"content":{"venue":{"value":"MM2024 Oral"},"abstract":{"value":"In the domain of generative multimedia and interactive experiences, generating realistic and accurate full-body poses from sparse tracking is crucial for many real-world applications, while achieving sequence modeling and efficient motion generation remains challenging. Recently, state space models (SSMs) with efficient hardware-aware designs (i.e., Mamba) have shown great potential for sequence modeling, particularly in temporal contexts. However, processing motion data is still challenging for SSMs. Specifically, the sparsity of input conditions makes motion generation an ill-posed problem. Moreover, the complex structure of the human body further complicates this task. To address these issues, we present Motion Mamba Diffusion (MMD), a novel conditional diffusion model, which effectively utilizes the sequence modeling capability of SSMs and the robust generation ability of diffusion models to track full-body poses accurately. In particular, we design a bidirectional Temporal Mamba Module (TMM) to model motion sequence. Additionally, a Spatial Mamba Module (SMM) is further proposed for feature enhancement within a single frame. Extensive experiments on the large motion capture dataset (AMASS) demonstrate that our proposed approach outperforms the latest methods in terms of accuracy and smoothness and achieves new state-of-the-art performance. Moreover, our approach runs in real-time, making it ideal for employment in practical applications. The source code will be made public upon acceptance of this paper."},"relevance_to_conference":{"value":"We propose Motion Mamba Diffusion,  a novel conditional generation framework that combines the robust generation capabilities of diffusion models with the sequence modeling proficiency of state space models. This innovative approach enables the effective and precise capture and generation of realistic human movements from sparse tracking, thereby greatly enhancing the user experience in interactive multimedia applications. Furthermore, the high-quality human body movements produced by Motion Mamba Diffusion have the potential to enrich artistic and cultural expressions through digital artworks and interactive multimedia installations. In summary, Motion Mamba Diffusion represents a significant advancement in human-computer interaction, multimedia applications, and the intersection of art and culture, aligning closely with various themes of ACM multimedia."},"_bibtex":{"value":"@inproceedings{\ndong2024realistic,\ntitle={Realistic Full-Body Motion Generation from Sparse Tracking with State Space Model},\nauthor={Kun Dong and Jian Xue and Zehai Niu and Xing Lan and Ke Lv and Qingyuan Liu and Xiaoyu Qin},\nbooktitle={ACM Multimedia 2024},\nyear={2024},\nurl={https://openreview.net/forum?id=dA6yat7UNM}\n}"},"title":{"value":"Realistic Full-Body Motion Generation from Sparse Tracking with State Space Model"},"secondary_subject_area":{"value":["[Generation] Generative Multimedia","[Experience] Multimedia Applications","[Experience] Art and Culture"]},"pdf":{"value":"/pdf/4c38630b5b420d223f14e40ed6afcba140e8662a.pdf"},"venueid":{"value":"acmmm.org/ACMMM/2024/Conference"},"paperhash":{"value":"dong|realistic_fullbody_motion_generation_from_sparse_tracking_with_state_space_model"},"primary_subject_area":{"value":"[Experience] Interactions and Quality of Experience"},"authorids":{"value":["~Kun_Dong4","~Jian_Xue2","~Zehai_Niu1","~Xing_Lan1","~Ke_Lv2","~Qingyuan_Liu2","~Xiaoyu_Qin2"]},"authors":{"value":["Kun Dong","Jian Xue","Zehai Niu","Xing Lan","Ke Lv","Qingyuan Liu","Xiaoyu Qin"]}},"tmdate":1721528151110,"pdate":1721513506919,"tcdate":1712285355795,"writers":["acmmm.org/ACMMM/2024/Conference","acmmm.org/ACMMM/2024/Conference/Submission2195/Authors"],"signatures":["acmmm.org/ACMMM/2024/Conference/Submission2195/Authors"],"forum":"dA6yat7UNM","license":"CC BY 4.0","number":2195,"cdate":1712285355795,"readers":["everyone"],"invitations":["acmmm.org/ACMMM/2024/Conference/-/Submission","acmmm.org/ACMMM/2024/Conference/-/Post_Submission","acmmm.org/ACMMM/2024/Conference/Submission2195/-/Revision","acmmm.org/ACMMM/2024/Conference/-/Edit"],"mdate":1721528151110,"odate":1721513506919,"domain":"acmmm.org/ACMMM/2024/Conference","id":"dA6yat7UNM","version":2},{"content":{"summary":{"value":"The paper introduces a customized version of the Mamba framework within multimodal large language models (MLLMs). This framework has three core components: a visual long sequence, a Mamba projector, and a Mamba LLM. Experimental results on various benchmarks suggest improved performance and speed compared to several existing methods."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"See Weakness"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"The framework is concise and clear, making the proposed approach easy to understand."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1):  Limited Novelty in Technical Contribution: While the paper proposes a \"visual long sequence\" as part of the framework, a significant body of literature already exists on augmenting visual features using ensembles of different visual encoders, as demonstrated in works such as [A-D]. The design of the Mamba projector, specifically its cross-stitch scanning scheme that concatenates four scanning paths, seems heuristic rather than theoretically grounded.\n\n2): Unclear Motivation for Mamba Projector: The Mamba projector, the primary technical contribution of this paper, has an unclear motivation. The 1x1 convolutional MLP layer can be treated as a full attention layer, suggesting that the Mamba projector is an approximation. Lines 250–253 argue that \"a simple MLP layer may not be able to accomplish sufficient vision-language alignment and interaction of different visual features. Therefore, we devise a lightweight mamba projector…\" However, this rationale does not sufficiently justify the addition of the Mamba projector.\n\n3): Unfair Experimental Comparisons: For instance, in Table 4, using a longer visual sequence generally increases latency. Models such as TinyLLaVA and MobileVLMv2 should be substituted with the Mamba LLM. In Table 2, MambaVLM shows superior performance, largely attributed to encoder ensembling—a common approach in the literature.\n\n4): Presentation Quality: The paper’s overall clarity and presentation could benefit from further refinement.\n\n\n\nReferences:\n\n[A]: BRAVE: Broadening the Visual Encoding of Vision-Language Models, ArXiv.\n\n[B]: Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, CVPR 2024.\n\n[C]: Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders, ArXiv.\n\n[D]: Law of Vision Representation in MLLMs, ArXiv."}},"nonreaders":[],"tmdate":1731428096571,"tcdate":1730731505393,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4294/Reviewer_MzvA"],"signatures":["ICLR.cc/2025/Conference/Submission4294/Reviewer_MzvA"],"forum":"0A6f1b66pE","number":3,"license":"CC BY 4.0","cdate":1730731505393,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4294/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428096571,"domain":"ICLR.cc/2025/Conference","replyto":"0A6f1b66pE","id":"jYYBBuLfcy","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Vision-Language Models; Mamba;"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"While emerging multi-modal large language models (MLLM) have demonstrated impressive advances, the quadratic complexity of their Transformer-based LLMs (3B or larger) inevitably leads to considerable computational overhead. On the other hand, the recently proposed selective state space model (i.e., Mamba) enjoys both model capacity and computational efficiency, making it an ideal component to enhance MLLM's efficiency and performance. However, recent attempts to introduce Mamba into MLLMs simply replace their LLMs with Mamba, ignoring the unique characteristics of either side. We argue that such a naive combination cannot exhibit the potential of Mamba in MLLMs. In this paper, we delve into harnessing Mamba's unique properties, and propose tailored designs from both multi-modal input and architectural perspectives to unleash its true power. First, we fully utilize Mamba's linear complexity to construct visual long sequences for a thorough perception at a minor efficiency burden. To integrate the scanning mechanism with the built visual long sequence, we devise a novel cross-stitch scanning approach to capture and fuse spatial and semantic properties simultaneously, enhancing the interaction of visual information and the vision-language alignment. Built upon these designs, we propose MambaVLM, a simple yet effective MLLM framework that exhibits highly competitive results across multiple benchmarks. Moreover, our framework is also compatible with Transformer-based LLMs (e.g., Vicuna), demonstrating remarkable training and inference efficiency. Notably, with only 0.66M data and 14 hours training on a single A800 node, our MambaVLM outperforms LLaVA-1.5 by significant margins and performs on par or even better than the 1.4B data trained Qwen-VL. The appealing results from both effectiveness and efficiency aspects indicate the promising prospects of Mamba in MLLMs."},"_bibtex":{"value":"@misc{\nchen2024unleashing,\ntitle={Unleashing the Power of Selective State Space Models in Vision-Language Models},\nauthor={Honghao Chen and Yibing Song and Shoufa Chen and Chongjian GE and Kaiqi Huang},\nyear={2024},\nurl={https://openreview.net/forum?id=0A6f1b66pE}\n}"},"title":{"value":"Unleashing the Power of Selective State Space Models in Vision-Language Models"},"pdf":{"value":"/pdf/1183a88cc6949631a83ff61a73451d323b4592a6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"chen|unleashing_the_power_of_selective_state_space_models_in_visionlanguage_models"},"authorids":{"value":["~Honghao_Chen1","~Yibing_Song1","~Shoufa_Chen1","~Chongjian_GE1","~Kaiqi_Huang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Honghao Chen","Yibing Song","Shoufa Chen","Chongjian GE","Kaiqi Huang"]}},"version":2},{"content":{"summary":{"value":"The authors seek to incorporate domain priors and in-context reasoning into linear RNNs.\nThey show that (i) linear RNNs can act as effective hypernetworks, (ii) these RNNs can engage in in-context learning or physics-informed modeling, and (iii) these hypernetworks exhibit significant improvements over baseline models.\nModels are evaluated on forecasting / reconstruction, classification, and in-context linear regression tasks.\nBaseline models include gated RNNs, linear SSMs (S4), and transformers."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"I suggest the following:\n- Traditional long-range dependency experiments like Long-Range Arena benchmarks.\n- Text prediction and classification experiments to support claims of expressivity.\n- Connections to biology, especially in the main text, should be clarified.\n- As stated by the authors, future work should explore ways to reduce the density of hidden-state transition matrices used in WARP. \nComplex diagonal recurrences may be fruitful in this regard."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":4},"strengths":{"value":"- Fantastically written, very clear.\n- Related work and appendix are thorough.\n- Experiments show clear improvements of the proposed WARP RNN over baseline models, both in accuracy and compute.\n- Authors are clear about limitations, especially the important limitation that the hidden state transition matrix is dense."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The introduction does not (although the related work does) mention selective SSMs (e.g., Mamba), which have re-introduced nonlinearities into linear RNNs.\n- CelebA reconstruction seems a bit contrived as a task: the annotated S4 article referenced does indeed reconstruct MNIST and similar image data, but then proceeds to spoken digit data, which the authors did not try reconstructing.\nMoreover, CelebA face reconstructions are substantially corrupted.\nWhile I do not deny WARP's improvement in image reconstruction over baseline, I am unsure about the salience of image reconstruction as a metric for evaluating RNNs.\nI think there are more relevant tasks to measure long-range dependencies, such as Long-Range Arena benchmarks.\n- There are no experiments on text prediction or classification, which are some of the most relevant tasks for evaluating new RNN architectures - does WARP have limitations or inductive biases that prevent it from expressively processing text?\nIf so, the authors should state it, as this is an important limitation, especially to claims of in-context learning.\nIf not, then the authors should evaluate their model with text to support the expressivity of WARP.\n- WARP's connections to biology are not sufficiently explained in the main text, although the authors do discuss it further in the appendix."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764357964128,"tcdate":1761190017162,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4784/Reviewer_rzHm"],"signatures":["ICLR.cc/2026/Conference/Submission4784/Reviewer_rzHm"],"forum":"zHdKaF3ZM7","number":1,"license":"CC BY 4.0","cdate":1761190017162,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4784/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764357964128,"domain":"ICLR.cc/2026/Conference","replyto":"zHdKaF3ZM7","id":"MIiGSYD6YS","forumContent":{"TLDR":{"value":"We introduce a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["physics-informed machine learning","weight-space learning","meta-learning","deep sequence modeling","linear recurrence","test-time training"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"We introduce WARP (**W**eight-space **A**daptive **R**ecurrent **P**rediction), a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into fixed-dimensional hidden states, WARP explicitly parametrizes its hidden state as the weights and biases of a distinct auxiliary neural network, and uses input differences to drive its recurrence. This brain-inspired formulation enables efficient gradient-free adaptation of the auxiliary network at test-time, in-context learning abilities, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state-of-the-art baselines on diverse classification tasks, featuring in the top three in 4 out of 6 real-world challenging datasets. Furthermore, extensive experiments across sequential image completion, multivariate time series forecasting, and dynamical system reconstruction demonstrate its expressiveness and generalization capabilities. Remarkably, a physics-informed variant of our model outperforms the next best model by more than 10x. Ablation studies confirm the architectural necessity of key components, solidifying weight-space linear RNNs as a transformative paradigm for adaptive machine intelligence."},"_bibtex":{"value":"@inproceedings{\nnzoyem2026weightspace,\ntitle={Weight-Space Linear Recurrent Neural Networks},\nauthor={Roussel Desmond Nzoyem and Nawid Keshtmand and Enrique Crespo-Fernandez and Idriss TSAYEM and Raul Santos-Rodriguez and David A.W. Barton and Tom Deakin},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zHdKaF3ZM7}\n}"},"title":{"value":"Weight-Space Linear Recurrent Neural Networks"},"pdf":{"value":"/pdf/c01f4c1dbdaf1c6e821cde510623f871a07c3404.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"nzoyem|weightspace_linear_recurrent_neural_networks"},"authorids":{"value":["~Roussel_Desmond_Nzoyem1","~Nawid_Keshtmand1","~Enrique_Crespo-Fernandez1","~Idriss_TSAYEM1","~Raul_Santos-Rodriguez1","~David_A.W._Barton1","~Tom_Deakin1"]},"authors":{"value":["Roussel Desmond Nzoyem","Nawid Keshtmand","Enrique Crespo-Fernandez","Idriss TSAYEM","Raul Santos-Rodriguez","David A.W. Barton","Tom Deakin"]}},"version":2},{"content":{"venue":{"value":"CoRR 2015"},"pdf":{"value":"http://arxiv.org/pdf/1506.04917v2"},"venueid":{"value":"dblp.org/journals/CORR/2015"},"paperhash":{"value":"crochemore|lineartime_sequence_comparison_using_minimal_absent_words_applications"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Maxime_Crochemore:","https://dblp.org/search/pid/api?q=author:Gabriele_Fici:","https://dblp.org/search/pid/api?q=author:Robert_Mercas:","~Solon_P._Pissis1"]},"html":{"value":"http://arxiv.org/abs/1506.04917"},"_bibtex":{"value":"@article{DBLP:journals/corr/CrochemoreFMP15,\n  publtype={informal},\n  author={Maxime Crochemore and Gabriele Fici and Robert Mercas and Solon P. Pissis},\n  title={Linear-Time Sequence Comparison Using Minimal Absent Words & Applications},\n  year={2015},\n  cdate={1420070400000},\n  journal={CoRR},\n  volume={abs/1506.04917},\n  url={http://arxiv.org/abs/1506.04917}\n}\n"},"abstract":{"value":"Sequence comparison is a prerequisite to virtually all comparative genomic analyses. It is often realized by sequence alignment techniques, which are computationally expensive. This has led to increased research into alignment-free techniques, which are based on measures referring to the composition of sequences in terms of their constituent patterns. These measures, such as $q$-gram distance, are usually computed in time linear with respect to the length of the sequences. In this article, we focus on the complementary idea: how two sequences can be efficiently compared based on information that does not occur in the sequences. A word is an {\\em absent word} of some sequence if it does not occur in the sequence. An absent word is {\\em minimal} if all its proper factors occur in the sequence. Here we present the first linear-time and linear-space algorithm to compare two sequences by considering {\\em all} their minimal absent words. In the process, we present results of combinatorial interest, and also extend the proposed techniques to compare circular sequences."},"title":{"value":"Linear-Time Sequence Comparison Using Minimal Absent Words & Applications"},"authors":{"value":["Maxime Crochemore","Gabriele Fici","Robert Mercas","Solon P. Pissis"]}},"tmdate":1739212188691,"pdate":1420070400000,"tcdate":1739212091304,"writers":["~"],"signatures":["~Solon_P_Pissis1"],"forum":"E0FlxF8ZBa","license":"CC BY-SA 4.0","number":317097,"cdate":1420070400000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1739212188691,"domain":"DBLP.org","id":"E0FlxF8ZBa","version":2},{"content":{"venue":{"value":"LATIN 2016"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-662-49529-2_25.pdf"},"venueid":{"value":"dblp.org/conf/LATIN/2016"},"paperhash":{"value":"crochemore|lineartime_sequence_comparison_using_minimal_absent_words_applications"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Maxime_Crochemore:","https://dblp.org/search/pid/api?q=author:Gabriele_Fici:","https://dblp.org/search/pid/api?q=author:Robert_Mercas:","~Solon_P._Pissis1"]},"html":{"value":"https://doi.org/10.1007/978-3-662-49529-2_25"},"_bibtex":{"value":"@inproceedings{DBLP:conf/latin/CrochemoreFMP16,\n  author={Maxime Crochemore and Gabriele Fici and Robert Mercas and Solon P. Pissis},\n  title={Linear-Time Sequence Comparison Using Minimal Absent Words & Applications},\n  year={2016},\n  cdate={1451606400000},\n  pages={334-346},\n  url={https://doi.org/10.1007/978-3-662-49529-2_25},\n  booktitle={LATIN},\n  crossref={conf/latin/2016}\n}\n"},"abstract":{"value":"Sequence comparison is a prerequisite to virtually all comparative genomic analyses. It is often realized by sequence alignment techniques, which are computationally expensive. This has led to increased research into alignment-free techniques, which are based on measures referring to the composition of sequences in terms of their constituent patterns. These measures, such as q-gram distance, are usually computed in time linear with respect to the length of the sequences. In this article, we focus on the complementary idea: how two sequences can be efficiently compared based on information that does not occur in the sequences. A word is an absent word of some sequence if it does not occur in the sequence. An absent word is minimal if all its proper factors occur in the sequence. Here we present the first linear-time and linear-space algorithm to compare two sequences by considering all their minimal absent words. In the process, we present results of combinatorial interest, and also extend the proposed techniques to compare circular sequences."},"title":{"value":"Linear-Time Sequence Comparison Using Minimal Absent Words & Applications"},"authors":{"value":["Maxime Crochemore","Gabriele Fici","Robert Mercas","Solon P. Pissis"]}},"tmdate":1739212133653,"pdate":1451606400000,"tcdate":1739212089700,"writers":["~"],"signatures":["~Solon_P_Pissis1"],"forum":"GEP0YVgZSl","license":"CC BY-SA 4.0","number":317039,"cdate":1451606400000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1739212133653,"domain":"DBLP.org","id":"GEP0YVgZSl","version":2},{"content":{"TLDR":{"value":"We propose a 8-bit static quantization method for selective state space models."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["State Space Models","Model quantization"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"State Space Models (SSMs) have emerged as an appealing alternative to Transformers for large language models, achieving state-of-the-art accuracy with constant memory complexity which allows for holding longer context lengths than attention-based networks. The superior computational efficiency of SSMs in long sequence modeling positions them favorably over Transformers in many scenarios. However, improving the efficiency of SSMs on request-intensive cloud-serving and resource-limited edge applications is still a formidable task. SSM quantization is a possible solution to this problem, making SSMs more suitable for wide deployment, while still maintaining their accuracy. Quantization is a common technique to reduce the model size and to utilize the low bit-width acceleration features on modern computing units, yet existing quantization techniques are poorly suited for SSMs. Most notably, SSMs have highly sensitive feature maps within the selective scan mechanism (i.e., linear recurrence) and massive outliers in the output activations which are not present in the output of token-mixing in the self-attention modules. To address this issue, we propose a static 8-bit per-tensor SSM quantization method which suppresses the maximum values of the input activations to the selective SSM for finer quantization precision and quantizes the output activations in an outlier-free space with Hadamard transform. Our 8-bit weight-activation quantized Mamba 2.8B SSM benefits from hardware acceleration and achieves a 1.72 $\\times$ lower generation latency on an Nvidia Orin Nano 8G, with only a 0.9\\% drop in average accuracy on zero-shot tasks. When quantizing Jamba, a 52B parameter SSM-style language model, we observe only a $1\\%$  drop in accuracy, demonstrating that our SSM quantization method is both effective and scalable for large language models, which require appropriate compression techniques for deployment. The experiments demonstrate the effectiveness and practical applicability of our approach for deploying SSM-based models of all sizes on both cloud and edge platforms."},"_bibtex":{"value":"@inproceedings{\nchiang2025quamba,\ntitle={Quamba: A Post-Training Quantization Recipe for Selective State Space Models},\nauthor={Hung-Yueh Chiang and Chi-Chih Chang and Natalia Frumkin and Kai-Chiang Wu and Diana Marculescu},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=mnna9LUg7P}\n}"},"title":{"value":"Quamba: A Post-Training Quantization Recipe for Selective State Space Models"},"pdf":{"value":"/pdf/0c4ea9cb96dd9e16a33f3931fb1a1347d00b5817.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"chiang|quamba_a_posttraining_quantization_recipe_for_selective_state_space_models"},"authorids":{"value":["~Hung-Yueh_Chiang1","~Chi-Chih_Chang1","~Natalia_Frumkin1","~Kai-Chiang_Wu1","~Diana_Marculescu4"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hung-Yueh Chiang","Chi-Chih Chang","Natalia Frumkin","Kai-Chiang Wu","Diana Marculescu"]}},"tmdate":1740884080633,"pdate":1737562754690,"tcdate":1727454526308,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11122/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission11122/Authors"],"forum":"mnna9LUg7P","license":"CC BY 4.0","number":11122,"cdate":1727454526308,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission11122/-/Full_Submission","ICLR.cc/2025/Conference/Submission11122/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit","ICLR.cc/2025/Conference/Submission11122/-/Camera_Ready_Revision"],"mdate":1740884080633,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"mnna9LUg7P","version":2},{"content":{"summary":{"value":"This paper studies the link prediction problem on continuous-time dynamic graphs. The main technique this paper used is the Mamba state space model (SSM)."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please check the questions mentioned in the weaknesses."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"S1. The introduction to the proposed method is detailed and useful for the readers to understand every module and architecture.\nS2. The baseline methods used in this paper is state-of-the-art and recent.\nS3. This paper includes 7 datasets with 3 different negative sampling strategies (NSS) which makes the experimental section comprehensive."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. Seems that the main body of the model is based on a mature model Mamba, e.g., Eqs. 4a-4d, and Eqs. 6a-6d. Looks like there are existing works using Mamba on graphs, e.g, [1], which undermines the novelty of the proposed method.\n\nW2. Some of the model designs are not reasonable or questionable to me. I will detail them as follows.\n\nW2.1 Lines 182-190 introduce the counting of interaction frequency. However, it is still pretty vague why the frequency features are formed like that in lines 184 and 185. E.g., why there are 5 rows in F_u^t, and why both F_u^t and F_v^t have 2 columns. In addition, why the encoding of 2 columns of F_u^t uses a shared MLP (in line 189)? In other words, why not use an MLP: from R^2 to R^{d_f}?\n\nW2.2 Lines 191 and 192 mentioned that it can save computation resources. If I understood correctly, by such a patching, the time complexity to encode the X_u^t will increase because the matrix has more columns (features), but the SSMs module can be faster because there are fewer rows (in lines 198 and 199), right?\n\nW2.3 The section named “Dynamic Information Selection with Temporal Patterns” (line 255) seems to be the core module of the proposed method, which is related to the “edge-specific temporal pattern”. However, I think the edge information has been included in the Node-Level SSM Block (Eq. 3). Then, why repeatedly use the edge information, and what is any rationale regarding this?\n\nW3. Regarding the experimental results, looks like under the “random” negative sampling strategy (NSS) the proposed method performs very well, but under the historical and inductive NSS setting, the proposed method can only beat ~ half of the SOTAs. \n\nW4. The writing of this paper is somewhat sloppy. I detail them as follows.\nW4.1 As some modules of this paper are modified from the standard S4 and Mamba SSM, I checked the dimension and introduction for the backbone model between lines 125 and 140. However, a lot of them are confusing. (i) In Eq 1a, do matrices Az(\\tau) and Bq(\\tau) with the same dimensions? Looks like Bq(\\tau) is of the shape 1 x d1 but the matrix A is of the shape d1 x d1, so the matrix Az(\\tau) is of the shape d1 x (?), which cannot have the same shape as Bq(\\tau). The same problem to r(\\tau), which is a scalar, but from the second eq in Eq 1a, r(\\tau)=Cz(\\tau), and the shape of the matrix C is d1 x 1; in other words, it is not clear whether r(\\tau) is a scalar or vector.\n\nW4.2 In line 136, I think d2 was not mentioned in previous content. Also, why is the model in a SISO fashion when d_2>1? I am not familiar with this backbone model, but this statement is not intuitive to me.\n\nW4.3 The end of Line 161 should be |Nei_u^t| but not Nei_u^t.\n\n[1] Behrouz, Ali, and Farnoosh Hashemi. \"Graph mamba: Towards learning on graphs with state space models.\" Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024."}},"nonreaders":[],"tmdate":1732284474415,"tcdate":1730468483560,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5146/Reviewer_1QFu"],"signatures":["ICLR.cc/2025/Conference/Submission5146/Reviewer_1QFu"],"forum":"f3gCs2a4ZD","number":2,"license":"CC BY 4.0","cdate":1730468483560,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5146/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732284474415,"domain":"ICLR.cc/2025/Conference","replyto":"f3gCs2a4ZD","id":"f4ptq7YFDu","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["dynamic graph","state space model"]},"supplementary_material":{"value":"/attachment/2dc1a3502a1d53b635f3c846cad8277b987ac60b.zip"},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Learning useful representations for continuous-time dynamic graphs (CTDGs) is challenging, due to the concurrent need to span long node interaction histories and grasp nuanced temporal details. In particular, two problems emerge: (1) Encoding longer histories requires more computational resources, making it crucial for CTDG models to maintain low computational complexity to ensure efficiency; (2) Meanwhile, more powerful models are needed to identify and select the most critical temporal information within the extended context provided by longer histories. To address these problems, we propose a CTDG representation learning model named DyGMamba, originating from the popular Mamba state space model (SSM). DyGMamba first leverages a node-level SSM to encode the sequence of historical node interactions. Another time-level SSM is then employed to exploit the temporal patterns hidden in the historical graph, where its output is used to dynamically select the critical information from the interaction history. We validate DyGMamba experimentally on the dynamic link prediction task. The results show that our model achieves state-of-the-art in most cases. DyGMamba also maintains high efficiency in terms of computational resources, making it possible to capture long temporal dependencies with a limited computation budget."},"_bibtex":{"value":"@misc{\nding2025dygmamba,\ntitle={Dy{GM}amba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models},\nauthor={Zifeng Ding and Yifeng Li and Yuan He and Antonio Norelli and Jingcheng Wu and Volker Tresp and Yunpu Ma and Michael M. Bronstein},\nyear={2025},\nurl={https://openreview.net/forum?id=f3gCs2a4ZD}\n}"},"title":{"value":"DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models"},"pdf":{"value":"/pdf/0921c754d017d250e0d2f4b59a779766e47b30a6.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"ding|dygmamba_efficiently_modeling_longterm_temporal_dependency_on_continuoustime_dynamic_graphs_with_state_space_models"},"authorids":{"value":["~Zifeng_Ding1","~Yifeng_Li4","~Yuan_He5","~Antonio_Norelli2","~Jingcheng_Wu3","~Volker_Tresp1","~Yunpu_Ma1","~Michael_M._Bronstein1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zifeng Ding","Yifeng Li","Yuan He","Antonio Norelli","Jingcheng Wu","Volker Tresp","Yunpu Ma","Michael M. Bronstein"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Tramba, a traffic speed forecasting framework that combines a Mamba-based temporal encoder for long-range memory with an adaptive attention module that captures non-local spatial dependencies based on temporal similarity. The model integrates these two signals via a gating mechanism to balance temporal continuity and spatial alignment. Experiments on PEMS-BAY, METR-LA, and Seoul TOPIS show consistent improvements over strong Transformer- and Mamba-based baselines, with ablations highlighting the importance of adaptive attention."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. The paper should provide a clearer motivation for introducing Mamba and explain why it is necessary in this setting. What specific limitations of attention-based models does Mamba address, and how does it complement or outperform attention in this task?\n2. The current experiments compare mainly Transformer- and Mamba-based models. It would strengthen the paper to include more diverse baselines, such as LLM-based and diffusion-based approaches, to better position the method relative to the state of the art. Additionally, the paper notes that some methods model spatial dependencies using fixed adjacency matrices, but it is unclear which baselines fall into this category and which do not. Please clarify these distinctions and discuss how they affect the comparison.\n3. The paper describes the components of the proposed model but lacks sufficient explanation for why each part is designed this way. In particular, the rationale for deriving spatial dependencies from temporal similarity, and for the design of the “adaptive attention” mechanism, should be discussed in more depth."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The attempt to apply a new modeling paradigm like Mamba to traffic speed forecasting is interesting and contributes to methodological diversity in this area.\n2. The release of code and datasets improves reproducibility and supports further research.\n3. The paper is generally well-organized, and the experimental design is clear, making it easy to follow and understand the proposed approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The motivation for introducing Mamba is not clearly articulated, and its role relative to the attention module remains vague. Moreover, the so-called “adaptive attention” relies on similarity across temporal sequences to generate weights, which limits its adaptiveness and makes its ability to capture spatial dependencies questionable. The paper does not provide sufficient justification for this design choice or demonstrate its clear advantages over standard attention mechanisms.\n2. The paper’s claim that existing methods neglect spatial dependencies is inaccurate, as many prior work has already modeled spatial relationships in different flexible ways. The proposed adaptive attention lacks a clear justification for why its approach is more effective, and deriving spatial relations purely from temporal similarity is insufficiently motivated.\n3. The evaluation compares only Transformer- and Mamba-based models, omitting key state-of-the-art approaches such as LLM-based and diffusion-based methods. This narrow selection limits the credibility of the reported improvements and makes it difficult to assess the method’s generality and competitiveness."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764359985498,"tcdate":1760883670197,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16390/Reviewer_yQoX"],"signatures":["ICLR.cc/2026/Conference/Submission16390/Reviewer_yQoX"],"forum":"DjlVwQFRMb","number":1,"license":"CC BY 4.0","cdate":1760883670197,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16390/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764359985498,"domain":"ICLR.cc/2026/Conference","replyto":"DjlVwQFRMb","id":"dDd4C0AVRQ","forumContent":{"TLDR":{"value":"We propose Tramba, a novel time-series forecasting framework that integrates a selective state-space model with an adaptive attention mechanism for enhanced predictive accuracy and dynamic spatial reasoning in complex urban networks."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["traffic speed forecasting","adaptive attention","selective state-space model","time series"]},"primary_area":{"value":"learning on time series and dynamical systems"},"abstract":{"value":"We introduce \\textbf{Tramba}, a novel deep learning model for traffic speed forecasting in complex urban road networks. Unlike conventional methods that rely heavily on short-term trends or local spatial proximity (e.g., upstream and downstream links), Tramba captures dynamic, long-range dependencies across both time and space. It does so by integrating two key components: a Mamba-based temporal encoder that models long-term historical patterns of the target link, and an adaptive attention mechanism that learns temporally similar patterns from non-adjacent road links across the network. We evaluate Tramba on a real-world dataset from Gangnam-gu, Seoul, comprising 5-minute interval speed measurements across 366 road segments. Tramba is tested over forecasting horizons from 1 to 36 steps and compared with six strong baselines. It consistently outperforms all alternatives, achieving an average MAPE of 10.17\\%, MAE of 2.80~km/h, and MSE of 20.50~(km/h)$^2$ on the three datasets for 12-step forecasting. These results highlight Tramba’s ability to model long-range dependencies and detect non-local influences in complex urban networks, reducing prediction lag and improving robustness in dynamic traffic conditions. Code is available at~\\url{https://github.com/tr-anon-users/tramba-code}."},"_bibtex":{"value":"@misc{\nlee2026tramba,\ntitle={Tramba: Mamba with Adaptive Attention for Traffic Speed Forecasting},\nauthor={Eun Hak Lee and Hyunsoo Yun and Zhengzhong Tu},\nyear={2026},\nurl={https://openreview.net/forum?id=DjlVwQFRMb}\n}"},"title":{"value":"Tramba: Mamba with Adaptive Attention for Traffic Speed Forecasting"},"pdf":{"value":"/pdf/3689872be3bd2e79776d2739c2bd1d9e2b46dfa5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"lee|tramba_mamba_with_adaptive_attention_for_traffic_speed_forecasting"},"authorids":{"value":["~Eun_Hak_Lee1","~Hyunsoo_Yun1","~Zhengzhong_Tu1"]},"authors":{"value":["Eun Hak Lee","Hyunsoo Yun","Zhengzhong Tu"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2506.10915v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"huang|m4v_multimodal_mamba_for_texttovideo_generation"},"authorids":{"value":["~Jiancheng_Huang1","https://dblp.org/search/pid/api?q=author:Gengwei_Zhang:","https://dblp.org/search/pid/api?q=author:Zequn_Jie:","https://dblp.org/search/pid/api?q=author:Siyu_Jiao:","https://dblp.org/search/pid/api?q=author:Yinlong_Qian:","~Ling_Chen5","https://dblp.org/search/pid/api?q=author:Yunchao_Wei:","~Lin_Ma2"]},"html":{"value":"https://doi.org/10.48550/arXiv.2506.10915"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2506-10915,\n  publtype={informal},\n  author={Jiancheng Huang and Gengwei Zhang and Zequn Jie and Siyu Jiao and Yinlong Qian and Ling Chen and Yunchao Wei and Lin Ma},\n  title={M4V: Multi-Modal Mamba for Text-to-Video Generation},\n  year={2025},\n  month={June},\n  cdate={1748736000000},\n  journal={CoRR},\n  volume={abs/2506.10915},\n  url={https://doi.org/10.48550/arXiv.2506.10915}\n}\n"},"abstract":{"value":"Text-to-video generation has significantly enriched content creation and holds the potential to evolve into powerful world simulators. However, modeling the vast spatiotemporal space remains computationally demanding, particularly when employing Transformers, which incur quadratic complexity in sequence processing and thus limit practical applications. Recent advancements in linear-time sequence modeling, particularly the Mamba architecture, offer a more efficient alternative. Nevertheless, its plain design limits its direct applicability to multi-modal and spatiotemporal video generation tasks. To address these challenges, we introduce M4V, a Multi-Modal Mamba framework for text-to-video generation. Specifically, we propose a multi-modal diffusion Mamba (MM-DiM) block that enables seamless integration of multi-modal information and spatiotemporal modeling through a multi-modal token re-composition design. As a result, the Mamba blocks in M4V reduce FLOPs by 45% compared to the attention-based alternative when generating videos at 768$\\times$1280 resolution. Additionally, to mitigate the visual quality degradation in long-context autoregressive generation processes, we introduce a reward learning strategy that further enhances per-frame visual realism. Extensive experiments on text-to-video benchmarks demonstrate M4V's ability to produce high-quality videos while significantly lowering computational costs. Code and models will be publicly available at https://huangjch526.github.io/M4V_project."},"title":{"value":"M4V: Multi-Modal Mamba for Text-to-Video Generation"},"authors":{"value":["Jiancheng Huang","Gengwei Zhang","Zequn Jie","Siyu Jiao","Yinlong Qian","Ling Chen","Yunchao Wei","Lin Ma"]}},"tmdate":1768965092533,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2506-10915"],"tcdate":1758819603850,"writers":["~"],"signatures":["~Jiancheng_Huang1"],"forum":"vMRBRJXJCH","license":"CC BY-SA 4.0","number":629423,"cdate":1748736000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1768965092533,"domain":"DBLP.org","id":"vMRBRJXJCH","version":2},{"content":{"summary":{"value":"This paper presents MCO-E Net, a multimodal framework for event-assisted single-eye expression recognition that integrates RGB and event data through two novel modules: MCO-Mamba, which enables cross-modal joint optimization and semantic interaction based on the Mamba architecture, and HCE-MoE, a heterogeneous mixture-of-experts module combining deep, attention, and focal experts via an attention-guided router. The proposed model achieves state-of-the-art results on SEE and DSEE datasets, demonstrating strong robustness under challenging lighting conditions and fast inference speed."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1.How does the joint optimization function M(A,B) ensure stable gradients and avoid dominance by one modality?\n2.What is the computational overhead distribution among the three expert types in HCE-MoE? Any observed load imbalance or expert collapse?\n3.Have you tested the model’s transferability to other multimodal tasks (e.g., facial emotion recognition, driver monitoring)?\n4.Are training hyperparameters and hardware configurations fully reproducible from the released code repository?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"1. The combination of Mamba and heterogeneous MoE for multimodal feature fusion is creative and technically interesting.\n2. Experimental results are strong, showing consistent improvements over state-of-the-art methods across all lighting conditions with very low inference latency.\n3. The motivation is clear and addresses a real challenge of semantic misalignment between RGB and event modalities.\n4. Ablation studies are comprehensive, verifying the contribution of each component and parameter choice."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Although the model is complex from an engineering perspective, there is limited theoretical analysis of the convergence and dynamic equilibrium mechanisms of the MCO-Mamba joint optimization.\n2. Mamba-based multimodal fusion shares similarities with recent approaches (e.g., Sigma, MSFMamba, DepMamba, 2024–2025), where the innovations primarily arise from module combinations rather than fundamentally new principles.\n3. The validation is limited to two similar monocular datasets, without cross-dataset or real-world application testing, which makes it difficult to assess the generalization capability.\n4. Although the tables are comprehensive, statistical significance metrics (e.g., standard deviation or p-value) are not reported, suggesting that some improvements may fall within the noise range.\n5. Writing and notation clarity could be improved, with some overly long sections and insufficient explanation of key equations."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918963935,"tcdate":1761800849305,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6649/Reviewer_iv73"],"signatures":["ICLR.cc/2026/Conference/Submission6649/Reviewer_iv73"],"forum":"l0MXmQnMMv","number":2,"license":"CC BY 4.0","cdate":1761800849305,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6649/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918963935,"domain":"ICLR.cc/2026/Conference","replyto":"l0MXmQnMMv","id":"hBewEVDIBz","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Expression Recognition Tasks; Event-based  Vision Task; Cross-modal Fusion; Event Camera"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"In this paper, we proposed a Multi-modal Collaborative Optimization and Expansion Network (MCO-E Net), to use event modalities to resist challenges such as low light, high exposure, and high dynamic range in single-eye expression recognition tasks. The MCO-E Net introduces two innovative designs: Multi-modal Collaborative Optimization Mamba (MCO-Mamba) and Heterogeneous Collaborative and Expansion Mixture-of-Experts (HCE-MoE). MCO-Mamba, building upon Mamba, leverages dual-modal information to jointly optimize the model, facilitating collaborative interaction and fusion of modal semantics. This approach encourages the model to balance the learning of both modalities and harness their respective strengths. HCE-MoE, on the other hand, employs a dynamic routing mechanism to distribute structurally varied experts (deep, attention, and focal), fostering collaborative learning of complementary semantics. This heterogeneous architecture systematically integrates diverse feature extraction paradigms to comprehensively capture expression semantics. Extensive experiments demonstrate that our proposed network achieves competitive performance in the task of single-eye expression recognition, especially under poor lighting conditions."},"_bibtex":{"value":"@misc{\nhan2025multimodal,\ntitle={Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognition},\nauthor={Runduo Han and Hongchen Tan and Xiuping Liu},\nyear={2025},\nurl={https://openreview.net/forum?id=l0MXmQnMMv}\n}"},"title":{"value":"Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognition"},"pdf":{"value":"/pdf/79a6421036806a007dc4d662f5739d972802784d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"han|multimodal_collaborative_optimization_and_expansion_network_for_eventassisted_singleeye_expression_recognition"},"authorids":{"value":["~Runduo_Han1","~Hongchen_Tan1","~Xiuping_Liu1"]},"authors":{"value":["Runduo Han","Hongchen Tan","Xiuping Liu"]}},"version":2},{"content":{"summary":{"value":"The paper introduces MANAR, a drop-in replacement for MHA that combines local token attention with a retrieved, fixed-size Abstract Conceptual Representation (ACR) from a trainable external memory. The goal is to mitigate the quadratic time/space cost of standard attention. Reported results include up to 14.8× latency and 9.3× peak-memory reductions at 4k tokens, “linear” scaling for fixed windows, ImageNet-1K top-1 = 82.3% (DeiT-B capacity), and LibriSpeech WER 2.9/6.8.\n\nAlthough MANAR sounds promissing, the empirical evidence is not convincing. Experiments were conducted only on two tasks, including ImageNet classification and speech recognition. Missing comparison with other baselines e.g., sparse transformers, Linformer, or other linear attention methods. Experiments in the language domain would substantially strengthen the claim."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"- How is the accuracy vs latency curve across different values of $L$?  How does the number of retrieved memory concepts affect the performance?\n- The speedups rely on a custom fused Triton kernel. There is no ablation quantifying how much of the gain comes from kernel engineering rather than from the MANAR design itself. \n- The modified paper layout appears to create extra space; please adhere to the venue’s formatting rules."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The idea is clearly present: a unification of retrieved global context (ACR) with local attention to avoid all-pairs attention.\n- Efficiency: Substantial wall-clock and HBM savings in microbenchmarks and end-to-end DeiT-S at large resolutions, with improvements growing with sequence length.\n- MANAR enables quick adoption and a large reduction in trainable parameters/steps while retaining accuracy on vision and speech."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Modest accuracy gains:** On ImageNet-1K, improvements over DeiT-B are small (82.3% vs. 81.8%). For ASR, the paper claims SOTA, but test-clean 2.9 trails data2vec (2.8) and test-other is tied at 6.8.\n- **Related work gaps:** While Linformer/Performer and long-sequence families (Mamba/RetNet, KV-cache management) are cited, several key lines are missing or under-discussed: sparse attention baselines, Swin/local-window ViTs, Transformer-XL/Compressive Transformer, standard retrieval-augmented modeling for LMs, and Memory Transformer.\n- **Lack of empirical experiments:** See questions below."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919574211,"tcdate":1761812725653,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7461/Reviewer_6pDP"],"signatures":["ICLR.cc/2026/Conference/Submission7461/Reviewer_6pDP"],"forum":"9e9wHUkC8Q","number":1,"license":"CC BY 4.0","cdate":1761812725653,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7461/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919574211,"domain":"ICLR.cc/2026/Conference","replyto":"9e9wHUkC8Q","id":"uMwSctM0l8","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["brain inspired","memory-augmented architecture","attention","out-of-the-box thinking","imagenet","librispeech","quadratic complexity"]},"supplementary_material":{"value":"/attachment/b499e46c9b45ebe11b398880ad3ec4a736cb07ca.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Transformers - and their multi-head attention (MHA) core - power today's leading models across a broad application spectrum. Yet MHA contextualizes each token through explicit, pair-wise interactions with every other token, yielding quadratic time/space cost and an unbounded, linearly-growing context. This \\textit{direct all-to-all} modeling is both the source of attention's expressiveness and a barrier to scaling. We address this bottleneck by augmenting attention with a trainable external memory that stores both conceptual and relational general representations learned during training. For every input, lightweight scalable retrieval produces a fixed-size set of memory retrieved concepts whose values are fused into a compact Abstract Conceptual Representation (ACR). Tokens then attend jointly to (i) the global, concept-level ACR and (ii) a short local context, completely sidestepping all-to-all token interactions. The result is a non-convex pathway that provide the model with an \\textit{out-of-the-box thinking} contextualization - i.e., beyond the convex hull spanned by the input values - while reducing complexity to linear time and memory. Integrated as a drop-in replacement for MHA, our layer (MANAR) preserves accuracy on ImageNet-1K (82.3\\% top-1 with a DeIT-B backbone) and LibriSpeech (2.9/6.8\\% WER on test-clean/other) yet cuts inference latency by up to 14.8x and peak GPU memory by up to 9.3x as sequence length grows to 4K. A simple weight-copy knowledge-transfer procedure trims training cost by $\\approx$99\\% versus training from scratch. Finally, Convex Hull Membership (CHM) tests show that $>$50\\% of MANAR’s outputs lie outside the convex span of the input values, quantitatively confirming its out-of-the-box contextualization."},"_bibtex":{"value":"@misc{\njahshan2026manar,\ntitle={{MANAR}: Memory-augmented Attention with Navigational Abstract conceptual Representation},\nauthor={Zuher Jahshan and Ben Ben-Ishay and Leonid Yavits},\nyear={2026},\nurl={https://openreview.net/forum?id=9e9wHUkC8Q}\n}"},"title":{"value":"MANAR: Memory-augmented Attention with Navigational Abstract conceptual Representation"},"pdf":{"value":"/pdf/63ea5e4ceb3595103636cc7984d57c2115ee47b1.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"jahshan|manar_memoryaugmented_attention_with_navigational_abstract_conceptual_representation"},"authorids":{"value":["~Zuher_Jahshan1","~Ben_Ben-Ishay1","~Leonid_Yavits1"]},"authors":{"value":["Zuher Jahshan","Ben Ben-Ishay","Leonid Yavits"]}},"version":2},{"content":{"summary":{"value":"The authors presents a theoretical framework for modeling the latent dynamics of a neural network during sequence learning. A generalized Langevin equation is derived via time-dependent projections, and based on this, the neural wave architecture is introduced, having Mori-Zwanzig dynamics. This model demonstrates robust long-range recall, minimal memory dimension, and interpretable latent modes across synthetic and neuroscience datasets.  In particular, the work addresses a timely and important direction, given the growing interest in dynamical representation and the need for more interpretable machine learning models."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Is there a clear case where the MZ dynamics provide a practical performance benefit?\n\nSimilarly, is there a clear case where the MZ dynamics provides better interpretability?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The derivations are clear, and the problem is well motivated.\n\nMathematically, the difference between the derived GLE with the FFE-MZ is well explained\n\nThe long-range copy and selective copy tasks show improved results, especially when compared to other SSM architectures (such as Mamba) and transformers.  \n\nThe paper is well written and discusses limitations and potential future extensions of the work."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"When compared to other frameworks implementing MZ dynamics, it is not shown whether there is a practical performance benefit or how this architecture exhibits better interpretability.  \n\nWhile theoretically this framework has a lot of potential, the authors have not demonstrated in a clear way how this model contrasts to architectures having a state-dependent kernel other than requiring a marginally smaller latent state. \n\nChaotic time-series forecasting is mentioned in the abstract, but it is not shown in the main paper."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762941885065,"tcdate":1762517461983,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21675/Reviewer_eT8y"],"signatures":["ICLR.cc/2026/Conference/Submission21675/Reviewer_eT8y"],"forum":"RanDLuKUQT","number":4,"license":"CC BY 4.0","cdate":1762517461983,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission21675/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762941885065,"domain":"ICLR.cc/2026/Conference","replyto":"RanDLuKUQT","id":"MXKKnyuX2R","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We show that coupled dynamics between memory encoding and decoding give rise to emergent dynamical modes."},"keywords":{"value":["deep learning","traveling waves","neural oscillators","mori-zwanzig formalism","projection operator formalism"]},"supplementary_material":{"value":"/attachment/d5ebbfca9b9e19eec223602813c1ebcbbe55a784.zip"},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Memory in biological neural networks is often supported by coherent spatiotemporal patterns, such as traveling waves and neural activity confined to low dimensional manifolds, which are captured at mesoscopic scales by continuum neural field models. Substantial progress has been made in mechanistically analyzing both biological and artificial neural network architectures. Recent works obtain interpretable latent states by imposing traveling waves or low-dimensional invariant manifolds, but typically do not provide data-driven explanations for when and why such structures emerge during task training. We develop a theoretical framework for studying latent dynamics based on the Mori-Zwanzig projection-operator formalism. Our approach casts memory as a family of time-dependent projections that reveal how coupled dynamics support memory encoding and decoding. We instantiate a neural-field-inspired architecture, and evaluate it on both long-range benchmarks and neuroscience applications involving EEG and ECoG prediction. Across these tasks, we observe robust long-range accuracy and interpretable memory modes in the learned latent dynamics."},"_bibtex":{"value":"@misc{\nbaker2026on,\ntitle={On the Dynamics of Coherent Memory Structures in Neural Fields},\nauthor={Justin M. Baker and Harris Hardiman-Mostow and Yuer Tang and Andrea L. Bertozzi},\nyear={2026},\nurl={https://openreview.net/forum?id=RanDLuKUQT}\n}"},"title":{"value":"On the Dynamics of Coherent Memory Structures in Neural Fields"},"pdf":{"value":"/pdf/7585584225d9e167a5015b1e35c210d7391dcd03.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"baker|on_the_dynamics_of_coherent_memory_structures_in_neural_fields"},"authorids":{"value":["~Justin_M._Baker1","~Harris_Hardiman-Mostow1","~Yuer_Tang1","~Andrea_L._Bertozzi2"]},"authors":{"value":["Justin M. Baker","Harris Hardiman-Mostow","Yuer Tang","Andrea L. Bertozzi"]}},"version":2},{"content":{"decision":{"value":"Accept (spotlight)"},"comment":{"value":"This paper introduces VMamba, a new vision backbone model inspired by the Mamba state-space model, designed for efficient visual representation learning with linear computational complexity. The key innovation is the VSS block, which incorporates a 2D-Selective-Scan (SS2D) module, adapting the 1D selective scan of Mamba for 2D image data. Experiments show that VMamba outperforms existing models in image classification, object detection, instance segmentation, and semantic segmentation, demonstrating superior accuracy, scalability, and adaptability. All reviewers are satisfied with the authors' responses."},"title":{"value":"Paper Decision"}},"nonreaders":[],"tmdate":1730885517515,"tcdate":1727277632186,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Program_Chairs"],"signatures":["NeurIPS.cc/2024/Conference/Program_Chairs"],"forum":"ZgtLQQR1K7","number":1,"license":"CC BY 4.0","cdate":1727277632186,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission3213/-/Decision","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730885517515,"domain":"NeurIPS.cc/2024/Conference","replyto":"ZgtLQQR1K7","id":"WMNNY8QMpl","forumContent":{"TLDR":{"value":"This paper introduces a new architecture with promising performance on visual perception tasks, which is based on mamba."},"venue":{"value":"NeurIPS 2024 spotlight"},"keywords":{"value":["State Space Model","Transformer","Computer Vision","Foundation Model"]},"supplementary_material":{"value":"/attachment/550be5322feb4a54bbb9b3794247a515d062212f.zip"},"primary_area":{"value":"machine_vision"},"abstract":{"value":"Designing computationally efficient network architectures remains an ongoing necessity in computer vision. In this paper, we adapt Mamba, a state-space language model, into VMamba, a vision backbone with linear time complexity. At the core of VMamba is a stack of Visual State-Space (VSS) blocks with the 2D Selective Scan (SS2D) module. By traversing along four scanning routes, SS2D bridges the gap between the ordered nature of 1D selective scan and the non-sequential structure of 2D vision data, which facilitates the collection of contextual information from various sources and perspectives. Based on the VSS blocks, we develop a family of VMamba architectures and accelerate them through a succession of architectural and implementation enhancements. Extensive experiments demonstrate VMamba’s\n promising performance across diverse visual perception tasks, highlighting its superior input scaling efficiency compared to existing benchmark models. Source code is available at https://github.com/MzeroMiko/VMamba"},"_bibtex":{"value":"@inproceedings{\nliu2024vmamba,\ntitle={{VM}amba: Visual State Space Model},\nauthor={Yue Liu and Yunjie Tian and Yuzhong Zhao and Hongtian Yu and Lingxi Xie and Yaowei Wang and Qixiang Ye and Jianbin Jiao and Yunfan Liu},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=ZgtLQQR1K7}\n}"},"title":{"value":"VMamba: Visual State Space Model"},"pdf":{"value":"/pdf/496ccac4131e6ee0bc83b0ca167de274006e7b1a.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"liu|vmamba_visual_state_space_model"},"authorids":{"value":["~Yue_Liu19","~Yunjie_Tian1","~Yuzhong_Zhao1","~Hongtian_Yu1","~Lingxi_Xie1","~Yaowei_Wang1","~Qixiang_Ye1","~Jianbin_Jiao1","~Yunfan_Liu3"]},"authors":{"value":["Yue Liu","Yunjie Tian","Yuzhong Zhao","Hongtian Yu","Lingxi Xie","Yaowei Wang","Qixiang Ye","Jianbin Jiao","Yunfan Liu"]}},"version":2},{"content":{"summary":{"value":"To address the missing values in time series datasets, this work considers the imputation-free time series modeling. MCAR and periodic masking are considered in experiments by comparing with two stage baselines, consisting of imputation and modeling approaches."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"W1-W5"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"S1. Various imputation baselines are considered in experiments.\n\nS2. Time series modeling is important.\n\nS3. Time series forecasting and classification are considered in experiments."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. The introduction is not well-written, which seems more like the related work. It is suggested to further polish the manuscript, to improve the readability.\n\nW2. In the Introduction, the authors claim that all existing works for time-series modeling with missing values are two stage approaches, where missing values are first imputed, followed by feeding the imputed time-series to a modeling approach. Unfortunately, there already exists existing study [m1] considering the robust time series modeling over low quality data, where missing values and anomalies are associated with low importance in modeling, instead of treated as a two stage framework. Therefore, the motivation is unpersuasive, claiming that all the existing works are two stage approaches.\n\nW3. In practice, in addition to the missing values, there also exist various data quality problems, e.g., anomaly, errors, staleness, etc. Only focusing modeling the time series data with missing values is insufficient to serve real applications. Please discuss how the approach might be extended or adapted to handle other types of data quality problems, or explain why you chose to focus specifically on missing values.\n\nW4. MCAR and periodic masking are widely used operations in time series imputation, which are surprising to be proposed by this manuscript, at the end of the Introduction.\n\nW5. Various imputation methods are considered in the experiments, but the most related works, i.e., robust modeling methods such as [m1], are totally ignored.\n\nW6. The SOTA imputation baselines are ignored in the comparison, such as [m2]. Please include the comparison with it.\n\nW7. In addition to the MCAR and periodic masking, MAR and MNAR are also widely considered in the experiments.\n\n[m1] Cheng, Hao, et al. \"RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies.\" The Twelfth International Conference on Learning Representations.\n\n[m2] Tashiro, Yusuke, et al. \"Csdi: Conditional score-based diffusion models for probabilistic time series imputation.\" Advances in Neural Information Processing Systems 34 (2021): 24804-24816."}},"nonreaders":[],"tmdate":1731428703893,"tcdate":1730457925576,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12015/Reviewer_G7Mn"],"signatures":["ICLR.cc/2025/Conference/Submission12015/Reviewer_G7Mn"],"forum":"XBtDrlK1Qc","number":2,"license":"CC BY 4.0","cdate":1730457925576,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12015/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428703893,"domain":"ICLR.cc/2025/Conference","replyto":"XBtDrlK1Qc","id":"AfK6yUUc1P","forumContent":{"TLDR":{"value":"We propose a novel imputation-free approach of handling missing values in time series that can be trained in an end-to-end manner."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Time-Series","Deep Learning","Masked Autoencoders","Missing Data","Data Imputation"]},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Modeling time series is important in a variety of domains, yet it is  challenged by the presence of missing values in real-world time-series datasets. Traditional frameworks for modeling time-series with missing values typically involve a two-step process, where the missing values are first filled-in using some imputation technique, followed by a time-series modeling approach on the imputed time-series. However, existing two-stage approaches suffer from two major drawbacks: first, the propagation of imputation errors into subsequent time-series modeling performance, and second, the inherent trade-offs between imputation efficacy and imputation complexity. To this end, we propose a novel imputation-free approach for handling missing values in time series termed {Miss}ing Feature-aware {T}ime {S}eries {M}odeling ({MissTSM}) with two main innovations. {First}, we develop a novel embedding scheme that treats every combination of time-step and feature (or channel) as a distinct token, encoding them into a high-dimensional space. {Second}, we introduce a novel {Missing Feature-Aware Attention (MFAA) Layer} to learn latent representations at every time-step based on partially observed features. We evaluate the effectiveness of MissTSM  in handling missing values over multiple benchmark datasets using two synthetic masking techniques: masking completely at random (MCAR) and periodic masking, and a real-world missing-value dataset."},"_bibtex":{"value":"@misc{\nneog2025masking,\ntitle={Masking the Gaps: An Imputation-Free Approach to Time Series Modeling with Missing Data},\nauthor={Abhilash Neog and Arka Daw and Sepideh Fatemi Khorasgani and Aanish Pradhan and Mary E. Lofton and Bennett J. McAfee and Adrienne Breef-Pilz and Heather L. Wander and Dexter W Howard and Cayelan Carey and Paul Hanson and Anuj Karpatne},\nyear={2025},\nurl={https://openreview.net/forum?id=XBtDrlK1Qc}\n}"},"title":{"value":"Masking the Gaps: An Imputation-Free Approach to Time Series Modeling with Missing Data"},"pdf":{"value":"/pdf/5c4fcabcd0c8c3e8007b520506272684802e7bca.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"neog|masking_the_gaps_an_imputationfree_approach_to_time_series_modeling_with_missing_data"},"authorids":{"value":["~Abhilash_Neog1","~Arka_Daw1","~Sepideh_Fatemi_Khorasgani1","~Aanish_Pradhan1","~Mary_E._Lofton1","~Bennett_J._McAfee1","~Adrienne_Breef-Pilz1","~Heather_L._Wander1","~Dexter_W_Howard1","~Cayelan_Carey1","~Paul_Hanson1","~Anuj_Karpatne1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Abhilash Neog","Arka Daw","Sepideh Fatemi Khorasgani","Aanish Pradhan","Mary E. Lofton","Bennett J. McAfee","Adrienne Breef-Pilz","Heather L. Wander","Dexter W Howard","Cayelan Carey","Paul Hanson","Anuj Karpatne"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a memory-efficient and computationally light approach for 3D medical image segmentation, built upon a hybrid design that fuses convolutional layers with Structured State Space Models (SSMs), specifically Mamba-based blocks. By leveraging a Tri-oriented Mamba (ToM) mechanism and convolution mamba mixed blocks, the architecture captures both local and long-range dependencies in volumetric scans. It manages to substantially reduce memory consumption and computational load while preserving robust performance on multiple datasets (e.g., BraTS, BTCV, and Spleen)."},"final_rating":{"value":3},"justification_of_the_preliminary_rating":{"value":"This paper introduces a memory-efficient and computationally light approach for 3D medical image segmentation, built upon a hybrid design that fuses convolutional layers with Structured State Space Models (SSMs), specifically Mamba-based blocks.  However, due to the weakness mentioned, I recommend Broadline."},"strengths":{"value":"1.\tThe balanced integration of convolutional layers and Mamba blocks, which clearly addresses the local and global feature extraction issues in 3D volumes.\n2.\tSystematically evaluating on multiple datasets, including small-scale (Spleen) and more comprehensive ones (BTCV, BraTS), the authors convincingly show that SegResMamba maintains near state-of-the-art performance with significantly reduced GPU memory usage and overall computational cost."},"weaknesses":{"value":"1.\tThe paper does not compare its results with some state-of-the-art (SOTA) baselines, such as nnFormer or UNETR++.\n2.\tThe paper should also report parameters, FLOPs, and inference speed to demonstrate the model’s effectiveness in reducing computational complexity, memory usage, and training time.\n3.\tAlthough the authors mention the model’s environmental advantages, they do not provide details on fine-grained hyperparameter tuning or resource usage (e.g., training on multi-GPU setups or CPU-only scenarios)."},"confidence":{"value":5},"detailed_comments":{"value":"1.\tSee the weakness\n2.\tAdditional ablation on the number of Mamba blocks or the tri-oriented Mamba design (e.g., comparing forward-only vs. tri-oriented usage) could confirm the necessity of each orientation.\n3.\tSince Mamba-based blocks are relatively new, sharing code snippets or best practices would accelerate adoption."},"special_issue":{"value":"No"},"recommendation":{"value":["Poster"]},"questions_to_address_in_the_rebuttal":{"value":"1.\tCould you provide more details on what minor architectural or hyperparameter changes might close the small accuracy gap with SwinUNETR while preserving SegResMamba’s efficiency?\n2.\tPlease include comparisons with UNETR++ and nnFormer.\n3.\tReport the parameters, FLOPs, and inference speed of the models."},"justification_of_the_final_rating":{"value":"The authors do compare their method to some state-of-the-art (SOTA) approaches, but they do not include all of them (e.g., UNETR++). My main concern is that the proposed method does not offer significant improvements in either efficiency or effectiveness. Therefore, I maintain my borderline rating."},"preliminary_rating":{"value":3}},"nonreaders":[],"tmdate":1747661857550,"tcdate":1740113548521,"writers":["MIDL.io/2025/Conference","MIDL.io/2025/Conference/Submission111/Reviewer_29W9"],"signatures":["MIDL.io/2025/Conference/Submission111/Reviewer_29W9"],"forum":"zQLrITbcxJ","number":2,"license":"CC BY 4.0","cdate":1740113548521,"readers":["everyone"],"invitations":["MIDL.io/2025/Conference/Submission111/-/Official_Review","MIDL.io/2025/Conference/-/Edit","MIDL.io/2025/Conference/Submission111/Official_Review2/-/Review_Revision"],"mdate":1747661857550,"domain":"MIDL.io/2025/Conference","replyto":"zQLrITbcxJ","id":"0u3aopJYVj","forumContent":{"venue":{"value":"MIDL 2025 Poster"},"TLDR":{"value":"We propose SegResMamba, a memory and computation efficient Mamba-based model for 3D medical image segmentation."},"midl_latex_submission_checklist":{"value":["The paper compiles correctly using the pdflatex compiler.","Replace NNN with your OpenReview submission ID.","The LaTeX file includes the correct header commands before \\title.","Hyperref is preloaded; do not modify it.","If included, must contact program chairs immediately.","Format names correctly and avoid Unicode characters.","Title page, citations, repositories, data sources, and acknowledgements must reflect correct authorship.","Did not use \\begin{thebibliography} directly to insert references.","Avoid using \\scalebox; prefer \\resizebox{\\textwidth}{!}{...}.","Zip archive should only contain needed files.","Clean final version without visual artifacts.","Ensure proper LaTeX encoding of characters.","No separate supplementary PDF uploads.","Check page limit and content structure carefully."]},"keywords":{"value":["Mamba","Vision Transformer","Medical Image Segmentation"]},"abstract":{"value":"The Transformer architecture has opened a new paradigm in the domain of deep learning with its ability to model long-range dependencies and capture global context and has outpaced the traditional Convolution Neural Networks (CNNs) in many aspects. However, applying Transformer models to 3D medical image datasets presents significant challenges due to their high training time, and memory requirements, which not only hinder scalability but also contribute to elevated CO$_2$ footprint. This has led to an exploration of alternative models that can maintain or even improve performance while being more efficient and environmentally sustainable. Recent advancements in Structured State Space Models (SSMs) effectively address some of the inherent limitations of Transformers, particularly their high memory and computational demands. Inspired by these advancements, we propose an efficient 3D segmentation model for medical imaging called SegResMamba, designed to reduce computation complexity, memory usage, training time, and environmental impact while maintaining high performance. Our model uses less than half the memory during training compared to other state-of-the-art (SOTA) architectures, achieving comparable performance with significantly reduced resource demands."},"_bibtex":{"value":"@inproceedings{\ndas2025segresmamba,\ntitle={SegResMamba: An Efficient Architecture for 3D Medical Image Segmentation},\nauthor={Badhan Kumar Das and Ajay Singh and Saahil Islam and Gengyan Zhao and Andreas Maier},\nbooktitle={Medical Imaging with Deep Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=zQLrITbcxJ}\n}"},"title":{"value":"SegResMamba: An Efficient Architecture for 3D Medical Image Segmentation"},"latex_code":{"value":"/attachment/1043e16d50767cda6d590118fc2a95ceee7a7116.zip"},"paper_type":{"value":"Both"},"secondary_subject_area":{"value":"Application: Radiology"},"pdf":{"value":"/pdf/4020ee59db972b1ed9c95ffbfb6627b5d541ec0f.pdf"},"copyright_form":{"value":"/attachment/2b1823dee1afac163a808d46d40caab8a696c638.pdf"},"visa":{"value":"Yes"},"venueid":{"value":"MIDL.io/2025/Conference"},"paperhash":{"value":"das|segresmamba_an_efficient_architecture_for_3d_medical_image_segmentation"},"primary_subject_area":{"value":"Segmentation"},"authorids":{"value":["~Badhan_Kumar_Das1","ajay.s.singh@fau.de","~Saahil_Islam1","~Gengyan_Zhao1","~Andreas_Maier1"]},"registration":{"value":"Yes"},"authors":{"value":["Badhan Kumar Das","Ajay Singh","Saahil Islam","Gengyan Zhao","Andreas Maier"]}},"version":2},{"content":{"summary":{"value":"The work builds on Mamba state space models (SSMs) in vision tasks. State space models have been proposed as sequence models that can improve the limitations of quadratic complexity in self-attention by leveraging a framework that allows for parallelism in training, with sequential predictions in autoregressive tasks. In the field of vision, research on SSMs for vision has focused on remediating a central limitation of sequential modelling of spatial data; optimal ways for aligning data in a flattened two-way message passing setup. \n\nThe authors bring about an important point; given that spatial data have less of an inherent ordering than general sequence modelling tasks, it is not clear how a global receptive field can be achieved as handily as in vision transformers (ViTs). For transformers, a bidirectional approach simplifies to a global view where all tokens interact in parallel per self-attention block. Contrarily, bidirectionality in more classic sequence models still implies two-way message passing in a flattened sequence. Hence, work on SSMs for vision have rediscovered classic scan orders over patches to determine an appropriate extension to modelling spatial data as sequences. Currently, SSMs use more layers with lower throughput to achieve typically slightly worse results than ViTs.\n\nThe authors go on to propose an approach to provide global context for SSMs in vision by introducing tokenization by a pyramidal structure of low-pass filtered tokens from the top down. While the paper touches on a central modelling component in SSMs for vision, the authors end up modelling the pyramidal tokens at each level with classic raster scan ordering, referred to as “causal tokens”. The resulting method ends up being better, however, it is very unclear if this is due to the arguments in the paper. Instead, the gains could feasibly come from having many more tokens in the sequence, which generally improves performance, but increases complexity. **In other words, showing that the gains come from the ordering of low-pass filtered images rather than the feature pyramid with increased sequence length becomes imperative to demonstrate**, given the central argument provided by the authors. \n\n**In summary**, it seems that the tokenization method has benefits to other Mamba-based approaches in vision tasks. However, in this reviewer's opinion, the current draft does not adequately delineate the source of these performance gains. As it stands, it seems possible that gains can be attributed to more tokens, rather than a causal ordering of frequency bands in the image."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Q1: A central issue with the current paper is linking the approach to the argument of \"causal tokens\". Did you ablate the effect of the low-pass DCT filtering process during your experiments? In other words, can you show that this is the source of the gains in the model? Given the pyramidal structure, this reviewer struggles to find any particular reason why simply downscaling patches in the pyramid without applying a low-pass filter should fail to achieve exactly the same results.\n\nQ2: Given the extensive history of the techniques employed by the authors, are there any reason to omit references to these, or are the authors unaware of the overlap with classical image modelling approaches? In particular, Gaussian pyramids (Adelson et al. 1984) have a long history that is easily contextualised. The approach of sequentially scanning frequencies diagonally starting from the top right corner seems to draw inspiration from classic DCT compression regimes, such as JPEG (Wallace, 1992). \n\nQ3: Why is DEiT chosen as an appropriate baseline for comparison instead of the more recent DEiTv3? Notably, DEiT uses distillation from a convolutional model, while DEiTv3 is a pure transformer architecture.\n\nQ4: Can you provide examples of throughput (images / sec) for your models compared to baselines?"},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"S1: The method provides quite significantly stronger results compared to other Mamba-based SSNs, with results for both image level and dense prediction tasks and models for various model capacities.\n\nS2: Tokenization is an important and often overlooked component in both transformers and SSMs for vision tasks, and the approach has merit as a pyramidal tokenizer for both modelling paradigms (SSNs and ViTs). This reviewer appreciates the author's contribution to this field.\n\nS3: The approach utilises tried and true techniques in image processing, even if the manuscript in its current form does not reflect the extensive history in the background of the proposed method.\n\nS4: While the approach adds more tokens to the sequence, this does lean into the proposed benefits of longer sequences that SSNs can provide."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1: In contrast to how the method is motivated, the method does not subdivide the image into frequency bands, but instead employs a low-pass image pyramid. This has the effect of increasing the overall number of tokens in the sequence compared to other baselines, which is not mentioned in the paper. In the end, the tokenization method mimics classic Gaussian pyramids (which are not cited or referenced in the work).\n\nW2: While the authors argue that the approach provides  “causal tokens”, the grounding in causality is somewhat opaque. While it could be argued that the tokenization approach is more “global” through the pyramidal structure, it still ends up relying on a scan order in each level, which is arbitrary. \n\nW3: It is not clear that frequency subsampling in the DCT domain plays a significant role in the proposed pyramid structure, and thus the arguments on causality in the frequency domain are substantially weaker. Instead, the increase in performance seems more likely to stem from including many more tokens in the sequence.\n\nW4: It must be noted that there is a degree of ahistoricity in the presentation of the proposed method. The paper makes little effort to place their work in the context of the classical methods they end up employing.  \n\n**More minor weaknesses**:\n\nW5: While the overall FLOPs of the method is provided, there is no mention of throughput, which could be a central factor for modellers looking to employ the proposed method. Lower FLOPs do not always translate to faster inference, particularly in the case of Mamba models.\n\nW6: The paper spends an inordinate amount of space outlining the fundamentals of the DCT, which should be clear to any reader with any tangentially relevant background in image processing. This could be moved to the appendix or supplementary material.\n\nW7: The approach relies on somewhat older ViT baselines. While it is appropriate to compare against models that are exclusively trained on ImageNet1k, there are stronger supervised models which would provide a more fair comparison to the proposed model available.\n\nW8: The paper has a few typos and missing references to figures. This is a very minor point and does not impact the review or score."}},"nonreaders":[],"tmdate":1731427717846,"tcdate":1730322127221,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission3036/Reviewer_zaAz"],"signatures":["ICLR.cc/2025/Conference/Submission3036/Reviewer_zaAz"],"forum":"XKQ2qzajbU","number":1,"license":"CC BY 4.0","cdate":1730322127221,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission3036/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427717846,"domain":"ICLR.cc/2025/Conference","replyto":"XKQ2qzajbU","id":"hhoADdkciB","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["GlobalMamba","Representation Learning","Frequency Domain"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Vision mambas have demonstrated strong performance with linear complexity to the number of vision tokens. Their efficiency results from processing image tokens sequentially. However, most existing methods employ patch-based image tokenization and then flatten them into 1D sequences for causal processing, which ignore the intrinsic 2D structural correlations of images. It is also difficult to extract global information by sequential processing of local patches. In this paper, we propose a global image serialization method to transform the image into a sequence of causal tokens, which contain global information of the 2D image. We first convert the image from the spatial domain to the frequency domain using Discrete Cosine Transform (DCT) and then arrange the pixels with corresponding frequency ranges. We further transform each set within the same frequency band back to the spatial domain to obtain a series of images before tokenization. We construct a vision mamba model with causal input format, GlobalMamba, based on the proposed global image serialization, which can better exploit the causal relations among image sequences. Extensive experiments demonstrate the effectiveness of our GlobalMamba, including image classification on ImageNet-1K, object detection on COCO, and semantic segmentation on ADE20K."},"_bibtex":{"value":"@misc{\nwang2024globalmamba,\ntitle={GlobalMamba: Global Image Serialization for Vision Mamba},\nauthor={Chengkun Wang and Wenzhao Zheng and Jie Zhou and Jiwen Lu},\nyear={2024},\nurl={https://openreview.net/forum?id=XKQ2qzajbU}\n}"},"title":{"value":"GlobalMamba: Global Image Serialization for Vision Mamba"},"pdf":{"value":"/pdf/95cb7db5ab7fbf4df0c2b2683b1c65af9205e545.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"wang|globalmamba_global_image_serialization_for_vision_mamba"},"authorids":{"value":["~Chengkun_Wang1","~Wenzhao_Zheng1","~Jie_Zhou3","~Jiwen_Lu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chengkun Wang","Wenzhao Zheng","Jie Zhou","Jiwen Lu"]}},"version":2},{"content":{"summary":{"value":"This paper introduces ODEBRAIN, a framework for modeling dynamic brain networks from multi-channel EEG. Unlike existing work modeling EEG dynamics in discrete-time manner, the method aims to learn the continuous-time representation of brain state evolution by leveraging Neural Ordinary Differential Equation (NODE). To address the challenges of applying NODEs to EEG, the authors propose a dual-encoder architecture to obtain robust initialization for ODE solver, combining spectral graph features with stochastic temporal signals. Then a novel objective function is proposed to capture underlying EEG dynamics. Experiments on two seizure detection datasets demonstrate model effectiveness."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"- Can we always assume that the data supports that f_theta is continuous and differentiable function? Some nonlinearity of EEG may not support this assumption.\n- Several Technical details of the proposed method are not clearly stated / explained. How do the the temporal descriptor \\Psi and the objective \\Omega are defined? How the pooling of latent continuous trajectory z_t for downstream task is conducted?\n- The paragraph “RQ3 concerns consistency in the graphs…” seems confusing. From Fig.4 I cannot interpret the similarity scores or similarity matrices from discrete or continuous predictor, and other discussion. It seems the figure misaligned with the content.\n- Hyperparameter selection is not discussed. Why the paper uses top 3 correlation neighbors for each node when constructing the graph? How about the effectiveness of different GNN architectures?\n- The complexity analysis for the model is missing. How is the training/inference time of ODEBRAIN compared to other baselines.\n- What is the motivation for objective loss to predict graph structure rather than EEG signal forecasting? The proposed loss shows effectiveness for the prediction task, how it would perform on the typical EEG signal forecasting?\n- The discussion of prior work on modeling EEG signals, capturing their nonlinearity for a variety of brain activity mining [1][3][4][8][14], modeling and control for analyzing EEG signals in the context of brain machine interfaces and epileptic seizures [5][6][12][13][15], modeling latent variables in EEG [7][11] and other brain activity data [9][10], as well as related papers using graph ODE for EEG data [1] are all missing. This needs improvement in both discussing existing methods and comparing against these methods."},"rating":{"value":4},"details_of_ethics_concerns":{"value":"No comments."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The paper addresses an important problem that of modeling brain network in continuous time (with NODE).  \n- The novel contributions mainly come from the proposed dual-encoder architecture and objective loss. \n- The learned dynamic field demonstrate potential for qualitative analyses of brain networks.\n- The proposed method shows some improvement, especially on TUSZ."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Several Technical details of the proposed method are not clearly stated / explained. How do the the temporal descriptor \\Psi and the objective \\Omega are defined? How the pooling of latent continuous trajectory z_t for downstream task is conducted?\n- The paragraph “RQ3 concerns consistency in the graphs…” seems confusing. From Fig.4 I cannot interpret the similarity scores or similarity matrices from discrete or continuous predictor, and other discussion. It seems the figure misaligned with the content.\n- Hyperparameter selection is not discussed. Why the paper uses top 3 correlation neighbors for each node when constructing the graph? How about the effectiveness of different GNN architectures?\n- The complexity analysis for the model is missing. How is the training/inference time of ODEBRAIN compared to other baselines.\n- What is the motivation for objective loss to predict graph structure rather than EEG signal forecasting? The proposed loss shows effectiveness for the prediction task, how it would perform on the typical EEG signal forecasting?\n- The discussion of prior work on modeling EEG signals, capturing their nonlinearity for a variety of brain activity mining [1][3][4][8][14], modeling and control for analyzing EEG signals in the context of brain machine interfaces and epileptic seizures [5][6][12][13][15], modeling latent variables in EEG [7][11] and other brain activity data [9][10], as well as related papers using graph ODE for EEG data [1] are all missing. This needs improvement in both discussing existing methods and comparing against these methods.\n[1] JPM. Pijn et al., \"Nonlinear dynamics of epileptic seizures on basis of intracranial EEG recordings.\" Brain topography 9, no. 4 (1997): 249-270.\n[2] Y. Chen et al., \"EEG emotion recognition based on ordinary differential equation graph convolutional networks and dynamic time wrapping.\" Applied Soft Computing 152 (2024): 111181.\n[3] Y. Xue et al., \"Minimum number of sensors to ensure observability of physiological systems: A case study.\" In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1181-1188. IEEE, 2016.\n[4] K. Lehnertz et al. \"Seizure prediction by nonlinear EEG analysis.\" IEEE Engineering in Medicine and Biology Magazine 22, no. 1 (2003): 57-63.\n[5] G. Gupta et al., \"Re-thinking EEG-based non-invasive brain interfaces: Modeling and analysis.\" In 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS), pp. 275-286. IEEE, 2018.\n[6] V. Tzoumas et al., \"Selecting sensors in biological fractional-order systems.\" IEEE Transactions on Control of Network Systems 5, no. 2 (2018): 709-721.\n[7] G. Gupta et al., \"Learning latent fractional dynamics with unknown unknowns.\" In 2019 American Control Conference (ACC), pp. 217-222. IEEE, 2019.\n[8] K. Lehnertz, \"Epilepsy and nonlinear dynamics.\" Journal of biological physics 34, no. 3 (2008): 253-266.\n[9] R. Yang et al., \"Data-driven perception of neuron point process with unknown unknowns.\" In Proceedings of the 10th ACM/IEEE International Conference on Cyber-Physical Systems, pp. 259-269. 2019\n[10] R. Yang et al., \"Spiking dynamics of individual neurons reflect changes in the structure and function of neuronal networks.\" Nature Communications 16, no. 1 (2025): 6994\n[11] G. Gupta et al., \"Dealing with unknown unknowns: Identification and selection of minimal sensing for fractional dynamics with unknown inputs.\" In 2018 Annual American Control Conference (ACC), pp. 2814-2820. IEEE, 2018.\n[12] X. Lu, \"Detection and classification of epileptic EEG signals by the methods of nonlinear dynamics.\" Chaos, Solitons & Fractals 151 (2021): 111032.\n[13] R. Martis et al. \"Epileptic EEG classification using nonlinear parameters on different frequency bands.\" Journal of Mechanics in Medicine and Biology 15, no. 03 (2015): 1550040.\n[14] M. Mercier et al. \"The value of linear and non-linear quantitative EEG analysis in paediatric epilepsy surgery: a machine learning approach.\" Scientific reports 14, no. 1 (2024): 10887.\n[15] G. Lepeu et al. \"The critical dynamics of hippocampal seizures.\" Nature communications 15, no. 1 (2024): 6945.\nClaiming that the novelty of this approach is that it captures the connectivity of spatio-temporal graphs to capture nonstationary changes is incorrect as can be seen from existing works that do just that.\n- The paper states that the proposed approach “combines deterministic graph-based features with stochastic EEG representations to produce a robust initial state” but it is unclear how the stochastic EEG is accounted for.\n- Can we always assume that the data supports that f_theta is continuous and differentiable function? Some nonlinearity of EEG may not support this assumption.\n- The parameters N, d and T are barely mentioned in passing in section 4.2 but how they are selected in the case study and some guidelines to select them are missing."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915819801,"tcdate":1762015707572,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1575/Reviewer_GEpk"],"signatures":["ICLR.cc/2026/Conference/Submission1575/Reviewer_GEpk"],"forum":"9EjZGs8ube","number":3,"license":"CC BY 4.0","cdate":1762015707572,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission1575/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915819801,"domain":"ICLR.cc/2026/Conference","replyto":"9EjZGs8ube","id":"2V910sIyss","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["EEG","ODE","Brain network","continuous dynamics"]},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Modeling neural population dynamics is crucial for foundational neuroscientific research and various clinical applications. Conventional latent variable methods typically model continuous brain dynamics through discretizing time with recurrent architecture, which necessarily results in compounded cumulative prediction errors and failure of capturing instantaneous, nonlinear characteristics of EEGs. We propose ODEBrain, a Neural ODE latent dynamic forecasting framework to overcome these challenges by integrating spatio-temporal-frequency features into spectral graph nodes, followed by a Neural ODE modeling the continuous latent dynamics. Our design ensures that the latent representations can capture stochastic variations of complex brain states at any given time point. Extensive experiments verify that ODEBrain can improve significantly over existing methods in forecasting EEG dynamics with enhanced robustness and generalization capabilities."},"_bibtex":{"value":"@inproceedings{\njia2026odebrain,\ntitle={{ODEB}rain: Continuous-Time {EEG} Graph for Modeling Dynamic Brain Networks},\nauthor={Haohui Jia and Zheng Chen and Lingwei Zhu and Rikuto Kotoge and Jathurshan Pradeepkumar and Yasuko Matsubara and Jimeng Sun and Yasushi Sakurai and Takashi Matsubara},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=9EjZGs8ube}\n}"},"title":{"value":"ODEBrain: Continuous-Time EEG Graph for Modeling Dynamic Brain Networks"},"pdf":{"value":"/pdf/5bcec84aad7f3fd4ee53c884a2ca0b00a7a9a97b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jia|odebrain_continuoustime_eeg_graph_for_modeling_dynamic_brain_networks"},"authorids":{"value":["~Haohui_Jia2","~Zheng_Chen9","~Lingwei_Zhu1","~Rikuto_Kotoge1","~Jathurshan_Pradeepkumar1","~Yasuko_Matsubara2","~Jimeng_Sun3","~Yasushi_Sakurai1","~Takashi_Matsubara1"]},"authors":{"value":["Haohui Jia","Zheng Chen","Lingwei Zhu","Rikuto Kotoge","Jathurshan Pradeepkumar","Yasuko Matsubara","Jimeng Sun","Yasushi Sakurai","Takashi Matsubara"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2504.10449v3"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"wang|m1_towards_scalable_testtime_compute_with_mamba_reasoning_models"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Junxiong_Wang:","https://dblp.org/search/pid/api?q=author:Wen-Ding_Li:","https://dblp.org/search/pid/api?q=author:Daniele_Paliotta:","https://dblp.org/search/pid/api?q=author:Daniel_Ritter_0007:","~Alexander_M._Rush1","https://dblp.org/search/pid/api?q=author:Tri_Dao:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2504.10449"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2504-10449,\n  publtype={informal},\n  author={Junxiong Wang and Wen-Ding Li and Daniele Paliotta and Daniel Ritter and Alexander M. Rush and Tri Dao},\n  title={M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models},\n  year={2025},\n  month={April},\n  cdate={1743465600000},\n  journal={CoRR},\n  volume={abs/2504.10449},\n  url={https://doi.org/10.48550/arXiv.2504.10449}\n}\n"},"abstract":{"value":"Effective reasoning is crucial to solving complex mathematical problems. Recent large language models (LLMs) have boosted performance by scaling test-time computation through long chain-of-thought reasoning. However, transformer-based models are inherently limited in extending context length due to their quadratic computational complexity and linear memory requirements. In this paper, we introduce a novel hybrid linear RNN reasoning model, M1, built on the Mamba architecture, which allows memory-efficient inference. Our approach leverages a distillation process from existing reasoning models and is further enhanced through RL training. Experimental results on the AIME and MATH benchmarks show that M1 not only outperforms previous linear RNN models but also matches the performance of state-of-the-art Deepseek R1 distilled reasoning models at a similar scale. We also compare our generation speed with a highly performant general purpose inference engine, vLLM, and observe more than a 3x speedup compared to a same size transformer. With throughput speedup, we are able to achieve higher accuracy compared to DeepSeek R1 distilled transformer reasoning models under a fixed generation time budget using self-consistency voting. Overall, we introduce a hybrid Mamba reasoning model and provide a more effective approach to scaling test-time generation using self-consistency or long chain of thought reasoning."},"title":{"value":"M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models"},"authors":{"value":["Junxiong Wang","Wen-Ding Li","Daniele Paliotta","Daniel Ritter","Alexander M. Rush","Tri Dao"]}},"tmdate":1769967456855,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2504-10449"],"tcdate":1769967437141,"writers":["~"],"signatures":["~Alexander_M_Rush1"],"forum":"s2qdBAsD5s","license":"CC BY-SA 4.0","number":817279,"cdate":1743465600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769967456855,"domain":"DBLP.org","id":"s2qdBAsD5s","version":2},{"content":{"TLDR":{"value":"A hybrid linear RNN achieves same quality as R1 distilled reasoning models and performs better in test time scaling"},"venue":{"value":"NeurIPS 2025 ER Workshop Oral"},"pdf":{"value":"/pdf/36f974de8874cd1136c79597e73c73cbf35f56c3.pdf"},"keywords":{"value":["Mamba","Reasoning","Test time scaling"]},"venueid":{"value":"NeurIPS.cc/2025/Workshop/ER"},"paperhash":{"value":"wang|m1_towards_scalable_testtime_compute_with_mamba_reasoning_models"},"authorids":{"value":["~Junxiong_Wang1","~Wen-Ding_Li1","~Daniele_Paliotta1","~Daniel_Ritter2","~Alexander_M_Rush1","~Tri_Dao1"]},"abstract":{"value":"Effective reasoning is crucial to solving complex mathematical problems.Recent large language models (LLMs) have boosted performance by scaling test-time computation through long chain-of-thought reasoning. However, transformer-based models are inherently limited in extending context length due to their quadratic computational complexity and linear memory requirements. In this paper, we introduce a novel hybrid linear RNN reasoning model, M1, built on the Mamba architecture, which allows memory-efficient inference. Our approach leverages a distillation process from existing reasoning models and is further enhanced through RL training. Experimental results on the AIME and MATH benchmarks show that M1 not only outperforms previous linear RNN models but also matches the performance of state-of-the-art Deepseek R1 distilled reasoning models at a similar scale. We also compare our generation speed with a highly performant general purpose inference engine, vLLM, and observe more than a 3x speedup compared to the same-sized transformer. With throughput speedup, we are able to achieve higher accuracy compared to DeepSeek R1 distilled transformer reasoning models under a fixed generation time budget using self-consistency voting. Overall, we introduce the hybrid Mamba reasoning model and provide a more effective approach to scaling test-time generation using self-consistency or long chain of thought reasoning."},"_bibtex":{"value":"@inproceedings{\nwang2025m,\ntitle={M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models},\nauthor={Junxiong Wang and Wen-Ding Li and Daniele Paliotta and Daniel Ritter and Alexander M Rush and Tri Dao},\nbooktitle={NeurIPS 2025 Workshop on Efficient Reasoning},\nyear={2025},\nurl={https://openreview.net/forum?id=bOnhqVefxk}\n}"},"title":{"value":"M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models"},"authors":{"value":["Junxiong Wang","Wen-Ding Li","Daniele Paliotta","Daniel Ritter","Alexander M Rush","Tri Dao"]}},"tmdate":1765083721437,"pdate":1760647614556,"tcdate":1757393235284,"writers":["NeurIPS.cc/2025/Workshop/ER","NeurIPS.cc/2025/Workshop/ER/Submission148/Authors"],"signatures":["NeurIPS.cc/2025/Workshop/ER/Submission148/Authors"],"forum":"bOnhqVefxk","license":"CC BY 4.0","number":148,"cdate":1757393235284,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Workshop/ER/-/Submission","NeurIPS.cc/2025/Workshop/ER/-/Post_Submission","NeurIPS.cc/2025/Workshop/ER/-/Edit","NeurIPS.cc/2025/Workshop/ER/Submission148/-/Revision"],"mdate":1765083721437,"odate":1762800846769,"domain":"NeurIPS.cc/2025/Workshop/ER","id":"bOnhqVefxk","version":2},{"content":{"comment":{"value":"We sincerely appreciate the reviewer’s thoughtful feedback and would like to address the concerns raised with the following clarifications:\n\n## 1. Component Selection Rationale\n\nWe appreciate the reviewer's detailed feedback regarding the justification of our component integration. We respectfully disagree that the integration lacks rigorous justification, and we would like to clarify several key points. We recognize that time series data often contain diverse anomalies stemming from different causes, making it impractical to identify all anomalies in a single step or immediately recognize normal patterns across an entire sequence. Our architecture addresses this through a progressive detection approach, where KAN efficiently identifies physically implausible anomalies first, followed by refined detection through attention and MAMBA to capture more subtle deviations. This gradual approach enables more reliable reconstruction of normal patterns. Each component in our architecture serves distinct yet complementary roles: KAN functions as a rapid filter for physically implausible anomalies by enforcing data consistency through learnable univariate functions with minimal parameters; attention mechanisms capture global dependencies and long-range correlations crucial for detecting pattern deviations across the entire sequence; and MAMBA addresses the challenges of long sequence modeling and distribution shifts through its selective state space modeling, particularly important as anomalies may manifest differently across different time periods.\n\nThe selection of these specific components was driven by concrete challenges in time series anomaly detection. KAN was chosen for its unique ability to establish stable functional relationships while reducing parameters, making it ideal for rapid physical constraint checking (demonstrated in Table 3). Attention mechanisms were selected for their proven effectiveness in capturing global dependencies (supported by ablation results in Table 5). MAMBA was incorporated for its efficient handling of long sequences and distribution shifts, crucial for adapting to varying anomaly patterns over time (validated through comparative experiments in Table 4).\n\nWhile we acknowledge that visualization alone is not sufficient justification, our architectural choices are supported by comprehensive evidence including extensive ablation studies across 8 datasets (Tables 4-5), comparative analysis of different component orderings (Table 10), detailed performance metrics showing consistent improvements across multiple datasets, and statistical significance of the improvements (5% increase in F1 score). Each component addresses specific challenges: KAN efficiently handles the initial screening of physically implausible anomalies with minimal parameters, attention mechanisms resolve the challenge of capturing long-term dependencies, and MAMBA specifically addresses the dual challenges of long sequence modeling and distribution shift adaptation.\n\nThrough this comprehensive integration and extensive experimental validation, we demonstrate that our architecture effectively addresses the multifaceted challenges of time series anomaly detection, with each component playing a crucial and well-justified role in the overall system's superior performance."}},"tmdate":1732783456918,"tcdate":1732783456918,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4490/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission4490/Authors"],"forum":"XmnTfSX5Az","number":13,"license":"CC BY 4.0","cdate":1732783456918,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4490/-/Official_Comment"],"mdate":1732783456918,"domain":"ICLR.cc/2025/Conference","replyto":"FTN90l37AR","id":"jAvMN80lwp","forumContent":{"TLDR":{"value":"We introduce KambaAD, a framework combining MAMBA and KAN to enhance time series anomaly detection, outperforming state-of-the-art models across multiple datasets."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Anomaly Detection","Mamba","KAN","Time series"]},"supplementary_material":{"value":"/attachment/a7877f9569c65354bff9efad6c6e06a7477733f9.zip"},"primary_area":{"value":"learning on time series and dynamical systems"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Time series anomaly detection is critical in numerous practical applications, yet existing deep learning methods often fall short of real-world demands. These models fail to swiftly filter out physically implausible anomalies, insufficiently address distributional shifts, and lack a comprehensive approach that integrates both global and local perspectives for anomaly detection. Moreover, most successful models rely on channel-dependent methods that tend to treat all features at the same timestamp as a single token and then focus on finding relationships between these tokens. This approach overlooks the unique periodicities, trends, and lagged relationships between different features, leading to suboptimal performance. To address these limitations, we propose KambaAD, a model comprised of an Encoder and Reconstructor. The Encoder integrates the strengths of the Kolmogorov-Arnold Network (KAN), the attention mechanism, and the Selective Structured State Space Model (MAMBA). Specifically, KAN is employed to swiftly enforce data consistency, enabling rapid detection of anomalies that violate physical laws. The attention mechanism ensures balanced processing of global information while enhancing the representation of key data characteristics. We leverage MAMBA's capability as a sequence model to capture anomalies caused by local variations. Additionally, its internal selection mechanism allows the model to effectively handle distribution shifts, ensuring robustness and adaptability in the presence of changing data distributions. Additionally, the framework incorporates a time-series-specific Reconstructor, which reduces computational complexity through patch-based operations that exploit local consistency in time series data. It also employs channel-independent linear reconstruction to prevent interference between different features. Through extensive experiments on multiple multivariate datasets, KambaAD consistently outperforms state-of-the-art models, demonstrating its superior performance in anomaly detection."},"_bibtex":{"value":"@misc{\nchen2025kambaad,\ntitle={Kamba{AD}: Enhancing State Space Models with Kolmogorov{\\textendash}Arnold for time series Anomaly Detection},\nauthor={Yiyun Chen and Jia Guo and Xinchun Yu and Defu Cao and Xiao-Ping Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=XmnTfSX5Az}\n}"},"title":{"value":"KambaAD: Enhancing State Space Models with Kolmogorov–Arnold for time series Anomaly Detection"},"pdf":{"value":"/pdf/91e241263b9d771776b2d96ac63471b17016dc63.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chen|kambaad_enhancing_state_space_models_with_kolmogorovarnold_for_time_series_anomaly_detection"},"authorids":{"value":["~Yiyun_Chen2","~Jia_Guo9","~Xinchun_Yu1","~Defu_Cao1","~Xiao-Ping_Zhang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yiyun Chen","Jia Guo","Xinchun Yu","Defu Cao","Xiao-Ping Zhang"]}},"version":2},{"content":{"summary":{"value":"This manuscript describes a modification to vision state space models. To improve the processing speed of mamba-based vision models, the authors proposed to 1) auxilliary patch swapping for bidirectional information flow, and 2) batch folding with periodic state reset. Experiments shows positive results on these components."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"+ In the manuscript of MambaVision, they utilized the same hardware, but they reported a throughput of  3670 img/s for MambaVision-B. However, in this manuscript, the speed is downgraded to 2974 img/s. Why? If jittering exists, please report mean and std over multiple test runs.\n+  Why Mamba kernel requires 32 parallel threads? If current mamba kernel is not suitable for short sequence of image classification, why do we need parallel scan?  Why do we need mamba? The performance drops severly in Tab.3 if attention are removed.\n+ In Table 3, why the inference troughput drop when attention is removed? If that so, the inference scheme may not be appropriate. Please report performance under large batch size, say, 2048 as in SHViT and EfficientViT.\n+  Can the proposed method be applied to other mamba-based model, say, Vim, VMamba and their variants?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"+ The manuscript is well-presented and easy to follow.\n+ It is good to see analysis on the inference speed of the mamba based model.\n+ The reset trick is interesting."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"+ The proposed method is only testifed on MambaVision. It should be applied to more mamba-based models to support the claim of \"Rethinking State Space Model for Vision\".\n+ Swapping last token is not equivant to bi-directional scan, and the author failed to prove the superiority of swapping last token. As in table 3, if the attention is removed, swapping last token worse than parallel bi-scan and even series bi-scan. Since attention itself is a undirectional operation, this seems that switching to swapping last token is not working but attention works."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917964351,"tcdate":1761895167126,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5237/Reviewer_MDas"],"signatures":["ICLR.cc/2026/Conference/Submission5237/Reviewer_MDas"],"forum":"X4KsowemNB","number":4,"license":"CC BY 4.0","cdate":1761895167126,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5237/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917964351,"domain":"ICLR.cc/2026/Conference","replyto":"X4KsowemNB","id":"XQo0O4l2VT","forumContent":{"TLDR":{"value":"A high-performance, high-speed vision encoder based on improved information flow and GPU parallelism"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mamba","State space model","vision encoder"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"The realm of Mamba for vision has been advanced in recent years to strike for the alternatives of Vision Transformers (ViTs) that suffer from the quadratic complexity.\nWhile the recurrent scanning mechanism of Mamba offers computational efficiency, it inherently limits non-causal interactions between image patches.\nPrior works have attempted to address this limitation through various multi-scan strategies; however, these approaches suffer from inefficiencies due to suboptimal scan designs and frequent data rearrangement. Moreover, Mamba exhibits relatively slow computational speed under short token lengths, commonly used in visual tasks.\nIn pursuit of a truly efficient vision encoder, we rethink the scan operation for vision and the computational efficiency of Mamba.\nTo this end, we propose SF-Mamba, a novel visual Mamba with two key proposals: auxiliary patch swapping for encoding bidirectional information flow under an unidirectional scan and batch folding with periodic state reset for advanced GPU parallelism.\nExtensive experiments on image classification, object detection, and instance and semantic segmentation consistently demonstrate that our proposed SF-Mamba significantly outperforms state-of-the-art baselines while improving throughput across different model sizes.\nWe will release the source code after publication."},"_bibtex":{"value":"@misc{\nyoshimura2026sfmamba,\ntitle={{SF}-Mamba: Rethinking State Space Model for Vision},\nauthor={Masakazu Yoshimura and Teruaki Hayashi and Yuki Hoshino and Wei-Yao Wang and Takeshi Ohashi},\nyear={2026},\nurl={https://openreview.net/forum?id=X4KsowemNB}\n}"},"title":{"value":"SF-Mamba: Rethinking State Space Model for Vision"},"pdf":{"value":"/pdf/9683eabb7692d6dc64ba96b03dee4200513e80d4.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yoshimura|sfmamba_rethinking_state_space_model_for_vision"},"authorids":{"value":["~Masakazu_Yoshimura1","~Teruaki_Hayashi1","~Yuki_Hoshino1","~Wei-Yao_Wang1","~Takeshi_Ohashi1"]},"authors":{"value":["Masakazu Yoshimura","Teruaki Hayashi","Yuki Hoshino","Wei-Yao Wang","Takeshi Ohashi"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2412.08114v4"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"fujiwara|modeling_latent_nonlinear_dynamical_system_over_time_series"},"authorids":{"value":["~Ren_Fujiwara1","https://dblp.org/search/pid/api?q=author:Yasuko_Matsubara:","https://dblp.org/search/pid/api?q=author:Yasushi_Sakurai:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2412.08114"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-08114,\n  publtype={informal},\n  author={Ren Fujiwara and Yasuko Matsubara and Yasushi Sakurai},\n  title={Modeling Latent Non-Linear Dynamical System over Time Series},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2412.08114},\n  url={https://doi.org/10.48550/arXiv.2412.08114}\n}\n"},"abstract":{"value":"We study the problem of modeling a non-linear dynamical system when given a time series by deriving equations directly from the data. Despite the fact that time series data are given as input, models for dynamics and estimation algorithms that incorporate long-term temporal dependencies are largely absent from existing studies. In this paper, we introduce a latent state to allow time-dependent modeling and formulate this problem as a dynamics estimation problem in latent states. We face multiple technical challenges, including (1) modeling latent non-linear dynamics and (2) solving circular dependencies caused by the presence of latent states. To tackle these challenging problems, we propose a new method, Latent Non-Linear equation modeling (LaNoLem), that can model a latent non-linear dynamical system and a novel alternating minimization algorithm for effectively estimating latent states and model parameters. In addition, we introduce criteria to control model complexity without human intervention. Compared with the state-of-the-art model, LaNoLem achieves competitive performance for estimating dynamics while outperforming other methods in prediction."},"title":{"value":"Modeling Latent Non-Linear Dynamical System over Time Series"},"authors":{"value":["Ren Fujiwara","Yasuko Matsubara","Yasushi Sakurai"]}},"tmdate":1747123391452,"pdate":1704067200000,"tcdate":1747123387485,"writers":["~"],"signatures":["~Ren_Fujiwara1"],"forum":"vVNCdVRj40","license":"CC BY-SA 4.0","number":441513,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1747123391452,"domain":"DBLP.org","id":"vVNCdVRj40","version":2}],"count":10000}