IVERI-HYPERION: An Entropy-Driven, Byte-Level State-Space Architecture with Associative Memory for Sub-Quadratic Sequence Modeling

Abstract

Transformer-based sequence models rely on self-attention, whose computational cost grows quadratically with sequence length and whose autoregressive key-value cache grows linearly. This paper introduces IVERI-HYPERION, an Entropy-Driven Neural State Network (ENSN) that combines a diagonal selective state-space model, decayed matrix-valued associative memory, a causal convolutional entropy predictor, and raw-byte input representations. We also implement a fused Triton kernel for the selective state-space recurrence and evaluate its performance against a sequential PyTorch reference. In the evaluated 200-step raw-byte TinyStories benchmark (9.985M parameters, n = 3 seeds), IVERI-HYPERION achieves a mean validation loss of 1.2102 ± 0.0168 nats/byte vs. 3.1286 ± 0.0299 for Dense Transformer and 1.2462 ± 0.0017 for Mamba+FFN. The difference from Dense Transformer is statistically significant, whereas the difference from Mamba+FFN is not (paired p = 0.069, Welch p = 0.064). The fused kernel reduces isolated forward execution time from 14.88 ms to 0.60 ms over sequential PyTorch (an isolated 24.8× kernel speedup, delivering 1,367 tokens/sec training throughput vs. 318 unfused); this isolated kernel speedup does not establish an overall training-speed advantage over optimized Transformers (2,840 tokens/sec). During 500 measured decoding steps, the recurrent runtime buffer remained approximately 0.30 MB, excluding static model weights, with no net growth in the measured allocation. However, zero-shot synthetic passkey retrieval failed, and the primary multi-seed evaluation covers only a short training horizon. These results support further investigation of state-space and associative-memory hybrids but do not yet establish broad architectural superiority or reliable long-context retrieval.

Introduction

The text introduces IVERI-HYPERION, an Entropy-Driven Neural State Network (ENSN) designed to overcome the computational, memory, and efficiency limitations of traditional Transformer-based language models, particularly in small-scale and resource-constrained environments.

Transformers have become the foundation of modern natural language processing because their self-attention mechanism captures relationships between tokens effectively and supports parallel computation. However, traditional self-attention has significant limitations, including quadratic computational complexity with sequence length, increasing memory requirements for key-value (KV) caches, uniform computation across all tokens, and large vocabulary embedding tables. Alternative architectures, such as linear attention and state-space models, improve efficiency but may struggle to retain detailed information for complex retrieval and long-context tasks.

To address these challenges, the proposed IVERI-HYPERION architecture combines four major components:

1. Raw-byte processing: The model processes text directly as bytes using a compact vocabulary of 259 tokens. This reduces embedding parameters and avoids dependence on large subword vocabularies.

2. Selective State Space Model (S4D): This component tracks temporal information and sequence progression efficiently without calculating pairwise attention scores.

3. Associative Matrix Memory: Inspired by the Titans architecture, this component stores and retrieves key-value associations using a decayed matrix-based memory, helping preserve information needed for contextual recall.

4. Entropy-driven computation: A causal convolutional predictor estimates the information content of incoming tokens and dynamically adjusts computational effort, reducing unnecessary processing for predictable tokens while allocating more computation to informative ones.

The model integrates these components into a unified architecture intended to provide efficient sequence processing, improved memory management, and flexible computational allocation.

The study also introduces a custom GPU-optimized Triton kernel that combines state-space recurrence with its reverse-adjoint backward computation. According to the reported experiments, this kernel achieves a 24.8-fold forward-kernel speedup over a sequential PyTorch implementation, while full-model training throughput improves by 4.3 times to 1,367 tokens per second. However, the model remains slower than a tuned dense Transformer on short sequences, which achieves 2,840 tokens per second in the reported comparison.

The evaluation compares IVERI-HYPERION with Dense Transformers, Linear Attention, and Mamba-based models on the TinyStories English dataset. Under approximately equal parameter budgets of 9.985 million parameters, the proposed model demonstrates early sample-efficiency advantages over the convolutionless Dense Transformer in the reported 200-step benchmark. It also maintains a flat recurrent-state memory footprint as the generated sequence grows, avoiding the growing KV cache used by conventional autoregressive Transformers. These findings are limited to the reported experimental setting and do not establish superiority across all language tasks.

Conclusion

In this work, we introduced IVERI-HYPERION, an Entropy-Driven Neural State Network (ENSN) prototype designed to explore sub-quadratic trade-offs relative to pairwise self-attention in a compact 10M parameter setting. By unifying continuous-time diagonal state spaces (S4D SSM) with test-time decayed associative neural memory (Titans Memory) and strictly causal entropy-guided compute modulation, the architecture provides a preliminary proof-of-concept investigating the integration of sequential tracking, associative retention, and bounded inference memory scaling. Our preliminary findings and theoretical analysis indicate that: 1. Sub-Quadratic Dual-State Sequence Representation: By decoupling continuous temporal tracking (diagonal S4D SSM) from discrete associative memory (decayed matrix memory), IVERI-HYPERION operates with strictly linear sequence compute O(S) during recurrent inference. Recurrent updates are executed via a fused GPU register loop (with O(S) sequential steps per warp during training and O(1) during decoding), eliminating quadratic attention compute and KV-cache explosion. 2. Avoiding Expanding Key-Value Cache Footprints: Maintaining a fixed recurrent runtime buffer ( measured allocation, excluding static model weights), IVERI-HYPERION exhibits zero net allocation growth in dynamic context state over 500 decoding steps ( delta), preserving constant dynamic state memory. At 32,768 tokens, this yields an estimated reduction strictly in dynamic context-dependent storage relative to theoretical Key-Value caching (0.30 MB buffer vs. 234.88 MB theoretical KV-cache, excluding static weights).

References

[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, ?. Kaiser, and I. Polosukhin, \"Attention is all you need,\" in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, pp. 5998–6008, 2017. [2] A. Gu and T. Dao, \"Mamba: Linear-time sequence modeling with selective state spaces,\" arXiv preprint arXiv:2312.00752, 2023. [3] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, \"Transformers are RNNs: Fast autoregressive transformers with linear attention,\" in Proc. Int. Conf. Machine Learning (ICML), pp. 5156–5165, 2020. [4] R. Eldan and Y. Li, \"TinyStories: How small can language models be and still speak coherent English?,\" arXiv preprint arXiv:2305.07759, 2023. [5] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, \"FlashAttention: Fast and memory-efficient exact attention with IO-awareness,\" in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 16344–16359, 2022. [6] P. Tillet, H. T. Kung, and D. Cox, \"Triton: An intermediate language and compiler for tiled neural network computations,\" in Proc. ACM SIGPLAN Workshop on Machine Learning and Programming Languages (MAPL), pp. 10–19, 2019. [7] A. Gu, K. Goel, and C. Ré, \"Efficiently modeling long sequences with structured state spaces,\" in Proc. Int. Conf. Learning Representations (ICLR), 2022. [8] A. Gu, A. Gupta, K. Goel, and C. Ré, \"On the parameterization and initialization of diagonal state space models,\" in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 35971–35983, 2022. [9] A. Behrouz, M. Pezeshki, et al., \"Titans: Learning to memorize at test time,\" arXiv preprint arXiv:2501.00663, 2025. [10] B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. Grover, et al., \"RWKV: Reinventing RNNs for the Transformer era,\" in Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048–14077, 2023. [11] S. Hochreiter and J. Schmidhuber, \"Long short-term memory,\" Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. [12] J. L. Ba, J. R. Kiros, and G. E. Hinton, \"Layer normalization,\" arXiv preprint arXiv:1607.06450, 2016. [13] I. Loshchilov and F. Hutter, \"Decoupled weight decay regularization,\" in Proc. Int. Conf. Learning Representations (ICLR), 2019, pp. 1–11. [14] K. He, X. Zhang, S. Ren, and J. Sun, \"Deep residual learning for image recognition,\" in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016. [15] B. Zhang and R. Sennrich, \"Root mean square layer normalization,\" in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019. [16] N. Kitaev, ?. Kaiser, and A. Levskaya, \"Reformer: The efficient Transformer,\" in Proc. Int. Conf. Learning Representations (ICLR), 2020. [17] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, \"Efficient Transformers: A survey,\" ACM Computing Surveys, vol. 55, no. 6, pp. 1–28, 2022. [18] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, \"Scaling laws for neural language models,\" arXiv preprint arXiv:2001.08361, 2020. [19] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., \"Training compute-optimal large language models,\" in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 30016–30030, 2022. [20] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., \"LLaMA: Open and efficient foundation language models,\" arXiv preprint arXiv:2302.13971, 2023. [21] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, \"Language models are unsupervised multitask learners,\" OpenAI Technical Report, 2019. [22] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., \"Language models are few-shot learners,\" in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877–1901, 2020. [23] R. Sennrich, B. Haddow, and A. Birch, \"Neural machine translation of rare words with subword units,\" in Proc. 54th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1715–1725, 2016. [24] N. Kalchbrenner, L. Espeholt, K. Simonyan, A. van den Oord, A. Graves, and K. Kavukcuoglu, \"Neural machine translation in linear time,\" arXiv preprint arXiv:1610.10099, 2016. [25] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin, \"Convolutional sequence to sequence learning,\" in Proc. Int. Conf. Machine Learning (ICML), pp. 1243–1252, 2017. [26] ?. Kaiser and I. Sutskever, \"Neural GPUs learn algorithms,\" in Proc. Int. Conf. Learning Representations (ICLR), 2016. [27] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, \"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,\" in Proc. Int. Conf. Learning Representations (ICLR), 2017. [28] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré, \"HiPPO: Recurrent memory with optimal polynomial projections,\" in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1474–1487, 2020. [29] O. Press and L. Wolf, \"Using the output embedding to improve language models,\" in Proc. 15th Conf. European Chapter of the Association for Computational Linguistics (EACL), pp. 157–163, 2017. [30] F. Chollet, \"Xception: Deep learning with depthwise separable convolutions,\" in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1251–1258, 2017. [31] D. Raposo, S. Ritter, A. Santoro, et al., \"Mixture-of-Depths: Dynamically allocating compute in language models,\" arXiv preprint arXiv:2404.02258, 2024. [32] T. Dao and A. Gu, \"Transformers are SSMs: Generalized models and unified frameworks,\" arXiv preprint arXiv:2405.21060, 2024. [33] M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, S. Ermon, and C. Ré, \"Hyena hierarchy: Towards larger convolutional language models,\" in Proc. Int. Conf. Machine Learning (ICML), pp. 28043–28078, 2023. [34] D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Ré, \"Hungry Hungry Hippos: Towards language modeling with state space models,\" in Proc. Int. Conf. Learning Representations (ICLR), 2023. [35] A. Graves, \"Generating sequences with recurrent neural networks,\" arXiv preprint arXiv:1308.0850, 2013.

Copyright

Copyright © 2026 Nishant Datta, Ishwari Bhoyar, Gunjan Nandeshwar, Nazish Khan, Prof. Archana Potnurwar, Prof. Lalita Charpe. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.