Deep Learning-Based Detection Algorithm for the Multi-User MIMO-NOMA System

Abstract

The increasing demand for high data rates and massive connectivity in next-generation wireless systems has led to the adoption of non-orthogonal multiple access (NOMA) techniques. However, conventional successive interference cancellation (SIC)-based receivers suffer from performance degradation due to error propagation and nonlinear interference effects, particularly in multi-user MIMO environments. To address these limitations, this work proposes a feedback-based deep neural network (FDNN) receiver for signal detection in MIMO-NOMA systems. The proposed model integrates deep learning capabilities with iterative interference cancellation, enabling improved nonlinear signal separation and robustness against noise and inter-user interference. A comprehensive simulation framework is developed using Python to evaluate system performance under varying signal-to-noise ratios (SNR) and power allocation conditions. The results demonstrate that the FDNN-based receiver achieves significantly lower bit error rates (BER) compared to traditional SIC methods, especially in scenarios with small power differences between users. This study highlights the effectiveness of deep learning in enhancing receiver design for future wireless communication systems.

Introduction

The rapid growth of wireless communication technologies has been fueled by the increasing number of connected devices and the demand for high-speed data transmission in applications such as the Internet of Things (IoT), smart cities, and real-time multimedia services. Traditional Orthogonal Multiple Access (OMA) techniques allocate separate time or frequency resources to each user, which limits spectral efficiency and network capacity. To overcome these limitations, Non-Orthogonal Multiple Access (NOMA) has emerged as a promising multiple access technique that allows multiple users to share the same time and frequency resources by allocating different transmission power levels. At the receiver, Successive Interference Cancellation (SIC) is used to separate user signals. Although SIC is computationally efficient, its performance is highly dependent on accurate power allocation and channel conditions. Problems such as inter-user interference, the near-far effect, and error propagation significantly reduce detection accuracy. When NOMA is integrated with Multiple-Input Multiple-Output (MIMO) technology to improve capacity and reliability, signal detection becomes even more challenging due to increased interference and system complexity.

Recent advances in deep learning have opened new opportunities for wireless communication systems. Deep Neural Networks (DNNs) can learn complex nonlinear relationships directly from data without requiring explicit mathematical models, making them well suited for signal detection in MIMO-NOMA environments where conventional linear methods often perform poorly. Motivated by these advantages, the proposed work introduces a Feedback-Based Deep Neural Network (FDNN) receiver that replaces conventional linear detection with a neural network capable of nonlinear signal estimation. The FDNN incorporates a feedback mechanism that iteratively reconstructs and removes detected interference, reducing error propagation and improving overall detection accuracy. The study develops a novel FDNN receiver architecture, proposes a data-driven detection framework, compares its performance with the conventional SIC receiver using Bit Error Rate (BER), and investigates the effects of Signal-to-Noise Ratio (SNR) and user power allocation.

The system model considers an uplink power-domain NOMA system where multiple users simultaneously transmit signals to a base station using the same time-frequency resources. Each user's transmitted signal is assigned a specific power level, and the received signal consists of the superposition of all user signals combined with Additive White Gaussian Noise (AWGN). Users are ordered according to their channel gains, enabling the receiver to decode stronger signals first. The Signal-to-Interference-plus-Noise Ratio (SINR) determines each user's achievable data rate based on Shannon's capacity theorem. Although NOMA improves spectral efficiency and supports massive connectivity, its performance depends heavily on effective power allocation and accurate signal detection under fading, interference, and noise.

The conventional Successive Interference Cancellation (SIC) receiver detects user signals sequentially according to their received power. The strongest user's signal is detected first, reconstructed, and subtracted from the received signal before detecting the remaining users. Linear detection methods such as Zero-Forcing (ZF) and Minimum Mean Square Error (MMSE) are commonly used during this process. While SIC reduces interference compared to direct linear detection, it suffers from significant limitations. Detection errors made during early stages propagate to later users, leading to accumulated errors and degraded overall performance. Furthermore, linear detectors cannot effectively model the nonlinear interference characteristics encountered in practical wireless channels, motivating the need for deep learning-based alternatives.

To overcome these limitations, the proposed Feedback Deep Neural Network (FDNN) introduces nonlinear signal detection into the SIC framework. The FDNN architecture consists of an input layer, multiple fully connected hidden layers with nonlinear activation functions, and an output layer. The network receives the residual received signal together with channel information as input and learns a nonlinear mapping between received and transmitted signals. Unlike traditional receivers, the FDNN estimates transmitted symbols directly using learned representations, making it more robust to interference and channel impairments.

The proposed detection framework operates iteratively. At each stage, the FDNN estimates one user's transmitted symbol from the current residual signal. The estimated symbol is then reconstructed using the corresponding channel coefficient and transmission power. This reconstructed signal is subtracted from the received signal to generate an updated residual signal, which becomes the input for detecting the next user. By combining neural-network-based nonlinear estimation with iterative interference cancellation, the FDNN significantly reduces error propagation while maintaining the sequential detection structure of SIC.

Simulation studies are conducted using a Python-based framework for a two-user uplink NOMA system employing Quadrature Phase Shift Keying (QPSK) modulation over an AWGN channel. The simulations evaluate performance across SNR values ranging from 0 to 20 dB with unequal power allocation between users. The FDNN consists of two hidden layers containing 16 and 32 neurons with ReLU activation functions, while the output layer uses Softmax classification. The network is trained using cross-entropy loss with approximately 200,000 training samples.

Performance evaluation demonstrates that the proposed FDNN consistently outperforms the conventional SIC receiver. The Bit Error Rate (BER) decreases for both methods as SNR increases; however, the FDNN achieves significantly lower BER across the entire SNR range. At low SNR values (0–5 dB), both receivers experience relatively high error rates due to noise, but the FDNN shows greater robustness as SNR increases. At 20 dB, the FDNN achieves a BER of approximately 10?³, while the SIC receiver remains around 10?², indicating a substantial improvement in detection accuracy.

The impact of user power allocation is also investigated by varying the received power difference (ΔSNR). When users have similar power levels, the SIC receiver struggles to separate signals, resulting in higher BER. In contrast, the FDNN maintains stable performance by learning nonlinear decision boundaries that effectively distinguish user signals even under small power differences. Although both methods improve as ΔSNR increases, the FDNN consistently delivers superior performance. However, excessively large power differences reduce the weaker user's performance due to insufficient received signal strength.

Training analysis shows that the FDNN converges efficiently, with the training loss decreasing rapidly during the initial epochs before stabilizing. Appropriate batch size selection provides a balance between convergence speed and stability, whereas excessively large or small batch sizes negatively affect training efficiency.

References

[1] W. Saad, M. Bennis, and M. Chen, “A vision of 6G wireless systems: Applications, trends, technologies, and open research problems,” IEEE Network, vol. 34, no. 3, pp. 134–142, 2020. [2] H. Zhang, T. Zhou, T. Xu, and H. Hu, “Remote interference discrimination testbed employing AI ensemble algorithms for 6G TDD networks,” Sensors, vol. 23, no. 4, p. 2264, 2023. [3] I. F. Akyildiz, A. Kak, and S. Nie, “6G and beyond: The future of wireless communication systems,” IEEE Access, vol. 8, pp. 133995–134030, 2020. [4] J. Wu et al., “Feature-based spectrum sensing of NOMA system for cognitive IoT networks,” IEEE Internet of Things Journal, vol. 10, no. 1, pp. 801–814, 2023. [5] Y. Saito et al., “Non-orthogonal multiple access (NOMA) for cellular future radio access,” in Proc. IEEE VTC Spring, 2013. [6] Z. Ding et al., “A survey on non-orthogonal multiple access for 5G networks,” IEEE Journal on Selected Areas in Communications, vol. 35, no. 10, pp. 2181–2195, 2017. [7] E. G. Larsson et al., “Massive MIMO for next generation wireless systems,” IEEE Communications Magazine, vol. 52, no. 2, pp. 186–195, 2014. [8] M. Agiwal, A. Roy, and N. Saxena, “Next generation 5G wireless networks: A comprehensive survey,” IEEE Communications Surveys & Tutorials, vol. 18, no. 3, pp. 1617–1655, 2016. [9] S. Tweneboah-Koduah et al., “Performance of cooperative relay NOMA with large antenna transmitters,” Electronics, vol. 11, no. 21, p. 3482, 2022. [10] F. Alraddady, I. Ahmed, and F. Habtemicail, “Robust hybrid beamforming for NOMA in massive MIMO downlink,” Electronics, vol. 11, no. 1, p. 75, 2022. [11] P. K. Gkonis et al., “Non-orthogonal multiple access in multiuser MIMO configurations,” Electronics, vol. 9, no. 8, p. 1330, 2020. [12] S. N. Sur et al., “Hybrid precoding algorithm for mmWave massive MIMO-NOMA systems,” Electronics, vol. 11, no. 14, p. 2198, 2022. [13] W. L. Xie et al., “Downlink MIMO-NOMA system for 6G IoT,” Electronics, vol. 11, no. 20, p. 3233, 2022. [14] M. M. El-Gayar and M. N. Ajour, “Resource allocation in UAV-enabled NOMA networks,” Electronics, vol. 12, no. 24, p. 5033, 2023. [15] L. Dai et al., “Non-orthogonal multiple access for 5G: Solutions and challenges,” IEEE Communications Magazine, vol. 53, no. 9, pp. 74–81, 2015. [16] S. A. H. Mohsan et al., “Deep learning-based NOMA: A survey,” Sensors, vol. 23, no. 6, p. 2946, 2023.

Copyright

Copyright © 2026 Dr. Dilip Kumar Sharma, Nidhi Sisodiya. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.