arises from the adaptive correction mechanism via AO and PSO, which stabilizes beam power and mitigates fluctuations, ensuring robust demodulation and decoding for high-speed links. Wavelength (m) vs. Power (dBm) graph evaluates WDM performance in OAM-based MDM by mapping power distribution across multiple wavelength channels, often in the mid-IR spectral region. This ensures sufficient signal strength for reliable multi-channel, high-capacity transmission.
PdBm = 10log10 ( P(λ) 1mW ) (74) In Eq. (74), PdBm represents the optical power of the signal at a specific wavelength λ, and P(λ) is the measured optical power of the signal at the wavelength. Figure 14 presents the results. The proposed model shows significant power enhancement, with peak values reaching -18 dBm at 1.55µm and rising to 0 dBm at 1.61µm, outperforming MDM-FSO and DFE by up to 10 dB. This gain stems from efficient metasurface-based OAM demultiplexing and optimized spectrum usage, which enhances channel separation, improves SNR, and enables energy-efficient communication. OAM modes with different topological charges are orthogonal, enabling MDM to transmit multiple data streams simultaneously. The analysis examines the impact of adding OAM modes on BER, particularly under AT, where crosstalk may occur.
BER(NOAM) = f (NOAM, Turbulence Strength) (75) In Eq. (75), BER(NOAM) denotes the BER as a function of the number of OAM modes used, NOAM is the number of distinct OAM modes being multiplexed, and Turbulence Strength is a parameter that represents the severity of AT. Fig. 13. Numerical Outcomes of Time (s) vs. Signal (V).
Fig. 12. Numerical Outcomes of Time (s) vs. Error (%).
Scientific Reports | (2026) 16:8921 22 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
Figure 15 shows that the proposed model maintains a low and stable BER ( 0.00025 to 0.00050 for up to 10 OAM modes), whereas MDM-FSO and DFE exhibit higher and fluctuating BER, especially at higher mode counts. This demonstrates that DNFIS-based equalization effectively mitigates inter-modal interference and preserves OAM orthogonality. Combined with AO, it supports scalable multi-mode transmission, crucial for high-bandwidth applications such as inter-satellite links. Thus, Fig. 15 provides a crucial sensitivity analysis on OAM mode count, demonstrating the framework’s capacity-robustness trade-off and its consistent advantage over baselines across multiplexed channels. AT, caused by refractive index fluctuations from pressure and temperature variations, induces beam wander, scintillation, and phase distortions that impair signal quality. In this study, turbulence is varied across weak, moderate, and strong regimes, with performance assessed using SNR and received power. The objective is to demonstrate that the proposed approaches, including structured light beams and AO, can sustain or even improve signal quality under increasing turbulence conditions.
QSignal = Q (Turbulence Strength) (76) In Eq. (76), QSignal, is a quantitative measure of the received signal’s integrity. Figure 16 shows the proposed framework consistently outperforms baseline methods across all turbulence regimes, exhibiting improved robustness against turbulence-induced signal degradation compared to MDM- FSO and DFE. Although the proposed framework significantly mitigates turbulence-induced degradation, residual performance loss is still observed under strong turbulence conditions, reflecting the fundamental limits imposed by atmospheric channel impairments. Figure 16 provides the sensitivity analysis for turbulence strength. Collectively, Figures 10b, 11, 15, and 16 evaluate system performance against propagation distance, pointing error, OAM mode count, and turbulence strength, confirming that the reported performance gains are robust across the operational parameter space. Fig. 15. Number of OAM modes vs. BER.
Fig. 14. Wavelength (m) vs. Power (dBm).
Scientific Reports | (2026) 16:8921 23 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
Computational complexity and real-time performance The practical deployment of intelligent compensation requires an assessment of computational overhead. The complexity of the core algorithms was evaluated. The DCNN-TCSGm, with the architecture detailed in Table , requires approximately 2.5 Giga-FLOPS (GFLOPs) per inference. Based on standard benchmarks for embedded AI platforms (e.g., NVIDIA Jetson AGX), this complexity translates to a projected processing latency of < 10ms per frame. The DNFIS equalizer exhibits a complexity that is quadratic in the number of OAM modes, O(M2), enabling sub-millisecond symbol-rate adaptation. The total projected system latency encompassing wavefront sensing, AO correction, PSO optimization, DCNN inference, and DNFIS equalization is therefore estimated to be under 20 ms. This is critically compared to the characteristic coherence time of atmospheric turbulence, which is typically on the order of 1 −10ms for terrestrial links. Therefore, the proposed processing pipeline is estimated to operate within the required real- time window, enabling effective compensation before the turbulent channel state decorrelates. Conclusion This study proposes an advanced FSO communication framework integrating structured light beams, intelligent optimization, AO, and deep learning-based turbulence mitigation to overcome key limitations of conventional FSO systems. By leveraging the self-healing and non-diffracting properties of Bessel, Airy, and optimized vortex beams, the system inherently enhances robustness against AT. Adaptive wavefront correction, combined with PSO and real-time compensation using a DNFIS, further improves signal fidelity in long-distance MDM scenarios. To address temporal channel variations, a DCNN-TCSGm is implemented to predict and compensate turbulence effects in real-time. Simulation results demonstrate up to 55% reduction in BER, 22% improvement in received signal voltage, and up to 10 dB power gain over baseline MDM-FSO and DFE systems. Additionally, the integration of WDM with metasurface-based OAM multiplexing significantly enhances data throughput, particularly for inter-satellite communication. The proposed framework demonstrates significant robustness, scalability, and resilience under challenging atmospheric conditions, highlighting its potential for next- generation optical communication networks. Future work may explore advanced ML for real-time turbulence mitigation and hybrid multiplexing strategies to further enhance throughput, scalability, and reliability. Limitations and future validation This study establishes a comprehensive simulation-based proof-of-concept. A key limitation is the absence of experimental validation under real atmospheric conditions. Future work will therefore focus on implementing a laboratory-scale prototype. This experimental phase will utilize a programmable turbulence phase screen, SLMs for dynamic beam generation, and high-bandwidth coherent detectors to empirically validate the predicted performance gains. Furthermore, future experimental and deployment efforts will also investigate practical integration challenges, such as the stringent alignment requirements for high-order OAM modes and the integration of adaptive optics within robust pointing, acquisition, and tracking PAT subsystems. It will also serve to identify and address practical integration challenges related to hardware latency, alignment precision, and component non-idealities not captured in simulation. Data availability All data generated or analysed during this study are included in this article Received: 9 January 2026; Accepted: 16 February 2026 Fig. 16. Normalized signal quality versus Rytov variance (σ2 R) under atmospheric turbulence.
Scientific Reports | (2026) 16:8921 24 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
References 1. Hassan, H., Althunibat, S., Al-Mbaideen, A., Hasna, M. & Qaraqe, K. A survey on hybrid free space optical and radio frequency systems: Classification, progress, observations and challenges. IEEE Access (2025). 2. Aboelala, O., Lee, I. E. & Chung, G. C. A survey of hybrid free space optics (fso) communication networks to achieve 5g connectivity for backhauling. Entropy 24, 1573 (2022). 3. Jeon, H.-B. et al. Free-space optical communications for 6g wireless networks: Challenges, opportunities, and prototype validation. IEEE Commun. Mag. 61, 116–121 (2023). 4. Altakhaineh, A. T. et al. Outdoor free space optical systems: Motivations, challenges, contributions in environmental conditions, and future directions-a systematic survey. IEEE Access (2025). 5. Singh, H. et al. Designing an optimized free space optical (fso) link for terrestrial commercial applications under turbulent channel conditions. Opt. Quantum Electron. 55, 532 (2023). 6. Guiomar, F. P., Fernandes, M. A., Nascimento, J. L., Rodrigues, V. & Monteiro, P. P. Coherent free-space optical communications: Opportunities and challenges. J. Lightwave Technol. 40, 3173–3186 (2022). 7. Álvarez-Roa, C. et al. Performance analysis of a vertical fso link with energy harvesting strategy. Sensors 22, 5684 (2022). 8. Elsayed, E. E. Performance analysis and modeling: Atmospheric turbulence and crosstalk of wdm-fso network. J. Opt. 1–18 (2024). 9. Hayal, M. R. et al. Modeling and investigation on the performance enhancement of hovering uav-based fso relay optical wireless communication systems under pointing errors and atmospheric turbulence effects. Opt. Quantum Electron. 55, 625 (2023). 10. Le, H. D. & Pham, A. T. Link-layer retransmission-based error-control protocols in fso communications: A survey. IEEE Commun. Surv. Tutori. 24, 1602–1633 (2022). 11. Rawshan Habib, M. et al. Performance analysis of a fso link considering different atmospheric turbulence. In Inventive Computation and Information Technologies: Proceedings of ICICIT 2021, 509–519 (Springer, 2022). 12. Monisha, M. A. & Geetha, M. Resilient adaptive polarized optical transmission for mitigating atmospheric turbulence in free space optics communication. Optik 326, 172262 (2025). 13. Zhang, G. et al. A review of variable-beam divergence angle fso communication systems. Photonics 10, 756 (2023). 14. Freitas, M. M., Fernandes, M. A., Monteiro, P. P., Guiomar, F. P. & Fernandes, G. M. Requirements and solutions for robust beam alignment in fiber-coupled free-space optical systems. Photonics 10, 394 (2023). 15. Raj, A. A. B. et al. A review-unguided optical communications: Developments, technology evolution, and challenges. Electronics 12, 1922 (2023). 16. Hayat, B. et al. Broadband oam vortex beams generating through transmitarray for millimeter wave applications. Opt. Exp. 33, 10036–10046 (2025). 17. Ahmad, M., Wang, Z., Fang, M., Huang, Z. & Xie, G. Securing and optimizing optical transmission in quantum wells using oam and advanced modulation techniques. Sci. Rep. 15, 29096 (2025). 18. Noor, S. K. et al. A review of orbital angular momentum vortex waves for the next generation wireless communications. Ieee Access 10, 89465–89484 (2022). 19. Khulbe, M. & Parthasarathy, H. Orbital angular momentum wave generation and multiplexing: Experiments and analysis using classical and quantum optics. Wirel. Commun. Mobile Comput. 2022, 5355854 (2022). 20. Fang, L., Luo, L., Zhao, R. & Liu, B. Measuring the orbital angular momentum of light based on optical differentiation. IEEE Photonics J. 15, 1–7 (2023). 21. Du, T. et al. Orbital angular momentum modes recognition method based on transfer learning. IEEE Photonics J. (2025). 22. Chen, J. et al. High-order orbital angular momentum mode-based phase shift-keying communication using phase difference modulation. Opt. Exp. 31, 44353–44363 (2023). 23. Xiang, Y., Zeng, L., Wu, M., Luo, Z. & Ke, Y. Deep learning recognition of orbital angular momentum modes over atmospheric turbulence channels assisted by vortex phase modulation. IEEE Photonics J. 14, 1–9 (2022). 24. Zhao, Y., Wang, Z., Lu, Y. & Guan, Y. L. Multimode oam convergent transmission with co-divergent angle tailored by airy wavefront. IEEE Trans. Antennas Propag. 71, 5256–5265 (2023). 25. Hou, J. et al. Time-varying propagation model and dynamic-feedback-phase correction for multiplexed orbital angular momentum beams in atmospheric turbulence. Opt. Exp. 32, 11079–11091 (2024). 26. Nong, L. et al. Orbital angular momentum mode diversity gain in optical communication. Opt. Exp. 30, 27482–27496 (2022). 27. Elsayed, E. E. et al. Coding techniques for diversity enhancement of dense wavelength division multiplexing mimo-fso fault protection protocols systems over atmospheric turbulence channels. IET Optoelectron. 18, 11–31 (2024). 28. Alamayreh, A., Qasem, N. & Rahhal, J. S. Pre-coding oam based mimo system for multi-user communications. IEEE Access 10, 125411–125420 (2022). 29. Willner, A. E. et al. Free-space mid-ir communications using wavelength and mode division multiplexing. Opt. Commun. 541, 129518 (2023). 30. Elfikky, A. et al. Spatial diversity-based fso links under adverse weather conditions: performance analysis. Opt. Quantum Electron. 56, 826 (2024). 31. Zhang, C., Jiang, X., He, J., Li, Y. & Ta, D. Spatiotemporal acoustic communication by a single sensor via rotational doppler effect. Adv. Sci. 10, 2206619 (2023). 32. Singh, M., Atieh, A., Grover, A. & Barukab, O. Performance analysis of 40 gb/s free space optics transmission based on orbital angular momentum multiplexed beams. Alex. Eng. J. 61, 5203–5212 (2022). 33. Zhang, Y. & Jiang, L. Suppressing white-noise interference for orbital angular momentum waves via the forward-backward dynamic mode decomposition. IEEE Trans. Antennas Propag. 71, 2879–2884 (2022). 34. Dabiri, M. T., Ghanbari, M. & Hasna, M. Advancing oam-based fso systems: Tackling pointing errors for next-generation space and terrestrial links. IEEE Photonics J. (2025). 35. Bulygin, A. D., Geints, Y. E. & Geints, I. Y. Vortex beam in a turbulent kerr medium for atmospheric communication. Photonics 10, 856 (2023). 36. Wen, W. & Zhang, X. Laser beam quality of airy beam in the jet engine exhaust induced turbulence. Atmosphere 14, 1374 (2023). 37. Ju, P. et al. Atmospheric turbulence effects on the performance of orbital angular momentum multiplexed free-space optical links using coherent beam combining. Photonics 10, 634 (2023). 38. Almogahed, A., Amphawan, A., Mohammed, F. & Alawadhi, A. Performance improvement of mode division multiplexing free space optical communication system through various atmospheric conditions with a decision feedback equalizer. Cogent Eng. 9, 2034268 (2022). 39. Agarwal, H., Mishra, D. & Kumar, A. A deep-learning approach for turbulence correction in free space optical communication with laguerre-gaussian modes. Opt. Commun. 556, 130249 (2024). 40. Singh, K. et al. Investigations on mode-division multiplexed free-space optical transmission for inter-satellite communication link. Wirel. Netw. 28, 1003–1016 (2022). 41. Gao, H. et al. Metasurface-based orbital angular momentum multi-dimensional demultiplexer and decoder. Laser Photonics Rev. 17, 2300393 (2023). 42. Elsayed, E. E. Atmospheric turbulence mitigation of mimo-rf/fso dwdm communication systems using advanced diversity multiplexing with hybrid n-sm/omi m-ary spatial pulse-position modulation schemes. Opt. Commun. 562, 130558 (2024). 43. Elsayed, E. E. Performance enhancement in fso relay systems with miso via multi-hop m-ary ppm integrating and spatial modulation over gamma-gamma channels. J. Opt. 54, 3364–3379 (2025). Scientific Reports | (2026) 16:8921 25 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/