Robust high-capacity free-space optical communication using OAM-based structured light and intelligent adaptive signal processing Muhammad Ahmad1,2,3, Babar Hayat4, Ming Fang1,2,3, Chao Wang1,2,3, Guoda Xie1,2,3 & Zhixiang Huang1,2,3 Free-space optical (FSO) communication enables high-speed, secure, and bandwidth-efficient data transmission for terrestrial and inter-satellite networks, outperforming traditional radio frequency (RF) systems in interference immunity and directional security. Atmospheric turbulence (AT), which causes beam distortion, intensity fading, and intermodal interference, remains a significant limitation for long-distance links. Existing approaches, such as Gaussian beam transmission, static equalization, and basic convolutional models, fail to provide real-time, adaptive resilience to these challenges. To overcome these limitations, this study proposes a novel hybrid FSO framework combining resilient structured light beams (Bessel, Airy, and orbital angular momentum (OAM) modes), adaptive optics (AO), and intelligent signal processing. A Dynamic Neural Fuzzy Inference System (DNFIS) provides robust equalization, and a Deep Convolutional Neural Network with Time-domain Correlation Sequence Generation (DCNN-TCSGm) predicts and compensates for turbulence effects in real time. Furthermore, the framework models compact optical metasurface-based OAM multiplexing combined with Wavelength Division Multiplexing (WDM) in the mid-infrared range to enhance spectral and spatial throughput. Simulation results demonstrate a 55% reduction in Bit Error Rate (BER), a 22% improvement in signal voltage stability, and up to 10 dB power gain over conventional Mode Division Multiplexing (MDM)-FSO and Decision Feedback Equalizer (DFE) systems, highlighting the proposed framework’s robustness and scalability under challenging atmospheric conditions. The rapid growth of multimedia applications has intensified the demand for high-speed wireless networks 1. Free-space optical (FSO) communication has emerged as a promising solution for high-speed terrestrial links, inter-satellite communications, and last-mile access, offering large unlicensed bandwidth, high data rates, and enhanced security compared to traditional radio frequency (RF) systems2–4. Unlike optical fiber, FSO transmits data via modulated laser beams through the atmosphere or vacuum 5, allowing for rapid and cost-effective deployment in areas where fiber installation is impractical or expensive 6. However, atmospheric turbulence (AT) poses significant challenges to FSO performance. Fluctuations in the refractive index due to temperature and pressure variations cause beam wander, intensity scintillation, and phase distortions, which degrade signal quality7–9. These impairments increase bit error rates (BER) and reduce link availability, particularly under adverse weather conditions such as fog, rain, and haze10,11. Overcoming these challenges is critical for advancing FSO technology across academic, industrial, and commercial domains 12. Conventional FSO systems typically employ Gaussian beams, which are straightforward to generate but highly susceptible to atmospheric distortions, resulting in signal degradation13,14. To address these limitations, structured light beams, especially those carrying Orbital Angular Momentum (OAM), have garnered increasing interest. OAM beams exhibit a helical phase front that enhances resilience against turbulence15–18. This work focuses on OAM, Bessel, and Airy beams, which 1The Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Anhui University, Hefei 230601, China. 2The Information Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui University, Hefei 230601, China. 3The Key Laboratory of Electromagnetic Environmental Sensing of Anhui Higher Education Institutes, Anhui University, Hefei 230601, China. 4School of Information Engineering, Xi’an Eurasia University, Xi’an 710065, Shaanxi, China. email: [email protected] OPEN Scientific Reports | (2026) 16:8921 1| https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports
possess properties such as non-diffraction, self-healing, and self-acceleration, helping maintain signal integrity over longer distances and through obstructions. Further performance enhancements can be achieved through adaptive optics (AO) for real-time wavefront correction and intelligent optimization techniques such as Particle Swarm Optimization (PSO). Advanced signal processing methods, including Decision Feedback Equalizers (DFE) and Deep Convolutional Neural Networks (DCNN), are also essential for reliable signal recovery in turbulent conditions. Moreover, integrating Wavelength Division Multiplexing (WDM) with OAM and Mode Division Multiplexing (MDM) can significantly increase data throughput, making these technologies vital for future terrestrial and inter-satellite FSO communication systems. Literature survey This section reviews structured light and OAM-based multiplexing and demultiplexing techniques for enhancing FSO communication. Authors in19 examine OAM waveform generation methods, comparing them with communication requirements and performance metrics. They highlight OAM’s potential for energy- efficient transmission and low error rates, including indirect fiber-to-atmosphere transmission suited for short- range wireless links. The study emphasizes optical communication’s role in meeting broadband demands and its application in secure quantum key distribution (QKD). In20, a simple and green detection method for OAM states is proposed using optical differentiation with weak measurements. Experiments on Laguerre-Gaussian beams (l =5 ,l = 8) show reduced sensitivity to disturbances, enabling direct detection with a simplified setup. While promising for FSO applications, real-time detection of multiple vortex modes remains a challenge. Authors in 21 introduce an optimized convolutional neural network (CNN) with transfer learning for recognizing OAM modes in distorted beams. Results show higher accuracy with reduced training time compared to conventional CNNs. The study also reports that turbulence and longer propagation distances gradually reduce recognition accuracy. In22, a novel modulation method is presented that exploits the phase differences between superposed OAM modes. Small phase shifts alter interference patterns, creating additional parameters for encoding information. This approach exponentially increases encoding capacity, and a neural network-based decoder enables accurate recovery of transmitted data from light intensity patterns. A vortex modulation technique enhances minor variations among closely spaced OAM states, enabling CNNs to distinguish features more accurately and improve topological charge identification 23. Simulations demonstrate high recognition accuracy under strong AT and long transmission distances, with future work exploring alternative modulation schemes and neural network architectures for high-resolution fractional OAM modes. Building on this, co-scale reception with convergent beams24 ensures OAM modes transmit with equal divergence, while a ring-shaped Airy compensation focuses energy onto a compact receiving region. Experiments demonstrate tunable OAM modes with improved received power and minimal crosstalk, while simulations indicate that lower-order OAM modes fluctuate less under weak time-varying AT (TV-AT), whereas higher-order modes perform better under stronger TV-AT . This highlights the need for accurate TV propagation models and phase-correction strategies. T o mitigate turbulence-induced degradation in real-time systems, predictive diversity combining (PDC) with optimal dynamic channel processing (ODCP) maintains BER25. Complementary diversity gains techniques optimize capacity while reducing turbulence-induced fading by assigning channel weights and applying digital synchronization, with optical fiber delay lines enabling effective signal combination26. These methods are validated in Multiple-Input Multiple-Output (MIMO) DWDM-FSO links using diversity coding, where an 8-channel, 2.5 Gbps per channel, 1500 m system performs efficiently under turbulent conditions27. Multi-OAM-mode transmission from a main transmitter to distributed receivers employs planar arrays and precoding to deliver multiple channels simultaneously, though scalability limits the number of achievable OAM modes. High-speed communication is further enhanced by combining optical code division multiple access (OCDMA) with OAM modes28. Two LG beams transmit three 10 Gbps channels each, with spatial diversity and beamforming reducing BER under turbulence, although dense fog and dust remain limiting factors29. Real-time spatiotemporal acoustic communication using a single sensor and the rotational Doppler effect demonstrates efficient multiplexing of multiple OAM channels 30, illustrating the versatility of OAM-based systems. T errestrial OAM-FSO links achieve up to 40 Gb/s using four OAM beams. Comparisons of Alternate Mark Inversion (AMI), Return to Zero (RZ), and Non-Return to Zero (NRZ) encodings indicate NRZ is optimal over 64–800 m31. Forward-backward dynamic mode decomposition (FBDMD) mitigates white noise interference, accurately recovering OAM topological charges and reducing crosstalk32, although real-time, high-data-rate applications requires significant computational resources. For UA V-to-ground communications, a 4-level quadrature amplitude modulation (4-QAM)-OFDM-FSO architecture33 and a comprehensive pointing error model 34 demonstrate trade-offs between mode number and modulation order. Simulations show that moderate laser pulse powers preserve phase singularities under cubic nonlinearity, mitigating divergence, while higher powers accelerate transverse spreading 35. Partially coherent Airy beam analysis under jet exhaust turbulence shows that beam quality improves with smaller structure constants and outer scales36. The authors in37 proposed coherent laser arrays with discrete vortices (CLA-DV) to reduce crosstalk in OAM transmission under turbulence, improving stability, though signal quality still degraded at longer distances. In38, the authors introduced DFE with minimum mean square error (MMSE) optimization across Hermite–Gaussian channels, enhancing BER performance, but effectiveness decreased under strong turbulence. The authors in 39 developed the deep-learning model cGULnet to extract phase information from multiplexed Laguerre–Gaussian modes, lowering BER, while high computational cost limited real-time use. In40, the authors demonstrated an 80 Gbps inter-satellite MDM-FSO link over 35,000 km, with potential gains using wavelength division multiplexing (WDM), though system complexity remained a challenge. The authors in 41 presented single-layer dielectric metasurfaces for compact multi-dimensional demultiplexing of wavelength, spin angular momentum (SAM), and OAM across 132 channels, offering scalable integration. However, the integration issues and performance degradation under real atmospheric conditions still limit practical implementation. Recent advancements Scientific Reports | (2026) 16:8921 2| https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/ continue to explore hybrid techniques for turbulence mitigation. For instance, Elsayed42 integrated OAM with spatial modulation and L-ary PPM in a DWDM-MIMO FSO system to enhance throughput. Similarly, Elsayed43 employed multi-hop MIMO with spatial modulation and M-ary PPM to improve spectral efficiency and combat turbulence-induced BER, highlighting the trend towards complex, co-optimized physical-layer designs. Table 1 lists the summary of the literature. Motivation and contributions FSO communication faces significant challenges from air turbulence, which severely degrades signal quality and image transmission, especially over long distances. Increasing air turbulence and transmission distance result in higher error rates and reduced data quality. Current equalization methods are insufficient to combat diverse AT in long-distance MDM-FSO links. Existing deep learning approaches for turbulence mitigation also lack optimal speed and accuracy. OAM beam transmission using CLA-DV37 reduces crosstalk under severe turbulence, but image quality still degrades with increasing turbulence and distance. In MDM-FSO systems, the MMSE across Hermite-Gaussian channels38 optimized by DFE is insufficient for diverse long-haul conditions highlighting the need for more advanced schemes. Deep learning models like cGULnet39 have shown promise in accurately extracting phase information from spatially multiplexed LG modes, but improvements are still required in speed Ref Objectives Model/techniques used Key limitations 19 Explore OAM waveform generation and communication performance. Comparative analysis of OAM generation, energy efficiency, BER, and fiber-to-atmosphere transition. Lacks experimental validation; focused on short- range systems. 20 Develop simplified detection for OAM states using weak optical differentiation. Differential measurements on LG beams with topological charges (l=5, l=8). Limited real-time differentiation of vortex modes; small-scale detection only. 21 Improve distorted vortex beam recognition using CNN transfer learning. Transfer learning-based CNN for OAM mode classification under turbulence. Accuracy drops with stronger turbulence/distance; limited hardware testing. 22 Introduce modulation using the phase-difference degree of freedom. Phase-interference encoding with NN-based decoding. Requires precise phase control; complexity grows with more modes. 23 Enhance OAM state separation under turbulence. CNN-based vortex modulation and image classification. Fractional OAM recognition is still weak; noise- sensitive. 24 Implement co-scale reception with Airy compensation. Experimental setup with an adjustable ring-shaped Airy wavefront. Mode power fluctuation reversal under strong V-AT isn’t resolved. 25 Maximize diversity gain and minimize fading in OAM multiplexed links. Diversity gains modeling, channel assignment, and fiber delay line. Synchronization complexity; optical-domain combination needed. 26 Evaluate MIMO-DWDM-FSO under turbulence with coding. MIMO, MMSE, STC, STBC, QO-STBC. Ignores pointing error turbulence; limited FEC flexibility. 27 Distribute multiple OAM channels using precoding. Grid array antennas with OAM precoding, interference mitigation. Limited scalability; large arrays required. 28 Propose an OCDMA-OAM hybrid for high-speed FSO. FRS coding with LG modes for multi-channel allocation. High turbulence sensitivity; lacks AI/ML compensation. 29 Reduce BER in FSO using spatial diversity + beamforming. Spatial diversity and beamforming under turbulence. Range reduced in fog/dust; weak environmental robustness. 30 Demonstrate real-time spatiotemporal acoustic OAM. Single-sensor Doppler detection of harmonic OAM waveforms. Needs high spatial resolution and synchronization; hardware limits. 31 Study OAM multiplexing in terrestrial FSO under weather effects. Simulations with AMI, RZ, and NRZ at 40 Gb/s. Simulation only; no experimental results. 32 Mitigate noise in OAM mode extraction. FBDMD matrix synthesis to reduce crosstalk. Computationally heavy; limited real-time scalability. 33 Enable UAV-to-ground OAM-FSO with 4-OAM-OFDM. 4-OAM-OFDM integration in dynamic UAV-ground topology. OFDM complexity under mobility isn’t fully addressed. 34 Design a robust OAM pointing error model for short-range FSO. Crosstalk-optimized BER model vs modulation/mode count. Short-range only; lacks multi-user/multi-beam validation. 35 Assess vortex beam propagation in nonlinear Kerr turbulence channels. Nonlinear Schrödinger equation with turbulence index model. Energy alignment isn’t addressed. 36 Assess the resilience of Airy beams against incident turbulence. Exhaust-flow turbulence tracking with beam modulation. Limited generalization to other turbulence types. 37 Reduce crosstalk in OAM transmission under turbulence. CLA-DV. Signal quality still degrades at longer distances. 38 Strengthen BER performance in MDM-FSO systems using Hermite–Gaussian channels. PTFB with MMSE optimization across Hermite– Gaussian channels. Effectiveness decreases under strong turbulence. 39 Extract phase information from distorted Hermite– Gaussian modes. Deep-learning model using GDL(VggNet). High computational cost limits real-time applicability. 40 Achieve liquid data-rate inter-satellite communications. MDM with potential WDM integration. System complexity and synchronization remain challenging. 41 Compact multi-dimentional demultiplexing (wavelength, SAM, OAM) Single layer dielectric metasurfaces Practical integration of nanophotonic components remains challenging. 42 To reduce signal degradation and BER caused by atmospheric turbulence. N-encoded SM with L-ary PPM in a DWDM-MIMO FSO High system complexity 43 To mitigate atmospheric turbulence and alignment-related signal degradation. Multi-hop MIMO with SM and M-ary PPM The Practical deployment may be constrained by hardware and relay station requirements. Table 1. Literature summary.
Scientific Reports | (2026) 16:8921 3 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/