Simulation results and comparative analysis This section presents a simulation-based performance evaluation of the proposed framework. Due to the complexity and cost of real-world multi-component FSO deployment, simulations provide a controlled and repeatable methodology essential for analyzing the complex interactions within such an integrated system. The performance was evaluated under various atmospheric turbulence conditions using established and widely accepted models for turbulence and beam propagation. The results demonstrate significant improvements in signal quality, data rate, and link reliability compared to conventional systems. All simulations were conducted in MATLAB R2023a. A discussion on the necessary path toward experimental validation is provided in the Conclusion. Experimental visualizations and results This section presents extensive visualizations of the simulations, detailing the results of the simulated experiments and highlighting the performance of the proposed OAM-based FSO communication system in the following subsections. (a) OAM-FSO Communication Model Implementation Figure 4 shows the spatial intensity and phase profile of OAM Mode 1 at a propagation distance of Z = 4.00m. The intensity map in the x–y plane exhibits the characteristic doughnut-shaped pattern of vortex beams, while the phase front confirms the helical structure associated with OAM. This baseline visualization verifies beam fidelity and symmetry in a controlled FSO channel, serving as a reference for evaluating the effects of turbulence and compensation mechanisms during transmission. Fig. 4. Implemented OAM-based Free-Space Optical (FSO) communication model.
Parameter SLM Dielectric metasurface Notes Phase control Programmable (8–12 bit resolution) Static (pre-fabricated) SLM offers dynamic reconfiguration; the metasurface profile is fixed. Refresh rate ≤ 1 kHz Not applicable SLM refresh limits real-time switching speed. Insertion loss 10–20% 20–40% Metasurface loss varies with material and design. Crosstalk (SXR) 15–20 dB 15–25 dB Higher-order OAM modes show increased sensitivity to imperfections. Field of view Limited (pixel-constrained) Wide (typically >30◦) SLM field of view is restricted by pixel count and fill factor. Cost Significantly higher Potentially lower at scale Metasurfaces benefit from batch fabrication and semiconductor-process compatibility. Integration potential Moderate High Metasurfaces enable ultra-thin, planar integration with detectors. Alignment sensitivity High (µrad-level precision) Moderate SLM requires precise angular and translational alignment. Table 3. Comparison of OAM Manipulation Technologies for FSO Terminals.
Scientific Reports | (2026) 16:8921 16 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
(b) Error Vector Magnitude (EVM) Analysis
The EVM graph illustrates modulation accuracy by comparing in-phase and quadrature (I–Q) deviations under four polarization and interference scenarios: No Crosstalk (EVM = 0.0007), Single Polarization (EVM = 0.0022), Dual Polarization–X (EVM = 0.0028), and Dual Polarization–Y (EVM = 0.0032). The tight clustering of symbols near the ideal constellation points indicates superior signal fidelity, while the low EVM values confirm the system’s robustness against polarization-induced distortions and inter-mode crosstalk, ensuring reliable high- quality demodulation in FSO links. Figure 5 presents the EVM graph as a function of SNR, turbulence strength, and propagation distance, with curves for different mitigation strategies and OAM modes. Results show consistently low EVM under varying AT conditions, confirming the proposed system’s robustness and reliable demodulation in FSO links. (c) Integration of Structured Light, AO, and PSO for Turbulence Mitigation Fig. 5. Error Vector Magnitude (EVM) analysis graph.
Scientific Reports | (2026) 16:8921 17 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
The integration of resilient structured light beams (Bessel, Airy, and optimized OAM vortex beams) with AO and PSO effectively mitigates AT. Structured beams inherently resist image degradation, while AO provides real-time wavefront correction, and PSO optimizes system parameters intelligently. Experimental results demonstrate that this combined approach preserves signal integrity and significantly reduces AT-induced distortions over long- distance FSO links. Figure 6a–d illustrates the integrated system architecture, high- lighting the joint use of structured beams, AO, and PSO for enhanced turbulence compensation. (d) BER Performance as a Function of OAM Modes The system’s BER performance was analyzed with respect to the number of OAM modes employed for MDM under varying AT conditions. BER was measured before and after applying the DNFIS equalization scheme, showing significant improvements in error correction and signal recovery. While conventional FSO systems experience notable BER degradation with increasing turbulence and mode count, the proposed system, which integrates structured light and advanced equalization, maintains a consistently low BER across multiple OAM modes. Figure 7 shows the BER performance, where the final BER (after DNFIS) is plotted against initial BER, with separate curves representing different OAM modes. (e) Real-time Turbulence Compensation using DCNN-TCSGm Fig. 7. BER based on OAM modes graph.
Fig. 6. (a) Turbulence limits OAM image quality, (b) Bessel beam, (c) Airy beam, and (d) Vortex beam .
Scientific Reports | (2026) 16:8921 18 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
The DCNN-TCSGm was implemented for real-time compensation of turbulence-induced effects. The training curve plots RMSE and loss across iterations. Initially, high RMSE values indicate poor prediction of turbulence degradation; however, as training progresses, the model learns temporal patterns in the channel, enabling accurate prediction and compensation. The sharp decline in RMSE and loss demonstrates fast convergence, strong generalization, and more effective distortion reduction compared to traditional equalization. Figure 8 shows the training performance of the proposed DCNN-TCSGm framework in terms of RMSE and loss over 500 iterations. The RMSE drops rapidly during the first 50 iterations and stabilizes below 0.05 after about 100 iterations, while the training loss decreases from 0.18 to nearly 0.01. The smooth convergence without significant fluctuations indicates stable learning and the absence of overfitting, confirming that the network effectively captures the underlying signal features. Table 4 summarizes the performance before and after applying DCNN- TCSGm. The BER decreases from 0.0032 to 0.00144, representing a 55% improvement. The MSE is reduced from 0.015 to 0.0117, yielding a 22% gain in signal stability. In addition, the SNR improves from 18.5 dB to 28.5 dB, providing a 10 dB power gain. These results clearly demonstrate the effectiveness of DCNN-TCSGm in mitigating channel impairments and enhancing the reliability of OAM-based optical transmission. (f) Performance of Combined WDM-MDM with Optical Metasurface-based OAM Multiplexing/Demulti plexing The simulation results demonstrate that integrating WDM with OAM-based MDM in the mid-infrared spectrum significantly enhances data transmission capacity for inter-satellite FSO links. Modeled compact dielectric metasurfaces enable efficient OAM multiplexing and demultiplexing, supporting multiple channels across different wavelengths and OAM modes. Figure 9 shows the enhanced throughput, illustrating multiplexed wavelengths carrying multiple OAM modes and the simulated metasurface demultiplexing incoming beams into distinct spatial positions based on their topological charge. Parameters Pre-mitigation Post-mitigation (DCNN-TCSGm) Improvement Bit error rate (BER) 0.0032 0.0014 55% Mean square error (MSE) Signal stability 0.0150 0.0117 22% Signal to noise ratio (SNR) Power gain (dB) 18.5000 28.5000 10% Table 4. DCNN-TCSGm performance summary.
Fig. 8. Mitigate the data using DCNN-TCSGm methods image.
Scientific Reports | (2026) 16:8921 19 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
OAM mode purity under turbulence To evaluate the robustness of OAM modes under turbulent conditions, we analyze the mode purity as defined in Eq. ( 13). Figure 10a plots the mode purity against the topological charge (ℓ) for three turbulence strengths, demonstrating the proposed framework’ s compensation capabilities. It represents Mode purity versus topological charge ℓ for the proposed framework under weak ( C2 n ≈ 1 × 10−14 m−2/3) , moderate ( C2 n ≈ 3 × 10−14 m−2/3) , and strong ( C2 n =5 × 10−14 m−2/3) turbulence conditions. Error bars represent Monte Carlo simulation uncertainty (±1.5 − 2.5%). Dashed lines indicate performance thresholds: P > 0.90 (excellent), P > 0.75 (good), P > 0.50 (marginal). Under strong turbulence, the framework maintains exceptional purity for low-order modes (ℓ =0 : P=0 .90, ℓ = ±1: P=0 .85) while exhibiting graceful parabolic degradation for higher-order modes, with ℓ = ±5 retaining P=0 .42. This demonstrates the system’ s ability to support up to five reliably usable channels (|ℓ|≤ 2, P> 0.75) even under severe turbulence conditions. Figure 10b OAM mode purity degradation with propagation distance for different topological charges |ℓ| =1 –5 under moderate atmospheric turbulence (C2 n =3 × 10−14 m−2/3). Mode purity shows approximately linear degradation with distance, with higher-order modes exhibiting faster degradation rates. Lower-order modes (|ℓ|≤ 2) maintain Pℓ > 0.70 at 10 km, supporting long-distance transmission, while |ℓ| =5 degrades to Pℓ =0 .38 at the same distance. These findings highlight a critical trade-off in free-space optical communication that employs OAM multiplexing. While higher-order modes offer greater spectral efficiency, their susceptibility to both turbulence and propagation loss makes lower-order modes substantially more robust for long-distance links. These findings highlight a critical trade-off in free-space optical communication that employs OAM multiplexing. While higher-order modes offer greater spectral efficiency, their susceptibility to both turbulence and propagation loss makes lower-order modes substantially more robust for long-distance links. This analysis Fig. 10. OAM mode purity characterization under atmospheric turbulence.
Fig. 9. Multiplexed and Demultiplexed output image.
Scientific Reports | (2026) 16:8921 20| https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/ also establishes the framework’s sensitivity to propagation distance, quantifying the graceful degradation of mode purity essential for long-haul link planning. Impact of pointing errors on system performance We evaluate BER degradation under the combined effects of turbulence and pointing errors. Figure 11 shows BER versus the normalized pointing jitter σp/ωz under moderate turbulence. Without compensation, BER exhibits approximately exponential growth with pointing error, reaching 0.095 at σp/ωz = 0.5. With our proposed hybrid mitigation, where tip/tilt correction is handled by the adaptive optics subsystem and residual distortions are compensated by the DCNN-TCSGm equalizer, the BER is significantly suppressed to 0.018 at the same pointing error level, representing an 81% improvement. This demonstrates the system’s resilience to misalignment, a critical factor for long-distance terrestrial links. Comparative analysis To evaluate the improvements of the proposed methodology, we compare its performance with existing approaches, namely DFE38 and conventional MDM-FSO44 systems. This comparison highlights enhancements in signal quality, BER, and data throughput under varying atmospheric conditions. The Time (s) versus Error (%) metric is used to assess how error percentage varies with time, reflecting the impact of dynamic AT on FSO links. A reduced and more stable error profile demonstrates the effectiveness of the proposed mitigation techniques.
Error(%)= Number of Erroneous Bits (t) Total Number of Transmitted Bits (t) × 100% (72) In Eq. (72), Error (%) denotes the ratio of the Number of Erroneous Bits (t), i.e., bits incorrectly received within a time interval around t, to the Total Number of Transmitted Bits (t) during the same interval. The error percentage over time reflects the system’s temporal stability and robustness under dynamically varying atmospheric conditions. As shown in Fig. 12, the proposed method starts with an error rate of 1.8 % at 0.1s, gradually increasing to 4.5 % at 1.0s. In contrast, MDM-FSO shows a sharp increase from 6.1 % to 9.5%, while DFE rises from 5.3% to 8.0%. This consistently lower error trend highlights the superior turbulence adaptation achieved through the integration of structured light beams, AO, and DCNN-TCSGm, which collectively enhance signal clarity and resilience against rapid environmental fluctuations. Time (s) vs. Signal (V) graph examines the variation of received optical signal voltage under turbulence- induced refractive index fluctuations, leading to intensity scintillation. Since scintillation is time-dependent, tracking the signal voltage over time reflects the effectiveness of AO and mitigation techniques in maintaining stable power for reliable decoding. A relatively flat signal trace indicates effective suppression of turbulence effects.
Signal (V) = Vreceived(t) (73) In Eq. (73), Signal (V) is the instantaneous voltage of the received optical signal at time (t) and Vreceived is the measured voltage of the signal at the receiver. Figure 13 shows the numerical results of time vs. signal. The proposed system demonstrates steady voltage growth from 0.93V to 1.02V within 1s, indicating strong resilience to atmospheric fading. By contrast, MDM- FSO starts at 0.70V and peaks at 0.80V, while DFE marginally increases from 0.75V to 0.84V. This improvement Fig. 11. BER vs. normalized pointing error for different compensation schemes.
Scientific Reports | (2026) 16:8921 21 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/