Data collection A robust training dataset for turbulence mitigation was generated by simulating structured OAM beam propagation under varying AT using a multi-phase screen model. Each screen represented Kolmogorov- spectrum turbulence, parameterized by Rytov variance σ2 R, refractive index structure constant C2 n, wind speed, and propagation distance. Simulations spanned 500m to 5km, covering short to long-distance terrestrial and inter-satellite dynamics were captured via frozen flow and Taylor hypothesis models, yielding frame-by-frame distortion sequences. In total, 5000 OAM beam sequences across 10 turbulence profiles and 5 beam types (Gaussian, Bessel, Airy, Vortex, hybrid) with up to 50 frames per sequence were formatted as tensors for DCNN- TCSGm training. MATLAB R2023a with custom wave optics modules was used. This dataset enables the model to learn both spatial features and temporal fading patterns of turbulence-distorted beams. Turbulence limits OAM image quality The Proposed methodology addresses FSO communication challenges by first tackling turbulence limits on OAM image quality. To counter image degradation and unreliability under strong AT, we implement a multi- pronged solution. This applies to resilient, self-repairing structured beams, such as Airy and Bessel beams, as well as optimized Vortex OAM beams, which perform well under turbulent conditions. Combined with AO for real-time wavefront correction and PSO for intelligent parameter optimization, the system adapts to changing atmospheric conditions and preserves data over long distances. The proposed approach integrates three structured light beams: Bessel, Airy, and Vortex OAM. Implementing these beams requires modifying a conventional Gaussian-beam terminal. The key change is the addition of a wavefront-shaping element, such as a spatial light modulator (SLM) or a spiral phase plate, that imprints a specific spatial phase pattern (e.g., a helical profile for OAM) onto the laser output. This increases terminal complexity, cost, and alignment sensitivity. Fig. 1. Overall architecture of the proposed FSO framework. The optical domain comprises structured beam generation, multiplexing, atmospheric propagation, and wavefront sensing. The DSP domain at the receiver performs signal recovery and turbulence mitigation using deep learning and adaptive optimization modules, including DCNN, DNFIS, and PSO.
Scientific Reports | (2026) 16:8921 7 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
For future deployment, particularly in size, weight, and power-constrained (SWaP) applications like satellite communications, dielectric metasurfaces (discussed in section 3.5c) offer a promising path toward compact, integrated transceivers that can generate and manipulate structured light efficiently. these beams are selected for their unique ability to adapt to varying conditions and maintain data didelity across extended transmission ranges. (a) Besel-Gauss Beams (BGBs) Bessel-Gauss beams are non-diffracting, meaning they can propagate over a finite range while maintaining their transverse intensity profile without divergence. A key trait is their self-reconstruction property, allowing them to reform after encountering obstacles, thus maintaining continuous signal propagation. The electric field distribution of a Bessel beam in cylindrical coordinates (r, ϕ, z) is generally expressed as a solution of the Helmholtz equation, as shown in Eq. (19).
E(r, ϕ, z) = A0Jl(ktr) · exp(ikzz) · exp(ilϕ) (19) In this expression, A0 is the constant amplitude, Jl is the Bessel function of the first kind of order l (the topological charge), k_t is the transverse wave vector, kz is the longitudinal wave vector, and r and ϕ are the radial and azimuthal coordinates, respectively. (b) Finite-Energy Airy Beams (FABs) Airy beams are unique for their self-accelerating behavior, allowing them to follow curvilinear paths, and for their strong resistance to diffraction. They also possess self-healing properties, enabling reconstruction after distortions during propagation through turbulence. Under the paraxial approximation, where light rays are nearly parallel to the optical axis, their propagation is governed by a specific form of the diffraction equation.
i∂ψ ∂ς + 1 2 ∂2ψ ∂σ2 = 0 (20) In Eq. (20), i denotes the imaginary unit, while ψ represents the complex amplitude of the light field, capturing both phase and magnitude. The variable ς is the normalized propagation distance along the beam axis, and σ is the normalized transverse spatial coordinate, which renders the equation dimensionless for easier analysis. The exact non-diffracting solution for an ideal Airy beam, given by Eq. (21).
ψ(σ, ς) = Ai ( σ −ς2 4 ) exp ( i ( σς 2 −ς3 12 )) (21) Here, Ai denotes the Airy function, defining the characteristic beam shape. The term ς2/4 describes the parabolic trajectory, illustrating its self-accelerating property. Exponential terms capture phase evolution during propagation. (c) Vortex Beams (VBs) Vortex beams carry OAM via a helical phase front, producing a distinctive doughnut-shaped intensity profile due to a central phase singularity. Their OAM property is crucial for multiplexing data channels, increasing capacity, and enhancing the security of FSO systems. The complex electric field of a Vortex beam can be represented in terms of its azimuthal dependence as:
Efield(ϕ) = A0 exp(ilϕ) (22) In Eq. (22), Efield is the complex electric field of the beam, and A0 is its constant amplitude. The term exp(ilϕ) represents the helical phase variation, where l is the topological charge (an integer) defining the phase winding and the OAM carried per photon. The symbol ϕ denotes the azimuthal angle in the plane perpendicular to propagation. The implementation also incorporates a multi-faceted security strategy, including beam generation, shaping, and adaptive beam selection and switching. Spatial Light Modulators (SLMs) serve as reconfigurable optical elements, converting Gaussian input beams into desired structured profiles while precisely controlling amplitude, phase, and polarization. A dynamic switching mechanism adapts the transmitted beam type based on current atmospheric conditions and pre-agreed communication protocols, enhancing data protection and mitigating electromagnetic interference and signal attenuation in FSO links. The total attenuation, αtotal, experienced by a structured beam is the sum of contributions from individual atmospheric conditions and can be estimated to assess weather-induced signal loss, as detailed in Eq. (23).
αtotal = αmist + αflurry + αdownpower + αdispersion (23) The total attenuation, αtotal, represents the overall signal intensity reduction over a given distance. Individual contributions include αmist (fog or haze), αflurry (snowfall), αdownpower (rainfall), and αdispersion (scattering by atmospheric particles), which redirects light away from its intended path. Recovering the original Gaussian Scientific Reports | (2026) 16:8921 8 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
beam from the incoming structured light is typically a computational process at the receiver, compensating for structural changes and atmospheric wavefront distortions. The retrieval process can be expressed in Eq. (24).
Goutput = F{Eincident} ⊗F{TSLM} (24) Here, Goutput is the complex electric field of the reconstructed Gaussian beam at the output, and F denotes the Fourier transform from the spatial to the frequency domain. Eincident is the complex amplitude of the received structured light beam, and TSLM is the complex transmittance of the SLM, specifying how it modifies amplitude and phase. The symbol ⊗ represents a convolution operation, combining two functions to quantify their overlap. This approach enhances security and applies to various structured beams. According to experimental data, Bessel and Vortex beams maintain integrity over > 3 km, while Airy beams sustain integrity over ∼2 km under similar conditions. (d) PSO for System Parameter Tuning The PSO algorithm is employed to intelligently optimize key parameters of the FSO system, enhancing its resilience to atmospheric turbulence. The proposed PSO algorithm dynamically tunes parameters such as the beam waist (w0), divergence angle, and AO correction factors in real-time. The objective is to maximize the SNR and minimize the BER under turbulent conditions. The PSO algorithm was implemented with the following parameters: swarm size N = 50, inertia weight w = 0.8, and acceleration coefficients c1 = c2 = 1.5. These values were selected based on preliminary grid search trials across representative turbulence scenarios to balance exploration and exploitation. Particle velocities were clamped to ±20% of the parameter search range to prevent divergence. The optimization typically converged within 100 iterations, with termination triggered when the global best fitness improvement remained below 10−4 for 10 consecutive iterations. The fitness function (F) for the PSO is formulated to maximize a composite performance score based on received signal power and stability. For a set of system parameters X = [w0, θdiv, αAO, . . .], the fitness is evaluated as shown in Eq. (25)
F(X) = w1 · SNRavg(X) + w2 · (1 −BERavg(X)) −w3 · σ2 I(X) (25) Where SNRavg(X) is the average SNR ratio, BERavg(X) is the average bit error rate and σ2 I(X) is the variance of the signal intensity, which PSO seeks to minimize. The coefficients w1, w2, w3 are weighting coefficients that prioritize the different performance metrics. A swarm of particles, each representing a candidate solution X , explores the parameter space. The position Xi and velocity Vi of each particle i are updated iteratively based on their personal best position Pbest,i and the global best potion found by the swarm Gbest, given by Eq. (26).
V(k+1) i = wV(k) i
- c1r1(Pbest,i −X(k) i ) + c2r2(Gbest −X(k) i ) (26)
X(k+1) i = X(k) i
- V(k+1)
i
(27)
Here, w is the inertia weight, c1 and c2 are the acceleration coefficient, and r1, r2 are random values. By
converging towards the parameter set Gbest the maximizes F(X), the PSO algorithm ensures the FSO system is
optimally configured to maintain link integrity, compensating for dynamic atmospheric effects more effectively
than static parameter settings.
Dynamic neural-fuzzy equalization for MDM-FSO channels
To combat the nonlinear and time-varying distortions in long-haul MDM-FSO links, we implement a DNFIS as
an intelligent equalizer. Unlike static equalizers, the DNFIS adapts in real-time to the dynamics of the atmospheric
channels. Its function is to estimate and cancel the inter-modal crosstalk and nonlinear phase distortion
introduced by turbulence, thereby recovering the original transmitted symbols s(t) = [s−L(t), . . . , sL(t)]T .
(a) System inputs and outputs
The DNFIS is integrated at the receiver after OAM mode demultiplexing. For each received OAM mode (ℓ), the
equalizer takes two primary inputs derived from the receiver signal:
- Instantaneous normalized power (m1): The receiver power in mode ℓ, normalized by the average power. Thhis is a direct indicator of intensity scintillation, given by Eq. (28).
m1(t) = |rℓ(t)|2 E[|rℓ(t)|2] (28) 2. Phase gradient (m2): The rate of change of the unwrapped phase of rℓ(t), which captures phase fluctuations and beam wander effects, given by Eq. (29).
m2(t) = d dt arg(rℓ(t)) (29) Scientific Reports | (2026) 16:8921 9 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
The output of the DNFIS for the mode ℓ is a complex-valued correction factor, vf nn,ℓ(t), which is applied to the received signal to produce the equalized symbol. (b) Fuzzy rule-based for FSO channel The fuzzy logic core of the DNFIS uses rules that directly map atmospheric conditions to correction strategies. The input m1 and m2 are fuzzified into linguistic variables, such as Low, Medium, and High for power, and Stable, Fluctuating, and Erratic for phase gradient. (c) Neural network adaptation The neural network component continuously optimizes the membership functions and rule weights based on the error between the equalized output and the known training symbols (or decisions from a preliminary equalizer). The cost functionJ(t) is the mean squared error between the equalized symbol and the target symbol, as shown in Eq. (30) J(t)= 1 2 |ˆsℓ(t) − sℓ(t)|2 (30) The learning algorithm for adjusting the weights ( W) of the output layer, considering a learning rate µ and a momentum factor ω for faster convergence, is given by Eq. (31). ∆W (t)= −µ ∂J (t) ∂W = µe(t)L3 (31) The updated weight can be expressed as in Eq. (32). W (t)= W (t − 1 )+∆ W (t)+ ω[W (t − 1) − W (t − 2)] (32) T o modify the core parameters of the association function for the hidden layer, the learning method, as given by Eq. (33), is used.
∆ηij = −µ ∂J (t) ∂ηij = µe(t)WL 3 2(mi − ηij ) V 2 j (33) The updated center parameter is given by Eq. (34). ηij (t)= ηij (t − 1 )+∆ ηij (t)+ ω[ηij (t − 1) − ηij (t − 2)] (34) (d) Integration with Overall System The DNFIS operates in tandem with the DCNN-TCSGm. While the DCNN-TCSGm provides a predictive, frame-level correction, the DNFIS performs a sample-by-sample, adaptive equalization, making it highly effective against rapid scintillation and phase noise that characterize long-distance FSO links. This dual approach ensures robust signal recovery across both slow and fast fading conditions. Enhanced deep learning turbulence mitigation The proposed approach employs advanced deep learning for turbulence mitigation in FSO communication. To enhance the speed and efficiency of turbulence compensation, we utilize a DCNN-TCSGm framework. The DCNN, integrated with a generative component, learns temporal correlations of fading channels in AT, enabling accurate prediction and compensation of complex dynamic correlations, especially under slower fading conditions. This results in faster convergence, improved resilience, and higher system throughput. The architecture of the proposed DCNN-TCSGm method is shown in Fig. 2. Model architecture specifications The DCNN TCSGm is engineered to process sequences of turbulence-distorted OAM beam images, extracting spatial features through convolutional layers. In contrast, its dedicated temporal convolution layers learn how turbulence evolves over consecutive frames. This enables predictive compensation, anticipating distortion before it fully develops, which is fundamental to its accuracy advantage over conventional reactive equalizers. The DNFIS provides complementary, sample-by-sample adaptive equalization. Table 2 illustrates the model architecture specifications for DCNN-TCSGm and DNFIS. (a) DCNN Architecture for Turbulence Compensation To enhance the real-time turbulence mitigation in FSO Communication, we implement a DCNN that learns to correct phase and intensity distortions from TV-AT. The DCNN takes as input a sequence of distorted OAM beam images and produces compensated beam profiles, enabling robust signal recovery and reduced BER. Let X ∈ RT ×H×W be the input tensor representing T sequential OAM beam intensity frames of size H × W , captured under turbulent conditions. Each convolutional layer extracts feature maps using the operation defined in Eq. (35) Scientific Reports | (2026) 16:8921 10| https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/ Component DCNN-TCSGm DNFIS Input 10 sequential frames (256 × 256 pixels) 2 features: normalized power, phase gradient Core layers 5 convolutional layers Filters: 32 → 64 → 128 → 64 → 32 Kernel: 3 × 3, stride 1, ReLU activation Batch normalization after each layer 9 fuzzy rules (3 × 3 combinations) Takagi-Sugeno-Kang inference 3 Gaussian MFs per input Temporal processing 1D temporal convolution Kernel size: 5 Captures turbulence evolution across frames Adaptation Learning rate: µ = 0.01 Momentum: α = 0.9 Training Adam optimizer (β1 = 0.9, β2 = 0.999) Learning rate: 0.001 Batch size: 32 Early stopping on validation loss Gradient descent with momentum Decision-directed training Key function Predictive compensation: learns temporal patterns to anticipate turbulence distortion Adaptive equalization: sample- by-sample correction based on instantaneous channel state Table 2. Model Architecture Specifications for DCNN-TCSGm and DNFIS.
Fig. 2. DCNN-TCSGm.
Scientific Reports | (2026) 16:8921 11 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/