and performance. High-data-rate inter-satellite FSO links using MDM have achieved high speeds over long distances40, though system complexity and integration remain bottlenecks. Additionally, single-layer dielectric metasurfaces allow compact multi-dimensional demultiplexing of wavelength, SAM, and OAM across 132 channels41, yet the practical integration of nanophotonic components remains a significant hurdle. To address the aforementioned issues, this paper proposes a high-capacity and resilient optical communication system that integrates self-healing structured light beams with advanced adaptive equalization and deep learning techniques to enhance signal integrity over long distances. This paper aims to significantly improve the robustness, data rate, and adaptability of FSO links for both terrestrial and inter-satellite applications. The key objectives include mitigating nonlinear atmospheric distortions, enabling real-time turbulence prediction and compensation, and developing highly integrated multiplexing solutions. The main contributions of this paper are given below: • A novel, intelligent FSO framework that synergistically integrates adaptive optics (AO) with particle swarm optimization (PSO) for dynamic wavefront correction, and structured OAM beams for inherent turbulence resilience. • A closed-loop intelligent signal processing core that uniquely co-integrates a Deep Convolutional Neural Net work (DCNN-TCSGm) for predictive turbulence channel estimation with a Dynamic Neural-Fuzzy Inference System (DNFIS) for adaptive, nonlinear equalization. This core enables real-time, proactive compensation. • The unified integration of this intelligent core with compact, dielectric metasurface-based OAM multiplexers/ demultiplexers in the mid-infrared spectrum, proposing a pathway to highly integrated transceivers. • A hybrid WDM-MDM (wavelength and mode division multiplexing) scheme for the mid-infrared spectrum, significantly enhancing the spectral-spatial capacity for inter-satellite links. The integration of these components provides a novel, intelligent, and turbulence-resilient optical transmission system that advances the state-of-the-art in channel robustness, data capacity, and system adaptability. Paper organization The remainder of this paper is structured as follows. Section 2 describes the system model. Section 3 details the proposed methodology. Section 4 presents and discusses the simulation results and comparative analysis. Finally, Section 5 provides the conclusion. System model The base of our OAM-based FSO communication framework relies on a formalized mathematical systems model that captures the physics of structured beam propagation through turbulence and signal degradation due to atmospheric impairments. This model also serves as a baseline for incorporating smart signal processing, equalization, and adaptive corrections discussed later in the methodology. We begin by modeling the structured light beam produced by the transmitter, which carries information encoded onto separate OAM modes. An optical field carrying an OAM mode with topological charge (El) can be mathematically described by Eq. (1)
El(r, ϕ, z) = A(r, z) · eilϕ (1) In this Equation, r is the radial distance from the beam’s center, A(r, z) denotes the radial amplitude distribution of the beam, typically defined by its type (e.g., Gaussian, Bessel, Airy). The azimuthal angle ϕ and propagation distance z determine the phase along the vortex beam. The azimuthally varying exponential term eilϕ represents the helically structured wavefront, producing the central “doughnut” intensity profile characteristic of OAM beams. In a multiplexed system, multiple OAM modes can be combined to carry parallel data streams. The resulting optical field at the transmitter, Etx, is expressed as in Eq. (2).
Etx(r, ϕ, z) = L ∑ l=−L sl(t) · El(r, ϕ, z) (2) Where sl(t) is the symbol transmitted on the mode l, and 2L + 1 represents the total number of OAM modes used. This expression captures the concept of MDM, exploiting the orthogonality of OAM modes to transmit multiple independent data streams over a single optical channel. During propagation, these structured beams experience phase distortions and intensity fluctuations due to spatial and temporal variations in the atmospheric refractive index. The accumulated phase perturbation Φ(r, ϕ, z) over the propagation distance is modeled by the integral given in Eq. (3).
Φ(r, ϕ, z) = 2π λ ˆ z 0 δn(r, ϕ, z′)dz′ (3) In this expression, δn(r, ϕ, z′) represents the stochastic fluctuations in the refractive index due to turbulence, and λ is the wavelength of the optical beam. This integral accounts for the total accumulated phase distortion along the propagation distance, leading to wavefront incoherence, which manifests as beam wandering, scintillation, and ultimately, mode crosstalk. To quantify turbulence severity, the Rytov variance σ2 R is introduced, defined in Eq. (4).
σ2 R = 1.23C2 nk7/6z11/6 (4) Scientific Reports | (2026) 16:8921 4 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
Here, C2 n denotes the atmospheric structure constant characterizing turbulence strength, k =2 π/λ is the optical wave number, and z the link distance. The Rytov variance directly reflects intensity scintillation, with values above one indicating strong turbulence, and signal loss could be expected. Atmospheric turbulence strength is categorized using the Rytov variance σ2 R (Eq. 4): weak turbulence (σ2 R < 0.3), moderate turbulence (0.3 ≤ σ2 R ≤ 5), and strong turbulence (σ2 R > 5). The Fried parameter, r0 = (0.423k 2C2 nz) −3/5 , quantifies the transverse coherence length of atmospheric phase distortions. For terrestrial FSO links in the mid-infrared regime (λ ∼ 3–5 µm) over distances of 1–5 km, typical r0 values range from approximately 2 cm under strong turbulence to 10–20 cm under weak turbulence. The distortion severity scales as (D/r0 )5/3 , guiding AO requirements. Intensity fluctuations are characterized by the scintillation index, which for a plane wave is expressed as in Eq. (5).
σ2 I = exp ( 0.49σ2 R ( 1+1 .11σ12/5 R ) 7/6 + 0.51σ2 R ( 1+0 .69σ12/5 R ) 5/6 ) − 1 (5) The turbulence spectrum includes the inner scale l0 (∼ 1 m to 100 m) and outer scale L0 (∼ 1 m to 100 m) effects via the modified V on Kármán spectrum. W eather-induced attenuation follows established models: fog/ haze (Kim model), rain (α = aRb), and snowfall, which are incorporated into the total path loss αtotal in Eq. (23). At the receiver, the structured beam is further degraded by mode-dependent fading, and the received signal r(t), as shown in Eq. (6).
r(t)= L∑ l=−L (hl(t)sl(t)+ n(t)) (6) Where hl(t)= αl(t)eiθl (t) is the complex fading coefficient for the mode l, incorporating both amplitude attenuation and phase shift, and n(t) is Additive White Gaussian Noise (AWGN). This model represents the TV effects of atmospheric conditions and serves as the input for equalization and error correction. Turbulence- induced aberrations diminish OAM mode orthogonality, leading to inter-modal interference; thus, the system response is best expressed in channel matrix form, as defined by Eq. (7) Hm,n = ⟨Erecv m | Etrans n ⟩ (7) Here H ∈ CM ×M is the MDM channel matrix, with each element Hm,n representing the crosstalk between the transmitted mode n and received mode m. The coupling coefficient is defined by the inner product projection. In practical FSO systems, the received signal is impaired by the combined effects of path loss, AT-induced fading, and pointing errors. The composite channel coefficient for a single mode is modeled as the product: h = hl · ha · hp, (8) Where hl is the deterministic path loss, ha is the atmospheric turbulence fading coefficient, and hp is the pointing error loss factor. Pointing Error Model: Misalignment due to platform vibration and tracking jitter is modeled by a Rayleigh- distributed radial displacement r:
fr (r)= r σ2s exp ( − r2 2σ2s ) ,r ≥ 0, (9) Where σ2 s is the variance of the underlying Gaussian jitter. For a circular aperture of radius (a) and a Gaussian beam with a waist ωz at the receiver, the fraction of power collected (the pointing loss factor) is approximated by:
hp ≈ A0 exp ( − 2r2 ω2z ) , with A0 = [ erf ( √πa√ 2ωz )]2 . (10) Atmospheric Turbulence Model: For moderate-to-strong turbulence conditions, the intensity scintillation I is well-characterized by the Gamma-Gamma distribution:
fI (I)= 2(αβ)(α+β)/2 Γ(α)Γ(β) I(α+β)/2−1Kα−β ( 2 √ αβI ) , I> 0, (11) Where α and β are the effective numbers of large-scale and small-scale eddies, respectively, and Kv is the modified Bessel function of the second kind. T o rigorously quantify turbulence-induced impairments, we define the inter-modal crosstalk coefficient Xm,n as the normalized power coupled from the transmitted mode n to received mode m: Scientific Reports | (2026) 16:8921 5| https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/
Xm,n(z, C2 n) = |Hm,n|2 ∑ k |Hm,k|2 , m ̸= n. (12) This coefficient increases with propagation distance z and the refractive index structure constant C2 n. The OAM mode purity Pl for a mode with topological charge l is the fraction of power remaining in the intended mode after propagation through turbulence:
Pl(z, C2 n) = |Hl,l|2 ∑ k |Hl,k|2 . (13) The degradation of Pl is influenced by the turbulence spectrum, including the effects of inner l0 and outer L0 scales, which differentially impair the phase coherence of higher-order OAM modes. To mitigate these distortions, an AO correction phase is applied, given as.
Ecorr(r, ϕ, z) = El(r, ϕ, z)e−iΦAO(r,ϕ) (14) In Eq. (14), ΦAO(r, ϕ) is the phase correction applied by the AO system. The correction is iteratively optimized using PSO to minimize wavefront error and maximize signal quality at the receiver. System performance is ultimately evaluated through the signal-to-noise ratio (SNR) and BER. The instantaneous SNR for the OAM mode l is expressed as in Eq. (15).
SNRl = |hl(t)|2E[|sl(t)|2] σ2 (15) Here, the numerator denotes the signal power collected in that mode, while σ2 represents the noise variance. A higher SNR corresponds to more reliable data reception and lower error probability. For Binary Phase Shift Keying (BPSK) and Quadrature Phase Shift Keying (QPSK) modulation, the average BER per mode is expressed using the Q-function, as shown in Eq. (16).
BERl = Q( √ 2SNRl) (16) These expressions describe the tail probability of a Gaussian distribution exceeding a given value, providing an analytic means to evaluate link reliability and compare network performance. Despite the TV-AT effects, it is important to characterize the temporal dynamics of fading. The autocorrelation function (ACF) of the fading coefficient hl(t) is modeled as in Eq. (17).
Rl(τ) = E[hl(t)h∗ l (t + τ)] (17) Given in Eq. (17), which captures how the channel condition for a particular OAM mode evolves in delay τ. For wind-driven turbulence, the ACF follows a Gaussian decay model as described by Eq. (18),
Rl(τ) = exp(−(πWτ)2 4D2 ) (18) where W is the transverse wind speed and D is the aperture diameter. These time-domain characteristics are crucial for training deep learning models, such as the proposed DCNN-TCSGm, for real-time turbulence prediction and compensation. Modeling framework distinction Our framework delineates two distinct optical link scenarios with fundamentally different physical impairments. Uplink (Ground-to-Satellite) Channel: Employs multi-phase screen AT modeling with Rytov variance σ2 R, Fried parameter r0, and weather-dependent attenuation. Pointing errors arise from ground platform vibration and atmospheric tip-tilt. Inter Satellite Link Channel: Assume negligible atmospheric turbulence C2 n ≈0. The primary impairment is pointing error due to spacecraft vibration and jitter, modeled via statistical pointing error distributions. Background radiation (solar, cosmic) is included in the noise budget. For scenarios with significant jitter, the radial pointing error is modeled as a Rayleigh-distributed variable, and the normalized pointing jitter is defined as σp/wz , where σp is the standard deviation of the jitter and w_z is the receiver beam waist. While the core adaptive signal processing algorithms (DCNN TCSGm, DNFIS) remain applicable to both, the physical layer impairment models differ substantially, as reflected in our simulation parameters. Proposed methodology The proposed methodology enhances FSO performance by strategically combining the inherent resilience of structured light beams with advanced mitigation strategies. Its integrated approach leverages AO, intelligent optimization, and sophisticated signal processing to overcome AT. The overall architecture of this research is illustrated in Fig. 1. The following subsection outlines the step-by-step actions taken as part of this work. Scientific Reports | (2026) 16:8921 6 | https://doi.org/10.1038/s41598-026-40704-2 www.nature.com/scientificreports/