The 2012 demonstration of polarization-multiplexed single-carrier 1024-QAM (60 Gbit/s, 3 Gsymbol/s per polarization, 150 km, 4.05 GHz optical bandwidth) proved that dense square constellations can be transmitted when transmitter frequency-domain equalization, pilot-tone optical phase locking, and offline split-step digital back-propagation are combined, but it left a power penalty of about 7 dB at the FEC threshold and a set of acknowledged practical limitations. The problem is whether this capability can be turned into a practical transmission technology: (i) reduce the residual implementation penalty of ultra-high-order single-carrier QAM toward back-to-back performance, identifying and suppressing its physical origins; (ii) establish the real-time implementability and complexity trade-offs of digital nonlinear compensation in transceiver hardware; and (iii) realize multi-channel (WDM) operation at the projected ~13.8 bit/s/Hz net spectral efficiency, including inter-channel nonlinear interactions. Later work by the same group and by others raised the demonstrated multiplicity to 2048- and 4096-QAM and identified guided-acoustic-wave Brillouin scattering phase noise as a dominant error-floor source, yet as of 2026 the transceiver implementation ceiling — not the fiber — still caps the achievable QAM order, real-time nonlinear compensation remains unrealized at these constellation sizes, and a full WDM-grid demonstration is absent.

Background

Coherent optical transmission detects the phase and amplitude of the optical field (rather than only its intensity) by mixing the received signal with a local-oscillator laser, which enables digital signal processing (DSP) of the recovered field. In quadrature amplitude modulation (QAM), each symbol carries a point on a square grid of in-phase and quadrature amplitudes; 1024-QAM places 32×32=102432\times32=1024 points, so each symbol carries 10 bits, and neighboring points are separated by only a small amplitude step and a phase difference of order one degree. The modulation therefore demands very high optical signal-to-noise ratio (OSNR, the ratio of signal power to amplified-spontaneous-emission noise in a reference bandwidth), extremely low laser phase noise, and near-ideal optoelectronic hardware: the tolerance of the 1024-QAM constellation to residual phase fluctuations is about 0.95 degrees, and transmitter modulator ripple or digitizer quantization noise of only a percent already corrupts the constellation.

Koizumi, Toyoda, Yoshida and Nakazawa (Optics Express 20, 12508, 2012; DOI 10.1364/oe.20.012508, https://doi.org/10.1364/oe.20.012508) reported the first single-carrier 1024-QAM coherent transmission: 60 Gbit/s polarization-multiplexed (the same symbol stream sent on two orthogonal fiber polarizations), 3 Gsymbol/s per polarization, over 150 km of large-effective-area fiber, within an optical bandwidth of only 4.05 GHz. To reach this they (a) pre-compensated the MHz-period ripple of the lithium-niobate (LiNbO3_3) in-phase/quadrature (IQ) modulator with a frequency-domain equalizer (FDE, a digital filter that flattens the modulator's frequency response) of sub-MHz resolution, which improved the back-to-back (transmitter connected directly to the receiver, with no fiber span) error vector magnitude (EVM, the normalized deviation of the received symbols from their ideal constellation points) from 1.10% to 0.79%; (b) phase-locked the receiver local oscillator to a co-propagating pilot tone with an optical phase-locked loop (OPLL, a control loop that tracks the phase difference between the pilot tone and the local oscillator), reaching 0.46 degrees of residual integrated phase noise after 150 km; and (c) applied digital back-propagation (DBP), the method introduced by Ip and Kahn (2008; DOI 10.1109/jlt.2008.927791, https://doi.org/10.1109/jlt.2008.927791), which inverts fiber propagation by numerically solving the coupled Manakov equations — the standard averaged model of fiber propagation that combines chromatic dispersion and the Kerr nonlinearity under rapid random polarization coupling — backwards from the receiver using a split-step Fourier scheme with 9.375-km steps, compensating chromatic dispersion together with self- and cross-polarization-phase modulation; DBP raised the optimum launch power from -6 to -1 dBm. The Nyquist raised-cosine pulse-shaping roll-off factor (the fractional excess bandwidth of the Nyquist pulse shape) was set to 0.35 (occupied bandwidth = symbol rate × 1.35 = 4.05 GHz), a conservative choice preserving clock recovery (symbol timing extraction from the received waveform) and eye opening (distinguishability of neighboring symbols in the time-domain trace). After 150 km both polarizations met the forward-error-correction (FEC, added redundancy that lets the receiver correct residual symbol errors) threshold of a bit-error ratio (BER) =2×10−3=2\times10^{-3}, but with a power penalty of approximately 7 dB relative to back-to-back, and the transmission was a single channel processed offline. The paper projected a net multi-channel spectral efficiency of 13.8 bit/s/Hz once 7% FEC overhead is included, assuming wavelength-division-multiplexed (WDM, many wavelength channels carried on one fiber) neighbors spaced exactly at 4.05 GHz with no guard band.

The question this raises is the practical completion of that demonstration. The penalty of ~7 dB at the FEC threshold means the system operates far from theory (the OSNR margin above the ~27.8 dB requirement is spent on residual impairments); the DSP, in particular the nonlinear compensation, ran offline at fixed computational settings; and the projected spectral efficiency assumed inter-channel nonlinear interference between WDM neighbors could be neglected, which single-channel DBP does not compensate. Later ultra-high-order-QAM work introduced the further identified impairment of guided acoustic-wave Brillouin scattering (GAWBS), in which thermally excited acoustic modes of the glass core phase-modulate the signal and produce a floor of phase noise that scales with fiber length and is severe exactly at constellation sizes of 1024 and above.

Problem Statement

The 2012 single-carrier 1024-QAM demonstration achieved BER below the FEC threshold of 2×10−32\times10^{-3} over 150 km only at the cost of an approximately 7 dB power penalty at that threshold, with offline DSP, a single wavelength channel, and a conservative Nyquist roll-off of 0.35. The problem, in the source's own scope, is to complete this demonstration into a practical transmission capability by resolving its three coupled limitations:

  1. Identify and suppress the residual impairments — laser and OPLL phase noise, quantization noise, uncompensated fiber nonlinearity, and (as later established) GAWBS scattering phase noise — so that the power penalty of 1024-QAM single-carrier transmission at the FEC threshold is brought to within about 1 dB of back-to-back performance, down from the demonstrated ~7 dB.
  2. Establish the scalability, real-time implementability, and computational-complexity trade-offs of split-step digital back-propagation (and equivalent nonlinear compensators) in real-time transceiver hardware at the step resolution and constellation density the demonstration required, rather than in offline processing.
  3. Demonstrate multi-channel (WDM) operation of single-carrier 1024-QAM, including the inter-channel nonlinear interactions between wavelength-division-multiplexed neighbors spaced at or near the single-channel bandwidth, and validate the projected net spectral efficiency of approximately 13.8 bit/s/Hz with 7% FEC under such multi-channel conditions.

The source also asked how far the transmitter Nyquist roll-off factor can be reduced below 0.35 while preserving clock extraction and eye opening, in order to shrink the occupied optical bandwidth below 4.05 GHz for a 3-Gsymbol/s signal; later shaped-constellation demonstrations — probabilistically shaped 4096-QAM at roll-off 0.01 and geometrically shaped 2048-QAM at 15 GBd — have largely superseded that sub-question, so it is retained as source context only and is not among the three limitations to be resolved.

The common objective uniting these three asks is a real-time, multi-channel, spectrally efficient ultra-high-order single-carrier QAM transmission system whose implementation penalty approaches its back-to-back limit, or a rigorous demonstration that such a system is unattainable with realistic transceiver hardware.

References

  1. Yuki Koizumi, Kazushi Toyoda, Masato Yoshida, and Masataka Nakazawa, "1024 QAM (60 Gbit/s) single-carrier coherent optical transmission over 150 km," Optics Express 20(11), 12508 (2012). DOI: 10.1364/oe.20.012508, https://doi.org/10.1364/oe.20.012508
  2. Ezra Ip and Joseph M. Kahn, "Compensation of Dispersion and Nonlinear Impairments Using Digital Backpropagation," Journal of Lightwave Technology 26(20), 3416–3425 (2008). DOI: 10.1109/jlt.2008.927791, https://doi.org/10.1109/jlt.2008.927791
  3. Masato Yoshida, Toshihiko Hirooka, Keisuke Kasai, and Masataka Nakazawa, "Adaptive 4~64 QAM real-time coherent optical transmission over 320 km with FPGA-based transmitter and receiver," Optics Express 22(13), 16520 (2014). DOI: 10.1364/OE.22.016520, https://doi.org/10.1364/OE.22.016520
  4. Shohei Beppu, Keisuke Kasai, Masato Yoshida, and Masataka Nakazawa, "2048 QAM (66 Gbit/s) single-carrier coherent optical transmission over 150 km with a potential SE of 15.3 bit/s/Hz," Optics Express 23(4), 4960 (2015). DOI: 10.1364/oe.23.004960, https://doi.org/10.1364/oe.23.004960
  5. Masato Yoshida, Shohei Beppu, Keisuke Kasai, Toshihiko Hirooka, and Masataka Nakazawa, "1024 QAM, 7-core (60 Gbit/s x 7) fiber transmission over 55 km with an aggregate potential spectral efficiency of 109 bit/s/Hz," Optics Express 23(17), 20760 (2015). DOI: 10.1364/oe.23.020760, https://doi.org/10.1364/oe.23.020760
  6. Samuel L. I. Olsson, Junho Cho, Sethumadhavan Chandrasekhar, Xi Chen, Peter J. Winzer, and Sergejs Makovejs, "Probabilistically shaped PDM 4096-QAM transmission over up to 200 km of fiber using standard intradyne detection," Optics Express 26(4), 4522 (2018). DOI: 10.1364/OE.26.004522, https://doi.org/10.1364/OE.26.004522
  7. Masaki Terayama, Seiji Okamoto, Keisuke Kasai, Masato Yoshida, and Masataka Nakazawa, "4096 QAM (72 Gbit/s) Single-Carrier Coherent Optical Transmission with a Potential SE of 15.8 bit/s/Hz in All-Raman Amplified 160 km Fiber Link," Optical Fiber Communication Conference (OFC) 2018, paper Th1F.2 (2018). DOI: 10.1364/ofc.2018.th1f.2, https://doi.org/10.1364/ofc.2018.th1f.2
  8. Edson P. da Silva, Frederik Klejs, Mads Lillieholm, Shajeel Iqbal, Metodi P. Yankov, Julio C. M. Diniz, Toshio Morioka, Leif K. Oxenlowe, and Michael Galili, "Experimental Characterization of 10 × 8 GBd DP-1024QAM Transmission with 8-bit DACs and Intradyne Detection," 2018 European Conference on Optical Communication (ECOC), pp. 1–3 (2018). DOI: 10.1109/ECOC.2018.8535205, https://doi.org/10.1109/ECOC.2018.8535205
  9. Masato Yoshida, Naoya Takefushi, Keisuke Kasai, Toshihiko Hirooka, and Masataka Nakazawa, "Suppression of large error floor in 1024 QAM digital coherent transmission by compensating for GAWBS phase noise," Optics Express 27(25), 36691 (2019). DOI: 10.1364/oe.27.036691, https://doi.org/10.1364/oe.27.036691
  10. Xi Chen, Sethumadhavan Chandrasekhar, Junho Cho, and Peter Winzer, "Transmission of 30-GBd polarization-multiplexed probabilistically shaped 4096-QAM over 50.9-km SSMF," Optics Express 27(21), 29916 (2019). DOI: 10.1364/oe.27.029916, https://doi.org/10.1364/oe.27.029916
  11. Xi Chen, Junho Cho, and Di Che, "Experimental Quantification of Implementation Penalties from Laser Phase Noise for Ultra-High-Order QAM Signals," 2020 European Conference on Optical Communications (ECOC), pp. 1–4 (2020). DOI: 10.1109/ECOC48923.2020.9333253, https://doi.org/10.1109/ECOC48923.2020.9333253
  12. Frederik Klejs, Edson P. da Silva, Mads Lillieholm, Metodi P. Yankov, Toshio Morioka, Leif K. Oxenløwe, and Michael Galili, "Spectrally Efficient DP-1024QAM 640 Gb/s Long Haul Transmission using a Frequency Comb," Optical Fiber Communication Conference (OFC) 2020, paper M4K.4 (2020). DOI: 10.1364/ofc.2020.m4k.4, https://doi.org/10.1364/ofc.2020.m4k.4
  13. Yuta Wakayama, Thomas Gerard, Eric Sillekens, Lídia Galdino, Domaniç Lavery, Robert I. Killey, and Polina Bayvel, "2048-QAM transmission at 15 GBd over 100 km using geometric constellation shaping," Optics Express 29(12), 18743 (2021). DOI: 10.1364/OE.423361, https://doi.org/10.1364/OE.423361
  14. Meng Yang, Aiying Yang, Peng Guo, and Wenkai Wan, "Low-complexity clustered raised-cosine-optimized FIR time-domain digital back-propagation for coherent optical transmission systems," Optics Express 33(16), 36770 (2025). DOI: 10.1364/oe.570278, https://doi.org/10.1364/oe.570278
  15. Md Ghulam Saber and Zhiping Jiang, "System-Level Limits of Higher-Order QAM in Hollow-Core Fiber Systems," Journal of Lightwave Technology 44, 8069–8083 (2026). DOI: 10.1109/JLT.2026.3718469, https://doi.org/10.1109/JLT.2026.3718469