Complexity Reduction for Deep Machine Learning-Based Optical Transmission Systems

Abstract

Over the past decade, there has been a massive increase in demand for bandwidth to serve bandwidth-hungry applications, for example, video calls, the Internet of Things (IoT), and 5G/6G. Many of these applications require not only high speed but also low latency. It is widely known that the majority of digital data is transmitted over optical fibers, resulting in a national and international infrastructure. However, fiber nonlinearity (e.g., the Kerr effect) imposes significant limitations on the optical launch power; as a result, it constrains the information rate in modern coherent transmission systems. To address these challenges, the development of innovative system designs is required, for instance, advanced modulation formats, wideband transmission, new fiber types and enhanced digital signal processing (DSP) techniques to mitigate fiber nonlinearity. Mitigating fiber nonlinearity is essential to achieve higher transmission rates and improved signal quality, without the need for new infrastructure. Various techniques have been proposed, including traditional methods like digital backpropagation (DBP) and the Volterra series-based approach. However, the computational complexity is still the main challenge, encouraging the researchers to seek an alternative approach like machine learning (ML). ML, especially neural networks (NNs), has demonstrated its capability in a wide range of applications due to the universal approximation capability of NNs. NNs have been intensively studied for the optical channel post-equalization, because they can accurately approximate the inverse optical channel transfer function and reverse the nonlinear distortions. Despite their promising equalization performance, the limitations of the NN-based equalizers in real implementation still remain. The major challenges include the computational complexity, the parallelizability, and the generalizability. This thesis investigates the integration of NN-based equalizers for nonlinear impairment mitigation in coherent optical long-haul communication systems. By leveraging NNs, this work aims to improve transmission quality while focusing on the three major aspects of the challenges in NN-based equalizers. This thesis contains some key contributions: i) the investigation of computational complexity reduction techniques, including weight clustering, and activation function approximation; ii) parallelization strategies using knowledge distillation to facilitate real-time inference; iii) the application of multi-task learning frameworks to improve model flexibility and adaptability in dynamic network conditions; and iv) the validation of these methods based on theoretical and experimental data. The comprehensive analysis of this thesis highlights the performance complexity trade-offs, practical feasibility, and potential of NN-based equalizers. Finally, the results show that the NN-based equalizers can improve the quality of transmission, while keeping the complexity the same or lower than the traditional DSP algorithm, offering a promising approach for future optical networks.