Low-complexity machine learning-based equalisation in optical communication systems

Abstract

Kerr nonlinearity has a significant degrading effect on the performance of high-speed optical transmission systems. Hence, to achieve reliable and high-quality optical communication, compensating for fibre nonlinearity is crucial. However, many existing techniques like Digital back-propagation (DBP), Inverse Volterra series transfer functions (IVSTF), Nonlinear Fourier Transform (NFT), and Optical phase conjugation (OPC) have drawbacks, such as high computational complexity, marginal performance benefits, difficulty in reconfiguring. In recent years, machine learning (ML) has gained popularity as a promising solution for the compensation of nonlinear fibre effects. This is primarily due to the fact that Machine Learning (ML) techniques are universal approximations, allowing them to reverse the channel propagation function and effectively mitigate fibre impairments. Additionally, data science approaches like ML in optical communication applications excel due to the high data availability. However, a major challenge faced by most ML-based equalisation implementations is the high computational complexity, which imposes significant demands on device speed and energy consumption during equalisation operations. This becomes more prominent during training, requiring much energy and training data. This thesis explores low-complexity machine learning implementations, including Support Vector Machine (SVM), Support Vector Regression (SVR), and low-complexity Neural Networks (NN), particularly complex-valued multilayer perceptron. The impact of these approaches is evaluated through bit error rate (BER) performance and resulting computational complexity. Moreover, the thesis investigates different strategies for simplifying the resulting equalisers or the training process while maintaining adequate performance. Techniques such as weight pruning, integration with optical solutions (like optical phase conjugation and dispersion management), and crowd equalisation based on committee learning are examined. The findings provide valuable insights into their performance, computational complexity, and potential enhancements, thereby contributing to the development of efficient and effective equalisation techniques in optical communication.