Neuromorphic Computing and Machine Learning for Nonlinearity Mitigation in Optical Communication Systems

Abstract

Nonlinear fibre propagation and transceiver impairments limit the performance of coherent optical communication systems. Established digital signal processing techniques such as digital back-propagation and Volterra equalisation can mitigate Kerr-induced nonlinear distortions, but their computational complexity and energy consumption hinder real-time deployment. This thesis develops machine-learning and neuromorphic approaches to optical channel equalisation, focusingon performance–complexity trade-offs and practical hardware feasibility. A unified training and benchmarking framework is introduced for coherent transmission scenarios, employing Q-factor and bit-error-rate alongside implementation measures including operation counts, memory footprint, and inference latency. Within this framework, a broad class of equalisers is analysed, including multi-layer perceptrons, convolutional and recurrent neural networks, and complex-valued neural networks that process in-phase and quadrature components. Model compression is studied: pruning, quantisation, and weight clustering are jointly optimised using Bayesian optimisation to identify Pareto-efficient configurations that preserve equalisation performance while reducing computational load, memory usage, and latency. Experimental evaluations on edge-device platforms demonstrate feasibility under realistic receiver constraints. The thesis also explores hardware–software co-design. Optical phase conjugation is integrated with neural equalisers to offload part of the nonlinearity compensation to the optical domain, enabling smaller models and lower digital complexity. To address analogue noise in photonic neuromorphic computing, robustness is assessed under additive and signal-dependent noise; noise-aware training, stochastic-resonance neurons, and ensemble-based “crowd equalisation” are proposed to enhance resilience. Finally, a neuromorphic equaliser combining spiking neural networks with a streaming RWKV time-mixing module is introduced. By exploiting event-driven sparsity and constant-memory sequential processing, this architecture achieves competitive equalisation performance with reduced computational and energy requirements compared to conventional deep learning models.