Performance versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems


We present the results of the comparative performance-versus-complexity analysis for the several types of artificial neural networks (NNs) used for nonlinear channel equalization in coherent optical communication systems. The comparison is carried out using an experimental set-up with the transmission dominated by the Kerr nonlinearity and component imperfections. For the first time, we investigate the application to the channel equalization of the convolution layer (CNN) in combination with a bidirectional long short-term memory (biLSTM) layer and the design combining CNN with a multi-layer perceptron. Their performance is compared with the one delivered by the previously proposed NN-based equalizers: one biLSTM layer, three-dense-layer perceptron, and the echo state network. Importantly, all architectures have been initially optimized by a Bayesian optimizer. First, we present the general expressions for the computational complexity associated with each NN type; these are given in terms of real multiplications per symbol. We demonstrate that in the experimental system considered, the convolutional layer coupled with the biLSTM (CNN+biLSTM) provides the largest Q-factor improvement compared to the reference linear chromatic dispersion compensation (2.9 dB improvement). Then, we examine the trade-off between the computational complexity and performance of all equalizers and demonstrate that the CNN+biLSTM is the best option when the computational complexity is not constrained, while when we restrict the complexity to some lower levels, the three-layer perceptron provides the best performance.

Publication DOI:
Divisions: College of Engineering & Physical Sciences > Aston Institute of Photonics Technology (AIPT)
College of Engineering & Physical Sciences
Additional Information: This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see Funding: This paper was supported by the EU Horizon 2020 program under the Marie Sklodowska-Curie grant agreement 813144 (REAL-NET). YO acknowledges the support of the SMARTNET EMJMD programme (Project number - 586686-EPP-1-2017-1-UK-EPPKA1-JMD-MOB). JEP is supported by Leverhulme Trust, Grant No. RP-2018-063. SKT acknowledges support of the EPSRC project TRANSNET.
Uncontrolled Keywords: Bayesian optimizer,Neural network,coherent detection,computational complexity,digital signal processing,nonlinear equalizer,optical communications,Atomic and Molecular Physics, and Optics
Publication ISSN: 0733-8724
Last Modified: 14 May 2024 07:26
Date Deposited: 15 Jul 2021 08:58
Full Text Link:
Related URLs: https://ieeexpl ... cument/9483687/ (Publisher URL)
http://www.scop ... tnerID=8YFLogxK (Scopus URL)
PURE Output Type: Article
Published Date: 2021-10-01
Published Online Date: 2021-07-13
Accepted Date: 2021-07-07
Authors: Freiredecarvalhosouza, Pedro Jorge (ORCID Profile 0000-0003-3145-1018)
Osadchuk, Yevhenii
Spinnler, Bernhard
Napoli, Antonio
Schairer, Wolfgang
Costa, Nelson Manuel Simes da
Prilepsky, Jaroslaw (ORCID Profile 0000-0002-3035-4112)
Turitsyn, Sergei K. (ORCID Profile 0000-0003-0101-3834)



Version: Accepted Version

License: Creative Commons Attribution

| Preview

Export / Share Citation


Additional statistics for this record