Neural Networks-based Equalizers for Coherent Optical Transmission: Caveats and Pitfalls


This paper performs a detailed, multi-faceted analysis of key challenges and common design caveats related to the development of efficient neural networks (NN) based nonlinear channel equalizers in coherent optical communication systems. The goal of this study is to guide researchers and engineers working in this field. We start by clarifying the metrics used to evaluate the equalizers' performance, relating them to the loss functions employed in the training of the NN equalizers. The relationships between the channel propagation model's accuracy and the performance of the equalizers are addressed and quantified. Next, we assess the impact of the order of the pseudo-random bit sequence used to generate the-numerical and experimental-data as well as of the DAC memory limitations on the operation of the NN equalizers both during the training and validation phases. Finally, we examine the critical issues of overfitting limitations, the difference between using classification instead of regression, and batch-size-related peculiarities. We conclude by providing analytical expressions for the equalizers' complexity evaluation in the digital signal processing (DSP) terms and relate the metrics to the processing latency.

Publication DOI:
Divisions: College of Engineering & Physical Sciences > Aston Institute of Photonics Technology (AIPT)
Additional Information: This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see Funding: This paper was supported by the EU Horizon 2020 program under the Marie Sklodowska-Curie grant agreement 813144 (REAL-NET). JEP is supported by Leverhulme Trust, Grant No. RP-2018-063. SKT acknowledges support of the EPSRC project TRANSNET. (Corresponding Author: Pedro J. Freire)
Uncontrolled Keywords: Artificial neural networks,Equalizers,Fiber nonlinear optics,Neural network,Optical fiber amplifiers,Optical fibers,Symbols,Training,classification,coherent detection,nonlinear equalizer,optical communications,overfitting,pitfalls,regression,Atomic and Molecular Physics, and Optics,Electrical and Electronic Engineering
Publication ISSN: 1558-4542
Last Modified: 12 Jun 2024 16:44
Date Deposited: 23 May 2022 12:14
Full Text Link:
Related URLs: https://ieeexpl ... cument/9773034/ (Publisher URL)
http://www.scop ... tnerID=8YFLogxK (Scopus URL)
PURE Output Type: Article
Published Date: 2022
Published Online Date: 2022-05-11
Accepted Date: 2022-05-01
Authors: Freire, Pedro Jorge (ORCID Profile 0000-0003-3145-1018)
Napoli, Antonio
Spinnler, Bernhard
Costa, Nelson Manuel Simoes da
Turitsyn, Sergei K. (ORCID Profile 0000-0003-0101-3834)
Prilepsky, Jaroslaw (ORCID Profile 0000-0002-3035-4112)



Version: Accepted Version

License: Creative Commons Attribution

| Preview

Export / Share Citation


Additional statistics for this record