Implementing Neural Network-Based Equalizers in a Coherent Optical Transmission System Using Field-Programmable Gate Arrays


In this work, we demonstrate the offline FPGA realization of both recurrent and feedforward neural network (NN)-based equalizers for nonlinearity compensation in coherent optical transmission systems. First, we present a realization pipeline showing the conversion of the models from Python libraries to the FPGA chip synthesis and implementation. Then, we review the main alternatives for the hardware implementation of nonlinear activation functions. The main results are divided into three parts: a performance comparison, an analysis of how activation functions are implemented, and a report on the complexity of the hardware. The performance in Q-factor is presented for the cases of bidirectional long-short-term memory coupled with convolutional NN (biLSTM + CNN) equalizer, CNN equalizer, and standard 1-StpS digital back-propagation (DBP) for the simulation and experiment propagation of a single channel dual-polarization (SC-DP) 16QAM at 34 GBd along 17×70 km of LEAF. The biLSTM+CNN equalizer provides a similar result to DBP and a 1.7 dB Q-factor gain compared with the chromatic dispersion compensation baseline in the experimental dataset. After that, we assess the Q-factor and the impact of hardware utilization when approximating the activation functions of NN using Taylor series, piecewise linear, and look-up table (LUT) approximations. We also show how to mitigate the approximation errors with extra training and provide some insights into possible gradient problems in the LUT approximation. Finally, to evaluate the complexity of hardware implementation to achieve 200 G and 400 G throughput, fixed-point NN-based equalizers with approximated activation functions are developed and implemented in an FPGA.

Publication DOI:
Divisions: College of Engineering & Physical Sciences > School of Informatics and Digital Engineering > Electrical and Electronic Engineering
College of Engineering & Physical Sciences > Aston Institute of Photonics Technology (AIPT)
Additional Information: This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
Uncontrolled Keywords: Artificial intelligence,Artificial neural networks,Equalizers,FPGA,Field programmable gate arrays,Hardware,Nonlinear optics,Software,Table lookup,coherent detection,computational complexity,neural network hardware,nonlinear equalizer,recurrent neural networks,Atomic and Molecular Physics, and Optics
Publication ISSN: 1558-2213
Full Text Link:
Related URLs: https://ieeexpl ... ument/10113728/ (Publisher URL)
http://www.scop ... tnerID=8YFLogxK (Scopus URL)
PURE Output Type: Article
Published Date: 2023-06-15
Published Online Date: 2023-05-01
Accepted Date: 2023-04-18
Authors: Freire, Pedro J.
Srivallapanondh, Sasipim
Anderson, Michael
Spinnler, Bernhard
Bex, Thomas
Eriksson, Tobias A.
Napoli, Antonio
Schairer, Wolfgang
Costa, Nelson
Blott, Michaela
Turitsyn, Sergei K. (ORCID Profile 0000-0003-0101-3834)
Prilepsky, Jaroslaw E. (ORCID Profile 0000-0002-3035-4112)



Version: Published Version

License: Creative Commons Attribution

| Preview


Version: Accepted Version

Access Restriction: Restricted to Repository staff only

Export / Share Citation


Additional statistics for this record