A cross-entropy based direct policy search algorithm for multi-objective energy storage control

Abstract

Effective control of Energy Storage Systems (ESS) is crucial for the secure and profitable operation of microgrids. In this context, ESSs are essential for enhancing the overall grid resilience, balancing supply, and mitigating voltage and frequency variations. This paper presents a novel neuroevolutionary method, coupling a modified version of the Multi-Objective Evolutionary Policy Search (MEPS) algorithm with the Cross-Entropy method, aimed at optimizing an ESS control problem. The modified MEPS, named Cascade-MEPS, employs a cascade weights mutation operator to refine policies by focusing on the most recent hidden node, ensuring localized and non-disruptive adjustments. The resulting algorithm, referred to as cross-entropy Cascade-MEPS (CE-CMEPS), utilizes the cross-entropy method as a depth initialization strategy, conducting an initial exploration of the weights space to initialize the population prior to Cascade-MEPS execution. Experimental validation on a newly proposed multi-objective ESS control problem demonstrates the efficacy of CE-CMEPS, showcasing performance improvements and reduced variation compared to standalone MEPS. Our results show that CE-CMEPS is an effective ESS discharge controller and a sustainable multi-objective reinforcement learning solution.