In this paper, we investigate dynamic meta-parameter control for the covariance matrix adaptation evolution strategy (CMA-ES) through deep reinforcement learning (DRL). We formalize the online adaptation of meta-parameters as a Markov decision process (MDP), where each iteration corresponds to one decision step. Based on this formalization, we integrate a controller designed with proximal policy optimization (PPO) into CMA-ES that adapts the learning rate of the mean and the damping parameter of cumulative step-size adaptation (CSA) during the optimization run. Importantly, the controller preserves all known invariance properties of CMA-ES. We train the PPO model only on the Rastrigin function. The learned controller enables CMA-ES with default population size to solve a variety of multimodal functions: on the Rastrigin function, the PPO-CMA-ES is four times faster than IPOP-CMA-ES, and it is also highly competitive on a variety of other multimodal functions, while maintaining reasonable performance on unimodal functions.
@inproceedings{10.1007/978-3-032-36217-9_9,author={Oedingen, Marc and Girardin, Oskar and Hansen, Nikolaus and Chotard, Alexandre},editor={Iacca, Giovanni and Nadizar, Giorgia and Yaman, Anil and Bucur, Doina and Della Cioppa, Antonio and Hu, Ting and Medvet, Eric and Thomson, Sarah L.},title={Reinforcement Learning for Solving Multimodal Functions with CMA-ES},booktitle={Parallel Problem Solving from Nature -- PPSN XIX},year={2027},publisher={Springer Nature Switzerland},address={Cham},pages={136--151},location={Trento, Italy},isbn={978-3-032-36217-9}}
2025
Classification-Based Linear Surrogate Modeling of Constraints for AL-CMA-ES
Oskar Girardin, Nikolaus Hansen, Dimo Brockhoff, and Anne Auger
In Proceedings of the Genetic and Evolutionary Computation Conference, Malaga, Spain, 2025
We introduce linear surrogate functions for modeling inequality constraints to solve constrained blackbox optimization problems with the Augmented Lagrangian CMA-ES. Each surrogate is constructed from a binary classifier that predicts the sign of the constraint value. The classifier, and consequently the resulting algorithm, is invariant under sign preserving transformations of the constraint values and can handle binary, flat, and deceptive constraints. Somewhat surprisingly, we find that adopting a sign-based classification model of the constraints allows to solve classes of constrained problems which can not be solved with the original Augmented Lagrangian method using the true constraint value.
@inproceedings{10.1145/3712256.3726435,author={Girardin, Oskar and Hansen, Nikolaus and Brockhoff, Dimo and Auger, Anne},title={Classification-Based Linear Surrogate Modeling of Constraints for AL-CMA-ES},year={2025},isbn={9798400714658},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3712256.3726435},booktitle={Proceedings of the Genetic and Evolutionary Computation Conference},pages={728–736},location={Malaga, Spain},numpages={9},}
Benchmarking CMA-ES under Additive and Subtractive Noise on the BBOB Testbed
Oskar Girardin
In Proceedings of the Genetic and Evolutionary Computation Conference Companion, Malaga, Spain, 2025
We benchmark a non-elitist CMA-ES algorithm on the BBOB testbed with additive and subtractive noise. In particular, we consider the case where re-evaluated solutions produce the same observed function value. As a comparison, we benchmark a version of CMA-ES with resampling, which aims at reducing the effective noise level. We find CMA-ES to be more sensitive to subtractive noise than to additive noise in dimensions 2, 3, 5, 10, 20 and 40. Resampling for CMA-ES appears to be detrimental for low noise levels, while it is beneficial for high noise levels.
@inproceedings{10.1145/3712255.3734332,author={Girardin, Oskar},title={Benchmarking CMA-ES under Additive and Subtractive Noise on the BBOB Testbed},year={2025},isbn={9798400714641},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3712255.3734332},booktitle={Proceedings of the Genetic and Evolutionary Computation Conference Companion},pages={1867–1874},numpages={8},keywords={benchmarking, black-box optimization, randomized optimization},location={Malaga, Spain},series={GECCO '25 Companion},}
