Avoiding convergence in cooperative coevolution with novelty search

Abstract

Cooperative coevolution is an approach for evolving solutions composed of coadapted components. Previous research has shown, however, that cooperative coevolutionary algorithms are biased towards stability: they tend to converge prematurely to equilibrium states, instead of converging to optimal or near-optimal solutions. In single-population evolutionary algorithms, novelty search has been shown capable of avoiding premature convergence to local optima — a pathology similar to convergence to equilibrium states. In this study, we demonstrate how novelty search can be applied to cooperative coevolution by proposing two new algorithms. The first algorithm promotes behavioural novelty at the team level (NS-T), while the second promotes novelty at the individual agent level (NS-I). The proposed algorithms are evaluated in two popular multiagent tasks: predator-prey pursuit and keepaway soccer. An analysis of the explored collaboration space shows that (i) fitnessbased evolution tends to quickly converge to poor equilibrium states, (ii) NS-I almost never reaches any equilibrium state due to constant change in the individual populations, while (iii) NS-T explores a variety of equilibrium states in each evolutionary run and thus significantly outperforms both fitness-based evolution and NS-I.