IntroductionNeural networks have been used in many pattern recognition and regression applications. However, the success of neural networks largely depends on their architecture, their training algorithm, and the choice of features used in training. Unfortunately, determining the architecture of a neural network is a trial-and-error process; the learning algorithms must be carefully tuned to the data; and the relevance of features to the classification problem may not be known a priori.We are interested in robust algorithms to select the architecture of the neural networks, select relevant features and train the networks. Evolutionary algorithms, such as genetic algorithms and evolution strategies, have been used to address these problems successfully (Yao, 1999). In our research, we are exploring the effectiveness and scalability of different evolutionary algorithms in combination with neural networks. We have tested several methods to find relevant features, determine the architecture and train the weights of the network using a classification problem from astrophysics as benchmark (Cantu-Paz and Kamath, 2002, 2003). Our results with this problem suggest that using evolutionary algorithms to select the relevant features is the most effective method to increase the accuracy of classification. We performed additional experiments on public-domain benchmark problems. Many previous comparisons of algorithms have been done improperly, measuring the accuracy of the networks on the training data or without using proper statistical tests. We used a recently introduced methodology that is well accepted to compare different classification algorithms. We found that, in most cases, the combinations of evolutionary algorithms and neural nets perform equally well (in terms of accuracy) and were as accurate as hand-designed neural networks trained with backpropagation (Cantu-Paz and Kamath, 2005). However, some combinations of EAs and NNs performed much better for some data than the hand-designed networks or other EA/NN combinations. This suggests that in applications where accuracy is a premium it might pay off to experiment with EA and NN combinations. In different experiments, we used sophisticated distribution estimation algorithms, a relatively recent variant of evolutionary algorithms, to design the architecture of neural networks by pruning unneeded connections. The results suggest that simple genetic algorithms perform as well as the more sophisticated variants (Cantu-Paz, 2003). ReferencesCant-Paz, E. and C. Kamath, An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics, pages 915-927, 2005. UCRL-JC-151936. [pdf] Cantú-Paz, E. and C. Kamath. (2003). Evolving neural networks to identify bent-double galaxies in the FIRST survey. Neural Networks, 16 (3-4), 507-517. UCRL-JC-146705. [pdf] Yao, X. (1999). Evolving artificial neural networks. Proceedings of the IEEE, 87 (9), pp. 1423-1447. Cantú-Paz, E. and C. Kamath, (2002). "Evolving neural networks for the classification of galaxies," Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2002, pp. 1019-1026, Morgan Kaufmann Publishers, San Francisco, 2002, UCRL-JC-147020. Tied for the best paper award in the Real World Applications Category. [pdf] Cantú-Paz, E. (2003). Pruning Neural Networks with Distribution Estimation Algorithms. In Cant™-Paz, E. et al. (Eds.). Genetic and Evolutionary Computation Conference -- GECCO-2003. (pp. 790--800). Berlin: Springer Verlag. UCRL-JC-151521. [pdf] |
