Investigating the structural diversity within a committee of classifiers and their generalization performance

Abstract

This study investigates the measures of diversity within ensembles of classifiers. The use of neural networks is carried out in measuring ensemble diversity by the use of statistical and ecological methods and to some extent information theory. A new way of looking at ensemble diversity is proposed. This ensemble diversity is called ensemble structural diversity, for this study is concerned with the diversity within the structure of the individual classifiers forming an ensemble and not via the outcomes of the individual classifiers. Ensemble structural diversity was also induced within the ensemble by varying the structural parameters (learning parameters) of the artificial machines (classifiers). The importance or the use of these measures was judged by comparing the measure of structural diversity and the ensemble generalization performance. This was done so that comparisons can be made on the robustness of the idea of structural diversity and its relationship with ensemble generalization performance. It was found that diversity could be induced by having ensembles with different structural and implicit (e.g learning) parameters and that this diversity does influence the predictive ability of the ensemble. This was concurrent with literature even though within literature ensemble diversity was viewed from the output as opposed to the structure of the individual classifiers. As the structural diversity increased so did the generalization performance. However there was a point where structural diversity decreased the generalization performance of the ensemble, where from that point onwards as the structural diversity increased the generalization performance decreased. This makes sense because too much of diversity within the ensemble might mean no consensus is reached at all. The disadvantages of comparing structural diversity and the generalization performance (accuracy) of the ensemble are that: an ensemble can be structurally diverse even though all the classifiers within the ensemble approximate the same function which means in this case structural diversity is meaningless in terms of improving the accuracy of the ensemble. The use of ensemble structural diversity measures in developing efficient ensembles still remains to be explored. This study, however, has also shown that diversity can be measured from the structural parameters and moreover reducing the abstractness of diversity by being able to quantify structural diversity making it possible to map a relationship between structural diversity and accuracy. It was observed that structural diversity does improve the accuracy of the ensemble, however, within a limited region of structural diversity.