Performance measures in discrete supervised classification

Abstract

The evaluation of results in Cluster Analysis frequently appears in the literature, and a variety of evaluation measures have been proposed. On the contrary, in supervised classification, particularly in the discrete case, the subject of results’ evaluation is relatively scarce in this field of the literature. This is the motto underlying this study. The evaluation of the performance of any model of supervised classification is, generally, based on the number of cases correctly or incorrectly predicted by the model. However, these measures can lead to a misleading evaluation when the data is not balanced. More recently, other types of measures have been studied as association or agreement coefficients, the Huberty index, Mutual information, and even ROC curves. Exploratory studies were conducted in this study to understand the relationship between each measure and data characteristics, namely, sample size, balance, and separability of classes. To this end, simulated data and a Beta regression model in the performance of the models were used.