12.4 Test Set Results

Models were build using the optimal parameter settings for the unconstrained and constrained GAs, and were applied to the test set. The area under the ROC curve for the unconstrained GA was 0.831, while the corresponding value for the constrained GA was 0.804. Both of these values are larger than their analogous resampled estimates, but the two models’ performance rank order in the same way. Figure 12.12 shows that the unconstrained model has uniformly better sensitivity and specificity across the test set cutoffs, although it is difficult to tell if these results are significantly different from each other.

Figure 12.12: Test set results for two models derived from genetic algorithms.