Why is the accuracy reported in the Classification Learner app much higher than the accuracy of the exported model on the held out validation set?

2 views (last 30 days)
Problem description
I used the default 5-fold cross-validation (CV) scheme in the Classification Learner app and trained all the available models. The best model (quadratic SVM) has 74.2% accuracy. I used
export model => generate code
and then ran the generated code, again examining the 5-fold CV accuracy. Surprisingly, the validation accuracy of this generated model was 66.8%, which is much lower than that reported by the app.
This has happened repeatedly - everytime I follow this scheme (with different datasets and models) I observe reduced accuracy for the exported model.
Potential explanations
I understand that not resetting the random number generation seed may lead to some variability between runs, but this effect seems too large and consistent to be explained by chance. I also understand that choosing the empirically-best model on the validation set, without having an additional test set, may lead to overestimation of its performance. If the exported model has a different CV partition, that could explain some of the reduction in accuracy, but I wonder if there might be some other explanation that I am missing.

Answers (1)

Jalaj Gambhir
Jalaj Gambhir on 6 Mar 2020
Hi,
You can have a look at the answer here. Hope it helps!
  1 Comment
asaf benjamin
asaf benjamin on 10 Mar 2020
Hi,
Thanks for the response, but that answer deals with a very different issue, where one mistakingly compares cross-validation accuracy to accuracy on the training data. Under no circumstances would I try to test my model's performance on the data on which it trained. Rather, I am comparing
  1. The 5-fold cross-validation accuracy reported by the app, to
  2. The 5-fold cross-validation accuracy reported by the code generated by the app.
These should be equal (or at least very close), as I understand it, but they are in fact very different...

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!