fitcauto

Automatically select classification model with optimized hyperparameters

Description

Given predictor and response data, fitcauto automatically tries a selection of classification model types with different hyperparameter values. The function uses Bayesian optimization to select models and their hyperparameter values, and computes the cross-validation classification error for each model. After the optimization is complete, fitcauto returns the model, trained on the entire data set, that is expected to best classify new data. You can use the predict and loss object functions of the returned model to classify new data and compute the test set classification error, respectively.

Use fitcauto when you are uncertain which classifier types best suit your data. For information on alternative methods for tuning hyperparameters of classification models, see Alternative Functionality.

example

Mdl = fitcauto(Tbl,ResponseVarName) returns a classification model Mdl with tuned hyperparameters. The table Tbl contains the predictor variables and the response variable, where ResponseVarName is the name of the response variable.

Mdl = fitcauto(Tbl,formula) uses formula to specify the response variable and the predictor variables to consider among the variables in Tbl.

Mdl = fitcauto(Tbl,Y) uses the predictor variables in table Tbl and the class labels in vector Y.

example

Mdl = fitcauto(X,Y) uses the predictor variables in matrix X and the class labels in vector Y.

Mdl = fitcauto(___,Name,Value) specifies options using one or more name-value pair arguments in addition to any of the input argument combinations in previous syntaxes. For example, use the HyperparameterOptimizationOptions name-value pair argument to specify how the Bayesian optimization is performed.

example

[Mdl,OptimizationResults] = fitcauto(___) additionally returns OptimizationResults, a BayesianOptimization object containing the results of the model selection and hyperparameter tuning process.

Examples

collapse all

Use fitcauto to automatically select a classification model with optimized hyperparameters, given predictor and response data stored in a table.

Load Data

Load the carbig data set, which contains measurements of cars made in the 1970s and early 1980s.

load carbig

Categorize the cars based on whether they were made in the USA.

Origin = categorical(cellstr(Origin));
Origin = mergecats(Origin,{'France','Japan','Germany', ...
    'Sweden','Italy','England'},'NotUSA');

Create a table containing the predictor variables Acceleration, Displacement, and so on, as well as the response variable Origin.

cars = table(Acceleration,Displacement,Horsepower, ...
    Model_Year,MPG,Weight,Origin);

Partition Data

Partition the data into training and test sets. Use approximately 80% of the observations for the model selection and hyperparameter tuning process, and 20% of the observations to test the performance of the final model returned by fitcauto. Use cvpartition to partition the data.

rng('default') % For reproducibility of the data partition
c = cvpartition(Origin,'Holdout',0.2);
trainingIdx = training(c); % Training set indices
carsTrain = cars(trainingIdx,:);
testIdx = test(c); % Test set indices
carsTest = cars(testIdx,:);

Run fitcauto

Pass the training data to fitcauto. By default, fitcauto determines appropriate model types to try, uses Bayesian optimization to find good hyperparameter values, and returns a trained model Mdl with the best expected performance. Additionally, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.

Expect this process to take some time. To speed up the optimization process, consider specifying to run the optimization in parallel, if you have a Parallel Computing Toolbox™ license. To do so, pass 'HyperparameterOptimizationOptions',struct('UseParallel',true) to fitcauto as a name-value pair argument.

Mdl = fitcauto(carsTrain,'Origin');
Warning: It is recommended that you first standardize all numeric predictors when optimizing the Naive Bayes 'Width' parameter. Ignore this warning if you have done that.
Learner types to explore: ensemble, knn, nb, svm, tree
Total iterations (MaxObjectiveEvaluations): 150
Total time (MaxTime): Inf

|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|    1 | Best   |    0.14154 |            10.563 |         0.14154 |         0.14154 |     ensemble | Method:                 Bag |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      201 |
|      |        |            |                   |                 |                 |              | MinLeafSize:              7 |
|    2 | Accept |    0.18269 |           0.57392 |         0.14154 |         0.14154 |          knn | NumNeighbors:             3 |
|    3 | Accept |    0.23397 |            0.1264 |         0.14154 |         0.14154 |          knn | NumNeighbors:            91 |
|    4 | Accept |    0.16308 |            12.867 |         0.14154 |         0.15468 |     ensemble | Method:          LogitBoost |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      274 |
|      |        |            |                   |                 |                 |              | MinLeafSize:             15 |
|    5 | Accept |    0.20833 |             0.124 |         0.14154 |         0.15468 |          knn | NumNeighbors:             4 |
|    6 | Accept |    0.22115 |          0.079641 |         0.14154 |         0.15468 |          knn | NumNeighbors:            28 |
|    7 | Accept |    0.16923 |           0.20013 |         0.14154 |         0.15468 |         tree | MinLeafSize:            105 |
|    8 | Accept |    0.37179 |           0.59222 |         0.14154 |         0.15468 |          svm | BoxConstraint:     0.022186 |
|      |        |            |                   |                 |                 |              | KernelScale:       0.085527 |
|    9 | Accept |    0.37179 |           0.11659 |         0.14154 |         0.15468 |          svm | BoxConstraint:     0.045899 |
|      |        |            |                   |                 |                 |              | KernelScale:      0.0024758 |
|   10 | Accept |    0.24615 |           0.98386 |         0.14154 |         0.15468 |           nb | DistributionNames:   kernel |
|      |        |            |                   |                 |                 |              | Width:               1.1327 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|   11 | Accept |    0.16923 |          0.079996 |         0.14154 |         0.15468 |         tree | MinLeafSize:             78 |
|   12 | Accept |    0.26923 |           0.10923 |         0.14154 |         0.15468 |          svm | BoxConstraint:       11.063 |
|      |        |            |                   |                 |                 |              | KernelScale:         15.114 |
|   13 | Best   |    0.12923 |           0.11568 |         0.12923 |         0.15468 |         tree | MinLeafSize:              3 |
|   14 | Accept |    0.21154 |          0.084406 |         0.12923 |         0.15468 |          knn | NumNeighbors:             2 |
|   15 | Accept |    0.14154 |          0.080022 |         0.12923 |         0.15294 |         tree | MinLeafSize:              1 |
|   16 | Accept |    0.14769 |          0.092395 |         0.12923 |         0.15097 |         tree | MinLeafSize:              2 |
|   17 | Accept |    0.14154 |            10.869 |         0.12923 |         0.14872 |     ensemble | Method:                 Bag |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      208 |
|      |        |            |                   |                 |                 |              | MinLeafSize:             10 |
|   18 | Accept |    0.37179 |           0.12386 |         0.12923 |         0.14872 |          svm | BoxConstraint:       116.46 |
|      |        |            |                   |                 |                 |              | KernelScale:        0.52908 |
|   19 | Accept |    0.22769 |           0.15545 |         0.12923 |         0.14872 |           nb | DistributionNames:   normal |
|      |        |            |                   |                 |                 |              | Width:                  NaN |
|   20 | Accept |    0.22115 |          0.070813 |         0.12923 |         0.14872 |          knn | NumNeighbors:             8 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|   21 | Accept |    0.37179 |           0.11553 |         0.12923 |         0.14872 |          svm | BoxConstraint:       45.341 |
|      |        |            |                   |                 |                 |              | KernelScale:        0.76949 |
|   22 | Accept |    0.12923 |          0.080362 |         0.12923 |         0.14194 |         tree | MinLeafSize:              3 |
|   23 | Best   |    0.10154 |          0.079656 |         0.10154 |         0.13213 |         tree | MinLeafSize:              5 |
|   24 | Accept |    0.22769 |            0.2529 |         0.10154 |         0.13213 |           nb | DistributionNames:   kernel |
|      |        |            |                   |                 |                 |              | Width:              0.42571 |
|   25 | Accept |    0.11385 |          0.080085 |         0.10154 |          0.1289 |         tree | MinLeafSize:             11 |
|   26 | Accept |    0.13782 |          0.092228 |         0.10154 |          0.1289 |          svm | BoxConstraint:       9.7286 |
|      |        |            |                   |                 |                 |              | KernelScale:         293.41 |
|   27 | Accept |    0.22769 |          0.073346 |         0.10154 |          0.1289 |           nb | DistributionNames:   normal |
|      |        |            |                   |                 |                 |              | Width:                  NaN |
|   28 | Accept |    0.21795 |          0.074914 |         0.10154 |          0.1289 |          knn | NumNeighbors:            42 |
|   29 | Accept |    0.24308 |           0.27621 |         0.10154 |          0.1289 |           nb | DistributionNames:   kernel |
|      |        |            |                   |                 |                 |              | Width:               4.4662 |
|   30 | Accept |    0.16308 |            12.328 |         0.10154 |          0.1289 |     ensemble | Method:          LogitBoost |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      267 |
|      |        |            |                   |                 |                 |              | MinLeafSize:            131 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|   31 | Accept |    0.24308 |           0.22334 |         0.10154 |          0.1289 |           nb | DistributionNames:   kernel |
|      |        |            |                   |                 |                 |              | Width:              0.66296 |
|   32 | Accept |    0.22115 |          0.066711 |         0.10154 |          0.1289 |          knn | NumNeighbors:            28 |
|   33 | Accept |    0.13846 |          0.079934 |         0.10154 |         0.12465 |         tree | MinLeafSize:             25 |
|   34 | Accept |    0.21474 |          0.085438 |         0.10154 |         0.12465 |          knn | NumNeighbors:            14 |
|   35 | Accept |    0.16615 |             10.05 |         0.10154 |         0.12465 |     ensemble | Method:          LogitBoost |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      215 |
|      |        |            |                   |                 |                 |              | MinLeafSize:             13 |
|   36 | Accept |    0.14154 |            12.866 |         0.10154 |         0.12465 |     ensemble | Method:                 Bag |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      254 |
|      |        |            |                   |                 |                 |              | MinLeafSize:             31 |
|   37 | Accept |    0.22769 |          0.070251 |         0.10154 |         0.12465 |           nb | DistributionNames:   normal |
|      |        |            |                   |                 |                 |              | Width:                  NaN |
|   38 | Accept |    0.37179 |          0.077924 |         0.10154 |         0.12465 |          svm | BoxConstraint:    0.0073633 |
|      |        |            |                   |                 |                 |              | KernelScale:         774.33 |
|   39 | Accept |    0.16923 |          0.068411 |         0.10154 |         0.12552 |         tree | MinLeafSize:             82 |
|   40 | Accept |    0.20833 |          0.064563 |         0.10154 |         0.12552 |          knn | NumNeighbors:             4 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|   41 | Accept |    0.16308 |            12.932 |         0.10154 |         0.12552 |     ensemble | Method:          LogitBoost |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      274 |
|      |        |            |                   |                 |                 |              | MinLeafSize:            150 |
|   42 | Accept |    0.22462 |           0.24365 |         0.10154 |         0.12552 |           nb | DistributionNames:   kernel |
|      |        |            |                   |                 |                 |              | Width:               121.64 |
|   43 | Accept |    0.20308 |            11.027 |         0.10154 |         0.12552 |     ensemble | Method:                 Bag |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      229 |
|      |        |            |                   |                 |                 |              | MinLeafSize:            117 |
|   44 | Accept |    0.16923 |          0.069279 |         0.10154 |         0.12291 |         tree | MinLeafSize:             84 |
|   45 | Accept |    0.22769 |          0.078716 |         0.10154 |         0.12291 |           nb | DistributionNames:   normal |
|      |        |            |                   |                 |                 |              | Width:                  NaN |
|   46 | Accept |    0.22769 |          0.068458 |         0.10154 |         0.12291 |           nb | DistributionNames:   normal |
|      |        |            |                   |                 |                 |              | Width:                  NaN |
|   47 | Accept |    0.16615 |            9.9849 |         0.10154 |         0.12291 |     ensemble | Method:          LogitBoost |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      212 |
|      |        |            |                   |                 |                 |              | MinLeafSize:             49 |
|   48 | Accept |    0.14769 |            14.541 |         0.10154 |         0.12291 |     ensemble | Method:                 Bag |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      288 |
|      |        |            |                   |                 |                 |              | MinLeafSize:             25 |
|   49 | Accept |    0.23077 |           0.21379 |         0.10154 |         0.12291 |           nb | DistributionNames:   kernel |
|      |        |            |                   |                 |                 |              | Width:               73.249 |
|   50 | Accept |    0.37179 |          0.091937 |         0.10154 |         0.12291 |          svm | BoxConstraint:    0.0036501 |
|      |        |            |                   |                 |                 |              | KernelScale:         1.0504 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|   51 | Accept |    0.21474 |          0.098808 |         0.10154 |         0.12291 |          svm | BoxConstraint:       64.859 |
|      |        |            |                   |                 |                 |              | KernelScale:         23.779 |
|   52 | Accept |    0.37179 |           0.10415 |         0.10154 |         0.12291 |          svm | BoxConstraint:      0.16622 |
|      |        |            |                   |                 |                 |              | KernelScale:         4.4901 |
|   53 | Accept |    0.25846 |            0.2444 |         0.10154 |         0.12291 |           nb | DistributionNames:   kernel |
|      |        |            |                   |                 |                 |              | Width:             0.079498 |
|   54 | Accept |    0.21154 |          0.074835 |         0.10154 |         0.12291 |          knn | NumNeighbors:             2 |
|   55 | Accept |    0.13846 |            12.173 |         0.10154 |         0.12291 |     ensemble | Method:                 Bag |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      234 |
|      |        |            |                   |                 |                 |              | MinLeafSize:              8 |
|   56 | Accept |    0.36538 |           0.10958 |         0.10154 |         0.12291 |          svm | BoxConstraint:        271.6 |
|      |        |            |                   |                 |                 |              | KernelScale:          2.743 |
|   57 | Accept |    0.16615 |            11.482 |         0.10154 |         0.12291 |     ensemble | Method:          LogitBoost |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      248 |
|      |        |            |                   |                 |                 |              | MinLeafSize:            117 |
|   58 | Accept |    0.37179 |          0.095419 |         0.10154 |         0.12291 |          svm | BoxConstraint:       7.5785 |
|      |        |            |                   |                 |                 |              | KernelScale:      0.0066815 |
|   59 | Accept |    0.37179 |          0.097469 |         0.10154 |         0.12291 |          svm | BoxConstraint:    0.0017765 |
|      |        |            |                   |                 |                 |              | KernelScale:        0.86786 |
|   60 | Accept |    0.37179 |           0.11284 |         0.10154 |         0.12291 |          svm | BoxConstraint:     0.011465 |
|      |        |            |                   |                 |                 |              | KernelScale:        0.02747 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|   61 | Accept |    0.11692 |          0.077515 |         0.10154 |         0.12239 |         tree | MinLeafSize:             12 |
|   62 | Accept |    0.29167 |          0.091617 |         0.10154 |         0.12239 |          svm | BoxConstraint:       11.939 |
|      |        |            |                   |                 |                 |              | KernelScale:         11.002 |
|   63 | Accept |    0.21795 |          0.067171 |         0.10154 |         0.12239 |          knn | NumNeighbors:             6 |
|   64 | Accept |    0.18269 |          0.062887 |         0.10154 |         0.12239 |          knn | NumNeighbors:             3 |
|   65 | Accept |    0.12923 |          0.075704 |         0.10154 |         0.11989 |         tree | MinLeafSize:              3 |
|   66 | Accept |    0.16923 |          0.065889 |         0.10154 |         0.12048 |         tree | MinLeafSize:             56 |
|   67 | Accept |     0.1891 |          0.068215 |         0.10154 |         0.12048 |          knn | NumNeighbors:             1 |
|   68 | Accept |    0.13231 |            14.135 |         0.10154 |         0.12048 |     ensemble | Method:                 Bag |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      270 |
|      |        |            |                   |                 |                 |              | MinLeafSize:              4 |
|   69 | Accept |    0.22769 |          0.060902 |         0.10154 |         0.12048 |           nb | DistributionNames:   normal |
|      |        |            |                   |                 |                 |              | Width:                  NaN |
|   70 | Accept |    0.37231 |           0.24511 |         0.10154 |         0.12048 |           nb | DistributionNames:   kernel |
|      |        |            |                   |                 |                 |              | Width:               1629.5 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|   71 | Accept |    0.16923 |          0.069135 |         0.10154 |         0.11947 |         tree | MinLeafSize:             61 |
|   72 | Accept |    0.22769 |          0.060552 |         0.10154 |         0.11947 |           nb | DistributionNames:   normal |
|      |        |            |                   |                 |                 |              | Width:                  NaN |
|   73 | Accept |    0.16308 |            10.133 |         0.10154 |         0.11947 |     ensemble | Method:          LogitBoost |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      217 |
|      |        |            |                   |                 |                 |              | MinLeafSize:             70 |
|   74 | Accept |    0.13231 |            13.055 |         0.10154 |         0.11947 |     ensemble | Method:                 Bag |
|      |        |            |                   |                 |                 |              | NumLearningCycles:      257 |
|      |        |            |                   |                 |                 |              | MinLeafSize:              2 |
|   75 | Accept |    0.21474 |          0.069312 |         0.10154 |         0.11947 |          knn | NumNeighbors:            49 |
|   76 | Accept |    0.13846 |          0.083714 |         0.10154 |          0.1214 |         tree | MinLeafSize:             25 |
|   77 | Accept |       0.12 |          0.069041 |         0.10154 |         0.11923 |         tree | MinLeafSize:              6 |
|   78 | Accept |    0.10154 |          0.077399 |         0.10154 |          0.1118 |         tree | MinLeafSize:              5 |
|   79 | Accept |       0.12 |          0.076269 |         0.10154 |         0.11007 |         tree | MinLeafSize:              4 |
|   80 | Accept |    0.10154 |           0.09325 |         0.10154 |         0.10878 |         tree | MinLeafSize:              5 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|   81 | Accept |    0.10154 |          0.074541 |         0.10154 |         0.10737 |         tree | MinLeafSize:              5 |
|   82 | Accept |       0.12 |          0.069929 |         0.10154 |          0.1063 |         tree | MinLeafSize:              4 |
|   83 | Accept |    0.10154 |          0.072591 |         0.10154 |         0.10514 |         tree | MinLeafSize:              5 |
|   84 | Accept |    0.10154 |          0.071254 |         0.10154 |         0.10366 |         tree | MinLeafSize:              5 |
|   85 | Accept |    0.10154 |          0.077378 |         0.10154 |         0.10361 |         tree | MinLeafSize:              5 |
|   86 | Accept |    0.10154 |          0.070643 |         0.10154 |         0.10348 |         tree | MinLeafSize:              5 |
|   87 | Accept |       0.12 |          0.070551 |         0.10154 |         0.10286 |         tree | MinLeafSize:              4 |
|   88 | Accept |       0.12 |          0.078438 |         0.10154 |          0.1029 |         tree | MinLeafSize:              6 |
|   89 | Accept |    0.10154 |          0.072996 |         0.10154 |         0.10262 |         tree | MinLeafSize:              5 |
|   90 | Accept |    0.10154 |          0.078162 |         0.10154 |         0.10246 |         tree | MinLeafSize:              5 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|   91 | Accept |    0.10154 |           0.07038 |         0.10154 |         0.10267 |         tree | MinLeafSize:              5 |
|   92 | Accept |    0.10154 |           0.07752 |         0.10154 |         0.10257 |         tree | MinLeafSize:              5 |
|   93 | Accept |    0.10154 |          0.076155 |         0.10154 |         0.10217 |         tree | MinLeafSize:              5 |
|   94 | Accept |    0.10154 |          0.075983 |         0.10154 |         0.10221 |         tree | MinLeafSize:              5 |
|   95 | Accept |    0.10154 |           0.07407 |         0.10154 |         0.10211 |         tree | MinLeafSize:              5 |
|   96 | Accept |    0.10154 |          0.080633 |         0.10154 |         0.10207 |         tree | MinLeafSize:              5 |
|   97 | Accept |    0.10154 |          0.086164 |         0.10154 |         0.10205 |         tree | MinLeafSize:              5 |
|   98 | Accept |    0.10154 |          0.080264 |         0.10154 |         0.10191 |         tree | MinLeafSize:              5 |
|   99 | Accept |    0.12308 |          0.076015 |         0.10154 |          0.1021 |         tree | MinLeafSize:             17 |
|  100 | Accept |    0.10154 |          0.074579 |         0.10154 |          0.1019 |         tree | MinLeafSize:              5 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|  101 | Accept |    0.10154 |          0.072077 |         0.10154 |         0.10186 |         tree | MinLeafSize:              5 |
|  102 | Accept |    0.10154 |          0.071287 |         0.10154 |         0.10199 |         tree | MinLeafSize:              5 |
|  103 | Accept |    0.10154 |          0.080003 |         0.10154 |         0.10186 |         tree | MinLeafSize:              5 |
|  104 | Accept |       0.12 |           0.07462 |         0.10154 |         0.10189 |         tree | MinLeafSize:             13 |
|  105 | Accept |    0.10154 |          0.077244 |         0.10154 |         0.10198 |         tree | MinLeafSize:              5 |
|  106 | Accept |       0.12 |          0.080302 |         0.10154 |         0.10173 |         tree | MinLeafSize:              4 |
|  107 | Accept |    0.10154 |          0.071858 |         0.10154 |         0.10183 |         tree | MinLeafSize:              5 |
|  108 | Accept |    0.10154 |          0.076013 |         0.10154 |         0.10166 |         tree | MinLeafSize:              5 |
|  109 | Accept |    0.10154 |          0.079106 |         0.10154 |         0.10164 |         tree | MinLeafSize:              5 |
|  110 | Accept |    0.10154 |          0.075807 |         0.10154 |         0.10172 |         tree | MinLeafSize:              5 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|  111 | Accept |    0.10154 |          0.076903 |         0.10154 |         0.10159 |         tree | MinLeafSize:              5 |
|  112 | Accept |    0.10154 |          0.077409 |         0.10154 |         0.10163 |         tree | MinLeafSize:              5 |
|  113 | Accept |    0.10154 |          0.069569 |         0.10154 |         0.10165 |         tree | MinLeafSize:              5 |
|  114 | Accept |    0.12308 |          0.076744 |         0.10154 |         0.10156 |         tree | MinLeafSize:             18 |
|  115 | Accept |    0.19551 |          0.097443 |         0.10154 |         0.10156 |          svm | BoxConstraint:       1.2977 |
|      |        |            |                   |                 |                 |              | KernelScale:         33.654 |
|  116 | Accept |    0.26603 |          0.099474 |         0.10154 |         0.10156 |          svm | BoxConstraint:     0.082725 |
|      |        |            |                   |                 |                 |              | KernelScale:         139.98 |
|  117 | Accept |    0.37179 |          0.077589 |         0.10154 |         0.10156 |          svm | BoxConstraint:    0.0065156 |
|      |        |            |                   |                 |                 |              | KernelScale:         559.53 |
|  118 | Accept |    0.15385 |           0.08281 |         0.10154 |         0.10156 |          svm | BoxConstraint:       8.5337 |
|      |        |            |                   |                 |                 |              | KernelScale:         482.07 |
|  119 | Accept |    0.20833 |          0.074271 |         0.10154 |         0.10156 |          svm | BoxConstraint:       5.1729 |
|      |        |            |                   |                 |                 |              | KernelScale:         980.38 |
|  120 | Accept |    0.17949 |          0.081488 |         0.10154 |         0.10156 |          svm | BoxConstraint:       6.8028 |
|      |        |            |                   |                 |                 |              | KernelScale:         578.14 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|  121 | Accept |    0.22115 |          0.076901 |         0.10154 |         0.10156 |          svm | BoxConstraint:      0.19345 |
|      |        |            |                   |                 |                 |              | KernelScale:         367.75 |
|  122 | Accept |    0.17308 |          0.076171 |         0.10154 |         0.10156 |          svm | BoxConstraint:       10.344 |
|      |        |            |                   |                 |                 |              | KernelScale:         679.05 |
|  123 | Accept |      0.125 |           0.09187 |         0.10154 |         0.10156 |          svm | BoxConstraint:       72.626 |
|      |        |            |                   |                 |                 |              | KernelScale:         228.42 |
|  124 | Accept |    0.13462 |           0.15746 |         0.10154 |         0.10156 |          svm | BoxConstraint:        586.5 |
|      |        |            |                   |                 |                 |              | KernelScale:         176.02 |
|  125 | Accept |    0.13462 |          0.093366 |         0.10154 |         0.10156 |          svm | BoxConstraint:       83.771 |
|      |        |            |                   |                 |                 |              | KernelScale:         117.21 |
|  126 | Accept |    0.22436 |           0.10124 |         0.10154 |         0.10156 |          svm | BoxConstraint:       962.52 |
|      |        |            |                   |                 |                 |              | KernelScale:         20.898 |
|  127 | Accept |    0.15705 |            0.1133 |         0.10154 |         0.10156 |          svm | BoxConstraint:       29.038 |
|      |        |            |                   |                 |                 |              | KernelScale:         66.563 |
|  128 | Accept |    0.16346 |           0.10388 |         0.10154 |         0.10156 |          svm | BoxConstraint:        156.8 |
|      |        |            |                   |                 |                 |              | KernelScale:         62.775 |
|  129 | Accept |    0.13782 |          0.087555 |         0.10154 |         0.10156 |          svm | BoxConstraint:       94.357 |
|      |        |            |                   |                 |                 |              | KernelScale:         932.27 |
|  130 | Accept |    0.12821 |          0.093318 |         0.10154 |         0.10156 |          svm | BoxConstraint:       25.982 |
|      |        |            |                   |                 |                 |              | KernelScale:         247.06 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|  131 | Accept |    0.13462 |           0.08648 |         0.10154 |         0.10156 |          svm | BoxConstraint:       16.818 |
|      |        |            |                   |                 |                 |              | KernelScale:         352.46 |
|  132 | Accept |    0.14103 |          0.090667 |         0.10154 |         0.10156 |          svm | BoxConstraint:       15.817 |
|      |        |            |                   |                 |                 |              | KernelScale:         130.15 |
|  133 | Accept |    0.12179 |          0.089208 |         0.10154 |         0.10156 |          svm | BoxConstraint:       74.054 |
|      |        |            |                   |                 |                 |              | KernelScale:         555.66 |
|  134 | Accept |    0.21474 |          0.082634 |         0.10154 |         0.10156 |          svm | BoxConstraint:      0.80097 |
|      |        |            |                   |                 |                 |              | KernelScale:         239.08 |
|  135 | Accept |      0.125 |          0.088947 |         0.10154 |         0.10156 |          svm | BoxConstraint:       47.244 |
|      |        |            |                   |                 |                 |              | KernelScale:         214.51 |
|  136 | Accept |    0.13141 |           0.07954 |         0.10154 |         0.10156 |          svm | BoxConstraint:       13.389 |
|      |        |            |                   |                 |                 |              | KernelScale:         145.55 |
|  137 | Accept |    0.13782 |          0.091167 |         0.10154 |         0.10156 |          svm | BoxConstraint:       60.755 |
|      |        |            |                   |                 |                 |              | KernelScale:         131.88 |
|  138 | Accept |      0.125 |          0.087338 |         0.10154 |         0.10156 |          svm | BoxConstraint:       11.454 |
|      |        |            |                   |                 |                 |              | KernelScale:         176.64 |
|  139 | Accept |      0.125 |          0.080567 |         0.10154 |         0.10156 |          svm | BoxConstraint:        11.85 |
|      |        |            |                   |                 |                 |              | KernelScale:         160.69 |
|  140 | Accept |    0.16667 |          0.083852 |         0.10154 |         0.10156 |          svm | BoxConstraint:       11.155 |
|      |        |            |                   |                 |                 |              | KernelScale:         652.28 |
|=================================================================================================================================|
| Iter | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|=================================================================================================================================|
|  141 | Accept |      0.125 |          0.089769 |         0.10154 |         0.10156 |          svm | BoxConstraint:       10.933 |
|      |        |            |                   |                 |                 |              | KernelScale:         172.97 |
|  142 | Accept |    0.21474 |            0.1003 |         0.10154 |         0.10156 |          svm | BoxConstraint:       16.172 |
|      |        |            |                   |                 |                 |              | KernelScale:         22.373 |
|  143 | Accept |    0.13782 |            0.0833 |         0.10154 |         0.10156 |          svm | BoxConstraint:       7.6332 |
|      |        |            |                   |                 |                 |              | KernelScale:         178.51 |
|  144 | Accept |    0.13462 |           0.10562 |         0.10154 |         0.10156 |          svm | BoxConstraint:       6.7646 |
|      |        |            |                   |                 |                 |              | KernelScale:         160.72 |
|  145 | Accept |    0.14103 |          0.076631 |         0.10154 |         0.10156 |          svm | BoxConstraint:       6.8617 |
|      |        |            |                   |                 |                 |              | KernelScale:         189.52 |
|  146 | Accept |    0.15385 |          0.082538 |         0.10154 |         0.10156 |          svm | BoxConstraint:       6.0384 |
|      |        |            |                   |                 |                 |              | KernelScale:         332.89 |
|  147 | Accept |    0.14423 |          0.077025 |         0.10154 |         0.10156 |          svm | BoxConstraint:       5.7255 |
|      |        |            |                   |                 |                 |              | KernelScale:         182.19 |
|  148 | Accept |    0.13782 |          0.089308 |         0.10154 |         0.10156 |          svm | BoxConstraint:       70.212 |
|      |        |            |                   |                 |                 |              | KernelScale:         818.46 |
|  149 | Accept |    0.14103 |           0.08348 |         0.10154 |         0.10156 |          svm | BoxConstraint:        5.493 |
|      |        |            |                   |                 |                 |              | KernelScale:         230.68 |
|  150 | Accept |    0.13782 |          0.079744 |         0.10154 |         0.10156 |          svm | BoxConstraint:       5.3564 |
|      |        |            |                   |                 |                 |              | KernelScale:         111.96 |

__________________________________________________________
Optimization completed.
Total iterations: 150
Total elapsed time: 835.2167 seconds
Total time for training and validation: 193.4979 seconds

Best observed learner is a tree model with:
	MinLeafSize:              5
Observed validation loss: 0.10154
Time for training and validation: 0.079656 seconds

Best estimated learner (returned model) is a tree model with:
	MinLeafSize:              5
Estimated validation loss: 0.10156
Estimated time for training and validation: 0.076205 seconds

Documentation for fitcauto display

The final model returned by fitcauto corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (carsTrain), the listed Learner (or model) type, and the displayed hyperparameter values.

Evaluate Test Set Performance

Evaluate the performance of the model on the test set.

testAccuracy = 1 - loss(Mdl,carsTest,'Origin')
testAccuracy = 0.9143
confusionchart(carsTest.Origin,predict(Mdl,carsTest))

Use fitcauto to automatically select a classification model with optimized hyperparameters, given predictor and response data stored in separate variables.

Load Data

Load the humanactivity data set. This data set contains 24,075 observations of five physical human activities: Sitting (1), Standing (2), Walking (3), Running (4), and Dancing (5). Each observation has 60 features extracted from acceleration data measured by smartphone accelerometer sensors. The variable feat contains the predictor data matrix of the 60 features for the 24,075 observations, and the response variable actid contains the activity IDs for the observations as integers.

load humanactivity

Partition Data

Partition the data into training and test sets. Use 90% of the observations to select a model, and 10% of the observations to validate the final model returned by fitcauto. Use cvpartition to reserve 10% of the observations for testing.

rng('default') % For reproducibility of the partition
c = cvpartition(actid,'Holdout',0.10);
trainingIndices = training(c); % Indices for the training set
XTrain = feat(trainingIndices,:);
YTrain = actid(trainingIndices);
testIndices = test(c); % Indices for the test set
XTest = feat(testIndices,:);
YTest = actid(testIndices);

Run fitcauto

Pass the training data to fitcauto. By default, fitcauto determines appropriate model (or learner) types to try, uses Bayesian optimization to find good hyperparameter values for those models, and returns a trained model with the best expected performance. Specify to run the optimization in parallel (requires Parallel Computing Toolbox™). Return a second output OptimizationResults that contains the details of the Bayesian optimization.

Expect this model selection process to take some time. By default, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.

options = struct('UseParallel',true);
[Mdl,OptimizationResults] = fitcauto(XTrain,YTrain,'HyperparameterOptimizationOptions',options);
Warning: It is recommended that you first standardize all numeric predictors when optimizing the Naive Bayes 'Width' parameter. Ignore this warning if you have done that.
Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 6).
Copying objective function to workers...
Done copying objective function to workers.
Learner types to explore: ensemble, knn, nb, svm, tree
Total iterations (MaxObjectiveEvaluations): 150
Total time (MaxTime): Inf
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|    1 |       6 | Best   |    0.28088 |            48.752 |         0.28088 |         0.28088 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.22686 |
|      |         |        |            |                   |                 |                 |              | KernelScale:          330.4 |
|    2 |       6 | Best   |   0.036459 |            51.455 |        0.036459 |        0.036459 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      254 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           1786 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            12 |
|    3 |       6 | Best   |   0.025845 |            5.1379 |        0.025845 |        0.025845 |         tree | MinLeafSize:             59 |
|    4 |       6 | Best   |   0.006415 |            60.999 |        0.006415 |        0.021738 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      214 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              5 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            23 |
|    5 |       6 | Accept |   0.025845 |            4.8823 |        0.006415 |        0.021738 |         tree | MinLeafSize:             59 |
|    6 |       6 | Accept |   0.017768 |            5.4601 |        0.006415 |        0.021738 |         tree | MinLeafSize:              9 |
|    7 |       6 | Accept |   0.050212 |            1.1024 |        0.006415 |        0.021738 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|    8 |       6 | Accept |   0.050212 |           0.64183 |        0.006415 |        0.021738 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|    9 |       6 | Accept |   0.019568 |            151.32 |        0.006415 |        0.021152 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      218 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              2 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            63 |
|   10 |       6 | Accept |   0.026537 |            165.42 |        0.006415 |        0.022035 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      264 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              7 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            36 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   11 |       6 | Accept |    0.59166 |            29.107 |        0.006415 |        0.022035 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:           4.7212e-14 |
|   12 |       6 | Accept |   0.021645 |            50.626 |        0.006415 |        0.022122 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      243 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           1247 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            45 |
|   13 |       6 | Accept |   0.043567 |            24.583 |        0.006415 |        0.022122 |          knn | NumNeighbors:           144 |
|   14 |       6 | Accept |   0.028844 |            22.503 |        0.006415 |        0.022122 |          knn | NumNeighbors:            18 |
|   15 |       6 | Accept |    0.04389 |            157.12 |        0.006415 |        0.022122 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.068467 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         117.01 |
|   16 |       6 | Accept |   0.024598 |            20.721 |        0.006415 |        0.022122 |          knn | NumNeighbors:             7 |
|   17 |       6 | Accept |    0.03009 |            20.969 |        0.006415 |        0.022122 |          knn | NumNeighbors:            27 |
|   18 |       6 | Accept |   0.016753 |            6.5065 |        0.006415 |        0.021547 |         tree | MinLeafSize:              2 |
|   19 |       6 | Accept |   0.040059 |            4.3496 |        0.006415 |        0.022122 |         tree | MinLeafSize:            166 |
|   20 |       6 | Accept |   0.060319 |            2.2673 |        0.006415 |        0.022122 |         tree | MinLeafSize:           1881 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   21 |       6 | Accept |   0.050212 |           0.97596 |        0.006415 |        0.022122 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   22 |       6 | Accept |   0.036552 |            20.775 |        0.006415 |        0.022122 |          knn | NumNeighbors:            67 |
|   23 |       6 | Accept |   0.050212 |           0.55594 |        0.006415 |        0.022122 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   24 |       6 | Accept |    0.11076 |            36.229 |        0.006415 |        0.022122 |          knn | NumNeighbors:          2637 |
|   25 |       6 | Accept |    0.27884 |            66.582 |        0.006415 |        0.024532 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      287 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           4344 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            48 |
|   26 |       6 | Accept |    0.58127 |            31.861 |        0.006415 |        0.024532 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:           1.6293e-06 |
|   27 |       6 | Accept |    0.01583 |            5.7511 |        0.006415 |        0.020656 |         tree | MinLeafSize:              1 |
|   28 |       6 | Accept |   0.069319 |             1.806 |        0.006415 |         0.02077 |         tree | MinLeafSize:           2284 |
|   29 |       6 | Accept |    0.59166 |            352.55 |        0.006415 |         0.02077 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:        790.4 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.014348 |
|   30 |       6 | Accept |   0.043336 |            3.7865 |        0.006415 |        0.020133 |         tree | MinLeafSize:            432 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   31 |       6 | Accept |    0.10555 |            34.294 |        0.006415 |        0.020133 |          knn | NumNeighbors:          2430 |
|   32 |       6 | Accept |   0.021276 |             4.582 |        0.006415 |        0.018661 |         tree | MinLeafSize:             17 |
|   33 |       5 | Accept |   0.030829 |            159.57 |        0.006415 |        0.018642 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      288 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             45 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            23 |
|   34 |       5 | Accept |   0.014307 |            49.903 |        0.006415 |        0.018642 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      234 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            587 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            11 |
|   35 |       5 | Accept |   0.050212 |             1.025 |        0.006415 |        0.018642 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   36 |       6 | Accept |    0.74165 |            24.131 |        0.006415 |        0.018661 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      217 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           8856 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            36 |
|   37 |       6 | Accept |     0.4226 |            558.84 |        0.006415 |        0.018661 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:               72.906 |
|   38 |       6 | Accept |    0.57615 |            317.96 |        0.006415 |        0.018661 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       2.2347 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.16176 |
|   39 |       6 | Accept |    0.59166 |             26.64 |        0.006415 |        0.018661 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:            1.191e-07 |
|   40 |       6 | Accept |   0.087087 |            30.271 |        0.006415 |        0.018661 |          knn | NumNeighbors:          1634 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   41 |       6 | Accept |    0.73985 |            551.47 |        0.006415 |        0.018661 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:               1055.8 |
|   42 |       4 | Accept |   0.025983 |            146.08 |        0.006415 |        0.018661 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      234 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             73 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            43 |
|   43 |       4 | Accept |    0.02566 |            146.73 |        0.006415 |        0.018661 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      234 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             73 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            43 |
|   44 |       4 | Accept |   0.024922 |            124.65 |        0.006415 |        0.018661 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      206 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              5 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            39 |
|   45 |       6 | Accept |   0.025891 |            23.638 |        0.006415 |        0.018661 |          knn | NumNeighbors:             6 |
|   46 |       5 | Accept |   0.025891 |            24.038 |        0.006415 |        0.018661 |          knn | NumNeighbors:             6 |
|   47 |       5 | Accept |   0.025891 |            23.646 |        0.006415 |        0.018661 |          knn | NumNeighbors:             6 |
|   48 |       6 | Accept |    0.03009 |            188.48 |        0.006415 |        0.018661 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      299 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             31 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            25 |
|   49 |       6 | Accept |   0.017214 |            6.0889 |        0.006415 |        0.017935 |         tree | MinLeafSize:              4 |
|   50 |       6 | Accept |    0.01726 |            5.6027 |        0.006415 |        0.017303 |         tree | MinLeafSize:              5 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   51 |       6 | Accept |   0.037244 |             158.2 |        0.006415 |        0.017303 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       5.9571 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         840.87 |
|   52 |       6 | Accept |   0.046474 |            190.01 |        0.006415 |        0.017303 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       2.9119 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         12.771 |
|   53 |       6 | Accept |   0.032398 |            155.47 |        0.006415 |        0.017303 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      253 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             14 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            22 |
|   54 |       6 | Accept |   0.054135 |            3.0156 |        0.006415 |        0.017093 |         tree | MinLeafSize:            783 |
|   55 |       6 | Accept |   0.049797 |            28.421 |        0.006415 |        0.017093 |          knn | NumNeighbors:           331 |
|   56 |       6 | Accept |   0.046566 |            27.524 |        0.006415 |        0.017093 |          knn | NumNeighbors:           193 |
|   57 |       6 | Accept |    0.36307 |            711.57 |        0.006415 |        0.017093 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0011107 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.80966 |
|   58 |       6 | Accept |   0.022706 |            23.417 |        0.006415 |        0.017093 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       1.3505 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         127.55 |
|   59 |       6 | Accept |   0.028798 |            4.1232 |        0.006415 |         0.01733 |         tree | MinLeafSize:             84 |
|   60 |       6 | Accept |   0.041351 |            28.037 |        0.006415 |         0.01733 |          knn | NumNeighbors:           124 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   61 |       6 | Accept |   0.030044 |            26.265 |        0.006415 |         0.01733 |          knn | NumNeighbors:            26 |
|   62 |       6 | Accept |    0.11838 |            284.42 |        0.006415 |         0.01733 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0027874 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         33.944 |
|   63 |       6 | Accept |    0.47116 |            54.038 |        0.006415 |         0.01733 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:           4.7553e-05 |
|   64 |       6 | Accept |    0.26574 |            104.86 |        0.006415 |         0.01733 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:            0.0011441 |
|   65 |       6 | Accept |   0.050212 |           0.72243 |        0.006415 |         0.01733 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   66 |       6 | Accept |    0.01726 |            5.6362 |        0.006415 |        0.016846 |         tree | MinLeafSize:              5 |
|   67 |       6 | Accept |   0.077072 |            268.17 |        0.006415 |        0.016846 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:             0.026902 |
|   68 |       6 | Accept |   0.031613 |            25.909 |        0.006415 |        0.016846 |          knn | NumNeighbors:            33 |
|   69 |       6 | Accept |    0.02003 |            92.379 |        0.006415 |        0.016846 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       609.47 |
|      |         |        |            |                   |                 |                 |              | KernelScale:          39.88 |
|   70 |       6 | Accept |     0.1145 |            92.795 |        0.006415 |        0.016846 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      292 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           3870 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            33 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   71 |       6 | Accept |   0.012968 |            53.299 |        0.006415 |        0.016846 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      208 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              5 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            10 |
|   72 |       6 | Accept |   0.011999 |             53.54 |        0.006415 |        0.016846 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      215 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              5 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            11 |
|   73 |       6 | Accept |   0.011999 |            50.887 |        0.006415 |        0.012076 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      205 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              2 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            11 |
|   74 |       6 | Accept |   0.039921 |            40.808 |        0.006415 |        0.012733 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      200 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           1921 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            11 |
|   75 |       6 | Accept |    0.04232 |            98.684 |        0.006415 |        0.012754 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      220 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           1495 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            11 |
|   76 |       6 | Accept |   0.094702 |            397.88 |        0.006415 |        0.012754 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       671.76 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         4.6418 |
|   77 |       6 | Accept |   0.065165 |            44.801 |        0.006415 |        0.013179 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      233 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           2586 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            35 |
|   78 |       6 | Accept |    0.22503 |            164.64 |        0.006415 |        0.013179 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:        57.71 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         2.0632 |
|   79 |       6 | Accept |    0.03069 |             44.42 |        0.006415 |        0.013027 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      213 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           1619 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            95 |
|   80 |       6 | Accept |    0.11459 |             71.56 |        0.006415 |        0.012064 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      212 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           2669 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            31 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   81 |       6 | Accept |   0.044582 |            41.462 |        0.006415 |        0.013176 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      207 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           2211 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            56 |
|   82 |       6 | Accept |   0.014722 |             223.7 |        0.006415 |        0.013176 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       832.84 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         392.65 |
|   83 |       6 | Accept |   0.018322 |             23.34 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       322.92 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         542.76 |
|   84 |       6 | Accept |   0.016984 |            28.833 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       727.84 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         558.25 |
|   85 |       6 | Accept |    0.59166 |            2135.3 |        0.006415 |        0.013176 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.045413 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0034709 |
|   86 |       6 | Accept |   0.012461 |            39.005 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       806.52 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         244.64 |
|   87 |       6 | Accept |   0.016753 |            22.649 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       173.38 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         329.41 |
|   88 |       6 | Accept |   0.016845 |            28.292 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       242.37 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         62.644 |
|   89 |       6 | Accept |   0.016799 |            20.771 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       80.674 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         242.97 |
|   90 |       6 | Accept |   0.041259 |            175.27 |        0.006415 |        0.013176 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       2.0147 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         549.95 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   91 |       6 | Accept |   0.017953 |            19.778 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       63.367 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         263.72 |
|   92 |       6 | Accept |   0.019568 |            19.245 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       55.612 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         344.11 |
|   93 |       6 | Accept |   0.016061 |            19.511 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       60.019 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         185.49 |
|   94 |       6 | Accept |    0.11847 |             149.5 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       770.37 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         3.1188 |
|   95 |       6 | Accept |   0.017953 |            18.428 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:        43.94 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         234.89 |
|   96 |       6 | Accept |   0.025106 |            25.204 |        0.006415 |        0.013176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       2.1294 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         237.71 |
|   97 |       6 | Accept |   0.011676 |            70.358 |        0.006415 |        0.012446 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      283 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              4 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            10 |
|   98 |       6 | Accept |   0.031983 |            37.179 |        0.006415 |        0.012446 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:        1.832 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         449.73 |
|   99 |       6 | Accept |  0.0097379 |            76.639 |        0.006415 |        0.011402 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      299 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              4 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            12 |
|  100 |       6 | Accept |    0.22416 |            684.65 |        0.006415 |        0.011402 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       632.44 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.8647 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  101 |       6 | Accept |    0.11478 |            87.263 |        0.006415 |        0.011913 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      272 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:           3176 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            54 |
|  102 |       6 | Accept |  0.0081687 |            75.177 |        0.006415 |         0.01045 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      291 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              6 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            14 |
|  103 |       6 | Accept |   0.010753 |             70.01 |        0.006415 |        0.010258 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      284 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              1 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            11 |
|  104 |       6 | Accept |    0.36076 |                77 |        0.006415 |        0.010258 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0054235 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         149.67 |
|  105 |       6 | Accept |  0.0084456 |            62.748 |        0.006415 |       0.0092051 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      214 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              3 |
|      |         |        |            |                   |                 |                 |              | MaxNumSplits:            16 |
|  106 |       6 | Accept |   0.012738 |             32.34 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       529.51 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         161.83 |
|  107 |       6 | Accept |   0.031521 |            41.482 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       0.6738 |
|      |         |        |            |                   |                 |                 |              | KernelScale:          289.2 |
|  108 |       6 | Accept |   0.021368 |            38.604 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       567.35 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         36.573 |
|  109 |       6 | Accept |   0.050212 |           0.59557 |        0.006415 |       0.0092051 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|  110 |       6 | Accept |    0.57883 |            127.15 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0011506 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         211.98 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  111 |       6 | Accept |   0.050212 |             0.685 |        0.006415 |       0.0092051 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|  112 |       6 | Accept |   0.029583 |            33.504 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.49417 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         209.72 |
|  113 |       6 | Accept |   0.012138 |            41.159 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       967.85 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         153.85 |
|  114 |       6 | Accept |   0.030921 |            38.482 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.57771 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         262.15 |
|  115 |       6 | Accept |     0.1295 |            52.128 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.35331 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         329.77 |
|  116 |       6 | Accept |   0.028475 |            33.553 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.37306 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         157.01 |
|  117 |       6 | Accept |   0.050212 |           0.62316 |        0.006415 |       0.0092051 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|  118 |       6 | Accept |   0.030737 |                37 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.37996 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         204.24 |
|  119 |       6 | Accept |   0.021229 |             36.97 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       803.51 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         38.271 |
|  120 |       6 | Accept |   0.015184 |            24.004 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       119.72 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         145.99 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  121 |       6 | Accept |    0.06955 |            101.38 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       39.152 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         8.5572 |
|  122 |       6 | Accept |   0.028291 |            32.637 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.32264 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         134.73 |
|  123 |       6 | Accept |   0.041951 |             51.65 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.28203 |
|      |         |        |            |                   |                 |                 |              | KernelScale:          237.1 |
|  124 |       6 | Accept |   0.039921 |            216.38 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.049329 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         190.24 |
|  125 |       6 | Accept |   0.048828 |            91.888 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       180.12 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         12.235 |
|  126 |       6 | Accept |   0.030644 |            38.563 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.25725 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         171.19 |
|  127 |       6 | Accept |   0.029537 |            37.409 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.25269 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         145.85 |
|  128 |       6 | Accept |    0.02949 |            31.222 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.24034 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         59.755 |
|  129 |       6 | Accept |   0.029537 |            130.19 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       76.533 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         22.982 |
|  130 |       6 | Accept |   0.094702 |            174.14 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.043666 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         25.411 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  131 |       6 | Accept |   0.037382 |            46.207 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.23867 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         204.62 |
|  132 |       6 | Accept |   0.022152 |            40.588 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       887.62 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         32.987 |
|  133 |       6 | Accept |   0.032629 |            32.651 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.22511 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         52.294 |
|  134 |       6 | Accept |    0.02806 |            131.12 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.20346 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         122.85 |
|  135 |       6 | Accept |    0.59166 |              3480 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       658.37 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0016161 |
|  136 |       6 | Accept |    0.59166 |            3481.3 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       658.37 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0016161 |
|  137 |       6 | Accept |   0.032121 |            44.172 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.20648 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         162.33 |
|  138 |       6 | Accept |    0.30755 |              71.9 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.015886 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         120.23 |
|  139 |       6 | Accept |   0.040936 |            233.89 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.046614 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         213.78 |
|  140 |       5 | Accept |   0.040521 |            224.72 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.043998 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         195.58 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  141 |       5 | Accept |    0.22208 |            64.659 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.049755 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         151.01 |
|  142 |       6 | Accept |   0.029444 |            30.014 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.21789 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         64.023 |
|  143 |       6 | Accept |   0.046013 |            49.025 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.14805 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         178.98 |
|  144 |       6 | Accept |   0.013291 |            42.726 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       960.14 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         334.83 |
|  145 |       6 | Accept |    0.59166 |            3493.5 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       17.982 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0043756 |
|  146 |       6 | Accept |   0.089533 |            189.71 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       0.2326 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         15.411 |
|  147 |       6 | Accept |    0.03369 |            49.586 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.21482 |
|      |         |        |            |                   |                 |                 |              | KernelScale:          180.5 |
|  148 |       6 | Accept |   0.013384 |            42.876 |        0.006415 |       0.0092051 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       920.16 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         334.75 |
|  149 |       6 | Accept |   0.027368 |             117.6 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.22783 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         105.59 |
|  150 |       6 | Accept |   0.041674 |            234.61 |        0.006415 |       0.0092051 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.032335 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         182.17 |

__________________________________________________________
Optimization completed.
Total iterations: 150
Total elapsed time: 4534.2478 seconds
Total time for training and validation: 24883.8563 seconds

Best observed learner is an ensemble model with:
	Method:          AdaBoostM2
	NumLearningCycles:      214
	MinLeafSize:              5
	MaxNumSplits:            23
Observed validation loss: 0.006415
Time for training and validation: 60.9987 seconds

Best estimated learner (returned model) is an ensemble model with:
	Method:          AdaBoostM2
	NumLearningCycles:      214
	MinLeafSize:              3
	MaxNumSplits:            16
Estimated validation loss: 0.0092051
Estimated time for training and validation: 57.8146 seconds

Documentation for fitcauto display

The final model returned by fitcauto corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (XTrain and YTrain), the listed Learner (or model) type, and the displayed hyperparameter values.

Evaluate Test Set Performance

Evaluate the final model performance on the test data set.

testAccuracy = 1 - loss(Mdl,XTest,YTest)
testAccuracy = 0.9917

The final model correctly classifies over 99% of the observations.

Use fitcauto to automatically select a classification model with optimized hyperparameters, given predictor and response data stored in a table. Before passing data to fitcauto, perform feature selection to remove unimportant predictors from the data set.

Load and Partition Data

Read the sample file CreditRating_Historical.dat into a table. The predictor data consists of financial ratios and industry sector information for a list of corporate customers. The response variable consists of credit ratings assigned by a rating agency. Preview the first few rows of the data set.

creditrating = readtable('CreditRating_Historical.dat');
head(creditrating)
ans=8×8 table
     ID      WC_TA     RE_TA     EBIT_TA    MVE_BVTD    S_TA     Industry    Rating 
    _____    ______    ______    _______    ________    _____    ________    _______

    62394     0.013     0.104     0.036      0.447      0.142        3       {'BB' }
    48608     0.232     0.335     0.062      1.969      0.281        8       {'A'  }
    42444     0.311     0.367     0.074      1.935      0.366        1       {'A'  }
    48631     0.194     0.263     0.062      1.017      0.228        4       {'BBB'}
    43768     0.121     0.413     0.057      3.647      0.466       12       {'AAA'}
    39255    -0.117    -0.799      0.01      0.179      0.082        4       {'CCC'}
    62236     0.087     0.158     0.049      0.816      0.324        2       {'BBB'}
    39354     0.005     0.181     0.034      2.597      0.388        7       {'AA' }

Because each value in the ID variable is a unique customer ID, that is, length(unique(creditrating.ID)) is equal to the number of observations in creditrating, the ID variable is a poor predictor. Remove the ID variable from the table, and convert the Industry variable to a categorical variable.

creditrating = removevars(creditrating,'ID');
creditrating.Industry = categorical(creditrating.Industry);

Partition the data into training and test sets. Use approximately 85% of the observations for the model selection and hyperparameter tuning process, and 15% of the observations to test the performance of the final model returned by fitcauto on new data. Use cvpartition to partition the data.

rng('default') % For reproducibility of the partition
c = cvpartition(creditrating.Rating,'Holdout',0.15);
trainingIndices = training(c); % Indices for the training set
testIndices = test(c); % Indices for the test set
creditTrain = creditrating(trainingIndices,:);
creditTest = creditrating(testIndices,:);

Perform Feature Selection

Before passing the training data to fitcauto, find the important predictors by using the fscchi2 function. Visualize the predictor scores by using the bar function. Because some scores can be Inf, and bar discards Inf values, plot the finite scores first and then plot a finite representation of the Inf scores in a different color.

[idx,scores] = fscchi2(creditTrain,'Rating');
bar(scores(idx)) % Represents finite scores
hold on
veryImportant = isinf(scores);
finiteMax = max(scores(~veryImportant));
bar(finiteMax*veryImportant(idx)) % Represents Inf scores
hold off
xticklabels(strrep(creditTrain.Properties.VariableNames(idx),'_','\_'))
xtickangle(45)
legend({'Finite Scores','Inf Scores'})

Notice that the Industry predictor has a low score corresponding to a p-value that is greater than 0.05, which indicates that Industry might not be an important feature. Remove the Industry feature from the training and test data sets.

creditTrain = removevars(creditTrain,'Industry');
creditTest = removevars(creditTest,'Industry');

Run fitcauto

Pass the training data to fitcauto. The function uses Bayesian optimization to select models and their hyperparameter values, and returns a trained model Mdl with the best expected performance. Specify to try all available learner types and run the optimization in parallel (requires Parallel Computing Toolbox™). Return a second output Results that contains the details of the Bayesian optimization.

Expect this process to take some time. By default, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.

options = struct('UseParallel',true);
[Mdl,Results] = fitcauto(creditTrain,'Rating', ...
    'Learners','all','HyperparameterOptimizationOptions',options);
Warning: It is recommended that you first standardize all numeric predictors when optimizing the Naive Bayes 'Width' parameter. Ignore this warning if you have done that.
Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 6).
Copying objective function to workers...
Done copying objective function to workers.
Learner types to explore: discr, ensemble, kernel, knn, linear, nb, svm, tree
Total iterations (MaxObjectiveEvaluations): 240
Total time (MaxTime): Inf
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|    1 |       6 | Best   |    0.42716 |            3.0379 |         0.42716 |         0.42716 |        discr | Delta:           0.00046441 |
|      |         |        |            |                   |                 |                 |              | Gamma:               0.2485 |
|    2 |       4 | Accept |    0.74185 |            4.7899 |         0.24948 |         0.29794 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.48455 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         354.44 |
|    3 |       4 | Best   |    0.24948 |            5.0813 |         0.24948 |         0.29794 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          6.3551e-08 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|    4 |       4 | Accept |    0.29794 |            3.7295 |         0.24948 |         0.29794 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:       12 |
|      |         |        |            |                   |                 |                 |              | LearnRate:         0.063776 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            277 |
|    5 |       3 | Accept |    0.25097 |            9.2655 |         0.24948 |         0.25067 |       kernel | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | KernelScale:         7.8433 |
|      |         |        |            |                   |                 |                 |              | Lambda:          1.4468e-06 |
|    6 |       3 | Accept |    0.25067 |           0.81139 |         0.24948 |         0.25067 |          knn | NumNeighbors:           105 |
|      |         |        |            |                   |                 |                 |              | Distance:         minkowski |
|    7 |       6 | Accept |    0.52917 |            2.3362 |         0.24948 |         0.25067 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.002417 |
|      |         |        |            |                   |                 |                 |              | KernelScale:          356.9 |
|    8 |       3 | Accept |    0.55818 |           0.63908 |         0.24948 |         0.25067 |        discr | Delta:              0.98612 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.86519 |
|    9 |       3 | Accept |     0.3781 |            1.6777 |         0.24948 |         0.25067 |       linear | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | Lambda:          1.0412e-06 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|   10 |       3 | Accept |    0.43225 |           0.80766 |         0.24948 |         0.25067 |        discr | Delta:           0.00013711 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.60585 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   11 |       3 | Accept |    0.47712 |            3.3756 |         0.24948 |         0.25067 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       2.7347 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         24.465 |
|   12 |       6 | Accept |    0.25695 |            2.3709 |         0.24948 |         0.25067 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:             0.057566 |
|   13 |       4 | Accept |    0.26413 |           0.49941 |         0.24379 |         0.25067 |         tree | MinLeafSize:             30 |
|   14 |       4 | Accept |    0.42327 |           0.85101 |         0.24379 |         0.25067 |          knn | NumNeighbors:            56 |
|      |         |        |            |                   |                 |                 |              | Distance:            cosine |
|   15 |       4 | Best   |    0.24379 |            1.8084 |         0.24379 |         0.25067 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          5.9172e-05 |
|      |         |        |            |                   |                 |                 |              | Learner:                svm |
|   16 |       3 | Accept |    0.81544 |            4.9586 |         0.24379 |         0.25067 |       kernel | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0043375 |
|      |         |        |            |                   |                 |                 |              | Lambda:           0.0023789 |
|   17 |       3 | Accept |    0.45169 |           0.96723 |         0.24379 |         0.25067 |       linear | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | Lambda:           0.0028505 |
|      |         |        |            |                   |                 |                 |              | Learner:                svm |
|   18 |       6 | Accept |    0.33712 |           0.19972 |         0.24379 |         0.25695 |          knn | NumNeighbors:             1 |
|      |         |        |            |                   |                 |                 |              | Distance:         cityblock |
|   19 |       3 | Accept |     0.4834 |           0.38951 |         0.24379 |         0.25695 |          knn | NumNeighbors:            72 |
|      |         |        |            |                   |                 |                 |              | Distance:       correlation |
|   20 |       3 | Accept |    0.46336 |           0.78881 |         0.24379 |         0.25695 |       linear | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | Lambda:           0.0075732 |
|      |         |        |            |                   |                 |                 |              | Learner:                svm |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   21 |       3 | Accept |    0.82082 |            2.9223 |         0.24379 |         0.25695 |       kernel | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0042587 |
|      |         |        |            |                   |                 |                 |              | Lambda:           0.0014754 |
|   22 |       3 | Accept |    0.61292 |            1.8557 |         0.24379 |         0.25695 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:       13 |
|      |         |        |            |                   |                 |                 |              | LearnRate:         0.055349 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            910 |
|   23 |       6 | Accept |    0.43255 |           0.68689 |         0.24379 |         0.25695 |        discr | Delta:             0.016844 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.64466 |
|   24 |       4 | Accept |    0.28866 |            1.9017 |         0.24379 |         0.25695 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:       10 |
|      |         |        |            |                   |                 |                 |              | LearnRate:          0.11662 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            181 |
|   25 |       4 | Accept |    0.74185 |            1.3546 |         0.24379 |         0.25695 |          knn | NumNeighbors:          1314 |
|      |         |        |            |                   |                 |                 |              | Distance:           hamming |
|   26 |       4 | Accept |    0.42746 |           0.69002 |         0.24379 |         0.25695 |        discr | Delta:           2.2544e-06 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.87275 |
|   27 |       3 | Accept |    0.25606 |            12.143 |         0.24379 |         0.25498 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      132 |
|      |         |        |            |                   |                 |                 |              | LearnRate:          0.92674 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            127 |
|   28 |       3 | Accept |    0.25366 |            2.4732 |         0.24379 |         0.25498 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:              0.10033 |
|   29 |       6 | Accept |    0.66796 |           0.22938 |         0.24379 |         0.25498 |          knn | NumNeighbors:            77 |
|      |         |        |            |                   |                 |                 |              | Distance:           jaccard |
|   30 |       4 | Accept |    0.69488 |            1.9352 |           0.242 |         0.25498 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       8.4886 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         192.19 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   31 |       4 | Best   |      0.242 |            1.9467 |           0.242 |         0.25498 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:        5.425 |
|      |         |        |            |                   |                 |                 |              | KernelScale:          2.434 |
|   32 |       4 | Accept |    0.32306 |           0.48104 |           0.242 |         0.25498 |         tree | MinLeafSize:              2 |
|   33 |       4 | Accept |    0.43225 |           0.10463 |           0.242 |         0.25498 |        discr | Delta:           0.00015292 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.51045 |
|   34 |       4 | Accept |    0.32994 |            0.2065 |           0.242 |         0.25498 |         tree | MinLeafSize:              3 |
|   35 |       6 | Accept |    0.53814 |            2.8748 |           0.242 |         0.25498 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       6.8148 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         382.11 |
|   36 |       4 | Accept |    0.24529 |            105.62 |           0.242 |         0.25498 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.25488 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0037823 |
|   37 |       4 | Accept |    0.53814 |            3.3272 |           0.242 |         0.25498 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       6.8148 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         382.11 |
|   38 |       4 | Accept |    0.53814 |            3.8593 |           0.242 |         0.25498 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       6.8148 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         382.11 |
|   39 |       4 | Accept |    0.25965 |            14.211 |           0.242 |         0.25498 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      150 |
|      |         |        |            |                   |                 |                 |              | LearnRate:         0.014842 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             21 |
|   40 |       5 | Accept |    0.42656 |           0.16731 |           0.242 |         0.25498 |        discr | Delta:            0.0020866 |
|      |         |        |            |                   |                 |                 |              | Gamma:             0.091054 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   41 |       5 | Accept |    0.42656 |           0.11476 |           0.242 |         0.25498 |        discr | Delta:            0.0020866 |
|      |         |        |            |                   |                 |                 |              | Gamma:             0.091054 |
|   42 |       4 | Accept |     0.2767 |            21.389 |           0.242 |         0.25498 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      221 |
|      |         |        |            |                   |                 |                 |              | LearnRate:        0.0028588 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              1 |
|   43 |       4 | Accept |    0.29973 |            0.1848 |           0.242 |         0.25498 |         tree | MinLeafSize:              7 |
|   44 |       4 | Accept |    0.25935 |            20.084 |           0.242 |         0.25498 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      304 |
|      |         |        |            |                   |                 |                 |              | LearnRate:              NaN |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            100 |
|   45 |       3 | Accept |    0.24499 |            6.5071 |           0.242 |         0.26328 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.019387 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0047515 |
|   46 |       3 | Accept |    0.28059 |           0.19378 |           0.242 |         0.26328 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   47 |       6 | Accept |    0.27281 |           0.13181 |           0.242 |         0.26328 |          knn | NumNeighbors:             8 |
|      |         |        |            |                   |                 |                 |              | Distance:         chebychev |
|   48 |       3 | Accept |     0.2429 |            1.5474 |           0.242 |         0.26328 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.16719 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.11257 |
|   49 |       3 | Accept |     0.2423 |            1.5219 |           0.242 |         0.26328 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:            4.28e-07 |
|      |         |        |            |                   |                 |                 |              | Learner:                svm |
|   50 |       3 | Accept |    0.67125 |           0.68464 |           0.242 |         0.26328 |          knn | NumNeighbors:            86 |
|      |         |        |            |                   |                 |                 |              | Distance:           jaccard |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   51 |       3 | Accept |    0.46485 |            0.8417 |           0.242 |         0.26328 |         tree | MinLeafSize:            645 |
|   52 |       6 | Accept |    0.63147 |            3.1926 |           0.242 |         0.26328 |       kernel | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | KernelScale:          176.2 |
|      |         |        |            |                   |                 |                 |              | Lambda:          2.3903e-06 |
|   53 |       4 | Accept |    0.37571 |           0.11086 |           0.242 |         0.26706 |         tree | MinLeafSize:            473 |
|   54 |       4 | Accept |    0.29136 |           0.43908 |           0.242 |         0.26706 |          knn | NumNeighbors:           354 |
|      |         |        |            |                   |                 |                 |              | Distance:         euclidean |
|   55 |       4 | Accept |    0.28059 |           0.50642 |           0.242 |         0.26706 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   56 |       3 | Accept |    0.36375 |            5.2617 |           0.242 |         0.26706 |       kernel | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | KernelScale:         28.598 |
|      |         |        |            |                   |                 |                 |              | Lambda:          8.3238e-05 |
|   57 |       3 | Accept |    0.27251 |           0.13425 |           0.242 |         0.26706 |         tree | MinLeafSize:             20 |
|   58 |       6 | Accept |    0.43225 |          0.083255 |           0.242 |         0.26706 |        discr | Delta:             0.021467 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.66016 |
|   59 |       3 | Accept |    0.28059 |           0.10921 |           0.242 |         0.26106 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   60 |       3 | Accept |    0.42537 |           0.15885 |           0.242 |         0.26106 |        discr | Delta:             0.001728 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.89471 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   61 |       3 | Accept |    0.81484 |            2.9209 |           0.242 |         0.26106 |       kernel | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0063987 |
|      |         |        |            |                   |                 |                 |              | Lambda:           0.0075855 |
|   62 |       3 | Accept |    0.24948 |            2.4493 |           0.242 |         0.26106 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          1.3237e-07 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|   63 |       6 | Accept |    0.68531 |           0.40944 |           0.242 |         0.26106 |          knn | NumNeighbors:           260 |
|      |         |        |            |                   |                 |                 |              | Distance:           jaccard |
|   64 |       3 | Accept |    0.32426 |            9.8071 |           0.242 |         0.26007 |     ensemble | Method:            RUSBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      132 |
|      |         |        |            |                   |                 |                 |              | LearnRate:        0.0014516 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            104 |
|   65 |       3 | Accept |    0.55369 |           0.69919 |           0.242 |         0.26007 |          knn | NumNeighbors:           615 |
|      |         |        |            |                   |                 |                 |              | Distance:       correlation |
|   66 |       3 | Accept |    0.24319 |            1.5742 |           0.242 |         0.26007 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          4.3001e-09 |
|      |         |        |            |                   |                 |                 |              | Learner:                svm |
|   67 |       3 | Accept |    0.70894 |            1.9822 |           0.242 |         0.26007 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.45564 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         51.195 |
|   68 |       6 | Accept |    0.46485 |           0.17082 |           0.242 |         0.26007 |         tree | MinLeafSize:            611 |
|   69 |       5 | Accept |    0.74185 |            2.1172 |           0.242 |         0.26007 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.94197 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         524.56 |
|   70 |       5 | Accept |    0.83548 |            6.3585 |           0.242 |         0.26007 |       kernel | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0038762 |
|      |         |        |            |                   |                 |                 |              | Lambda:           2.288e-06 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   71 |       3 | Accept |    0.25905 |            16.096 |           0.242 |         0.26007 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      160 |
|      |         |        |            |                   |                 |                 |              | LearnRate:              NaN |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              2 |
|   72 |       3 | Accept |    0.28448 |            13.826 |           0.242 |         0.26007 |     ensemble | Method:            RUSBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      161 |
|      |         |        |            |                   |                 |                 |              | LearnRate:           0.7581 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             14 |
|   73 |       3 | Accept |    0.25695 |           0.24642 |           0.242 |         0.26007 |          knn | NumNeighbors:            14 |
|      |         |        |            |                   |                 |                 |              | Distance:         cityblock |
|   74 |       6 | Accept |    0.32396 |            1.8799 |           0.242 |         0.26007 |       kernel | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.86904 |
|      |         |        |            |                   |                 |                 |              | Lambda:             0.29724 |
|   75 |       4 | Accept |    0.32456 |           0.20481 |           0.242 |         0.26007 |         tree | MinLeafSize:              5 |
|   76 |       4 | Accept |    0.32994 |           0.35799 |           0.242 |         0.26007 |         tree | MinLeafSize:              3 |
|   77 |       4 | Accept |    0.26054 |            4.1734 |           0.242 |         0.26007 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:       32 |
|      |         |        |            |                   |                 |                 |              | LearnRate:          0.06853 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             19 |
|   78 |       4 | Accept |    0.43703 |           0.92964 |           0.242 |         0.24831 |       linear | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | Lambda:            0.013265 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|   79 |       4 | Accept |    0.31588 |            17.116 |           0.242 |         0.24831 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      201 |
|      |         |        |            |                   |                 |                 |              | LearnRate:        0.0012955 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            319 |
|   80 |       3 | Accept |    0.25277 |            7.8173 |           0.242 |         0.24831 |       kernel | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.78697 |
|      |         |        |            |                   |                 |                 |              | Lambda:          4.1197e-06 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   81 |       3 | Accept |    0.43015 |           0.10239 |           0.242 |         0.24831 |        discr | Delta:            0.0069822 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.49526 |
|   82 |       6 | Accept |    0.42208 |          0.096953 |           0.242 |         0.24831 |        discr | Delta:             0.057485 |
|      |         |        |            |                   |                 |                 |              | Gamma:             0.045714 |
|   83 |       3 | Accept |    0.52617 |            2.3915 |           0.242 |         0.24831 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.26869 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         17.595 |
|   84 |       3 | Accept |    0.43344 |            1.2875 |           0.242 |         0.24831 |          knn | NumNeighbors:           119 |
|      |         |        |            |                   |                 |                 |              | Distance:       mahalanobis |
|   85 |       3 | Accept |    0.30093 |            1.3003 |           0.242 |         0.24831 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:            0.047624 |
|      |         |        |            |                   |                 |                 |              | Learner:                svm |
|   86 |       3 | Accept |    0.42267 |           0.64506 |           0.242 |         0.24831 |          knn | NumNeighbors:            48 |
|      |         |        |            |                   |                 |                 |              | Distance:            cosine |
|   87 |       6 | Accept |    0.32905 |           0.26891 |           0.242 |         0.24831 |          knn | NumNeighbors:            65 |
|      |         |        |            |                   |                 |                 |              | Distance:        seuclidean |
|   88 |       4 | Accept |    0.24349 |            1.9282 |           0.242 |         0.24684 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0024196 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0082547 |
|   89 |       4 | Accept |    0.24499 |            1.6543 |           0.242 |         0.24684 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          3.3697e-06 |
|      |         |        |            |                   |                 |                 |              | Learner:                svm |
|   90 |       4 | Accept |    0.24469 |            2.1463 |           0.242 |         0.24684 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          6.5777e-05 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|   91 |       4 | Accept |    0.28059 |           0.15481 |           0.242 |         0.24684 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   92 |       5 | Accept |    0.28059 |           0.38138 |           0.242 |         0.24684 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   93 |       5 | Accept |    0.28059 |           0.15804 |           0.242 |         0.24684 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   94 |       4 | Accept |    0.29674 |            11.234 |           0.242 |         0.24684 |     ensemble | Method:            RUSBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      147 |
|      |         |        |            |                   |                 |                 |              | LearnRate:          0.95321 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            112 |
|   95 |       4 | Accept |    0.28059 |           0.12279 |           0.242 |         0.24684 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   96 |       4 | Accept |    0.29704 |            1.1016 |           0.242 |         0.24684 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:       11 |
|      |         |        |            |                   |                 |                 |              | LearnRate:        0.0072731 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             12 |
|   97 |       4 | Accept |    0.28059 |          0.099833 |           0.242 |         0.24684 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|   98 |       4 | Best   |     0.2411 |            1.3634 |          0.2411 |         0.24471 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          0.00056045 |
|      |         |        |            |                   |                 |                 |              | Learner:                svm |
|   99 |       4 | Accept |    0.74185 |           0.10441 |          0.2411 |         0.24471 |        discr | Delta:               39.281 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.77032 |
|  100 |       4 | Accept |    0.30093 |           0.11043 |          0.2411 |         0.24471 |         tree | MinLeafSize:            135 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  101 |       4 | Accept |    0.46844 |           0.17182 |          0.2411 |         0.24471 |          knn | NumNeighbors:             2 |
|      |         |        |            |                   |                 |                 |              | Distance:            cosine |
|  102 |       4 | Accept |    0.25426 |            3.1634 |          0.2411 |         0.24471 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:       38 |
|      |         |        |            |                   |                 |                 |              | LearnRate:              NaN |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:             28 |
|  103 |       4 | Accept |    0.24469 |            1.6558 |          0.2411 |         0.24558 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:           0.0012413 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|  104 |       4 | Accept |    0.43823 |            3.6572 |          0.2411 |         0.24558 |       kernel | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | KernelScale:         13.293 |
|      |         |        |            |                   |                 |                 |              | Lambda:           0.0031304 |
|  105 |       4 | Accept |    0.31977 |            1.3423 |          0.2411 |         0.24556 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:            0.040102 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|  106 |       6 | Accept |    0.24678 |            2.0849 |          0.2411 |         0.24529 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          3.9918e-05 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|  107 |       5 | Accept |    0.24678 |            1.9627 |          0.2411 |         0.24441 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          3.9918e-05 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|  108 |       5 | Accept |    0.24678 |            2.3413 |          0.2411 |         0.24441 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          3.9918e-05 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|  109 |       4 | Accept |    0.29435 |            14.698 |          0.2411 |         0.24441 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      249 |
|      |         |        |            |                   |                 |                 |              | LearnRate:              NaN |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            292 |
|  110 |       4 | Accept |    0.37152 |          0.098701 |          0.2411 |         0.24441 |         tree | MinLeafSize:            375 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  111 |       4 | Accept |    0.25666 |            5.7386 |          0.2411 |         0.24441 |     ensemble | Method:                 Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:       61 |
|      |         |        |            |                   |                 |                 |              | LearnRate:              NaN |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              3 |
|  112 |       4 | Accept |    0.28059 |           0.10645 |          0.2411 |         0.24441 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|  113 |       4 | Accept |    0.28059 |           0.10515 |          0.2411 |         0.24441 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|  114 |       6 | Accept |    0.74185 |            2.7447 |          0.2411 |         0.24441 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:               74.975 |
|  115 |       4 | Accept |    0.78552 |            10.495 |          0.2411 |         0.24441 |       kernel | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0050713 |
|      |         |        |            |                   |                 |                 |              | Lambda:          3.7406e-06 |
|  116 |       4 | Accept |    0.74185 |            2.7544 |          0.2411 |         0.24441 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:               74.975 |
|  117 |       4 | Accept |    0.74185 |            2.6565 |          0.2411 |         0.24441 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:               74.975 |
|  118 |       4 | Accept |    0.45797 |            11.614 |          0.2411 |         0.24441 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.94716 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.072905 |
|  119 |       4 | Accept |    0.65271 |            1.6037 |          0.2411 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010356 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.3521 |
|  120 |       4 | Accept |     0.3108 |            1.4953 |          0.2411 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.080708 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         2.7439 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  121 |       4 | Accept |     0.2414 |            1.2954 |          0.2411 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0048396 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.02413 |
|  122 |       4 | Accept |     0.5157 |            9.6657 |          0.2411 |         0.24441 |       kernel | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.071659 |
|      |         |        |            |                   |                 |                 |              | Lambda:          1.3447e-05 |
|  123 |       4 | Accept |    0.53186 |            390.32 |          0.2402 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       416.46 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0019087 |
|  124 |       4 | Best   |     0.2402 |            1.4571 |          0.2402 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.033468 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.073489 |
|  125 |       4 | Accept |     0.2402 |            1.3713 |          0.2402 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0040249 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.030373 |
|  126 |       3 | Accept |    0.24529 |            39.503 |          0.2402 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.32116 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0076281 |
|  127 |       3 | Accept |    0.25576 |            1.4031 |          0.2402 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.018022 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         0.3703 |
|  128 |       6 | Accept |     0.2426 |            1.3555 |          0.2402 |         0.24229 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.029999 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.046245 |
|  129 |       3 | Accept |    0.24499 |            9.5236 |          0.2402 |         0.24419 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.38346 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.01786 |
|  130 |       3 | Accept |    0.28059 |           0.12828 |          0.2402 |         0.24419 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  131 |       3 | Accept |     0.6886 |            1.4213 |          0.2402 |         0.24419 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:              3.0763 |
|      |         |        |            |                   |                 |                 |              | Learner:           logistic |
|  132 |       3 | Accept |     0.4487 |             3.111 |          0.2402 |         0.24419 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:        4.845 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.74028 |
|  133 |       6 | Accept |    0.24469 |            8.3042 |          0.2402 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.45335 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.02034 |
|  134 |       3 | Accept |    0.26204 |            1.7109 |          0.2402 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0011217 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.10651 |
|  135 |       3 | Accept |    0.74185 |            4.0388 |          0.2402 |         0.24441 |       kernel | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | KernelScale:         10.518 |
|      |         |        |            |                   |                 |                 |              | Lambda:             0.20458 |
|  136 |       3 | Accept |    0.29734 |            9.9598 |          0.2402 |         0.24441 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      107 |
|      |         |        |            |                   |                 |                 |              | LearnRate:        0.0023672 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:            197 |
|  137 |       3 | Accept |    0.44391 |            2.0399 |          0.2402 |         0.24441 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:           0.00025608 |
|  138 |       6 | Accept |    0.24559 |            3.2102 |          0.2402 |         0.24276 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       0.5211 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.04163 |
|  139 |       4 | Accept |    0.25067 |            1.4995 |          0.2402 |         0.24276 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.030306 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.37391 |
|  140 |       4 | Accept |    0.24559 |            1.6227 |          0.2402 |         0.24276 |       linear | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | Lambda:          3.0973e-07 |
|      |         |        |            |                   |                 |                 |              | Learner:                svm |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  141 |       4 | Accept |    0.74634 |            7.0945 |          0.2402 |         0.24276 |       kernel | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.01094 |
|      |         |        |            |                   |                 |                 |              | Lambda:           0.0013866 |
|  142 |       4 | Accept |    0.29076 |            1.3294 |          0.2402 |         0.24334 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.006875 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.38629 |
|  143 |       4 | Accept |    0.24379 |            1.4595 |          0.2402 |         0.24269 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010315 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0079737 |
|  144 |       4 | Accept |    0.24499 |            4.9134 |          0.2402 |         0.24226 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.62692 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.033311 |
|  145 |       4 | Accept |    0.24469 |            12.208 |          0.2402 |         0.24242 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010122 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0010167 |
|  146 |       4 | Accept |    0.38588 |            1.5037 |          0.2402 |         0.24388 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0011127 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.52838 |
|  147 |       4 | Accept |    0.24589 |            1.3026 |          0.2402 |         0.24228 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       0.1424 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.58035 |
|  148 |       4 | Accept |     0.2408 |            1.2582 |          0.2402 |         0.24165 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010168 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.032668 |
|  149 |       4 | Accept |    0.24469 |            1.2764 |          0.2402 |         0.24197 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.13008 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.53455 |
|  150 |       5 | Accept |    0.24469 |            6.7111 |          0.2402 |         0.24222 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.45516 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.024796 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  151 |       5 | Accept |    0.30422 |            1.5471 |          0.2402 |         0.24244 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.36805 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         4.7124 |
|  152 |       6 | Accept |    0.24559 |            1.5701 |          0.2402 |          0.2414 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.21039 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.63504 |
|  153 |       5 | Accept |    0.24529 |            66.311 |          0.2402 |         0.24243 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.45714 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0069546 |
|  154 |       5 | Accept |    0.24589 |             1.604 |          0.2402 |         0.24243 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010712 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.043866 |
|  155 |       5 | Accept |    0.29345 |            1.4823 |          0.2402 |         0.24262 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.032053 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.88175 |
|  156 |       6 | Accept |    0.25247 |            1.4792 |          0.2402 |         0.24289 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.14103 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.91489 |
|  157 |       6 | Accept |     0.2405 |            1.5732 |          0.2402 |           0.242 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010011 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.012195 |
|  158 |       6 | Accept |     0.2426 |            1.5232 |          0.2402 |         0.24223 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.52301 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.69511 |
|  159 |       6 | Accept |    0.25456 |            1.9467 |          0.2402 |         0.24253 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.63305 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         2.1073 |
|  160 |       5 | Accept |    0.24559 |            133.37 |          0.2402 |         0.24253 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.73914 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.005832 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  161 |       5 | Accept |    0.57314 |            8.1723 |          0.2402 |         0.24253 |       kernel | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.071267 |
|      |         |        |            |                   |                 |                 |              | Lambda:          1.4009e-06 |
|  162 |       5 | Accept |    0.24649 |            1.6878 |          0.2402 |         0.24325 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.19825 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.74742 |
|  163 |       5 | Accept |     0.2417 |            1.5824 |          0.2402 |         0.24297 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010619 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.01008 |
|  164 |       4 | Accept |    0.45528 |            140.46 |          0.2402 |         0.24441 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.00706 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.030909 |
|  165 |       4 | Accept |    0.31409 |            1.5573 |          0.2402 |         0.24441 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.56217 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         7.5961 |
|  166 |       4 | Accept |    0.46575 |            2.2317 |          0.2402 |         0.24441 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.10607 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.67446 |
|  167 |       4 | Accept |    0.24529 |            3.3215 |          0.2402 |         0.24334 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       1.4843 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.069679 |
|  168 |       4 | Accept |    0.25396 |            1.3297 |          0.2402 |         0.24353 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.52671 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.9394 |
|  169 |       5 | Accept |     0.2402 |            1.5257 |          0.2402 |         0.24313 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010378 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.012749 |
|  170 |       6 | Accept |     0.3464 |            1.8583 |          0.2402 |         0.24237 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.50591 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         9.2835 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  171 |       6 | Accept |    0.24499 |            12.298 |          0.2402 |         0.24203 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0018442 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0012506 |
|  172 |       6 | Accept |    0.30272 |            1.6031 |          0.2402 |         0.24186 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       2.8265 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         11.897 |
|  173 |       6 | Accept |    0.24349 |            1.4247 |          0.2402 |         0.24134 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010459 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.036404 |
|  174 |       6 | Accept |    0.29794 |            1.6502 |          0.2402 |         0.24299 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       5.4893 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         13.359 |
|  175 |       5 | Accept |    0.24529 |            109.43 |          0.2402 |         0.24345 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.88969 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0075436 |
|  176 |       5 | Accept |    0.25007 |            1.6406 |          0.2402 |         0.24345 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010212 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.071763 |
|  177 |       5 | Accept |    0.24619 |             2.171 |          0.2402 |         0.24259 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010341 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0039146 |
|  178 |       5 | Accept |    0.32576 |            1.6841 |          0.2402 |         0.24185 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       1.6102 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         14.532 |
|  179 |       5 | Accept |     0.8262 |            890.49 |          0.2402 |         0.24179 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.18649 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0010802 |
|  180 |       5 | Accept |    0.30212 |            1.5671 |          0.2402 |         0.24174 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       6.0833 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         15.913 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  181 |       5 | Accept |    0.24499 |            14.697 |          0.2402 |         0.24192 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0078062 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0020664 |
|  182 |       5 | Accept |    0.24529 |            61.944 |          0.2402 |          0.2414 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       1.0466 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.011265 |
|  183 |       5 | Accept |    0.24499 |            17.644 |          0.2402 |         0.24186 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0021279 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0010743 |
|  184 |       5 | Accept |    0.29794 |            1.6007 |          0.2402 |         0.24189 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010132 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.17669 |
|  185 |       4 | Accept |    0.24918 |            279.94 |          0.2402 |         0.24236 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       891.39 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.076868 |
|  186 |       4 | Accept |    0.26593 |            1.4006 |          0.2402 |         0.24236 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.14381 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.2859 |
|  187 |       4 | Accept |    0.24768 |             1.438 |          0.2402 |         0.24161 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.47763 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.4081 |
|  188 |       4 | Accept |    0.24589 |            2.5142 |          0.2402 |          0.2416 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010362 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0024579 |
|  189 |       4 | Accept |     0.2405 |            1.2662 |          0.2402 |         0.24215 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       11.232 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.4768 |
|  190 |       4 | Accept |    0.24678 |            1.2604 |          0.2402 |         0.24174 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.48152 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.1771 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  191 |       4 | Accept |    0.25426 |            1.2846 |          0.2402 |         0.24186 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.47194 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.7714 |
|  192 |       5 | Accept |    0.24768 |            1.3295 |          0.2402 |         0.24188 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.44397 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.3493 |
|  193 |       6 | Accept |     0.2414 |            1.3979 |          0.2402 |         0.24154 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       1.8796 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.4211 |
|  194 |       6 | Accept |    0.24499 |            18.984 |          0.2402 |         0.24225 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0018335 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0010104 |
|  195 |       6 | Accept |    0.25037 |            1.5112 |          0.2402 |         0.24212 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.41413 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.4211 |
|  196 |       6 | Accept |    0.24499 |             6.007 |          0.2402 |         0.24191 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010001 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0013198 |
|  197 |       6 | Accept |    0.24529 |            111.16 |          0.2402 |         0.24215 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.046722 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0016388 |
|  198 |       5 | Accept |    0.24499 |            18.688 |          0.2402 |         0.24186 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0033281 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0012057 |
|  199 |       5 | Accept |     0.2426 |            1.4789 |          0.2402 |         0.24186 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       3.0066 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.3889 |
|  200 |       6 | Accept |     0.2417 |             1.499 |          0.2402 |         0.24212 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       1.7327 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.3637 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  201 |       5 | Accept |    0.26473 |            277.53 |          0.2402 |         0.24176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.19781 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0010905 |
|  202 |       5 | Accept |    0.30332 |            1.5632 |          0.2402 |         0.24176 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:     0.075228 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.8677 |
|  203 |       5 | Accept |     0.3108 |            1.5519 |          0.2402 |         0.24188 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       2.8086 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         16.078 |
|  204 |       5 | Accept |     0.2414 |            1.4413 |          0.2402 |         0.24147 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       3.1547 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.57043 |
|  205 |       5 | Accept |     0.3108 |            1.5576 |          0.2402 |         0.24195 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       3.0235 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         16.787 |
|  206 |       5 | Accept |     0.2411 |            1.3939 |          0.2402 |         0.24163 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010447 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.022855 |
|  207 |       5 | Accept |    0.29704 |            1.5568 |          0.2402 |         0.24155 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.11574 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.8573 |
|  208 |       5 | Accept |     0.2426 |            1.4114 |          0.2402 |         0.24195 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       2.2749 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.2395 |
|  209 |       5 | Accept |    0.24559 |             3.445 |          0.2402 |          0.2417 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       560.48 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.4181 |
|  210 |       4 | Accept |    0.24678 |            266.64 |          0.2402 |         0.24136 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       11.849 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0095181 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  211 |       4 | Accept |    0.24589 |            2.1278 |          0.2402 |         0.24136 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:        41.82 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.71147 |
|  212 |       3 | Accept |    0.95034 |            1042.9 |          0.2402 |         0.24153 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       10.672 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0013453 |
|  213 |       3 | Accept |    0.24649 |            2.0006 |          0.2402 |         0.24153 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       210.22 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.4654 |
|  214 |       6 | Accept |    0.26653 |            1.2521 |          0.2402 |         0.24136 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.16417 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.3626 |
|  215 |       4 | Accept |     0.2429 |            1.5969 |          0.2402 |         0.24136 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010295 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.008743 |
|  216 |       4 | Accept |    0.33353 |            1.2604 |          0.2402 |         0.24136 |           nb | DistributionNames:   kernel |
|      |         |        |            |                   |                 |                 |              | Width:            0.0017992 |
|  217 |       4 | Accept |    0.28059 |            0.2661 |          0.2402 |         0.24136 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|  218 |       4 | Accept |    0.24798 |            1.3024 |          0.2402 |         0.24173 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       66.132 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         16.829 |
|  219 |       4 | Accept |    0.24529 |             1.255 |          0.2402 |         0.24156 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       1.0858 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.3599 |
|  220 |       4 | Accept |     0.2408 |            1.2539 |          0.2402 |         0.24183 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       3.8001 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.3322 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  221 |       4 | Accept |    0.24589 |            1.8906 |          0.2402 |         0.24185 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       399.07 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         2.1791 |
|  222 |       4 | Accept |     0.2414 |            1.2821 |          0.2402 |         0.24187 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       4.8589 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.3542 |
|  223 |       4 | Accept |     0.2423 |            1.2804 |          0.2402 |         0.24131 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       2.6082 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.3565 |
|  224 |       4 | Accept |    0.24559 |            3.1158 |          0.2402 |         0.24116 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:        536.1 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         1.4679 |
|  225 |       4 | Accept |    0.25067 |            1.3228 |          0.2402 |         0.24178 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       74.127 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         18.528 |
|  226 |       3 | Accept |    0.28358 |               328 |          0.2402 |         0.24207 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:      0.47384 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.001006 |
|  227 |       3 | Accept |    0.24499 |            6.2167 |          0.2402 |         0.24207 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0026293 |
|      |         |        |            |                   |                 |                 |              | KernelScale:      0.0019141 |
|  228 |       6 | Accept |     0.2402 |            1.2501 |          0.2402 |         0.24173 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:    0.0010356 |
|      |         |        |            |                   |                 |                 |              | KernelScale:       0.012867 |
|  229 |       3 | Accept |    0.24589 |            1.4248 |          0.2402 |         0.24173 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       104.21 |
|      |         |        |            |                   |                 |                 |              | KernelScale:          17.45 |
|  230 |       3 | Accept |    0.43015 |           0.13765 |          0.2402 |         0.24173 |        discr | Delta:              0.00951 |
|      |         |        |            |                   |                 |                 |              | Gamma:              0.68613 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  231 |       3 | Accept |    0.47383 |            2.5354 |          0.2402 |         0.24173 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       1.2431 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.76632 |
|  232 |       3 | Accept |    0.67664 |           0.30377 |          0.2402 |         0.24173 |          knn | NumNeighbors:             8 |
|      |         |        |            |                   |                 |                 |              | Distance:           jaccard |
|  233 |       6 | Accept |     0.2414 |            1.2369 |          0.2402 |         0.24133 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       820.98 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         17.331 |
|  234 |       3 | Accept |    0.27759 |            1.4452 |          0.2402 |         0.24133 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       22.198 |
|      |         |        |            |                   |                 |                 |              | KernelScale:          17.97 |
|  235 |       3 | Accept |    0.28059 |           0.13623 |          0.2402 |         0.24133 |           nb | DistributionNames:   normal |
|      |         |        |            |                   |                 |                 |              | Width:                  NaN |
|  236 |       3 | Accept |    0.52408 |             1.503 |          0.2402 |         0.24133 |          knn | NumNeighbors:           351 |
|      |         |        |            |                   |                 |                 |              | Distance:       mahalanobis |
|  237 |       3 | Accept |    0.27789 |            5.8867 |          0.2402 |         0.24133 |     ensemble | Method:            RUSBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:       68 |
|      |         |        |            |                   |                 |                 |              | LearnRate:          0.18331 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              3 |
|  238 |       6 | Accept |    0.24499 |            1.2819 |          0.2402 |         0.24188 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       201.86 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         18.062 |
|  239 |       3 | Accept |    0.25097 |            1.4345 |          0.2402 |         0.24116 |          svm | Coding:            onevsone |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:        105.8 |
|      |         |        |            |                   |                 |                 |              | KernelScale:         23.315 |
|  240 |       3 | Accept |    0.30212 |           0.12275 |          0.2402 |         0.24116 |         tree | MinLeafSize:            122 |
|===========================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:       Value |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                             |
|===========================================================================================================================================|
|  241 |       3 | Accept |    0.25307 |            11.978 |          0.2402 |         0.24116 |     ensemble | Method:          AdaBoostM2 |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:      119 |
|      |         |        |            |                   |                 |                 |              | LearnRate:          0.60308 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:              1 |
|  242 |       3 | Accept |    0.44182 |            11.435 |          0.2402 |         0.24116 |          svm | Coding:            onevsall |
|      |         |        |            |                   |                 |                 |              | BoxConstraint:       6.2783 |
|      |         |        |            |                   |                 |                 |              | KernelScale:        0.19364 |

__________________________________________________________
Optimization completed.
Total iterations: 242
Total elapsed time: 1907.7403 seconds
Total time for training and validation: 4936.5339 seconds

Best observed learner is a multiclass svm model with:
	Coding (ECOC):     onevsone
	BoxConstraint:     0.033468
	KernelScale:       0.073489
Observed validation loss: 0.2402
Time for training and validation: 1.4571 seconds

Best estimated learner (returned model) is a multiclass svm model with:
	Coding (ECOC):     onevsone
	BoxConstraint:    0.0010378
	KernelScale:       0.012749
Estimated validation loss: 0.24116
Estimated time for training and validation: 1.5595 seconds

Documentation for fitcauto display

The final model returned by fitcauto corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (creditTrain), the listed Learner (or model) type, and the displayed hyperparameter values.

Evaluate Test Set Performance

The model Mdl corresponds to the best point in the Bayesian optimization according to the 'min-visited-mean' criterion. To gauge how the model will perform on new data, look at the observed cross-validation accuracy of the model (cvAccuracy) and its general estimated performance based on the Bayesian optimization (estimatedAccuracy).

[x,~,iteration] = bestPoint(Results,'Criterion','min-visited-mean');

cvError = Results.ObjectiveTrace(iteration);
cvAccuracy = 1 - cvError
cvAccuracy = 0.7598
estimatedError = predictObjective(Results,x);
estimatedAccuracy = 1 - estimatedError
estimatedAccuracy = 0.7588

Evaluate the performance of the model on the test set. Create a confusion matrix from the results, and specify the order of the classes in the confusion matrix.

testAccuracy = 1 - loss(Mdl,creditTest,'Rating')
testAccuracy = 0.7438
cm = confusionchart(creditTest.Rating,predict(Mdl,creditTest));
sortClasses(cm,{'AAA','AA','A','BBB','BB','B','CCC'})

Input Arguments

collapse all

Sample data, specified as a table. Each row of Tbl corresponds to one observation, and each column corresponds to one predictor. Optionally, Tbl can contain one additional column for the response variable. Multicolumn variables and cell arrays other than cell arrays of character vectors are not accepted.

If Tbl contains the response variable, and you want to use all remaining variables in Tbl as predictors, specify the response variable using ResponseVarName.

If Tbl contains the response variable, and you want to use only a subset of the remaining variables in Tbl as predictors, specify a formula using formula.

If Tbl does not contain the response variable, specify a response variable using Y. The length of the response variable and the number of rows in Tbl must be equal.

Data Types: table

Response variable name, specified as the name of a variable in Tbl.

You must specify ResponseVarName as a character vector or string scalar. For example, if the response variable Y is stored as Tbl.Y, then specify it as 'Y'. Otherwise, the software treats all columns of Tbl, including Y, as predictors when training the model.

The response variable must be a categorical, character, or string array, a logical or numeric vector, or a cell array of character vectors. If Y is a character array, then each element of the response variable must correspond to one row of the array.

A good practice is to specify the order of the classes by using the ClassNames name-value pair argument.

Data Types: char | string

Explanatory model of the response variable and a subset of the predictor variables, specified as a character vector or string scalar in the form 'Y~X1+X2+X3'. In this form, Y represents the response variable, and X1, X2, and X3 represent the predictor variables.

To specify a subset of variables in Tbl as predictors for training the model, use a formula. If you specify a formula, then the software does not use any variables in Tbl that do not appear in formula.

The variable names in the formula must be both variable names in Tbl (Tbl.Properties.VariableNames) and valid MATLAB® identifiers.

You can verify the variable names in Tbl by using the isvarname function. The following code returns logical 1 (true) for each variable that has a valid variable name.

cellfun(@isvarname,Tbl.Properties.VariableNames)
If the variable names in Tbl are not valid, then convert them by using the matlab.lang.makeValidName function.
Tbl.Properties.VariableNames = matlab.lang.makeValidName(Tbl.Properties.VariableNames);

Data Types: char | string

Class labels, specified as a numeric, categorical, or logical vector, a character or string array, or a cell array of character vectors.

  • If Y is a character array, then each element of the class labels must correspond to one row of the array.

  • The length of Y must be equal to the number of rows in Tbl or X.

  • A good practice is to specify the class order by using the ClassNames name-value pair argument.

Data Types: single | double | categorical | logical | char | string | cell

Predictor data, specified as a numeric matrix.

Each row of X corresponds to one observation, and each column corresponds to one predictor.

The length of Y and the number of rows in X must be equal.

To specify the names of the predictors in the order of their appearance in X, use the PredictorNames name-value pair argument.

Data Types: single | double

Note

The software treats NaN, empty character vector (''), empty string (""), <missing>, and <undefined> elements as missing data. The software removes rows of data corresponding to missing values in the response variable. However, the treatment of missing values in the predictor data X or Tbl varies among models (or learners).

Name-Value Pair Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside quotes. You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

Example: 'HyperparameterOptimizationOptions',struct('MaxObjectiveEvaluations',200,'Verbose',2) specifies to run 200 iterations of the optimization process (that is, try 200 model hyperparameter combinations), and to display information in the Command Window about the next model hyperparameter combination to be evaluated.
Optimization Options

collapse all

Types of classification models to try during the optimization, specified as the comma-separated pair consisting of 'Learners' and a value in the first table below or one or more learner names in the second table. Specify multiple learner names as a string or cell array.

ValueDescription
'auto'fitcauto automatically selects a subset of learners, suitable for the given predictor and response data. The learners can have model hyperparameter values that differ from the default. For more information, see Automatic Selection of Learners.
'all'fitcauto selects all possible learners.
'all-linear'fitcauto selects all linear learners: 'discr' (with a linear discriminant type), 'linear', and 'svm' (with a linear kernel).
'all-nonlinear'fitcauto selects all nonlinear learners: 'discr' (with a quadratic discriminant type), 'ensemble', 'kernel', 'knn', 'nb', 'svm' (with a Gaussian or polynomial kernel), and 'tree'.

Learner NameDescription
'discr'Discriminant analysis classifier
'ensemble'Ensemble classification model
'kernel'Kernel classification model
'knn'k-nearest neighbor model
'linear'Linear classification model
'nb'Naive Bayes classifier
'svm'Support vector machine classifier
'tree'Binary decision classification tree

Example: 'Learners','all'

Example: 'Learners','ensemble'

Example: 'Learners',{'svm','tree'}

Data Types: char | string | cell

Hyperparameters to optimize, specified as the comma-separated pair consisting of 'OptimizeHyperparameters' and 'auto' or 'all'. The optimizable hyperparameters depend on the model (or learner), as described in this table.

Learner NameHyperparameters for 'auto'Additional Hyperparameters for 'all'Notes
'discr'Delta, GammaDiscrimType

  • When the Learners value is 'all-linear', the fitcauto function chooses among the DiscrimType values of 'linear', 'diaglinear', and 'pseudolinear', regardless of the OptimizeHyperparameters value.

  • When the Learners value is 'all-nonlinear', the fitcauto function chooses among the DiscrimType values of 'quadratic', 'diagquadratic', and 'pseudoquadratic', regardless of the OptimizeHyperparameters value.

For more information, including hyperparameter search ranges, see OptimizeHyperparameters. Note that you cannot change hyperparameter search ranges when you use fitcauto.

'ensemble'Method, NumLearningCycles, LearnRate, MinLeafSizeMaxNumSplits, NumVariablesToSample, SplitCriterion

When the ensemble Method value is a boosting method, the ensemble NumBins value is 50.

For more information, including hyperparameter search ranges, see OptimizeHyperparameters. Note that you cannot change hyperparameter search ranges when you use fitcauto.

'kernel'KernelScale, Lambda, Coding (for three or more classes only)Learner, NumExpansionDimensionsFor more information, including hyperparameter search ranges, see OptimizeHyperparameters and OptimizeHyperparameters (for three or more classes only). Note that you cannot change hyperparameter search ranges when you use fitcauto.
'knn'Distance, NumNeighborsDistanceWeight, Exponent, StandardizeFor more information, including hyperparameter search ranges, see OptimizeHyperparameters. Note that you cannot change hyperparameter search ranges when you use fitcauto.
'linear'Lambda, Learner, Coding (for three or more classes only)RegularizationFor more information, including hyperparameter search ranges, see OptimizeHyperparameters and OptimizeHyperparameters (for three or more classes only). Note that you cannot change hyperparameter search ranges when you use fitcauto.
'nb'DistributionNames, WidthKernelFor more information, including hyperparameter search ranges, see OptimizeHyperparameters. Note that you cannot change hyperparameter search ranges when you use fitcauto.
'svm'BoxConstraint, KernelScale, Coding (for three or more classes only)KernelFunction, PolynomialOrder, Standardize

  • When the Learners value is 'all-linear', the fitcauto function does not optimize the KernelFunction or PolynomialOrder hyperparameters when the OptimizeHyperparameters value is 'all'.

  • When the Learners value is 'all-nonlinear', the fitcauto function chooses among the KernelFunction values of 'gaussian' and 'polynomial', regardless of the OptimizeHyperparameters value.

  • The Standardize value is true when the OptimizeHyperparameters value is 'auto'.

For more information, including hyperparameter search ranges, see OptimizeHyperparameters and OptimizeHyperparameters (for three or more classes only). Note that you cannot change hyperparameter search ranges when you use fitcauto.

'tree'MinLeafSizeMaxNumSplits, SplitCriterionFor more information, including hyperparameter search ranges, see OptimizeHyperparameters. Note that you cannot change hyperparameter search ranges when you use fitcauto.

Note

When 'Learners' is set to a value other than 'auto', the default values for the model hyperparameters not being optimized match the default fit function values, unless otherwise indicated in the table notes. When 'Learners' is set to 'auto', the optimized hyperparameter search ranges and nonoptimized hyperparameter values can vary, depending on the characteristics of the training data. For more information, see Automatic Selection of Learners.

Example: 'OptimizeHyperparameters','all'

Options for the optimization, specified as the comma-separated pair consisting of 'HyperparameterOptimizationOptions' and a structure. All fields in the structure are optional.

Field NameValuesDefault
MaxObjectiveEvaluationsMaximum number of iterations (objective function evaluations)30*L, where L is the number of learners (see Learners)
MaxTime

Time limit, specified as a positive real number. The time limit is in seconds, as measured by tic and toc. Run time can exceed MaxTime because MaxTime does not interrupt function evaluations.

Inf
ShowPlotsLogical value indicating whether to show plots. If true, this field plots the best observed and estimated objective function values (so far) against the iteration number.true
SaveIntermediateResultsLogical value indicating whether to save results. If true, this field overwrites a workspace variable named 'BayesoptResults' at each iteration. The variable is a BayesianOptimization object.false
Verbose

Display at the command line:

  • 0 — No iterative display

  • 1 — Iterative display

  • 2 — Iterative display with additional information about the next point to be evaluated

1
UseParallelLogical value indicating whether to run Bayesian optimization in parallel, which requires Parallel Computing Toolbox™. Due to the nonreproducibility of parallel timing, parallel Bayesian optimization does not necessarily yield reproducible results.false
Repartition

Logical value indicating whether to repartition the cross-validation at every iteration. If false, the optimizer uses a single partition for the optimization.

true usually gives the most robust results because this setting takes partitioning noise into account. However, for good results, true requires at least twice as many function evaluations.

false
Specify only one of the following three options.
CVPartitioncvpartition object, created by cvpartition'Kfold',5 if you do not specify any cross-validation field
HoldoutScalar in the range (0,1) representing the holdout fraction
KfoldInteger greater than 1

Example: 'HyperparameterOptimizationOptions',struct('UseParallel',true)

Data Types: struct

Classification Options

collapse all

Categorical predictors list, specified as the comma-separated pair consisting of 'CategoricalPredictors' and one of the values in this table.

ValueDescription
Vector of positive integersEach entry in the vector is an index value corresponding to the column of the predictor data (X or Tbl) that contains a categorical variable.
Logical vectorA true entry means that the corresponding column of predictor data (X or Tbl) is a categorical variable.
Character matrixEach row of the matrix is the name of a predictor variable. The names must match the entries in PredictorNames. Pad the names with extra blanks so each row of the character matrix has the same length.
String array or cell array of character vectorsEach element in the array is the name of a predictor variable. The names must match the entries in PredictorNames.
'all'All predictors are categorical.

By default, if the predictor data is in a table (Tbl), fitcauto assumes that a variable is categorical if it is a logical vector, categorical vector, character array, string array, or cell array of character vectors. However, learners that use decision trees assume that mathematically ordered categorical vectors are continuous variables. If the predictor data is a matrix (X), fitcauto assumes that all predictors are continuous. To identify any other predictors as categorical predictors, specify them by using the 'CategoricalPredictors' name-value pair argument.

For more information on how fitting functions treat categorical predictors, see Automatic Creation of Dummy Variables.

Note

  • fitcauto does not support categorical predictors for discriminant analysis classifiers. That is, if you want Learners to include 'discr' models, you cannot specify the 'CategoricalPredictors' name-value pair argument or use a table of sample data (Tbl) containing categorical predictors.

  • fitcauto does not support a mix of numeric and categorical predictors for k-nearest neighbor models. That is, if you want Learners to include 'knn' models, you must specify the 'CategoricalPredictors' value as 'all' or [].

Example: 'CategoricalPredictors','all'

Data Types: single | double | logical | char | string | cell

Names of classes to use for training, specified as the comma-separated pair consisting of 'ClassNames' and a categorical, character, or string array, a logical or numeric vector, or a cell array of character vectors. ClassNames must have the same data type as Y.

If ClassNames is a character array, then each element must correspond to one row of the array.

Use 'ClassNames' to:

  • Order the classes during training.

  • Specify the order of any input or output argument dimension that corresponds to the class order. For example, use 'ClassNames' to specify the order of the dimensions of Cost or the column order of classification scores returned by predict.

  • Select a subset of classes for training. For example, suppose that the set of all distinct class names in Y is {'a','b','c'}. To train the model using observations from classes 'a' and 'c' only, specify 'ClassNames',{'a','c'}.

The default value for ClassNames is the set of all distinct class names in Y.

Example: 'ClassNames',{'b','g'}

Data Types: categorical | char | string | logical | single | double | cell

Misclassification cost, specified as the comma-separated pair consisting of 'Cost' and a square matrix or structure array.

  • If you specify a square matrix Cost and the true class of an observation is i, then Cost(i,j) is the cost of classifying a point into class j. That is, rows correspond to the true classes and columns correspond to the predicted classes. To specify the class order for the corresponding rows and columns of Cost, also specify the ClassNames name-value pair argument.

  • If you specify a structure S, then it must have two fields:

    • S.ClassNames, which contains the class names as a variable of the same data type as Y

    • S.ClassificationCosts, which contains the cost matrix with rows and columns ordered as in S.ClassNames

The default value for Cost is ones(K) – eye(K), where K is the number of distinct classes.

Example: 'Cost',[0 1; 2 0]

Data Types: single | double | struct

Predictor variable names, specified as the comma-separated pair consisting of 'PredictorNames' and a string array of unique names or cell array of unique character vectors. The functionality of 'PredictorNames' depends on the way you supply the training data.

  • If you supply X and Y, then you can use 'PredictorNames' to assign names to the predictor variables in X.

    • The order of the names in PredictorNames must correspond to the column order of X. That is, PredictorNames{1} is the name of X(:,1), PredictorNames{2} is the name of X(:,2), and so on. Also, size(X,2) and numel(PredictorNames) must be equal.

    • By default, PredictorNames is {'x1','x2',...}.

  • If you supply Tbl, then you can use 'PredictorNames' to choose which predictor variables to use in training. That is, fitcauto uses only the predictor variables in PredictorNames and the response variable during training.

    • PredictorNames must be a subset of Tbl.Properties.VariableNames and cannot include the name of the response variable.

    • By default, PredictorNames contains the names of all predictor variables.

    • A good practice is to specify the predictors for training using either 'PredictorNames' or formula, but not both.

Example: 'PredictorNames',{'SepalLength','SepalWidth','PetalLength','PetalWidth'}

Data Types: string | cell

Prior probabilities for each class, specified as the comma-separated pair consisting of 'Prior' and a value in this table.

ValueDescription
'empirical'The class prior probabilities are the class relative frequencies in Y.
'uniform'All class prior probabilities are equal to 1/K, where K is the number of classes.
numeric vectorEach element is a class prior probability. Order the elements according to Mdl.ClassNames or specify the order using the ClassNames name-value pair argument. The software normalizes the elements to sum to 1.
structure

A structure S with two fields:

  • S.ClassNames contains the class names as a variable of the same type as Y.

  • S.ClassProbs contains a vector of corresponding prior probabilities. The software normalizes the elements to sum to 1.

Example: 'Prior',struct('ClassNames',{{'b','g'}},'ClassProbs',1:2)

Data Types: single | double | char | string | struct

Response variable name, specified as the comma-separated pair consisting of 'ResponseName' and a character vector or string scalar.

  • If you supply Y, then you can use 'ResponseName' to specify a name for the response variable.

  • If you supply ResponseVarName or formula, then you cannot use 'ResponseName'.

Example: 'ResponseName','response'

Data Types: char | string

Score transformation, specified as the comma-separated pair consisting of 'ScoreTransform' and a character vector, string scalar, or function handle.

This table summarizes the available character vectors and string scalars.

ValueDescription
'doublelogit'1/(1 + e–2x)
'invlogit'log(x / (1 – x))
'ismax'Sets the score for the class with the largest score to 1, and sets the scores for all other classes to 0
'logit'1/(1 + ex)
'none' or 'identity'x (no transformation)
'sign'–1 for x < 0
0 for x = 0
1 for x > 0
'symmetric'2x – 1
'symmetricismax'Sets the score for the class with the largest score to 1, and sets the scores for all other classes to –1
'symmetriclogit'2/(1 + ex) – 1

For a MATLAB function or a function you define, use its function handle for the score transform. The function handle must accept a matrix (the original scores) and return a matrix of the same size (the transformed scores).

Example: 'ScoreTransform','logit'

Data Types: char | string | function_handle

Observation weights, specified as the comma-separated pair consisting of 'Weights' and a positive numeric vector or the name of a variable in Tbl. The software weights each observation in X or Tbl with the corresponding value in Weights. The length of Weights must equal the number of rows in X or Tbl.

If you specify the input data as a table Tbl, then Weights can be the name of a variable in Tbl that contains a numeric vector. In this case, you must specify Weights as a character vector or string scalar. For example, if the weights vector W is stored as Tbl.W, then specify it as 'W'. Otherwise, the software treats all columns of Tbl, including W, as predictors or the response variable when training the model.

By default, Weights is ones(n,1), where n is the number of observations in X or Tbl.

The software normalizes Weights to sum to the value of the prior probability in the respective class.

Data Types: single | double | char | string

Output Arguments

collapse all

Trained classification model, returned as one of the classification model objects in this table.

Learner NameReturned Model Object
'discr'CompactClassificationDiscriminant
'ensemble'CompactClassificationEnsemble
'kernel'
'knn'ClassificationKNN
'linear'
'nb'CompactClassificationNaiveBayes
'svm'
'tree'CompactClassificationTree

Optimization results, returned as a BayesianOptimization object. For more information on the Bayesian optimization process, see Bayesian Optimization.

More About

collapse all

Verbose Display

When you set the Verbose field of the HyperparameterOptimizationOptions name-value pair argument to 1 or 2, the fitcauto function provides an iterative display of the optimization results.

The following table describes the columns in the display and their entries.

Column NameDescription
IterIteration number — You can set a limit to the number of iterations by using the MaxObjectiveEvaluations field of the 'HyperparameterOptimizationOptions' name-value pair argument.
Active workersNumber of active parallel workers — This column appears only when you run the optimization in parallel by setting the UseParallel field of the 'HyperparameterOptimizationOptions' name-value pair argument to true.
Eval result

One of the following evaluation results:

  • Best — The learner and hyperparameter values at this iteration give the minimum observed validation loss computed so far. That is, the Validation loss value is the smallest computed so far.

  • Accept — The learner and hyperparameter values at this iteration give meaningful (for example, non-NaN) observed and estimated validation loss values.

  • Error — The learner and hyperparameter values at this iteration result in an error (for example, a Validation loss value of NaN).

Validation lossValidation loss computed for the learner and hyperparameter values at this iteration — In particular, fitcauto computes the cross-validation classification error by default. You can change the validation scheme by using the CVPartition, Holdout, or Kfold field of the 'HyperparameterOptimizationOptions' name-value pair argument.
Time for training & validation (sec)Time taken to train and compute the validation loss for the model with the learner and hyperparameter values at this iteration (in seconds) — In particular, this value excludes the time required to update the objective function model maintained by the Bayesian optimization process. For more details, see Bayesian Optimization.
Observed min validation loss

Observed minimum validation loss computed so far — This value corresponds to the smallest Validation loss value computed so far in the optimization process.

By default, fitcauto returns a plot of the optimization that displays dark blue points for the observed minimum validation loss values. This plot does not appear when the ShowPlots field of the 'HyperparameterOptimizationOptions' name-value pair argument is set to false.

Estimated min validation loss

Estimated minimum validation loss — At each iteration, fitcauto updates an objective function model maintained by the Bayesian optimization process and uses this model to estimate the minimum validation loss. For more details, see Bayesian Optimization.

By default, fitcauto returns a plot of the optimization that displays light blue points for the estimated minimum validation loss values. This plot does not appear when the ShowPlots field of the 'HyperparameterOptimizationOptions' name-value pair argument is set to false.

LearnerModel type evaluated at this iteration — Specify the learners used in the optimization by using the 'Learners' name-value pair argument.
Hyperparameter: ValueHyperparameter values at this iteration — Specify the hyperparameters used in the optimization by using the 'OptimizeHyperparameters' name-value pair argument.

The display also includes a description of two models:

  • Best observed learner — This model, with the listed learner type and hyperparameter values, yields the final observed minimum validation loss.

  • Best estimated learner — This model, with the listed learner type and hyperparameter values, yields the final estimated minimum validation loss. fitcauto retrains the model on the entire training data set and returns it as the Mdl output.

Tips

  • Depending on the size of your data and the number of learners you specify, fitcauto can take some time to run. If you have a Parallel Computing Toolbox license, you can speed up computations by running the optimization in parallel. To do so, specify 'HyperparameterOptimizationOptions',struct('UseParallel',true). You can include other fields in the structure to control other aspects of the optimization. See HyperparameterOptimizationOptions.

Algorithms

collapse all

Automatic Selection of Learners

When you specify 'Learners','auto', the fitcauto function analyzes the predictor and response data in order to choose appropriate learners. The function considers whether the data set has any of these characteristics:

  • Categorical predictors

  • Missing values for more than 5% of the data

  • Imbalanced data, where the ratio of the number of observations in the largest class to the number of observations in the smallest class is greater than 5

  • More than 100 observations in the smallest class

  • Wide data, where the number of predictors is greater than or equal to the number of observations

  • High-dimensional data, where the number of predictors is greater than 100

  • Large data, where the number of observations is greater than 50,000

  • Binary response variable

  • Ordinal response variable

The selected learners are always a subset of those listed in the Learners table. However, the associated models tried during the optimization process can have different default values for hyperparameters not being optimized, as well as different search ranges for hyperparameters being optimized.

For example, when you specify Learners as 'auto', the fitcauto function tries SVM learners with the given KernelFunction value, based on the data set characteristics in this table.

Categorical PredictorsMissing ValuesWide DataSVM KernelFunction Value
YesNoEither yes or no'polynomial'
NoNoYes'linear'
NoNoNo'gaussian'

Bayesian Optimization

The goal of Bayesian optimization, and optimization in general, is to find a point that minimizes an objective function. In the context of fitcauto, a point is a learner type together with a set of hyperparameter values for the learner (see Learners and OptimizeHyperparameters), and the objective function is the cross-validation classification error, by default. The Bayesian optimization implemented in fitcauto internally maintains a multi-TreeBagger model of the objective function. That is, the objective function model splits along the learner type and, for a given learner, the model is a TreeBagger ensemble for regression. (This underlying model differs from the Gaussian process model employed by other Statistics and Machine Learning Toolbox™ functions that use Bayesian optimization.) Bayesian optimization trains the underlying model by using objective function evaluations, and determines the next point to evaluate by using an acquisition function ('expected-improvement'). For more information, see Expected Improvement. The acquisition function balances between sampling at points with low modeled objective function values and exploring areas that are not well modeled yet. At the end of the optimization, fitcauto chooses the point with the minimum objective function model value, among the points evaluated during the optimization. For more information, see the 'Criterion','min-visited-mean' name-value pair argument of bestPoint.

Alternative Functionality

  • If you are unsure which models work best for your data set, you can alternatively use the Classification Learner app. Using the app, you can perform hyperparameter tuning for different models, and choose the optimized model that performs best. Although you must select a specific model before you can tune the model hyperparameters, Classification Learner provides greater flexibility for selecting optimizable hyperparameters and setting hyperparameter values. However, you cannot optimize in parallel, choose 'linear' or 'kernel' learners, specify observation weights, or specify prior probabilities in the app. For more information, see Hyperparameter Optimization in Classification Learner App.

  • If you know which models might suit your data, you can alternatively use the corresponding model fit functions and specify the 'OptimizeHyperparameters' name-value pair argument to tune hyperparameters. You can compare the results across the models to select the best classifier. For an example of this process, see Moving Towards Automating Model Selection Using Bayesian Optimization.

Extended Capabilities

Introduced in R2020a