Main Content

ClassificationPartitionedModel

Cross-validated classification model

Description

ClassificationPartitionedModel is a set of classification models trained on cross-validated folds. Estimate the quality of classification by cross validation using one or more “kfold” methods: kfoldPredict, kfoldLoss, kfoldMargin, kfoldEdge, and kfoldfun.

Every “kfold” method uses models trained on in-fold observations to predict the response for out-of-fold observations. For example, suppose you cross validate using five folds. In this case, the software randomly assigns each observation into five roughly equally sized groups. The training fold contains four of the groups (i.e., roughly 4/5 of the data) and the test fold contains the other group (i.e., roughly 1/5 of the data). In this case, cross validation proceeds as follows:

  • The software trains the first model (stored in CVMdl.Trained{1}) using the observations in the last four groups and reserves the observations in the first group for validation.

  • The software trains the second model (stored in CVMdl.Trained{2}) using the observations in the first group and last three groups, and reserves the observations in the second group for validation.

  • The software proceeds in a similar fashion for the third to fifth models.

If you validate by calling kfoldPredict, it computes predictions for the observations in group 1 using the first model, group 2 for the second model, and so on. In short, the software estimates a response for every observation using the model trained without that observation.

Creation

Description

CVMdl = crossval(Mdl) creates a cross-validated classification model from a classification model (Mdl).

example

Alternatively:

  • CVDiscrMdl = fitcdiscr(X,Y,Name,Value)

  • CVKNNMdl = fitcknn(X,Y,Name,Value)

  • CVNetMdl = fitcnet(X,Y,Name,Value)

  • CVNBMdl = fitcnb(X,Y,Name,Value)

  • CVSVMMdl = fitcsvm(X,Y,Name,Value)

  • CVTreeMdl = fitctree(X,Y,Name,Value)

create a cross-validated model when Name is either 'CrossVal', 'KFold', 'Holdout', 'Leaveout', or 'CVPartition'. For syntax details, see fitcdiscr, fitcknn, fitcnet, fitcnb, fitcsvm, and fitctree.

Input Arguments

expand all

A classification model, specified as one of the following:

  • A classification tree trained using fitctree

  • A discriminant analysis classifier trained using fitcdiscr

  • A neural network classifier trained using fitcnet

  • A naive Bayes classifier trained using fitcnb

  • A nearest neighbor classifier trained using fitcknn

  • A support vector machine classifier trained using fitcsvm

Properties

expand all

This property is read-only.

Bin edges for numeric predictors, specified as a cell array of p numeric vectors, where p is the number of predictors. Each vector includes the bin edges for a numeric predictor. The element in the cell array for a categorical predictor is empty because the software does not bin categorical predictors.

The software bins numeric predictors only if you specify the 'NumBins' name-value argument as a positive integer scalar when training a model with tree learners. The BinEdges property is empty if the 'NumBins' value is empty (default).

You can reproduce the binned predictor data Xbinned by using the BinEdges property of the trained model mdl.

X = mdl.X; % Predictor data
Xbinned = zeros(size(X));
edges = mdl.BinEdges;
% Find indices of binned predictors.
idxNumeric = find(~cellfun(@isempty,edges));
if iscolumn(idxNumeric)
    idxNumeric = idxNumeric';
end
for j = idxNumeric 
    x = X(:,j);
    % Convert x to array if x is a table.
    if istable(x) 
        x = table2array(x);
    end
    % Group x into bins by using the discretize function.
    xbinned = discretize(x,[-inf; edges{j}; inf]); 
    Xbinned(:,j) = xbinned;
end
Xbinned contains the bin indices, ranging from 1 to the number of bins, for numeric predictors. Xbinned values are 0 for categorical predictors. If X contains NaNs, then the corresponding Xbinned values are NaNs.

Categorical predictor indices, specified as a vector of positive integers. CategoricalPredictors contains index values indicating that the corresponding predictors are categorical. The index values are between 1 and p, where p is the number of predictors used to train the model. If none of the predictors are categorical, then this property is empty ([]).

If Mdl is a trained discriminant analysis classifier, then CategoricalPredictors is always empty ([]).

Data Types: single | double

This property is read-only.

Unique class labels used in training, specified as a categorical or character array, logical or numeric vector, or cell array of character vectors. ClassNames has the same data type as the class labels Y. (The software treats string arrays as cell arrays of character vectors.) ClassNames also determines the class order.

Data Types: categorical | char | logical | single | double | cell

Square matrix, where Cost(i,j) is the cost of classifying a point into class j if its true class is i (i.e., the rows correspond to the true class and the columns correspond to the predicted class). The order of the rows and columns of Cost corresponds to the order of the classes in ClassNames. The number of rows and columns in Cost is the number of unique classes in the response.

If CVModel is a cross-validated ClassificationDiscriminant, ClassificationKNN, ClassificationNaiveBayes, or ClassificationNeuralNetwork model, then you can change its cost matrix to e.g., CostMatrix, using dot notation.

CVModel.Cost = CostMatrix;

Data Types: double

This property is read-only.

Name of the cross-validated model, returned as a character vector.

Data Types: char

Number of folds in the cross-validated model, returned as a positive integer.

Data Types: double

Parameters of the cross-validated model, returned as an object.

This property is read-only.

Number of observations in the training data, returned as a positive integer. NumObservations can be less than the number of rows of input data when there are missing values in the input data or response data.

Data Types: double

This property is read-only.

Partition used in cross-validation, returned as a CVPartition object.

This property is read-only.

Predictor names in order of their appearance in the predictor data X, specified as a cell array of character vectors. The length of PredictorNames is equal to the number of columns in X.

Data Types: cell

Prior probabilities for each class, returned as a numeric vector. The order of the elements of Prior corresponds to the order of the classes in ClassNames.

If CVModel is a cross-validated ClassificationDiscriminant or ClassificationNaiveBayes model, then you can change its vector of priors using dot notation. For example, if priorVector is a vector whose length is the number of classes,

CVModel.Prior = priorVector;

Data Types: double

Response variable name, specified as a character vector.

Data Types: char

Score transformation, specified as a character vector or function handle. ScoreTransform represents a built-in transformation function or a function handle for transforming predicted classification scores.

To change the score transformation function to function, for example, use dot notation.

  • For a built-in function, enter a character vector.

    Mdl.ScoreTransform = 'function';

    This table describes the available built-in functions.

    ValueDescription
    'doublelogit'1/(1 + e–2x)
    'invlogit'log(x / (1 – x))
    'ismax'Sets the score for the class with the largest score to 1, and sets the scores for all other classes to 0
    'logit'1/(1 + ex)
    'none' or 'identity'x (no transformation)
    'sign'–1 for x < 0
    0 for x = 0
    1 for x > 0
    'symmetric'2x – 1
    'symmetricismax'Sets the score for the class with the largest score to 1, and sets the scores for all other classes to –1
    'symmetriclogit'2/(1 + ex) – 1

  • For a MATLAB® function or a function that you define, enter its function handle.

    Mdl.ScoreTransform = @function;

    function must accept a matrix (the original scores) and return a matrix of the same size (the transformed scores).

Data Types: char | string | function_handle

The trained learners, returned as a cell array of compact classification models trained on cross-validation folds.

This property is read-only.

Scaled weights in the model, returned as a numeric vector. W has length n, the number of rows in the training data.

Data Types: double

This property is read-only.

Predictor values, returned as a real matrix or table. Each column of X represents one variable (predictor), and each row represents one observation.

Data Types: double | table

This property is read-only.

Class labels corresponding to the observations in X, returned as a categorical array, cell array of character vectors, character array, logical vector, or a numeric vector. Each row of Y represents the classification of the corresponding row of X.

Data Types: single | double | logical | char | string | cell | categorical

Object Functions

gatherGather properties of Statistics and Machine Learning Toolbox object from GPU
kfoldEdgeClassification edge for cross-validated classification model
kfoldLossClassification loss for cross-validated classification model
kfoldMarginClassification margins for cross-validated classification model
kfoldPredictClassify observations in cross-validated classification model
kfoldfunCross-validate function for classification

Examples

collapse all

Evaluate the k-fold cross-validation error for a classification tree model.

Load Fisher's iris data set.

load fisheriris

Train a classification tree using default options.

Mdl = fitctree(meas,species);

Cross validate the classification tree model.

CVMdl = crossval(Mdl);

Estimate the 10-fold cross-validation loss.

L = kfoldLoss(CVMdl)
L = 
0.0533

Estimate positive class posterior probabilities for the test set of an SVM algorithm.

Load the ionosphere data set.

load ionosphere

Train an SVM classifier. Specify a 20% holdout sample. It is good practice to standardize the predictors and specify the class order.

rng(1) % For reproducibility
CVSVMModel = fitcsvm(X,Y,'Holdout',0.2,'Standardize',true,...
    'ClassNames',{'b','g'});

CVSVMModel is a trained ClassificationPartitionedModel cross-validated classifier.

Estimate the optimal score function for mapping observation scores to posterior probabilities of an observation being classified as 'g'.

ScoreCVSVMModel = fitSVMPosterior(CVSVMModel);

ScoreSVMModel is a trained ClassificationPartitionedModel cross-validated classifier containing the optimal score transformation function estimated from the training data.

Estimate the out-of-sample positive class posterior probabilities. Display the results for the first 10 out-of-sample observations.

[~,OOSPostProbs] = kfoldPredict(ScoreCVSVMModel);
indx = ~isnan(OOSPostProbs(:,2));
hoObs = find(indx); % Holdout observation numbers
OOSPostProbs = [hoObs, OOSPostProbs(indx,2)];
table(OOSPostProbs(1:10,1),OOSPostProbs(1:10,2),...
    'VariableNames',{'ObservationIndex','PosteriorProbability'})
ans=10×2 table
    ObservationIndex    PosteriorProbability
    ________________    ____________________

            6                   0.17375     
            7                   0.89638     
            8                 0.0076573     
            9                   0.91602     
           16                  0.026709     
           22                4.6069e-06     
           23                   0.90241     
           24                2.4119e-06     
           38                0.00042666     
           41                   0.86429     

Tips

To estimate posterior probabilities of trained, cross-validated SVM classifiers, use fitSVMPosterior.

Extended Capabilities

Version History

Introduced in R2011a

expand all

Go to top of page