Trying to create ROC curves for n-class classification problem

Hi,

I have trained two classification models, a Naive Bayes model, and a Decision Tree, now I am looking to create ROC curves for each model but I get a "Positive class is not found in the input data." Error.

I have 11 factors and 6 classes, here is my code for the Naive Bayes ROC curves

%%
%ROC Curves for our final models
% 
% hypernormoutnb >Final NB model
% optctree >final DT model
% NBxtestx = partitioned X values for testing
% NBxtesty = Partitioned Y values for testing
[predictions,score1,cost1] = predict(hypernormoutNB,NBxtestx)

%Then you pass in scores from predictions on test set and generate metrics for each class. Each line will identify each class as the positive label.

[fpr0,tpr0,T0,AUC0,OPTROCPT0] = perfcurve(NBxtesty,score1(:,1:1),0);
[fpr1,tpr1,T1,AUC1,OPTROCPT1] = perfcurve(NBxtesty,score1(:,2:2),1);
[fpr2,tpr2,T2,AUC2,OPTROCPT2] = perfcurve(NBxtesty,score1(:,3:3),2);
[fpr3,tpr3,T3,AUC3,OPTROCPT3] = perfcurve(NBxtesty,score1(:,4:4),3);
[fpr4,tpr4,T4,AUC4,OPTROCPT4] = perfcurve(NBxtesty,score1(:,5:5),4);
[fpr5,tpr5,T5,AUC5,OPTROCPT5] = perfcurve(NBxtesty,score1(:,6:6),5);
figure
plot(fpr0,tpr0)
title('Naive Bayes ROC Curves')
xlabel('False Positive') 
ylabel('True Positive') 
hold on
plot(fpr1,tpr1)
plot(fpr2,tpr2)
plot(fpr3,tpr3)
plot(fpr4,tpr4)
plot(fpr5,tpr5)
plot(OPTROCPT0(1),OPTROCPT0(2),'ro')
plot(OPTROCPT1(1),OPTROCPT1(2),'ro')
plot(OPTROCPT2(1),OPTROCPT2(2),'ro')
plot(OPTROCPT3(1),OPTROCPT3(2),'ro')
plot(OPTROCPT4(1),OPTROCPT4(2),'ro')
plot(OPTROCPT5(1),OPTROCPT5(2),'ro')
hold off

I get the following error message:

Error using perfcurve>membership (line 705)

Positive class is not found in the input data.

Error in perfcurve (line 449)

[W,subYnames] = membership(cls(sorted),weights(sorted),...

Error in Final_Project (line 642)

[fpr0,tpr0,T0,AUC0,OPTROCPT0] = perfcurve(NBxtesty,score1(:,1:1),0);

Can anyone offer some insight into why I haven't defined a positive class?

thanks,

Jeremy

Answer 1

[Pranav Verma]

Hi Jeremy,

You have defined the positive class as scalar value in perfcurve (the last parameter in perfcurve). Please check if the parameters you have defined are the same as expected by the perfcurve function.

Documentation for perfcurve.

labels - numeric vector | logical vector | character matrix | string array | cell array of character vectors | categorical array

scores - vector of floating points

posclass - numeric scalar | logical scalar | character vector | string scalar | cell containing a character vector | categorical scalar

Also, please go through the the below question on similar lines:

https://www.mathworks.com/matlabcentral/answers/163600-roc-curve-with-matlab-using-svmtrain

https://www.mathworks.com/matlabcentral/answers/500114-i-had-the-following-error-message-positive-class-is-not-found-in-the-input-data

Thanks

[Jeremy Brecevic]

Ah so my positive value should be my class name? I have 6 classes which are all integers (3,4,5,6,7,8)

so it should be:

[fpr0,tpr0,T0,AUC0,OPTROCPT0] = perfcurve(NBxtesty,score1(:,1:1),3);

[fpr1,tpr1,T1,AUC1,OPTROCPT1] = perfcurve(NBxtesty,score1(:,2:2),4);

[fpr2,tpr2,T2,AUC2,OPTROCPT2] = perfcurve(NBxtesty,score1(:,3:3),5);

[fpr3,tpr3,T3,AUC3,OPTROCPT3] = perfcurve(NBxtesty,score1(:,4:4),6);

[fpr4,tpr4,T4,AUC4,OPTROCPT4] = perfcurve(NBxtesty,score1(:,5:5),7);

[fpr5,tpr5,T5,AUC5,OPTROCPT5] = perfcurve(NBxtesty,score1(:,6:6),8);

Or am I misunderstanding?