How to split matrix data into training, validation and testing data based on a categorical label?
5 views (last 30 days)
Show older comments
Kimberly Cardillo
on 20 Aug 2020
Answered: Srivardhan Gadila
on 23 Aug 2020
I have a 184 x 1 cell array called aefm_4 and each cell is 1 x 1599. Each row (184) has a corresponding categorical label in a separate variable called f and its either labeled 'crack-related' or 'noise'. I want to split up my data so that I have 70% training data, 10% validation data and 20% testing data. I want about even amounts of noise and crack related signals in each training, validation and testing dataset. How would I do that?
Right now, I have been manually separating them as seen below but I will be running this for more data and it is too time consuming to do this.
dataTrain = aefm_4([1:9,14:23,28:47,55:75,87:90,93:101,108:152,169:177],:); %127 hits (70%)
fTrain = f([1:9,14:23,28:47,55:75,87:90,93:101,108:152,169:177],:);
dataVal = aefm_4([12:13,26:27,53:54,83:86,92,105:107,163:168],:); %20 hits (10%)
fVal = f([12:13,26:27,53:54,83:86,92,105:107,163:168],:);
dataTest = aefm_4([10:11,24:25,48:52,76:82,91,102:104,151:162,178:184],:); %39 hits (20%)
fTest = aefm_4([10:11,24:25,48:52,76:82,91,102:104,151:162,178:184],:);
0 Comments
Accepted Answer
Srivardhan Gadila
on 23 Aug 2020
You can do something like below:
%% Create categorical label data
numLabels = 18
labels = randi([1 2],numLabels,1);
labelsCat = categorical(labels,[1 2],{'crack-related','noise'});
%% find lndices w.r.t each label
crackInd = find(labelsCat==categorical("crack-related"));
noiseInd = find(labelsCat==categorical("noise"));
%% divide into train, val, test for each label
[trainCrack,valCrack,testCrack] = dividerand(numel(crackInd),0.7,0.1,0.2);
[trainNoise,valNoise,testNoise] = dividerand(numel(noiseInd),0.7,0.1,0.2);
%% get final indices
trainInd = [crackInd(trainCrack); noiseInd(trainNoise)];
valInd = [crackInd(valCrack); noiseInd(valNoise)];
testInd = [crackInd(testCrack); noiseInd(testNoise)];
You can optimize the above code based on your requirement like, dividing the indices randomly or not. Refer to Divide Data for Optimal Neural Network Training, dividerand, divideblock, divideint, divideind for more information.
trainInd = trainInd(randperm(numel(trainInd)));
0 Comments
More Answers (0)
See Also
Categories
Find more on Statistics and Machine Learning Toolbox in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!