Classify and Count Fruits Using Arduino Nano 33 BLE Sense, ThingSpeak, and Machine Learning
This example shows how to collect RGB values of three types of fruits using the APDS9960 sensor embedded in an Arduino® Nano 33 BLE board and use the MATLAB® Classification Learner app to train a machine learning model with the RGB values to classify the fruits. This example also shows how to use the ThingSpeak™ platform for tracking the number of fruits in a box and setting up an email alert if the number of any one of the three types of fruits drops to zero.
This example createsa classification and counting sytem for three types of fruits, namely apple, guava, and banana.
To access all the files for this example, click the Open Live Script button and download the files.
This example uses an Arduino Nano BLE 33 Sense board that you can connect to your machine using a micro USB cable. The algorithm in the example completes these steps.
Capture the RGB values using the Arduino Nano BLE 33 Sense board and create a data set.
Use the Classification Learner app to determine the classification model and export the trained model.
Create a ThingsSpeak channel to track the number of fruits in a box.
Set up an email alert if the number of any one of the three types of fruits drops to zero.
You can use the trained model included with this example instead of training the model on your own.
If you are using the trained model, skip the Create Data Set from RGB Values of Fruits and Export Classification Algorithm section.
Create Data Set from RGB Values of Fruits
Place the Arduino Nano 33 BLE Sense board near the fruit for which you want to capture the RBG values. This image shows a person capturing the RBG data for an apple.
captureRGB.m file by clicking the Open Live Script button. Execute the file and complete the instructions that display in the Command Window. Once you complete executing the file, you can view the
trainingData variable of size
2001X3 in the MATLAB workspace. Add a fourth field,
fruit, to the
trainingData table. Change all the entries in the column under the new field to
trainingData.fruit(:) = "Apple" ;
Create a training data variable for apples.
trainingDataApple = trainingData ;
trainingDataApple variable of size
2001x4 is now visible in the MATLAB workspace.
To capture the RGB values for bananas, execute the
captureRBG.m file again. Create a new field called
Banana and a training data variable for bananas.
trainingData.fruit(:) = "Banana" ; trainingDataBanana = trainingData ;
To capture the RGB values for guavas, execute the
captureRBG.m file again. Create a new field called
Guava and a training data variable for guavas.
trainingData.fruit(:) = "Guava" ; trainingDataGuava = trainingData ;
trainingDataGuava variables of size
2001x4 are now visible in the MATLAB workspace.
Store the training data for the three fruits in a single variable
finalTrainingData variable must be of the size
finalTrainingData = [trainingDataApple; trainingDataBanana; trainingDataGuava];
Export Classification Algorithm
Use the Classification Learner app to select a classification model. In the Apps tab on the MATLAB toolstrip, search for the Classification Learner app.
In the app, click the New Session button, and then click From Workspace.
In the Data set pane, select the training variable
finalTrainingData and click Start Session.
On the app toolstrip, in the Models section, click All, and in the Train section, click Train All.
The app provides you with a list of classification models and their results. For this example, select the model with highest accuracy.
On the app toolstrip, click Export and then click Export Model.
Select Include the training data in the exported model and export the model.
Create ThingSpeak Channel and Add Widgets to Count Fruits
If you are using the trained model included with this example, skip the Create Data Set from RGB Values of Fruits and Export Classification Algorithm sections and start with this section.
To download the trained model included with this example, click the Open Live Script button. To load the trained model, type this command at the MATLAB command line.
ThingSpeak is an IoT analytics platform service that you can use to aggregate, visualize, and analyze live data streams in the cloud. For more information, see Get Started with ThingSpeak.
To track the number of fruits in the box using ThingSpeak™, sign in using your MathWorks® Account credentials, or create a new account. Create a new channel in ThingSpeak to collect and analyze the fruit count data. To create a new channel in ThingSpeak, on the Channels tab, click
My Channels. In the
My Channels window, click New Channel and enter these values to customize the channel.
ThingSpeak uses the Field 4 field to track if the number of any one of the three types of fruits drops to zero.
Click Save Channel to save the channel settings. In the Channels window, click Add Widgets to add three numeric display widgets to track the number of fruits.
Identify API Keys in ThingSpeak
Before you write the MATLAB code to track the number of fruits, identify the API keys of the ThingSpeak channel. These keys allow you to read data from and write data to your channel.
To identify the API keys, on the Channels tab, click API Keys.
Execute MATLAB Code to Classify and Count Fruits
To track the number of fruits in the box, whenever you take a fruit from the box, place the fruit near the sensor. The sensor then classifies the fruit and updates the fruit count in the ThingSpeak channel.
You can execute the following MATLAB code to read RGB data from the sensor and use this data to predict the type of fruit based on a pretrained model and publish the data to ThingSpeak using the channel ID and the Write API key of the channel. Run these commands, one after another, in the MATLAB Command Window. This example assumes that the box contains 50 guavas, 40 bananas. This example also assumes that 30 apples and the Arduino board is connected to the COM14 port.
% Create the arduino object with required library arduinoObj = arduino("COM14","Nano33BLE","Libraries","APDS9960"); % Create an APDS9960 object apds9960Obj = apds9960(arduinoObj, "Bus", 1, "BitRate", 100000); % Variable to track the number of apples, bananas, and guavas guavaCount = 50; bananaCount = 40; appleCount = 30; %the variable to track if the fruit count of one of the fruits drops to zero isEmpty = 0; %isEmpty variable changes to 1 if the fruit count of one of the fruits drops to zero if(guavaCount == 0 || bananaCount == 0 || appleCount == 0) isEmpty = 1; end % ThingSpeak configurations channelID = xxxxxxx; writeAPI = 'xxxxxxxxxxxxxxxx'; %Update this to the writeAPI key identified your Thingspeak Channels page % Write the value to ThingSpeak channel thingSpeakWrite(channelID,[appleCount,guavaCount,bananaCount,isEmpty],'WriteKey',writeAPI,'Fields',[1,2,3,4]); while(1) % Continuously read the proximity readings proximReadings = readProximity(apds9960Obj); flag = 0; while(proximReadings>0.1) if(flag == 0) % Alert to bring the object near the sensor disp("Bring object near sensor"); end proximReadings = readProximity(apds9960Obj); flag = 1; end % Turn on the built-in LED when the object is near writeDigitalPin(arduinoObj, 'D13', 1); % Read the RGB value from apds9960 colorData = readColor(apds9960Obj); % Convert colorData to table format colorDataInTableFormat = array2table(colorData); % Call predict function [predictedFruit,scores] = trainedModel.predictFcn(colorDataInTableFormat); message = ["The object detected is ",predictedFruit]; disp(message); % Update the fruit count according to the prediction if(predictedFruit=="Apple") appleCount = appleCount-1; elseif(predictedFruit=="Banana") bananaCount = bananaCount-1; elseif(predictedFruit=="Guava") guavaCount = guavaCount-1; end while(proximReadings <0.1) proximReadings = readProximity(apds9960Obj); end if(guavaCount == 0 || bananaCount == 0 || appleCount == 0) isEmpty = 1; end % Turn off the LED writeDigitalPin(arduinoObj, 'D13', 0); % Write data to ThingSpeak channel thingSpeakWrite(channelID,[appleCount,guavaCount,bananaCount,isEmpty],'WriteKey',writeAPI,'Fields',[1,2,3,4]); end
This GIF shows the channel dashboard when you remove one fruit from the box.
Set Up Email Alert
Configure ThingSpeak to alert you via email if the number of any one of the three types of fruits drops to zero. ThingSpeak sends the email to the email address associated with your ThingSpeak account.
To set up the email alert, on the Apps tab on the ThingSpeak toolstrip, click
Click New to select a code template.
Custom (no starter code) option and then click Create to create the template.
Set the Name to
RefillAlert and enter the code in the MATLAB Code window or copy the sample code from the
RefillAlert.m file provided with this example. To access the
RefillAlert.m file, click the Open Live Script button and download the file.
Specify the channel ID, Read API key, and Alerts API key corresponding to your channel in code. You can access the alerts key from your ThingSpeak profile page.
To set up an email alert for when the number of any one of the three types of fruits drops to zero, on the Apps tab in ThingSpeak, click Actions, and then click React. In the React window, set these parameters.
React Name — Enter a unique name for your alert. For this example, set it to
Fruit Empty Trigger.
Condition Type — Select
Test Frequency — Select
On Data Insertion.
Condition — Select the name of your channel and the field and set the condition for your alert. In this example, select field
is equal to 1.
Action — Select MATLAB Analysis. In the code to execute list, select the name of the MATLAB file where you stored the analysis code. For this example, select Refill
Options — Select
Run action only the first time the condition is met.
Click Save React to save your alert.
When the number of any one of the three types of fruits drops to zero, ThingSpeak sends an email to the registered email address. This is an image of a sample email sent by ThingSpeak.