Chapter 1: Exploring Data
- Create Data That Needs Preprocessing
- Preprocess Missing Values
- Clean Missing Data with a Live Editor Task
- Smooth Data
Chapter 1
Exploring Data
This ebook demonstrates the tasks involved to preprocess data used for machine learning algorithms in MATLAB®.
If you have worked with machine learning, you know that you must preprocess the data. You also know that this can be a tedious process when handled manually. You need to be able to make updates to preprocessing scripts within a framework that allows you to quickly evaluate the impact of changes on the accuracy of the machine learning model.
If you think back to early math classes, there were clear rules on the order of operations (PEMDAS), remembered with mnemonics like “Please Excuse My Dear Aunt Sally.” Whether your type of problem concerns apples or mortgage rates, you know that 2 * (6+4)2 = 200.
The order of operations for data preprocessing tasks is not so straightforward. There are few hard and fast rules when it comes to what order you should do tasks, but every problem has its own factors that may affect what comes first.
Before data can be preprocessed, you need to know which preprocessing tasks are needed. Querying, visualizing, and otherwise exploring your data will provide insight into where you should focus your effort and lead to an iterative workflow in which exploration informs preprocessing. Repeat this process until exploration no longer yields issues that need to be addressed.
![Workflow diagram](https://se.mathworks.com/campaigns/offers/next/preprocessing-data-for-machine-learning/_jcr_content/mainParsys/band_2102914970/mainParsys/lockedsubnav/mainParsys/image_copy.adapt.full.medium.jpg/1721861887241.jpg)
To keep track of the iterations through this workflow, it is helpful to keep track of preprocessing operations in a script. This also makes it easier to adjust the order of your preprocessing steps, as the resulting preprocessed data may vary depending on the order in which preprocessing steps were applied.
Let’s look at an example MATLAB script for preprocessing data.
Create Data That Needs Preprocessing
% Create a sine wave with noise % and remove some of the values % so they're missing. rng default; t = 0:0.01:5; y = sin(t) + randn(size(t)); missingIdx = randi([1 length(y)],100,1); y(missingIdx) = NaN; % Plot the data. You can see gaps % in the plot that represent the % missing data. figure; plot(t,y)
![Graph data with gaps](https://se.mathworks.com/campaigns/offers/next/preprocessing-data-for-machine-learning/_jcr_content/mainParsys/band_2102914970/mainParsys/lockedsubnav/mainParsys/columns_copy/19d24ab2-4b2f-4c9b-a69d-d0024c8b0470/image.adapt.full.medium.jpg/1721861887327.jpg)
Preprocess Missing Values
% Use the fillmissing function % with linear interpolation to % fill the gaps in the data. y_filled = fillmissing(y, "linear"); % Add the filled data to the % plot. You can see that the % gaps are now filled. hold on; plot(t, y_filled, ':r')
![Graph data with filled gaps](https://se.mathworks.com/campaigns/offers/next/preprocessing-data-for-machine-learning/_jcr_content/mainParsys/band_2102914970/mainParsys/lockedsubnav/mainParsys/columns_copy_copy/19d24ab2-4b2f-4c9b-a69d-d0024c8b0470/image.adapt.full.medium.png/1721861887408.png)
Clean Missing Data with a Live Editor Task
Alternatively, you could use a Live Editor task to perform this preprocessing. Live Editor tasks can be found in the MATLAB toolbar under the Live Editor > Task drop-down. The Clean Missing Data Live Editor task shows a plot of the cleaned data and missing values that were filled. It also presents the various methods that can be used to clean the data.
Smooth Data
In this script, the missing data was cleaned before the smoothing was applied. But this might not always be the best choice. Because all of the steps are captured in the script, it would be easy to go back and move sections around to see what effect changing the preprocessing order has on the results.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)
Asia Pacific
- Australia (English)
- India (English)
- New Zealand (English)
- 中国
- 日本Japanese (日本語)
- 한국Korean (한국어)