Remove non-finite values while decimating

I was able to decimate 8 other identical (or so I thought) files, but this one hit an error. Code and error below:

%% Set path
cd 'C:\folder'

%% Load data
data = readtable('data.csv');

%% Decimate and then convert to table format
data50_X = decimate(data{:,4},2);

Error using filtfilt
Expected input to be finite.

Error in filtfilt>efiltfilt (line 123)
    validateattributes(x,{'double','single'},{'finite','nonempty'},'filtfilt');

Error in filtfilt (line 102)
        y = efiltfilt(b,a,x);

Error in decimate (line 157)
    odata = filtfilt(b,a,idata);

Answer 1

[Image Analyst]

Try this:

data = readtable('data.csv');
goodRows = isfinite(data{:, 4});
data = data(goodRows, :); % Extract only the good/finite values rows.

[Taylor Azizeh]

Thank you! This fixed it, but what if I needed to keep the finite values from columns 4, 5, and 6??

[Image Analyst]

If you want to throw out any rows in which a nan or infinite value occurs in either column 4, 5, or 6, you could try

data = readtable('data.csv');
goodRows = all(isfinite(data{:, 4:6}));
data = data(goodRows, :); % Extract only the good/finite values rows.

[Taylor Azizeh]

This did something funny to the "data" table.

[Image Analyst]

@Taylor Azizeh, if you need more help, please attach your csv file and your code, or:

[Taylor Azizeh]

My CSV file is huge, so I can't attach it, but I will include a snippet below. But my code is as follows. Those last rows of code are when it seems to mess up the data variable.

%% First, set pathway
cd 'C:\path'

%% Load in your data (this may take a while, depdending on the size of the file)
data = readtable('data.csv');
beep

%% Decimate and then convert to table format
goodRows = all(isfinite(data{:, 4:6}));
data = data(goodRows, :); % Extract only the good/finite values rows

Data sample of what I'm trying to decimate (within the larger file, these are columns 4:6

data_XYZ = 124826901×3

-0.1250 -0.0630 1.0000

-0.1250 -0.0630 0.9390

-0.1230 -0.0630 1.0000

-0.0660 -0.0630 0.9970

-0.1250 -0.0630 0.9430

-0.1190 -0.0630 1.0000

-0.0700 -0.0630 0.9930

-0.1160 -0.0630 0.9470

-0.0630 -0.0630 1.0000

-0.0630 -0.0630 1.0000

[Image Analyst]

Just edit it so that it's less than 5 MB and attach it. I'll check for it tomorrow.

[Taylor Azizeh]

Here you go! It's only 1000 rows. But you can see here that despite the finite values in XYZ, the code only keeps three rows...

%% Load in your data (this may take  a while, depdending on the size of the file)
data = readtable('small_data.csv');
beep

%% Decimate and then convert to table format
goodRows = all(isfinite(data{:, 4:6}));
data = data(goodRows, :) % Extract only the good/finite values rows

data = 3×18 table
             TagID                Date            Time          X         Y       Z       Activity       Depth    Temp___C_    location_lat    location_lon    height_msl    ground_speed    satellites    hdop    signal_strength    SensorRaw    Metadata
    _______________________    ___________    ____________    ______    ______    _    ______________    _____    _________    ____________    ____________    __________    ____________    __________    ____    _______________    _________    ________

    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.000    -0.063    -0.063    1    {'Active/Dry'}       0       18.6           NaN             NaN            NaN            NaN            NaN        NaN           NaN             997         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.010    -0.063    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.020    -0.063    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   

[Image Analyst]

Sorry, you need another argument for all to make it apply across columns. Try this:

%% Load in your data (this may take  a while, depdending on the size of the file)
data = readtable('small_data.csv');
%% Decimate and then convert to table format
goodRows = all(isfinite(data{:, 4:6}), 2);
data = data(goodRows, :) % Extract only the good/finite values rows

data = 1000×18 table
             TagID                Date            Time          X         Y       Z       Activity       Depth    Temp___C_    location_lat    location_lon    height_msl    ground_speed    satellites    hdop    signal_strength    SensorRaw    Metadata
    _______________________    ___________    ____________    ______    ______    _    ______________    _____    _________    ____________    ____________    __________    ____________    __________    ____    _______________    _________    ________

    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.000    -0.063    -0.063    1    {'Active/Dry'}       0       18.6           NaN             NaN            NaN            NaN            NaN        NaN           NaN             997         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.010    -0.063    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.020    -0.063    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.030    -0.063    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.040    -0.003    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.050    -0.059    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.060    -0.004    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.070         0    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.080    -0.058    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.090    -0.063    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.100    -0.063    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.110    -0.063    -0.007    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.120    -0.008    -0.055    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.130    -0.054    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.140    -0.063    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   
    {'EP19_PEN19_EP9_B_S1'}    30-Oct-2019    19:49:07.150    -0.063    -0.063    1    {0×0 char    }     NaN        NaN           NaN             NaN            NaN            NaN            NaN        NaN           NaN             NaN         NaN   

[Taylor Azizeh]

Fixed!! Thank you so much for your help, you're a lifesaver.

Answer 2

[Matt J]

Likely, there are NaNs in your data that you have to remove.

find(~isfinite(data{:,4}))

[Taylor Azizeh]

That creates a new variable (407 x 1) with an output like this:

81501
1028902
1032803
1075404
1080505
1081706
1081807
1084408
1713709
1761010
1762411
2260910
2260912

But after that, my decimation code still doesn't run.

[Matt J]

Presumably you haven't removed any of the bad values from your data.

[Taylor Azizeh]

Can you explain how I would remove the bad values after running the

find(~isfinite(data{:,4}))

code? Thanks for your patience! I am still new to MATLAB.

[Matt J]

find(~isfinite(data{:,4})) gives the locations of the bad data. You can go there and see what those values are, and either remove them or replace them with something else.

[Taylor Azizeh]

Ohhhh gotcha. Thank you so much!