Filtering Columns of Array by Number of Row Contents

I have arrays that look like this:

A =

     1   NaN   NaN
     2     3     4
     2     5   NaN

I want to remove columns that contain rows with less than n non-NaN entries. In this case, with n=2, the first column of A would be removed, since the first row contains only 1 non-NaN value. Is there a compact way to do this? I hope this explanation makes sense!

Answer 1

[Ameer Hamza]

A = [1   NaN   NaN
     2     3     4
     2     5   NaN];
n = 2;
A(sum(~isnan(A), 2) < n, :) = [];

Result

A =
     2     3     4
     2     5   NaN

[Justin Delano]

Thank you so much! I knew I was close!

[Ameer Hamza]

Glad to be of help.

[Justin Delano]

I'm sorry, I just realized this wasn't the result I'm looking for. I'm looking for the result:

A = [NaN NaN
       3   4
       5 NaN]

Is there a solution for this problem?

[Ameer Hamza]

Are you looking for something like this

A = [1   NaN   NaN
     2     3     4
     2     5   NaN];
n = 2;
A(:, sum(~isnan(A), 2) < n) = [];

[Justin Delano]

Hello, I can give another example.

Input = 
    1     2   NaN
    10   NaN     5
     1     3     5
   NaN   NaN     6

Output = 
    1     2   
    10   NaN  
     1     3  
   NaN   NaN  

Column 3 was removed due to it having a value in row 4, which contains more than n=2 NaNs.

[Ameer Hamza]

try this:

A = [1     2   NaN
    10   NaN     5
     1     3     5
   NaN   NaN     6];

idx = sum(isnan(A),2) >= 2;
idx2 = find(~isnan(A(idx, :)));
A(:,idx2) = [];

[Justin Delano]

Thank you, that seems closer. I didn't expect this to be such a tough problem. If need be I'll do this with loops instead of vectorized functions. That attempt works on that test case, but throws an error on this one:

A =

     1   NaN   NaN
    10   NaN     5
     1     3     5
   NaN   NaN     6

[Ameer Hamza]

I realized I overlooked a things in my last code. Does following code work

A = [1   NaN   NaN
    10   NaN     5
     1     3     5
   NaN   NaN     6];

idx = sum(isnan(A),2) >= 2;
[~, idx2] = find(~isnan(A(idx, :)));
A(:,idx2) = [];

[Justin Delano]

Thank you, that is better. Unfortunately, for my data, there are a lot more NaNs. I have attached two example matrices. When I use your code, the first matrix is turned completely blank, which I don't think should be the case.

[Ameer Hamza]

I am still not entirely sure what is the correct rule to delete columns. Can you write a simple for loop version for a small matrix. I will try to see if it can be vactorized.

[Justin Delano]

Here is a rough loop version:

%scan each column of full matrix
for column = matrix_columns
    %scan each row of each column
    for entry = column_rows
        %assume entry is from row = entry_row in full matrix
        if ~isnan(entry)
            if sum(~isnan(entry_row)) < 10
                %delete this column from full matrix
                continue
            end
        end
    end
end

[Ameer Hamza]

Justin, there some to be some issue in this loop version too. For example If I apply it to your previous example, it still does not give the result you want

x =  [1     2   NaN
    10   NaN     5
     1     3     5
   NaN   NaN     6];

col_delete = [];
%scan each column of full matrix
for col = 1:size(x,2)
    %scan each row of each column
    for row = 1:size(x,1)
        entry = x(row, col);
        %assume entry is from row = entry_row in full matrix
        if ~isnan(entry)
            entry_row = x(row, :);
            if sum(~isnan(entry_row)) < 10
                %delete this column from full matrix
                col_delete = [col_delete col];
            end
        end
    end
end

x(:,unique(col_delete)) = [];

Final value of x should be

     1     2   
    10   NaN  
     1     3  
   NaN   NaN

But your loop version make it an empty matrix.

[Justin Delano]

I apologize, you are correct. I set the sum in the final if statement to be less than 10, but in reality it should be less than n, which we have set to 2.

[Ameer Hamza]

Ok. Now it is clear. Please try

x = fate1nums; % variable from file you shared
n = 10;
[r, c] = find(~isnan(x));
x(:, c(sum(~isnan(x(r,:)), 2) < n)) = [];

[Justin Delano]

That looks good! Thank you so much!

[Ameer Hamza]

Glad to be of help.