How to compare one row of four matrices with all other rows of these matrices?

In sum, I have four matrices, with a column and thousands of lines. It should now be checked whether the current row of all matrices is larger than the remaining rows.

This means that the first row of each matrix is compared to the remaining rows. Then the second line with the remaining rows etc. ...

If all values in the current row are larger than in the remaining rows, then this row should be removed.

In the end, only those lines should be contained in the four matrices, which at least contain one smaller value.

Furthermore, the indices of the non-rejected rows should be added to the output.

I have tried the following approach, but which does not work, since he does not compare the individual lines over all four matrices.

for n = 1:length(NSE2014);

        inrange = bsxfun(@lt,NSE2014(n),NSE2014(n+1)) & bsxfun(@gt,RMSE2014(n),RMSE2014(n+1)) & bsxfun(@gt,BIAS2014(n),BIAS2014(n+1)) & bsxfun(@gt,MAD2014(n),MAD2014(n+1));

% The same has to be done, for 2015 and 2016!

end

Does anyone have an idea how to solve this problem (with Matlab R2012a)?

[Andrei Bobrov]

Please attach small example of your matrices and expected result

[Glazio]

@Andrei Bobrov:

The following is a short excerpt from the example matrices.

NSE2014 =

    0.2733
    0.2930
    0.2930
    0.2930

RMSE2014 =

    0.6939
    0.6797
    0.6797
    0.6797

MAD2014 =

    0.4495
    0.4583
    0.4583
    0.4583

BIAS2014 =

    0.0826
    0.0124
    0.0124
    0.0124

In this test case, these matrices have only four lines, otherwise several thousands.

_ _ _ _

Expected Result: Each row of each matrix represents a run (This means the first line of NSE, RMSE, BIAS and MAD represents a run, the second line of ... and so on). It should now be checked whether the current row (current run) of all matrices is larger than the remaining rows (remaining runs).

In the end, only the pareto-optimal runs should be left. All lines that match the selection criterion (all values in the current row are larger than in the remaining rows) must be removed from the matrices.

Either write the remaining rows into new matrices NSE2014a, ... or into a single matrix with four columns.

As an addition, the index of the remaining rows / runs should be output (based on the original matrices).

Hope it is more understandable, where the problem lies.

Thanks

Answer 1

[Andrei Bobrov]

NSE2014 =[
    0.2733
    0.2930
    0.2930
    0.2930];

RMSE2014 =[
    0.6939
    0.6797
    0.6797
    0.6797];

MAD2014 =[
    0.4495
    0.4583
    0.4583
    0.4583];

BIAS2014 =[
    0.0826
    0.0124
    0.0124
    0.0124];

M = [NSE2014,RMSE2014,MAD2014,BIAS2014];
out = M(sum(all(M > permute(M,[3 2 1]),2),3) < size(M,1) - 1,:);% R2016b and later
out = M(sum(all(bsxfun(@gt,M,permute(M,[3 2 1])),2),3) < size(M,1) - 1,:);% R2016a and earlier

[Glazio]

@Andrei Bobrov:

Thanks for your answer, it seems as if the right results are delivered.

How can the indices be queried in this context (based on the original matrix)?

Can you explain the procedure in your code example, since it is not completely comprehensible for me?

Is it also possible, for example, to search for smaller solutions for the NSE and to filter the remaining three matrices for larger values?

[Andrei Bobrov]

t = sum(all(M > permute(M,[3 2 1]),2),3) < size(M,1) - 1;
index = find(t);
out = M(t,:);

explanation:

a = M > permute(M,[3 2 1]); %compare each element of row with corresponding elements of other rows
b = all(a,2); % find the rows that fulfill the conditions 
c = sum(b,3); % number of rows satisfying the conditions for each row
t = c < size(M,1) - 1; % result (logical array)
indices = find(t); % indices of rows
out = M(t,:); % result

[Glazio]

@Andrei Bobrov:

The first code line t= ... gives the following error message:

Error using > Number of array dimensions must match for binary array op.

Can you explain the procedure in your code example, since it is not completely comprehensible for me (e.g. why is permute necessary)?

Is it also possible, for example, to combine different case selections (e.g. NSE smaller than,... and RMSE, BIAS, MAD greater than)?

Thanks

[Glazio]

@Andrei Bobrov:

Thanks, for your explanation, now the context is clear to me. The code line a = M > permute(M,[]); also gives the error message:

Error using > Number of array dimensions must match for binary array op.

Where could the error lie (use MATLAB R2012a)? Found the error, was the wrong version, which applies only to newer Matlab versions.

[Glazio]

How can you exclude rows if there is a NaN in this row? Ie. Remove all lines that contain both a NaN and/or do not meet the criterion (as defined above).

I solved it with (even if there are more efficient solutions):

M2015(any(isnan(M2015),2),:)=[];

[Andrei Bobrov]

if M have NaN:

t0 = ~any(isnan(M),2);
M1 = M(t0,:);
t = sum(all(bsxfun(@gt,M1, permute(M1,[3 2 1])),2),3) < size(M1,1) - 1;
index = nonzeros(find(t0).*t);
out = M1(t,:);

[Glazio]

@Andrei Bobrov:

Thanks for your explanation and answer.

If I run this with a matrix of 1000 lines, no line is filtered out. The final matrix (out) has the same dimension and values.

There are too many lines in the final result (too many points in the Pareto Space). It seems to me as if not all criteria are correctly checked during the query. For example, the second line in this test case would have to be discarded, but remains in the matrix.

The algorithm (multi-objective optimization) should be based on a comparison of each run with all other runs (each row with all other rows) -> Ie. The preceding lines must also be compared. If the current run is larger in all objective functions (that means in all columns of a row) than another run it has to be removed from the set.

What could be the reason for that problem?

Furthermore, the provided indices do not allow conclusions to be drawn back to the original positions.

Example:

I have an example matrix like:

Matrix =

       1     1     1     2
       2     3     4     5
       0     0     0     0
       1     1     1     1
       6     7     8     9

If I use the following algorithm:

a = bsxfun(@gt,Matrix,permute(Matrix,[3 2 1])); %compare each element of row with corresponding elements of other rows
b = all(a,2); % find the rows that fulfill the conditions 
c = sum(b,3); % number of rows satisfying the conditions for each row
t = c < size(Matrix,1) - 1; % result (logical array)
indices = find(t); % indices of rows
outindex = Matrix(t,:); % result

The resulting matrix (outindex) is:

1  1  1  2
2  3  4  5
0  0  0  0
1  1  1  1

The algorithm removes only the last row, but not the second row which is in all elements larger than the first row. The third row could be kept because all values are smaller than in the previous row. The same applies to row 4, ...

[Andrei Bobrov]

Matrix =[1     1     1     2
       2     3     4     5
       0     0     0     0
       1     1     1     1
       6     7     8     9]
a = bsxfun(@gt,Matrix,permute(Matrix,[3 2 1]));
idx = squeeze(sum(all(a,2))) == size(Matrix,1) - 1;
out = Matrix(idx,:);
idx_out = find(~idx);

[Glazio]

@Andrei Bobrov:

Thanks for your answer and help. The above algorithm removes all entries, except the second column with 0 0 0 0.

I've tested it with a matrix that also contains NaN and it has been returned an empty matrix. What could be this problem?

If I try the above algorithm for the following Matrix:

Matrix =

       1     2     3     4
       2     4     5     6
       1     0     1     2
       0     0     0     3
       0     2     4     0

I get an empty idx_out.

However, the following lines must be returned:

- the third row

- the fourth row

- the fifth row

because this three rows are pareto efficient. The third row is in all elements smaller than the previous row. The fourth row is in the first element and third element smaller than the third row and could be kept. The fifth row is in the fourth element smaller than all the other rows. In this case only the first and second row have to be removed,..

For this example matrix:

Matrix =

    158.1000  150.0000  158.1000
    111.8000  100.0000  111.8000
    141.4000  111.8000  100.0000
     70.7000   50.0000   70.7000
    111.8000   70.7000   50.0000

I get also an empty idx_out. In fact, the fourth and fifth lines should remain in idx_out.

If the current run is larger in all objective functions (that means in all columns of a row) than another run it has to be removed from the set.

[Andrei Bobrov]

a = bsxfun(@gt,Matrix,permute(Matrix,[3 2 1]));
t = any(squeeze(all(a,2)),2);
out = Matrix(~t,:);
idx = find(t);

[Glazio]

@Andrei Bobrov: It seems to work on smaller matrices and to provide the desired results. Now I will apply your answer to the larger data sets.

Thank you for your help and patience.