How to calculate standar deviation and trend in each row of data

11 views (last 30 days)
Retno Purwaningsih
Retno Purwaningsih on 20 Oct 2021
Edited: dpb on 23 Oct 2021
I want to ask about my script,
I had a file with different number of column in each row, then calculate the mean, standar deviation and trend of data in each row from column 4 to end. The mean value is work but the std and trend still facing some error.
This is my script
clear;
clc;
format longG;
f = dir('D:\full_Data_Cryosat_c2p0004.txt');
A=1:length(f);
D = f(A).name;
Insufficient number of outputs from right hand side of equal sign to satisfy assignment.
ff=D(:,19:25)
data=readtable(D,'HeaderLines',0);
rata2=mean(data{:,4:end},2,'omitnan');
standar_deviation=std(data{:,4:end},2,'omitnan');
ud=trend(data{:,4:end},2,'omitnan');
ref=data{:,1};
lat=data{:,2};
lon=data{:,3};
fixed=[ref lat lon rata2 std ud]
file_name_data=['data_msl_' ff '.txt']
dlmwrite(file_name_data, fixed, 'delimiter', '\t');
Error using var (line 179)
W must be a vector of nonnegative weights, or a scalar 0 or 1.
Error in std (line 51)
y = sqrt(var(varargin{:}));
Error in script_msl_v2 (line 13)
standar_deviation=std(data{:,4:end},2,'omitnan');
Thanks for helping

Answers (1)

dpb
dpb on 20 Oct 2021
Edited: dpb on 20 Oct 2021
The input arguments for std include a flag for the weighting for the divisor and/or the option to compute a weighted standard deviation. Consequetnly, they don't mimic those of mean identically. This makes for a very error-prone user interface, but it was designed in the earliest incarnations of MATLAB and remains that way now for compatibility reasons.
You're far from the first (and certainly won't be the last) to have fallen into the trap, however, ... :)
A section of the help shows the parts of specific interest here and what tripped you up -- the two-argument forms are interpreted as either the normalization flag if it is a single value or the weight vector if an array, but you passed three arguments -- and in a form that doesn't match up to the three-argument form.
>> help std
std Standard deviation.
...
std normalizes Y by N-1 if N>1, where N is the sample size. This is
the sqrt of an unbiased estimator of the variance of the population
from which X is drawn, as long as X consists of independent,
identically distributed samples. For N=1, Y is normalized by N.
Y = std(X,1) normalizes by N and produces the square root of the second
moment of the sample about its mean. std(X,0) is the same as std(X).
Y = std(X,W) computes the standard deviation using the weight vector W.
W typically contains either counts or inverse variances. ...
...
Y = std(X,W,DIM) takes the standard deviation along the dimension DIM
of X.
Hence, you need
standar_deviation=std(data{:,4:end},[],2,'omitnan');
to compute the unbiased estimator over the rows dimension. The [] passes the empty argument placeholder for the weighting vector.
  3 Comments
dpb
dpb on 23 Oct 2021
trend isn't a base MATLAB function; there is one in the OPC Toolbox
but that doesn't seem appropriate here.
I suspect you were intending to use detrend instead to remove linear trend from the data?
If so, that would be
ud=detrend(data{:,4:end}.','omitnan').';
where you must explicitly transpose the input data array (and I transposed the result back to keep same orientation) because detrend does NOT support the dimension of the input array over which to operate as an optional input. In this way, it doesn't follow the canonical form, either.
MORAL: You must always consult the actual documentation for each function; particularly many of these long-standing ones that were developed very early in the history of MATLAB because they were written before the use of named parameters was well developed and the very first incarnations followed typical FORTRAN coding practices and these very early routines have not been modified since other than to graft on a few more recent niceties like the 'omitnan' flag on detrend that is quite recent in terms of the overall age of the function's initial incarnation.
If, OTOH, you were intending to compute the linear trend of each row in the data, then look at polyfit, but it and none of the other fitting toolsets if you have Curve Fitting or Statistics TB are set up for doing multiple fits on arrays; you have to iterate over the array by row or column.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!