How to find a point in a dataset at which graph is changing (aka whole graph derivative)

Pretty trivial problem from someone not strong in math:) I hava a dataset with intensity values over time. At a certain point intensity starts increasing. I want to find the time point at which intensity starts increasing. Due to the nature of the data there is some fluctuation in the baseline, therefore graph is not a smooth line. I tried finding the first derivative:

data = [1 2 1 3 1 2 4 7 8 15 6 4 3 1 2 1];
  time = [1:16]
  dydx = diff(data)./diff(time)
  end

but this returns the derivative for every single datapoint. How do I find the tangent point for the whole graph, and not just for a single point?

I tried approaching it from another angle by doing the following:

S = std(data(1:5)
logic = data > S

I take the first few points as the baseline and calculate the standard deviation of the baseline. And later I create a logical array where 1 is any value 1 SD above the baseline. This is pretty accurate, the instance where '0 1 0 0 1' changes into a continuous '1 1 1 1 1' indicates the point that I need, but I don't know how to extract an index of this changing point.

Attached is an image of a sample graph.

[jonas]

You should check out findchangepts (signal processing toolbox) and ischange. If you upload the data I can give you a working example.

[Madina Makhmutova]

Hello Jonas,

Thank you for your suggestion.

1)ischange is returning a logical array where almost every single value is '1', indicating that it considers a change almost at every single point. This is probably because my graph is nosy and has fluctuations at every point. So this option is not sensitive enough.

2)I tried 'findchangepts' and this one seems to work, it finds a single change point. My dataset is a matrix with intensity in columns over time in rows. So I expected to get an output as a vector, where it would give me a single change point for each column, however I'm getting a single point for the whole matrix. So I'm not sure how to interpret it.

I have attached the sample file to the question.

[jonas]

findchangepts has loads of input arguments whereas ischange is a bit more limited. However, both functions comes with the optional threshold value to limit the "sensitivity to noise".

[Image Analyst]

Is there some number of points at the beginning where you are assured that they are the initial, flat, non-increasing values that we can use as a threshold? If so I'd smooth the signal with sgolayfilt() and then just threshold at the proper value.

Answer 1

[jonas]

findchangepts has a number of optional inputs . Try this for example:

% change in std
findchangepts(data(:,3),'statistic','std') 

% constant segments
findchangepts(data(:,3),'statistic','mean','minthreshold',2e4) 

% linear segments
findchangepts(data(:,3),'statistic','linear','minthreshold',2e4)

To get the index, you simply specify the output arguments

[idx] = findchangepts(data(:,3),'statistic','linear','minthreshold',2e4)

Ideally, the first value in idx is your point of interest. You have to play around with the input arguments to find the best method for this. I would go for one of the above. Perhaps its better to remove minthreshold and use maxnumchanges instead.