Need help with Exponential Curve fitting

I'm not a experienced user with the curve fitting toolbox, I'm trying to fit the data I have and I'm a bit confused.

1. How is the Curve fitting application's Custom Equation operate differently from the exponential functions? The app gives a fitted result when I select 'Exponential' from the drop-down, but the solution is completely different when I switch to Custom equation and also doesn't plot anything? Why is this so?

2. I tried fitting the same data with an additional term and gave Initial estimates, but the solution flattens? I don't get it.

expfit = fittype('a -b*exp(-c*x)');     
f0 = fit(xdata,ydata,expfit,'StartPoint',[[ones(size(xdata)), -exp(-xdata)]\ydata; 1]);  %Guess with c=1
fitresult = f0(xdata);

My understanding of what I'm seeing:

1.Because my data is discrete, I also tried smooth function on it using sgolay and rlowess, which don't help.

2. From the results of the curve fit exponential tool, from my limited understanding, it looks like the gradient descent is stuck on a local solution and I need to find a global solution using Multistart. While I can do that, I don't think my problem is that complicated to begin with.

3. Because I extracted this data from a much larger dataset using filtering, the start and end of the data can be erraneous and I would like to have a ignore outlier data towards both the extremities of the data length.

Can someone help me on this? The x and ydata are attached.

Answer 1

[John D'Errico]

First, plot EVERYTHING. Then if you don't understand what you see, plot something more, or plot it in a different way.

load xdata
load ydata
plot(xdata,ydata,'.')

So, first, we see this is NOT a simple exponential. That overshoots makes it not so. What happens when we fit it with the default exponential?

fittedmdl = fit(xdata,ydata,'exp1')

Warning: NaN, Inf, or complex value detected in startpoint;
choosing random starting point instead.

fittedmdl = 
     General model Exp1:
     fittedmdl(x) = a*exp(b*x)
     Coefficients (with 95% confidence bounds):
       a =  -7.851e-12  (-1.96e-10, 1.803e-10)
       b =      0.0439  (-0.004165, 0.09197)

It fails, of course, because the random starting values given it are complete crap for this data.

As well, LOOK CAREFULLY at the form it is using. Do you see that you are trying to fit a THREE parameter model, this with constant offset in y, when you use the simple exponential fit? The exponential fit used by the CFTB is a TWO parameter model. So no matter what, this will fit like complete crap.

Next, I would strongly suggest that to make the numerical analysis better, you use a subtly different model.

mdl2 = fittype('a + b*exp(-c*(x - 440)/10)','indep','x');

Subtracting 440, and then dividing by 10 will make those numbers much smaller numbers. And that means the constants c and b will be closer to 1 when we fit them. And in turn, your model will be easier to estimate. Or, you can just hope and pray for a result that is not complete crap. It is your choice in the end.

fittedmdl2 = fit(xdata,ydata,mdl2,'start',[-.1 1 1])

fittedmdl2 = 
     General model:
     fittedmdl2(x) = a + b*exp(-c*(x - 440)/10)
     Coefficients (with 95% confidence bounds):
       a =    -0.02745  (-0.02941, -0.0255)
       b =      0.2038  (0.1953, 0.2122)
       c =       5.678  (5.497, 5.859)

plot(fittedmdl2)
hold on
plot(xdata,ydata,'b.')
ylim([-.1,1])
hold off

As you can see, such an exponential model will NEVER have any overshoot, as you see in your data. It simply is not capable of such behavior. However, in context of a poorly chosen model to fit that data, the parameters are not unreasonable.

Could we choose a better model? Well, possibly. But you need to provide VERY good starting values for such a fit to work. Better would simply bue to use a spline, here a smoothing spline might be a good idea.

fittedmdl3 = fit(xdata,ydata,'smoothingspline')

fittedmdl3 = 
     Smoothing spline:
       fittedmdl3(x) = piecewise polynomial computed from p
     Coefficients:
       p = coefficient structure

plot(fittedmdl3)
hold on
plot(xdata,ydata,'b.')
ylim([-.1,1])
grid on
hold off

Of course, a spline does not give you a model you can write down, looking very pretty. But it fits the data well. Your choice.

[Lalith Chandra]

Thanks John for taking the time to write up the long answer to make absolutely clear to me.

While I agree that choosing a spline form would better fit my data, it is of no use of to me, because essentially what I am trying by curvefitting is trying to estimate the underlying physics that gave this data as output. But this is helpful because now I know that the data is flawed.

If you look carefully, I had to further clean up the data and only take the data from the min(ydata). Doing that will fill my data (trimmed) to an exponential one, which is useful to me.

Now, I still have three questions:

1. The number '440' that you have used needs to change with the changing data. I don't see any Name value arguments in fittype to pass constants to the function. So how do I replace it by a variable that changes according to the data.
2. If I try and compare the expressions 'a + b*exp(-c*(x - d)/10)' and 'A + B*exp(-C*x), wouldn't they be the same, just that B = b*exp(44C) and C = c/10. So I should only be seeing a change in the coefficient values, but a successfully fitted curve, why is it no so? If I change the form to the the latter, why is the fitted result curve diffferent from the former one?
3. Could you comment on the first point in my question? Why would the custom equation give me a different result with the same form in the Exponential form selected from the dropdown? Do I need to make any changes to the default fit options for the Custom Equation for it to match?

Thanks

[Alex Sha]

How about the fitting function below, seems good enough:

y = p4*(1/(1+exp(p1*x+p7)))+p5*(1/(1+exp(p2*x+p8)))+p6*(1/(1+exp(p3*x+p9)))+p10

Sum Squared Error (SSE): 0.00936807773656548
Root of Mean Square Error (RMSE): 0.00358723272859852
Correlation Coef. (R): 0.999548796486555
R-Square: 0.999097796557721

Parameter	Best Estimate     	Std. Deviation    	Confidence Bounds[95%]
---------	-------------     	--------------    	--------------------------------
p1       	-1.00004115169767 	0.720830838313611 	[-2.41522920959232, 0.415146906196975]
p2       	-0.990934271030763	0.517094455003215 	[-2.00613209363795, 0.024263551576423]
p3       	-6.09360530222925 	0.853389148326652 	[-7.76904157278921, -4.4181690316693]
p4       	15.034090396609   	2101.62198242023  	[-4111.02458196559, 4141.0927627588]
p5       	-15.9062905526017 	2101.63036694188  	[-4141.98142402366, 4110.16884291846]
p6       	-0.21376224827022 	0.0353809509276216	[-0.283224730406622, -0.144299766133819]
p7       	440.618077870117  	324.146263714736  	[-195.769680585751, 1077.00583632598]
p8       	436.509997230971  	234.271300255541  	[-23.4286302627251, 896.448624724666]
p9       	2664.2455497613   	373.404450995785  	[1931.15049926343, 3397.34060025918]
p10      	1.06149065773623  	0.0320317517310523	[0.99860356951275, 1.12437774595971]

[Image Analyst]

@Lalith Chandra you say you're trying to use that exponential fit because you're trying to "estimate the underlying physics". So whatever process you're measuring, you say the theory says it should follow an exponential decay. However, in reality your process doesn't follow your expected theoretical formula. So why doesn't it follow theory?

1. Is there some artifact or source of measurement error in your experiment that causes the data to not follow the theory?
2. Or, conversely, is your experiment correct, but your assumption that the data should follow that exponential equation is wrong? Perhaps the theory says that it follows a damped sine wave or something.

What is this process actually? And what is the reference for the theoretical formula that you think this process should obey? Do you have some paper that says that the formula should be an exponential?