How to remove transient effect in the beginning of the filtered signal?

Hi all,

I have to filter a signal with a low pass filter but the filtered data has strong spikes at its beginning and its end as you can see in the following plots.

http://s27.postimg.org/adpqf54lv/filtered.jpg

The signal magnitude is around 15 for the whole time span, but it is not true for the filtered signal. Here my code:

Fs=1/data(2,1);
Fnorm=15/(Fs/2)
df = designfilt('lowpassfir','FilterOrder',70,'CutoffFrequency',Fnorm);
y=filter(df,data(:,3));

Why does this transient effect come from? How may I solve it?

Thank you

Best regards

Pietro

Answer 1

[Keqin Xu]

Here is my trick of dealing with the edge effect, it works perfectly for me.

Prefixing AND Appending the original signal with a small number (10-100) of the beginning AND ending sections of the signal, respectively. The sections are flipped and shifted in order to maintain continuity in signal level and slope at the joining points. After filtering, the prefixed AND the appended portions of the filtered signal are removed. Below is the process for a column signal x:

R=0.1; % 10% of signal

Nr=50;

N=size(x,1);

NR=min(round(N*R),Nr); % At most 50 points

For i=1:size(x,2)

    x1(:,i)=2*x(1,i)-flipud(x(2:NR+1,i));  % maintain continuity in level and slope

    x2(:,i)=2*x(end,i)-flipud(x(end-NR:end-1,i));

end

x=[x1;x;x2];

% Do filtering

x=myfilter(x,...);

x=x(NR+1:end-NR,:)

[Jim Riggs]

Nice idea.

Answer 2

[Image Analyst]

It looks like an edge effect, which happens when the filter "runs past" the end of your signal and assumes the signal is zero there. It's totally normal and expected. You can just crop off the last few elements, usually half the filter window width.

[pietro]

Thanks for your reply. May I solve the problem by setting the filter initial conditions? I tried to use filtic but I don't understand how to convert the filter designed as the following to the transposed direct form II.

df = designfilt('lowpassfir','FilterOrder',70,'CutoffFrequency',Fnorm);

[Image Analyst]

I'm not familiar with that function. Some functions have options for how to treat boundary/edge effects where the filter and the signal are not completely overlapped. For example with conv() and conv2() you can specify 'valid' to have it return only the fully overlapped part. I don't know about filtic() and designfilt() - I haven't used them or heard of them.

[pietro]

Hi ImageAnalyst,

would you please provide me the name of these function with options for how to treat the edge effects? How may I use conv() and conv2() for filtering?

Thank you

[Image Analyst]

You can use smooth(), lowess(), loess(), or rloess() (or names similar to those. Requires the toolboxes that they're in. They handle edge effects well.

[Zain]

How to get "half the filter window width"?

[Image Analyst]

You have to look at the function do determine where/when it stops scanning the filter window. Let's say you have a filter window width of 7, with the left element being 1, the middle element being 4, and the right element being 7. So when a filtering function has the right edge of the filter window hit the right edge of the data, all the valid data that can be filtered has been filtered. Let's say the array is 100 wide. So when element 7 is overlapped with element 100, it's at the edge. Now if we move one more to the right, element 7 is "off" the array and element 6 is overlapped with element 100. So, what do we do with element 7? One common way is to just assume the array is zero out there. There are other ways to handle "edge effects" but generally most people have such a large signal and a small filter window, and the "interesting" stuff is way in the middle so you can basically just ignore what happens out there near the edge.

Now, what happens if element 4 (where we put the output element) is also off the right side of the image? You can continue to use the filter with elements 1-3 being applied to elements 98,99, & 100, but the result would go to element 101 of the output image (actually a little farther because of edge effects on the left hand side of the image, but stick with me here). So the last possible place where there is any overlap at all is when element 1 is over element 100. In that case, element 4 (the middle and the output element location) is at element 103. That's 3 beyond the right hand side of the array. So the output array is 3 wider than the input array on the right, and another 3 wider on the left, so it's 106 wide instead of 100 wide. So you can crop out the middle, that is, if the filter does not have a 'same' option for it like conv() does which will do it for you.

halfWidth = floor(windowWidth/2);
ouputArray = outputArray(halfWidth + 1 : end - halfWidth); % Crop via indexing

Or for an input array of 100, a window width of 7, a half width of 3, and an output array of 106, you will extract elements 4 to 103 of the output array to crop it to the same size as the input array.

[Image Analyst]

I don't know - the poster took down the original image http://s27.postimg.org/adpqf54lv/filtered.jpg so I don't know what we're dealing with anymore.