averaging/smoothing data

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

averaging/smoothing data

Przemek Klosowski-7

[sorry for repost: maybe my original post gave an impression that I am not asking seriously, but I think it's an important and frequent use case, so here it is again]


Often, I need to smooth/average a data vector. I usually do a running average using filter(), e.g. for 5-value average:

    filter(ones(1:5)/5,eye(5,1),x)

I've been doing this for years, so I got used to it even though I have to think a little every time I need to do again, but maybe there's a better way? What do y'all use?



By the way, sometimes I'd use a Savitzky-Golay peak/moment-preserving filter coefficients instead of a simple rectangular averaging window:


# Copyright (C) 1993 Przemek Klosowski ; licensed under GPL
#
# You should have received a copy of the GNU General Public License
# along with Octave; see the file COPYING.  If not, write to the Free
# Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.

function retval = savgol (m,nl,nr)

# usage: savgol (m,n) or (m,nl,nr) 
# m - order -nl..nr - smoothing window
#
# Return the Savitzky-Golay filter coeffs for 2n window, order m. 
# Ref.: Num.Rec. (2nd edition) p. 650
# Try plot([savgol(2,8);savgol(4,8);savgol(6,8)]'); sometime

# filter coeffs are in first row of (A'*A)^-1 * A'

  if (nargin == 2)
    nr = nl; nl = nl;
  elseif (nargin!=3)
    error ("usage: savgol (n,m)");
  endif

  nmax = length(nl) * length(nr) * length(m);
  if (nmax == 1)
   if (nl<0 || nr<0 )
     error (" n must be positive in  savgol()");
   endif
   ii = -nl:nr;
   jj = 0:m;
   A  = repmat(ii',1,m+1).^repmat(jj,nl+nr+1,1) ;
   A=(A'*A)^-1 * A';
   retval=A(1,:);
  else
    error ("savgol: expecting scalar arguments, found something else");
  endif

endfunction


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: averaging/smoothing data

Dmitri A. Sergatskov
On Wed, Jul 13, 2016 at 11:58 AM, Przemek Klosowski <[hidden email]> wrote:

[sorry for repost: maybe my original post gave an impression that I am not asking seriously, but I think it's an important and frequent use case, so here it is again]


Often, I need to smooth/average a data vector. I usually do a running average using filter(), e.g. for 5-value average:

    filter(ones(1:5)/5,eye(5,1),x)

I've been doing this for years, so I got used to it even though I have to think a little every time I need to do again, but maybe there's a better way? What do y'all use?



By the way, sometimes I'd use a Savitzky-Golay peak/moment-preserving filter coefficients instead of a simple rectangular averaging window:


# Copyright (C) 1993 Przemek Klosowski ; licensed under GPL
#
# You should have received a copy of the GNU General Public License
# along with Octave; see the file COPYING.  If not, write to the Free
# Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.

function retval = savgol (m,nl,nr)




​There are also sgolay and agolayfilt in signal package.​
 

Dmi​tri.
--





_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: averaging/smoothing data

Colin Macdonald-2
In reply to this post by Przemek Klosowski-7
On 13/07/16 09:58, Przemek Klosowski wrote:
> Often, I need to smooth/average a data vector. I usually do a running
> average using filter(), e.g. for 5-value average:
>
>     filter(ones(1:5)/5,eye(5,1),x)

Did you mean:

filter(ones(1,5)/5, eye(5,1), x)

I don't understand what your example does, and it doesn't seem to do
anything nice to a random 1D x.

(I don't have a better way, but maybe that would explain why you didn't
much response).

Colin


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: averaging/smoothing data

Gordon Haverland
In reply to this post by Przemek Klosowski-7
On Wed, 13 Jul 2016 12:58:29 -0400
Przemek Klosowski <[hidden email]> wrote:

> Often, I need to smooth/average a data vector.

I have had the good fortune of seeing wide distributions and lots of
outliers over the years.  If there is a significant probability of
picking up data from some other distribution (such as blunders in GPS),
I like to use a double window, median filter.  I use a wider window to
find what data to ignore, and then I find the median within the inner
window.

Another use case associated with GPS, is when the combine harvestor
(farming) has finished cutting grain in the current path, and is
traversing ground that has already been cut and processed, the yield
monitor data is not useful.  This is when processing data as a time
series, and not processing it in conjunction with location.

Gord


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

RE: averaging/smoothing data

Allen.Windhorn-2
In reply to this post by Przemek Klosowski-7
Przemek,

From: Help-octave [mailto:help-octave-bounces+allen.windhorn=[hidden email]] On Behalf Of Przemek Klosowski

> Often, I need to smooth/average a data vector. I usually do a running
> average using filter(), e.g. for 5-value average:
>     filter(ones(1:5)/5,eye(5,1),x)
> I've been doing this for years, so I got used to it even though I have to
> think a little every time I need to do again, but maybe there's a better
> way? What do y'all use?

If you have "spiky" data it really helps to run it through a median filter
(signals package) before you do any linear filtering.

Yfilt = medfilt1(Y, n);

This gets rid of noise that only affects one reading, but doesn't affect
the shape of the signal very much.

Regards,
Allen
_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: averaging/smoothing data

Przemek Klosowski-7
In reply to this post by Colin Macdonald-2
On 07/13/2016 02:08 PM, Colin Macdonald wrote:
On 13/07/16 09:58, Przemek Klosowski wrote:
Often, I need to smooth/average a data vector. I usually do a running
average using filter(), e.g. for 5-value average:

    filter(ones(1:5)/5,eye(5,1),x)

Did you mean:

filter(ones(1,5)/5, eye(5,1), x)
Yes, of course; sorry for the slip-up.

Thanks for pointing out the filters in the signal package. I am still surprised that there's no built-in core function, although I understand how it'd be difficult to come up with an universally pleasing one.

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: averaging/smoothing data

Sergei Steshenko





>________________________________
> From: Przemek Klosowski <[hidden email]>
>To: [hidden email]
>Sent: Wednesday, July 13, 2016 9:31 PM

>Subject: Re: averaging/smoothing data
[snip]
> I am
   still surprised that there's no built-in core function, although I
   understand how it'd be difficult to come up with an universally
   pleasing one.
>
>
>


A working approach is to use polynomial approximation with proper polynomial order - too low order will miss real peaks/dips, too high order will try to approximate noise.

--Sergei.

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave