On Wed, Mar 28, 2018 at 06:46:47 +0000, vlad ionescu wrote:

> > I think what is hurting your precision here is the lazy evaluation of

> > ranges in Octave.

>

> This is the first time I see this, so I did some search. Correct me if

> I am wrong, but, apparently, it's about the way Octave stores the

> range. If it's without square braces, it uses less storage, which

> probably means it uses floats instead of doubles

Actually it's that Octave stores ranges as a (start, step, stop)

triplet. Let's say the value n is the range -16.5:1:16.5 as in your

example. Octave stores that as (start=-16.5, step=1, stop=16.5).

The expression 0.37*pi*n evaluates to another range, with the original

triplet values each multiplied by 0.37*pi. Each of the 34 values of the

range has not been evaluated yet.

It's only when the range is passed in to the sin function that it is

evaluated into an array of discrete doubles. So I think you can see how

this lazy evaluation can lead to differences when adding subsequent

increments of 0.37*pi to the start value of -0.37*pi*16.5, compared to

what you actually wanted.

> > FWIW the h I obtain above is not equal to the h I get from the attached

> > fir function. The error is worst at index 18, the same as the error you

> > are seeing.

>

> Could it be that different CPU architectures calculate differently the

> floats? I can't imagine why you would get different results. Did you

> use fir(33,[0 0.37])? Still, the eror is, again, at index 18, so maybe

> it really is about the floating point representation.

Turns out this was only because I had tested with n=(-16.5:16.5)', where

the transpose operator also expands the range into a column vector.

--

mike