Filling the gaps in data arrays

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Filling the gaps in data arrays

Mat086
Hello everybody,
I'm dealing with a little problem here and I'm a newbie with Octave,
although I used Matlab previously.

I have an [x y] dataset containing x positions and peaks. However, only data
for the peaks are presents and between them nothing. Let me explain better
with a simple example:

x = [1 2 3 4 5 6 7 8 9    27 28 29 30 31 32 33 34 35     56 57 58 59 60 61
62 63 64 ]
y = [0.1 0.5 1 5 25 5 1 0.5 .1     0.3 0.5 1.5 15 44 15 1.5 0.5 .3     0.3
0.2 3 9 23 9 3 0.2 .3 ]

As you can see, the step in x array is fixed to x(2)-x(1)=1 (in this
example), but there are gaps in both x and y in regions where background is
supposed to be flat.
However, In order to analyze such kind of data with another software I need
to create X and Y vectors without holes. Therefore, I should create an X
array X=[1:step:64], of length=64, and Y array of the same length, having
the peaks at the same positions of [x y] and zeros where there were gaps in
x. There was an in built function in Matlab ("fillmissing" if I remember
well), but I'm struggling to get this done in Octave.
Any help would be greatly appreciated!

thanks,
Matt



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Filling the gaps in data arrays

Doug Stewart-4


On Wed, Jul 11, 2018 at 10:37 AM, Mat086 <[hidden email]> wrote:
Hello everybody,
I'm dealing with a little problem here and I'm a newbie with Octave,
although I used Matlab previously.

I have an [x y] dataset containing x positions and peaks. However, only data
for the peaks are presents and between them nothing. Let me explain better
with a simple example:

x = [1 2 3 4 5 6 7 8 9    27 28 29 30 31 32 33 34 35     56 57 58 59 60 61
62 63 64 ]
y = [0.1 0.5 1 5 25 5 1 0.5 .1     0.3 0.5 1.5 15 44 15 1.5 0.5 .3     0.3
0.2 3 9 23 9 3 0.2 .3 ]

As you can see, the step in x array is fixed to x(2)-x(1)=1 (in this
example), but there are gaps in both x and y in regions where background is
supposed to be flat.
However, In order to analyze such kind of data with another software I need
to create X and Y vectors without holes. Therefore, I should create an X
array X=[1:step:64], of length=64, and Y array of the same length, having
the peaks at the same positions of [x y] and zeros where there were gaps in
x. There was an in built function in Matlab ("fillmissing" if I remember
well), but I'm struggling to get this done in Octave.
Any help would be greatly appreciated!

thanks,
Matt



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html





Here is one wah to do it;




x = [1 2 3 4 5 6 7 8 9 27 28 29 30 31 32 33 34 35 56 57 58 59 60 61 62 63 64 ]
y = [0.1 0.5 1 5 25 5 1 0.5 .1 0.3 0.5 1.5 15 44 15 1.5 0.5 .3 0.3 0.2 3 9 23 9 3 0.2 .3 ]

xx(64)=0
for k=1:27
xx(x(k))=y((k));
endfor
plot(xx)
xx





--
DAS



Reply | Threaded
Open this post in threaded view
|

Re: Filling the gaps in data arrays

Mat086
This post was updated on .
Doug Stewart-4 wrote
> Here is one wah to do it;
>
>
>
>
> x = [1 2 3 4 5 6 7 8 9 27 28 29 30 31 32 33 34 35 56 57 58 59 60 61 62 63
> 64 ]
> y = [0.1 0.5 1 5 25 5 1 0.5 .1 0.3 0.5 1.5 15 44 15 1.5 0.5 .3 0.3 0.2 3 9
> 23 9 3 0.2 .3 ]
>
> xx(64)=0
> for k=1:27
> xx(x(k))=y((k));
> endfor
> plot(xx)
> xx
>
> --
> DAS
>
> <https://linuxcounter.net/user/206392.html>

Hi and thanks!
This actually works fine only in this particular situation but it doesn't if
you have a non integer x-array like:
  x = [1.1:.1:1.9    2.7:.1:3.5     5.6:.1:6.4 ];
  y = [0.1 0.5 1 5 25 5 1 0.5 .1     0.3 0.5 1.5 15 44 15 1.5 0.5 .3     0.3
0.2 3 9 23 9 3 0.2 .3 ];
However, I got the hint from you and I modified the script into:

dx = x(2)-x(1);
Y = zeros(1,length([min(x):dx:max(x)]));
X = [min(x):dx:max(x)];
for k=1:length(x)
  indx = find(X==x(k));
  Y(k) = y(index);
endfor
plot(x,y,'b-'); hold on; plot(X,Y,'-o');

On the best of my knowledge, this should be working fine in Matlab, but with
octave I get the following error as soon as k=3:
......
index =  1
index =  2
index = [](1x0)
error: test_00: =: nonconformant arguments (op1 is 1x1, op2 is 1x0)
......

Any idea?

Many thanks,
Matt
Reply | Threaded
Open this post in threaded view
|

Re: Filling the gaps in data arrays

Przemek Klosowski-7
On 07/12/2018 04:36 AM, Mat086 wrote:
> dx = x(2)-x(1);
> Y = zeros(1,length([min(x):dx:max(x)]));
> X = [min(x):dx:max(x)];
> for k=1:length(x)
>    indx = find(X==x(k));
>    Y(k) = y(index);
> endfor
> plot(x,y,'b-'); hold on; plot(X,Y,'-o');

you mean  Y(k) = y(indx); , right?

And no, it won't work in Matlab either because you are doing exact
floating point comparisons on calculated values, so any roundoff will
trip you up---check for yourself, X==X(3) returns matches, but X==x(3)
probably doesn't.

You said that "In order to analyze such kind of data with another
software I need to create X and Y vectors without holes". What is
exactly the assumption there? Is the original dataset sampled on a
regular grid, and the non-zero values are dropped?

It seems to me that the best solution for you would be to interpolate
your data onto a regular grid, perhaps like this:

dx=x(2)-x(1);

plot(x,y,'x',X=min(x):dx:max(x),interp1(x,y,X))



Reply | Threaded
Open this post in threaded view
|

Re: Filling the gaps in data arrays

Mat086
Przemek Klosowski-7 wrote

> On 07/12/2018 04:36 AM, Mat086 wrote:
>> dx = x(2)-x(1);
>> Y = zeros(1,length([min(x):dx:max(x)]));
>> X = [min(x):dx:max(x)];
>> for k=1:length(x)
>>    indx = find(X==x(k));
>>    Y(k) = y(index);
>> endfor
>> plot(x,y,'b-'); hold on; plot(X,Y,'-o');
>
> you mean  Y(k) = y(indx); , right?
>
> And no, it won't work in Matlab either because you are doing exact
> floating point comparisons on calculated values, so any roundoff will
> trip you up---check for yourself, X==X(3) returns matches, but X==x(3)
> probably doesn't.
>
> You said that "In order to analyze such kind of data with another
> software I need to create X and Y vectors without holes". What is
> exactly the assumption there? Is the original dataset sampled on a
> regular grid, and the non-zero values are dropped?
>
> It seems to me that the best solution for you would be to interpolate
> your data onto a regular grid, perhaps like this:
>
> dx=x(2)-x(1);
>
> plot(x,y,'x',X=min(x):dx:max(x),interp1(x,y,X))


Hi Przemek,
thanks for your reply!
I don't need to interpolate the data, actually the best thing would be to
have NaN where the y-data doesn't match the X vector.
this script is doing what I needed to do:

dx = x(2)-x(1);
Y = zeros(1,length([min(x):dx:max(x)]));
X = [min(x):dx:max(x)];
st=1;
for k=1:length(X)
  if isempty(y(x==X(k)))
    Y(k) = 0;
  else
    Y(k) = y(st);
    st=st+1;
  end
end

thanks again,
Mat



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Filling the gaps in data arrays

Przemek Klosowski-7
On 07/26/2018 10:47 AM, Mat086 wrote:

> I don't need to interpolate the data, actually the best thing would be to
> have NaN where the y-data doesn't match the X vector.
> this script is doing what I needed to do:
>
> dx = x(2)-x(1);
> Y = zeros(1,length([min(x):dx:max(x)]));
> X = [min(x):dx:max(x)];
> st=1;
> for k=1:length(X)
>    if isempty(y(x==X(k)))
>      Y(k) = 0;
>    else
>      Y(k) = y(st);
>      st=st+1;
>    end
> end
You still have the problem with accuracy; I think you will zero out some
valid numbers because your calculated X values do not have to fall
exactly on your x values.
For a contrived example, x=[0 0.3333333 0.666667 1]
x =

    0.00000   0.33333   0.66667   1.00000

octave:14> dx = x(2)-x(1);
octave:15> Y = zeros(1,length([min(x):dx:max(x)]));
octave:16> X = [min(x):dx:max(x)]
X =

    0.00000   0.33333   0.66667   1.00000

octave:17> x==X
ans =

   1  1  0  0

As you see, x and X are so very close that they print as identical, but
they do not match. You can search for closeness:

my_eps=1e-6;
match = ( abs(x-X)<my_eps )

and then exploit the match vector to just transfer the matching y values:

dx = x(2)-x(1);
X = [min(x):dx:max(x)];
Y = zeros(1,length(X));
Y(find(match))= y

Octave/Matlab language has those 'vector' expressions that make it quite fast and concise. The cost is maybe the extra storage (e.g. the match vector, and the temporary vectors used by the find() operation) but in practice it is a big win and it's good to learn to write code this way.

BTW, if you want to really use NaNs, you could do this:
Y = repmat(NaN,1,length(X));






Reply | Threaded
Open this post in threaded view
|

Re: Filling the gaps in data arrays

Mat086

On 07/26/2018 10:47 AM, Mat086 wrote:

> I don't need to interpolate the data, actually the best thing would be to
> have NaN where the y-data doesn't match the X vector.
> this script is doing what I needed to do:
>
> dx = x(2)-x(1);
> Y = zeros(1,length([min(x):dx:max(x)]));
> X = [min(x):dx:max(x)];
> st=1;
> for k=1:length(X)
>    if isempty(y(x==X(k)))
>      Y(k) = 0;
>    else
>      Y(k) = y(st);
>      st=st+1;
>    end
> end
You still have the problem with accuracy; I think you will zero out some
valid numbers because your calculated X values do not have to fall
exactly on your x values.
For a contrived example, x=[0 0.3333333 0.666667 1]
x =

    0.00000   0.33333   0.66667   1.00000

octave:14> dx = x(2)-x(1);
octave:15> Y = zeros(1,length([min(x):dx:max(x)]));
octave:16> X = [min(x):dx:max(x)]
X =

    0.00000   0.33333   0.66667   1.00000

octave:17> x==X
ans =

   1  1  0  0

As you see, x and X are so very close that they print as identical, but
they do not match. You can search for closeness:

my_eps=1e-6;
match = ( abs(x-X)<my_eps )

and then exploit the match vector to just transfer the matching y values:

dx = x(2)-x(1);
X = [min(x):dx:max(x)];
Y = zeros(1,length(X));
Y(find(match))= y

Octave/Matlab language has those 'vector' expressions that make it quite
fast and concise. The cost is maybe the extra storage (e.g. the match
vector, and the temporary vectors used by the find() operation) but in
practice it is a big win and it's good to learn to write code this way.

BTW, if you want to really use NaNs, you could do this:
Y = repmat(NaN,1,length(X));
&lt;/quote>


Dear Przemek,
many thanks for this, it was very helpful!

M



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html