Importing columns as a matrix

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Importing columns as a matrix

opens
Hello everybody,

I basically want to import with GNU Octave a matrix of numbers.

It consists of 132651 values which equals 51x51x51 (later I want to reshape my matrix with
reshape(Matrix,51,51,51))

The problem is the format of my data file that I want to import:
it consists of 6 columns like the following example shows:

1.value      2.value      3.value      4.value    5.value    6.value
...
...
...
132649.value 132650.value 132651.value  no-value  no-value   no-value

So the last 3 values of my 6 x 22109 array are empty (6x22109-3=132651). If they would be ignored everything would be perfect.

My data-import command is the following:
my_textfile=load ("-ascii", "6-columns.txt");

But due to the fact that 3 values are missing, I get the following error:
error: load: 6-columns.txt: inconsistent number of columns near line 22109

Is there a simple workaround for that?
Every help is really appreciated! Thank you in advance!

Best regards
Arthur Niedermayr




_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Importing columns as a matrix

Juan Pablo Carbajal-2
On Fri, Jun 2, 2017 at 12:56 AM, "Niedermayr, Arthur"
<[hidden email]> wrote:

> Hello everybody,
>
> I basically want to import with GNU Octave a matrix of numbers.
>
> It consists of 132651 values which equals 51x51x51 (later I want to reshape
> my matrix with
> reshape(Matrix,51,51,51))
>
> The problem is the format of my data file that I want to import:
> it consists of 6 columns like the following example shows:
>
> 1.value      2.value      3.value      4.value    5.value    6.value
> ...
> ...
> ...
> 132649.value 132650.value 132651.value  no-value  no-value   no-value
>
> So the last 3 values of my 6 x 22109 array are empty (6x22109-3=132651). If
> they would be ignored everything would be perfect.
>
> My data-import command is the following:
> my_textfile=load ("-ascii", "6-columns.txt");
>
> But due to the fact that 3 values are missing, I get the following error:
> error: load: 6-columns.txt: inconsistent number of columns near line 22109
>
> Is there a simple workaround for that?
> Every help is really appreciated! Thank you in advance!
>
> Best regards
> Arthur Niedermayr
>
>
>
>
> _______________________________________________
> Help-octave mailing list
> [hidden email]
> https://lists.gnu.org/mailman/listinfo/help-octave
>
The simplest hack is to erase the last line of your data and add it
later manually. You could also use dlmread and read all except the
last row or to insert NA in the missing values.

1. first hack
sed '$ d' 6-columns.txt > 6-columns_new.txt

2. hack
x = dlmread ("6-columns.txt", " ", [0 0 22107 5])

dlmread can also put empty value son those missing ones

x = dlmread ("6-columns.txt", " ", "emptyvalue", NA)

the last row of x will be
132649.value 132650.value 132651.value  NA  NA  NA

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Importing columns as a matrix

opens
Hi,
thank you very much for your answer, it was very useful!

I have a further question regarding this special function:
A=dlmread("6-columns.txt"," ",[0 0 22107 5], "emptyvalue" , NA);

If line 22107 looks like this:
2.2 3.3 4.4
My array looks like this:
2.2 3.3 4.4 0.0 0.0 0.0

This functions gives me only NA if the line consists of letters,
eg.
2.2 3.3 4.4 x x x
Then:
2.2 3.3 4.4 NA NA NA

It means notexistent values give 0, letters NA.

Is there a way that i get always "NA" instead of "0.0" ? (Otherwise I have to do some matrix rearrangements which are not as trivial as they seem to be; they are also time consuming if the array is large)

Best regards and thank you in advance!
Arthur Niedermayr



-------- Original Message --------
Subject: Re: Importing columns as a matrix
Local Time: June 2, 2017 1:54 AM
UTC Time: June 1, 2017 11:54 PM
To: "Niedermayr, Arthur" <[hidden email]>

On Fri, Jun 2, 2017 at 12:56 AM, "Niedermayr, Arthur"
<[hidden email]> wrote:
> Hello everybody,
>
> I basically want to import with GNU Octave a matrix of numbers.
>
> It consists of 132651 values which equals 51x51x51 (later I want to reshape
> my matrix with
> reshape(Matrix,51,51,51))
>
> The problem is the format of my data file that I want to import:
> it consists of 6 columns like the following example shows:
>
> 1.value 2.value 3.value 4.value 5.value 6.value
> ...
> ...
> ...
> 132649.value 132650.value 132651.value no-value no-value no-value
>
> So the last 3 values of my 6 x 22109 array are empty (6x22109-3=132651). If
> they would be ignored everything would be perfect.
>
> My data-import command is the following:
> my_textfile=load ("-ascii", "6-columns.txt");
>
> But due to the fact that 3 values are missing, I get the following error:
> error: load: 6-columns.txt: inconsistent number of columns near line 22109
>
> Is there a simple workaround for that?
> Every help is really appreciated! Thank you in advance!
>
> Best regards
> Arthur Niedermayr
>
>
>
>
> _______________________________________________
> Help-octave mailing list
> https://lists.gnu.org/mailman/listinfo/help-octave
>
The simplest hack is to erase the last line of your data and add it
later manually. You could also use dlmread and read all except the
last row or to insert NA in the missing values.

1. first hack
sed '$ d' 6-columns.txt > 6-columns_new.txt

2. hack
x = dlmread ("6-columns.txt", " ", [0 0 22107 5])

dlmread can also put empty value son those missing ones

x = dlmread ("6-columns.txt", " ", "emptyvalue", NA)

the last row of x will be
132649.value 132650.value 132651.value NA NA NA


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Importing columns as a matrix

Juan Pablo Carbajal-2
On Sun, Jun 11, 2017 at 1:44 PM, "Niedermayr, Arthur"
<[hidden email]> wrote:

> Hi,
> thank you very much for your answer, it was very useful!
>
> I have a further question regarding this special function:
> A=dlmread("6-columns.txt"," ",[0 0 22107 5], "emptyvalue" , NA);
>
> If line 22107 looks like this:
> 2.2 3.3 4.4
> My array looks like this:
> 2.2 3.3 4.4 0.0 0.0 0.0
>
> This functions gives me only NA if the line consists of letters,
> eg.
> 2.2 3.3 4.4 x x x
> Then:
> 2.2 3.3 4.4 NA NA NA
>
> It means notexistent values give 0, letters NA.
>
> Is there a way that i get always "NA" instead of "0.0" ? (Otherwise I have
> to do some matrix rearrangements which are not as trivial as they seem to
> be; they are also time consuming if the array is large)
>
> Best regards and thank you in advance!
> Arthur Niedermayr
>
>
>
> -------- Original Message --------
> Subject: Re: Importing columns as a matrix
> Local Time: June 2, 2017 1:54 AM
> UTC Time: June 1, 2017 11:54 PM
> From: [hidden email]
> To: "Niedermayr, Arthur" <[hidden email]>
> [hidden email] <[hidden email]>
>
> On Fri, Jun 2, 2017 at 12:56 AM, "Niedermayr, Arthur"
> <[hidden email]> wrote:
>> Hello everybody,
>>
>> I basically want to import with GNU Octave a matrix of numbers.
>>
>> It consists of 132651 values which equals 51x51x51 (later I want to
>> reshape
>> my matrix with
>> reshape(Matrix,51,51,51))
>>
>> The problem is the format of my data file that I want to import:
>> it consists of 6 columns like the following example shows:
>>
>> 1.value 2.value 3.value 4.value 5.value 6.value
>> ...
>> ...
>> ...
>> 132649.value 132650.value 132651.value no-value no-value no-value
>>
>> So the last 3 values of my 6 x 22109 array are empty (6x22109-3=132651).
>> If
>> they would be ignored everything would be perfect.
>>
>> My data-import command is the following:
>> my_textfile=load ("-ascii", "6-columns.txt");
>>
>> But due to the fact that 3 values are missing, I get the following error:
>> error: load: 6-columns.txt: inconsistent number of columns near line 22109
>>
>> Is there a simple workaround for that?
>> Every help is really appreciated! Thank you in advance!
>>
>> Best regards
>> Arthur Niedermayr
>>
>>
>>
>>
>> _______________________________________________
>> Help-octave mailing list
>> [hidden email]
>> https://lists.gnu.org/mailman/listinfo/help-octave
>>
> The simplest hack is to erase the last line of your data and add it
> later manually. You could also use dlmread and read all except the
> last row or to insert NA in the missing values.
>
> 1. first hack
> sed '$ d' 6-columns.txt > 6-columns_new.txt
>
> 2. hack
> x = dlmread ("6-columns.txt", " ", [0 0 22107 5])
>
> dlmread can also put empty value son those missing ones
>
> x = dlmread ("6-columns.txt", " ", "emptyvalue", NA)
>
> the last row of x will be
> 132649.value 132650.value 132651.value NA NA NA
>
>
I am not sure this is the intended behavior and to me it looks like a bug.
However look at this data (I use comma instead of blanks to make it
clear, and \n is newline)
data.txt:
1,2,3,4,5,6\n
1,2,a,b,c,6\n
1,2,3,,,,\n
1,2,3,4,5,6\n
\n
1,2,3,4,5,6\n
1,2,3,,5,6\n
1,2,3,4,5,6\n

dlmread('data.txt', ', ', [0 0 8 5], 'emptyvalue', NA) would give you
     1     2     3     4     5     6
     1     2    NA    NA    NA     6
     1     2     3    NA    NA    NA
     1     2     3     4     5     6
     1     2     3     4     5     6
     1     2     3    NA     5     6
     1     2     3     4     5     6

Which is the correct output as you want. Note that 'emptyvalue' means
to consecutive separators, if your separator is a whitespace then your
file needs to have 5 whitespaces after the 3 column in the row with
only 3 values for dlmread to know those are empty values. Is your file
correctly formated? Are you sending the right options to dlmread? for
the latter read the help carefully.

Nevertheless, if dlmread is padding the row with zeros, it somehow
knows those are empty values, so I am assuming there is a bug here.
Could you please report it in the bug tracker to savanah?

At this point I would remind you that you shouldn't expect a
ready-to-use function that will read any kind of unstructured file.
You need either to curate your file or to write your own read function
using, for example, fscanf or textscan.

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Loading...