Problems with reading csv (textscan)

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with reading csv (textscan)

basic_coding
Hello there

I'm trying to read my measurement data on Octave in order to further analyze
them. My measurment device has large files like the one in the following
link as output:

Messungen1.csv
<https://octave.1599824.n4.nabble.com/file/t373453/Messungen1.csv>  

I got some problems with importing the variables. I used the following lines
of code:



It seemed to extract only the first line and most of the columns were empty.


If I try to write a shorter file similar to the one linked, I can read the
data with the following code:



This works for example for a file like this:


So the simple one works fine. Do you have an idea how I can read the large
file as well?

Best regards and thanks in advance

Kevin



--
Sent from: https://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Problems with reading csv (textscan)

nrjank
On Mon, Aug 19, 2019 at 10:46 AM basic_coding <[hidden email]> wrote:
Hello there

I'm trying to read my measurement data on Octave in order to further analyze
them. My measurment device has large files like the one in the following
link as output:

Messungen1.csv
<https://octave.1599824.n4.nabble.com/file/t373453/Messungen1.csv

I got some problems with importing the variables. I used the following lines
of code:



It seemed to extract only the first line and most of the columns were empty.


If I try to write a shorter file similar to the one linked, I can read the
data with the following code:



This works for example for a file like this:


So the simple one works fine. Do you have an idea how I can read the large
file as well?

Best regards and thanks in advance

Kevin



much of your code did not come through to the mailing list. Follow this Nabble archive link at the end to see what it looks like to everyone else).  

Please just copy/paste the code as plain text and resend to the list so we can see what you've tried and what output you're getting.

looking at your csv, I tried to do a textscan as a combination of strings and floats. I'm a ran into trouble getting the format string to handle the multiple spaces and semicolons, but I'm not overly familiar with formatted string parsing.

 


Reply | Threaded
Open this post in threaded view
|

Re: Problems with reading csv (textscan)

basic_coding
Thanks for quick answer.

So here is the simple 3 lines of code I used for the original file:
file = 'C:\......path_to_file....\messungen1'
fid = fopen(file, 'r');
[data] = textscan(fid, '%s %s %f %s %f %s %f %s %f %s %f %s %f %s %f %s %f'
,'delimiter',';');


As this didn't work I tried to create a simple csv with comparable content:
    2019-08-16-10:27:28 ;    value 1 ;    5.83 ;    value 2 ;    13 ;
    2019-08-16-10:27:30 ;    value 1 ;    6.73 ;    value 2 ;    14 ;


Using the follwoing code lines, I managed to read the data:
file = 'C:\......path_to_file....\filename.csv'
fid = fopen(file, 'r');
[data] = textscan(fid, '%s %s %f %s %f' ,'delimiter',';');

Sorry for the missing lines.



--
Sent from: https://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Problems with reading csv (textscan)

basic_coding
This post was updated on .
In reply to this post by nrjank
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Problems with reading csv (textscan)

nrjank


On Mon, Aug 19, 2019 at 12:34 PM basic_coding <[hidden email]> wrote:

>
> Thanks for quick answer.
>
> So here is the simple 3 lines of code I used for the original file:
> file = 'C:\......path_to_file....\messungen1'
> fid = fopen(file, 'r');
> [data] = textscan(fid, '%s %s %f %s %f %s %f %s %f %s %f %s %f %s %f %s %f'
> ,'delimiter',';');
>
>
> As this didn't work I tried to create a simple csv with comparable content:
>     2019-08-16-10:27:28 ;    value 1 ;    5.83 ;    value 2 ;    13 ;
>     2019-08-16-10:27:30 ;    value 1 ;    6.73 ;    value 2 ;    14 ;
>
>
> Using the follwoing code lines, I managed to read the data:
> file = 'C:\......path_to_file....\filename.csv'
> fid = fopen(file, 'r');
> [data] = textscan(fid, '%s %s %f %s %f' ,'delimiter',';');
>
> Sorry for the missing lines.
>
>

Thanks for the quick followup.  As I mentioned before, I also tried various combinations of options for textscan to see if it could pull in your data.  playing around with delimiter,  MultipleDelimsAsOne, the format string, etc. and I wasn't able to get it to output all of the data. it would always grab the first date string, sometimes the first few %s and first %f, but just empty cells after the first %f.  Maybe someone better with formatted strings can see what's causing the problem.


Reply | Threaded
Open this post in threaded view
|

Re: Problems with reading csv (textscan)

Dmitri A. Sergatskov
> Thanks for the quick followup.  As I mentioned before, I also tried various combinations of options for textscan to see if it could pull in your data.  playing around with delimiter,  MultipleDelimsAsOne, the format string, etc. and I wasn't able to get it to output all of the data. it would always grab the first date string, sometimes the first few %s and first %f, but just empty cells after the first %f.  Maybe someone better with formatted strings can see what's causing the problem.

There is some trailing junk before the ";" :
od -c Messungen1.csv | head
0000000                           2   0   1   9   -   0   8   -   1   6
0000020   -   1   0   :   2   7   :   2   8  \0   ;
0000040                                                           F   x
0000060       [   N   ]  \0   ;
0000100                                       -   3   .   6   9   8  \0
0000120   ;
0000140                   F   y       [   N   ]  \0   ;
0000160                                                           -   4
0000200   2   .   4   2   8  \0   ;


Dmitri.
--


Reply | Threaded
Open this post in threaded view
|

Re: Problems with reading csv (textscan)

PhilipNienhuis
In reply to this post by basic_coding
basic_coding wrote

> Thanks for quick answer.
>
> So here is the simple 3 lines of code I used for the original file:
> file = 'C:\......path_to_file....\messungen1'
> fid = fopen(file, 'r');
> [data] = textscan(fid, '%s %s %f %s %f %s %f %s %f %s %f %s %f %s %f %s
> %f'
> ,'delimiter',';');
>
>
> As this didn't work I tried to create a simple csv with comparable
> content:
>     2019-08-16-10:27:28 ;    value 1 ;    5.83 ;    value 2 ;    13 ;
>     2019-08-16-10:27:30 ;    value 1 ;    6.73 ;    value 2 ;    14 ;
>
>
> Using the follwoing code lines, I managed to read the data:
> file = 'C:\......path_to_file....\filename.csv'
> fid = fopen(file, 'r');
> [data] = textscan(fid, '%s %s %f %s %f' ,'delimiter',';');

In the io package there's a function called csv2cell that is much faster
than textscan. It'll return a cell array but it's easy to post-process the
numerical part into a numerical array using either:
- cell2mat () or
- parsecell(), also in the io package, which separates the text and
numerical parts into separate arrays w/o much hassle.

You'll need to specify the field delimiter (";") for csv2cell; type "help
csv2cell" and "help parsecell" for the relevant options.

Philip



--
Sent from: https://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Problems with reading csv (textscan)

Dmitri A. Sergatskov
In reply to this post by Dmitri A. Sergatskov
On Mon, Aug 19, 2019 at 12:18 PM Dmitri A. Sergatskov
<[hidden email]> wrote:

>
> > Thanks for the quick followup.  As I mentioned before, I also tried various combinations of options for textscan to see if it could pull in your data.  playing around with delimiter,  MultipleDelimsAsOne, the format string, etc. and I wasn't able to get it to output all of the data. it would always grab the first date string, sometimes the first few %s and first %f, but just empty cells after the first %f.  Maybe someone better with formatted strings can see what's causing the problem.
>
> There is some trailing junk before the ";" :
> od -c Messungen1.csv | head
> 0000000                           2   0   1   9   -   0   8   -   1   6
> 0000020   -   1   0   :   2   7   :   2   8  \0   ;
> 0000040                                                           F   x
> 0000060       [   N   ]  \0   ;
> 0000100                                       -   3   .   6   9   8  \0
> 0000120   ;
> 0000140                   F   y       [   N   ]  \0   ;
> 0000160                                                           -   4
> 0000200   2   .   4   2   8  \0   ;
>
>
> Dmitri.
> --
After I delete \0 and junked the remaining empty lines (result is attached)
at the end of the file, your code works just fine.


Dmitri.
--



t1.csv (618K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problems with reading csv (textscan)

basic_coding
Thanks a lot, Dmitri and Philip.

@PhilipNienhuis: With csv2cell I only managed to import the first column
(but this one properly). I couldn't find a way to extend this to the
complete csv. I guess the "\0"-stuff which Dmitri found causes problems.

@Dmitri: How can I get rid of this trailing junk in Octave? So far I
realized that with the following adaption at least the textscan works (I had
to add another %s):

[data] = textscan(fid, '%s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s
%s' ,'delimiter',';');

But now I can't convert the data to numbers, I assume the "\0" is still in
those strings and can't be converted. Do you know a workaround there?

Best regards,
Kevin



--
Sent from: https://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Problems with reading csv (textscan)

basic_coding
Okay, I found the solution on my own. Thanks for your help.
I just had to remove the last character of the string before converting with
str2double.

Cheers,

Kevin



--
Sent from: https://octave.1599824.n4.nabble.com/Octave-General-f1599825.html