Reading images from internal char string

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Reading images from internal char string

PhilipNienhuis
Hi,

AFAIK (pls correct me if I'm wrong) Octave can only read images from file (or URL, essentially a file as well).
I'd like to find a way to read images directly from a char string ("text file").

For some assignment I need to process a long series of rain radar images in PNG format.
I haul them in using urlread() in "raw" (compressed) form, IOW the very same format in memory as it would be on disk.  To make an image out of them I haven't found a better way yet than to first write them in raw form to to disk (fopen, fwrite, fclose), only then to be able turn them into an image using imread() to read them back in as an image.  
It would be a lot faster, and save disk space, if the output from urlread() could directly be interpreted and turned into an image while the "raw" image is still in memory, skipping disk I/O altogether.

Is this possible at all?

Stepping through imread.m I see that control finally passes to binary module __magickread__. All the way through there I only see references to real files.
I saw the url option in imageIO.m but that also results in a file on disk that is fed to __magickread__ in the end.

Thanks,

Philip



Reply | Threaded
Open this post in threaded view
|

Re: Reading images from internal char string

Pantxo
PhilipNienhuis wrote
Hi,

AFAIK (pls correct me if I'm wrong) Octave can only read images from file (or URL, essentially a file as well).
I'd like to find a way to read images directly from a char string ("text file").

For some assignment I need to process a long series of rain radar images in PNG format.
I haul them in using urlread() in "raw" (compressed) form, IOW the very same format in memory as it would be on disk.  To make an image out of them I haven't found a better way yet than to first write them in raw form to to disk (fopen, fwrite, fclose), only then to be able turn them into an image using imread() to read them back in as an image.  
It would be a lot faster, and save disk space, if the output from urlread() could directly be interpreted and turned into an image while the "raw" image is still in memory, skipping disk I/O altogether.

Is this possible at all?

Stepping through imread.m I see that control finally passes to binary module __magickread__. All the way through there I only see references to real files.
I saw the url option in imageIO.m but that also results in a file on disk that is fed to __magickread__ in the end.

Thanks,

Philip
Hi Phillip,

AFAIU Octave never has to deal with raw image headers/data, we rely on graphics magick to obtain image data in the form of RGB or indexed matrix.

To do the job, you'd have to either implement a png reader your self or use libpng from an oct file.

Pantxo
Reply | Threaded
Open this post in threaded view
|

Re: Reading images from internal char string

Richard Kirk
In reply to this post by PhilipNienhuis
If I had this job, I would cheat.

If you know all the images are the same size and uncompressed...
- save the first string to a file, & read it.
- get the image size from the file, and hence the number of bytes to the image data
- get the string, step back from the end by the number of bytes, fread, permute
- repeat for the following files.

A more advanced version would check you have got the right starting values to the
image, and add offsets for line and frame padding as necessary.
Reply | Threaded
Open this post in threaded view
|

Re: Reading images from internal char string

PhilipNienhuis
In reply to this post by Pantxo
Pantxo wrote
PhilipNienhuis wrote
Hi,

AFAIK (pls correct me if I'm wrong) Octave can only read images from file (or URL, essentially a file as well).
I'd like to find a way to read images directly from a char string ("text file").

For some assignment I need to process a long series of rain radar images in PNG format.
I haul them in using urlread() in "raw" (compressed) form, IOW the very same format in memory as it would be on disk.  To make an image out of them I haven't found a better way yet than to first write them in raw form to to disk (fopen, fwrite, fclose), only then to be able turn them into an image using imread() to read them back in as an image.  
It would be a lot faster, and save disk space, if the output from urlread() could directly be interpreted and turned into an image while the "raw" image is still in memory, skipping disk I/O altogether.

Is this possible at all?

Stepping through imread.m I see that control finally passes to binary module __magickread__. All the way through there I only see references to real files.
I saw the url option in imageIO.m but that also results in a file on disk that is fed to __magickread__ in the end.

Thanks,

Philip
Hi Phillip,

AFAIU Octave never has to deal with raw image headers/data, we rely on graphics magick to obtain image data in the form of RGB or indexed matrix.

To do the job, you'd have to either implement a png reader your self or use libpng from an oct file.
Ahhh... libpng, good suggestion. Hopefully that doesn't invoke file I/O either.

I was already setting up some .oct fle using libz (in my case the pixel data are deflated). The data chunks in a png "text" file are easily located and in the past I have hit other uses for in-memory uncompressing so maybe this will give the final nudge :-)

That GraphicsMagick always reads images from file is a bit misleading; deep down it cannot be an atomic procedure.  Either way it has to import file contents into memory, so  some raw read routine -possibly deeply buried- transports data into memory and some other (possibly equally deeply buried) routines process the data.
First writing to file then reading it back unaltered before processing the contents is just a plain waste of resources incl. time. In my opinion :-)

Thanks to all who responded.

Philip
Reply | Threaded
Open this post in threaded view
|

Re: Reading images from internal char string

Mike Miller-4
In reply to this post by PhilipNienhuis
On Thu, Jul 21, 2016 at 00:39:49 -0700, PhilipNienhuis wrote:

> Hi,
>
> AFAIK (pls correct me if I'm wrong) Octave can only read images from file
> (or URL, essentially a file as well).
> I'd like to find a way to read images directly from a char string ("text
> file").
>
> For some assignment I need to process a long series of rain radar images in
> PNG format.
> I haul them in using urlread() in "raw" (compressed) form, IOW the very same
> format in memory as it would be on disk.  To make an image out of them I
> haven't found a better way yet than to first write them in raw form to to
> disk (fopen, fwrite, fclose), only then to be able turn them into an image
> using imread() to read them back in as an image.  
> It would be a lot faster, and save disk space, if the output from urlread()
> could directly be interpreted and turned into an image while the "raw" image
> is still in memory, skipping disk I/O altogether.
>
> Is this possible at all?
>
> Stepping through imread.m I see that control finally passes to binary module
> __magickread__. All the way through there I only see references to real
> files.
> I saw the url option in imageIO.m but that also results in a file on disk
> that is fed to __magickread__ in the end.

I looked around a little, and as Pantxo said it seems that
GraphicsMagick will only read from a file. On unixy systems everything
is a file so that's not much of a limitation. I would just use a shared
memory file or a file in a tmpfs directory and it would be no different
from passing a memory buffer. I don't know if Windows has a concept of
being able to write a file in some way or in some directory so that it
is memory-backed instead of disk-backed.

--
mike

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave