Re: segmentation fault when load more then 2.1 GB

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: segmentation fault when load more then 2.1 GB

Daniel Heiserer

> | Why not changing ftell/fseek etc. to ftello/fseeko?
> | http://www.ece.utexas.edu/~luo/linux_lfs.html
> | then 64bit large files support can be enabled during compile time.
>
> That could allow you to read large files, but what will you do with
> the data once you have read it?  Octave does not currently support
> arrays larger than 2GB on systems that have 32-bit ints.  Or are you
> saying that the file is larger than 8GB, but it contains no individual
> matrix objects that are larger than about 2GB?

At least one object is larger then 2GB. I submitted a patch on February
12th 2003 which fixes that problem. I never got a feedback concerning
the question for the "adverse effects" also I haven't seen the patch
included till today. Unfortunately the people answering the thread
focused on the bad timing of the "rand" function I used to demonstrate
the usage :-(

The patch was:
http://www.octave.org/octave-lists/ archive/bug-octave.2003/msg00306.html

I am willing to help fixing all this "size" limitations quickly. The
main reason why this stuff didn't bother most people up to now was
simply that it takes a reasonable investment in hardware to deal with
large problems (more then 2GB memory and a 64 bit OS).
However dawn has approached for these restrictions. It will be only a
question of time when this limitation will become a major problem for
many people.
The reason that "the leading competitior" might have a problem with it
as well is for me not the reason of not fixing it.


-- daniel


Reply | Threaded
Open this post in threaded view
|

Re: segmentation fault when load more then 2.1 GB

John W. Eaton-6
On  6-Oct-2004, Daniel Heiserer <[hidden email]> wrote:

| At least one object is larger then 2GB. I submitted a patch on February
| 12th 2003 which fixes that problem. I never got a feedback concerning
| the question for the "adverse effects" also I haven't seen the patch
| included till today. Unfortunately the people answering the thread
| focused on the bad timing of the "rand" function I used to demonstrate
| the usage :-(
|
| The patch was:
| http://www.octave.org/octave-lists/archive/bug-octave.2003/msg00306.html

Your one-line change only allowed the >2GB allocation to succeed.
After that, you would surely run into some problems with indexing,
etc.

| I am willing to help fixing all this "size" limitations quickly.

Please see the recent discussion on the help list with the subject
"matrix sizes".

jwe


Reply | Threaded
Open this post in threaded view
|

Re: segmentation fault when load more then 2.1 GB

Daniel Heiserer
[hidden email] wrote:

> On  6-Oct-2004, Daniel Heiserer <[hidden email]> wrote:
>
> | At least one object is larger then 2GB. I submitted a patch on February
> | 12th 2003 which fixes that problem. I never got a feedback concerning
> | the question for the "adverse effects" also I haven't seen the patch
> | included till today. Unfortunately the people answering the thread
> | focused on the bad timing of the "rand" function I used to demonstrate
> | the usage :-(
> |
> | The patch was:
> | http://www.octave.org/octave-lists/archive/bug-octave.2003/msg00306.html
>
> Your one-line change only allowed the >2GB allocation to succeed.
> After that, you would surely run into some problems with indexing,
> etc.
>
> | I am willing to help fixing all this "size" limitations quickly.
>
> Please see the recent discussion on the help list with the subject
> "matrix sizes".

Did anyone create now an octave_idx or is there an agreement howto
implement a typedef for the indexing?
I cannot do this.
I could start crawling through the libraries and change them to this
index-type once the core maintainers made up the starting decision howto
do this. I would suggest that (one of) the next beta version has this
type defined.
I also suggest we do the same for the filepointer stuff. So we can
adress the big stuff and make it configurable before the compilation.
It shouldn't be that a big deal to write a check routine which includes
compiling and allocating and adressing these large entities.

-- daniel


Reply | Threaded
Open this post in threaded view
|

Re: segmentation fault when load more then 2.1 GB

Russell Standish
We are currently implementing this in Octave 2.1.57 (as typedef
idx_t). This is a big job, much bigger than than we anticipated when
we started. A lot of code needs to be changed (although usually in a
trivial way).

If you want to help out, you are welcome to join us as a developer.

This leaves a major unsolved task of how to fold these changes back
into John's CVS repository - as Octave has now marched onto 2.1.60...

                                                                Cheers

On Wed, Oct 06, 2004 at 10:52:34AM -0500, Daniel Heiserer wrote:

>
> Did anyone create now an octave_idx or is there an agreement howto
> implement a typedef for the indexing?
> I cannot do this.
> I could start crawling through the libraries and change them to this
> index-type once the core maintainers made up the starting decision howto
> do this. I would suggest that (one of) the next beta version has this
> type defined.
> I also suggest we do the same for the filepointer stuff. So we can
> adress the big stuff and make it configurable before the compilation.
> It shouldn't be that a big deal to write a check routine which includes
> compiling and allocating and adressing these large entities.
>
> -- daniel
--
*PS: A number of people ask me about the attachment to my email, which
is of type "application/pgp-signature". Don't worry, it is not a
virus. It is an electronic signature, that may be used to verify this
email came from me if you have PGP or GPG installed. Otherwise, you
may safely ignore this attachment.

----------------------------------------------------------------------------
A/Prof Russell Standish             Director
High Performance Computing Support Unit, Phone 9385 6967, 8308 3119 (mobile)
UNSW SYDNEY 2052                     Fax   9385 6965, 0425 253119 (")
Australia             [hidden email]            
Room 2075, Red Centre                    http://parallel.hpc.unsw.edu.au/rks
            International prefix  +612, Interstate prefix 02
----------------------------------------------------------------------------

attachment0 (189 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: segmentation fault when load more then 2.1 GB

John W. Eaton-6
On  7-Oct-2004, Russell Standish <[hidden email]> wrote:

| This leaves a major unsolved task of how to fold these changes back
| into John's CVS repository - as Octave has now marched onto 2.1.60...

The key is to use a typedef as Paul mentioned, and to send small sets
of changes often.  If you dump a multi-megabyte patch on me after you
finish with your changes, it will likely be so much work to apply that
we will not be able to use it.  So please, if you are working on a
change like this that will touch many files (which are also being
changed by other people for completely different reasons) then it is
important that you make your changes relative to the current CVS
sources, not some arbitrary version like 2.1.57, and that you submit
patches frequently for say one file at a time.

Thanks,

jwe


Reply | Threaded
Open this post in threaded view
|

Re: segmentation fault when load more then 2.1 GB

Daniel Heiserer
In reply to this post by Russell Standish
Were can I pick up your changes/version you started with?
Do we have something like a regression test-suite with which we can test
the code for reasonable/identical behaviour?

-- daniel


> We are currently implementing this in Octave 2.1.57 (as typedef
> idx_t). This is a big job, much bigger than than we anticipated when
> we started. A lot of code needs to be changed (although usually in a
> trivial way).
>
> If you want to help out, you are welcome to join us as a developer.
>
> This leaves a major unsolved task of how to fold these changes back
> into John's CVS repository - as Octave has now marched onto 2.1.60...
>
> Cheers
>
> On Wed, Oct 06, 2004 at 10:52:34AM -0500, Daniel Heiserer wrote:
>
>>Did anyone create now an octave_idx or is there an agreement howto
>>implement a typedef for the indexing?
>>I cannot do this.
>>I could start crawling through the libraries and change them to this
>>index-type once the core maintainers made up the starting decision howto
>>do this. I would suggest that (one of) the next beta version has this
>>type defined.
>>I also suggest we do the same for the filepointer stuff. So we can
>>adress the big stuff and make it configurable before the compilation.
>>It shouldn't be that a big deal to write a check routine which includes
>>compiling and allocating and adressing these large entities.
>>
>>-- daniel
>
>