memory() function for Octave

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

memory() function for Octave

Lars Kindermann
sorry - the first post went into a wrong thread...

Some time ago I wrote a mostly matlab compatible memory() function for
two main reasons:

- I often work with datasets far bigger than my pc memory and I have to
split them into chunks which just fit into ram.

- I wanted to track down a memory leak that occured in preparing 5.0.

So far I did not think about submitting it to the octave kernel because
it works on linux only. But I just realized that matlabs memory() is
available on windows only. So I think a linux only memory() for octave
is reasonable.

The function is octave only code, a single m-file which reads the linux
/proc/pid/statm, /proc/pid/status and /proc/meminfo pseudo files to
retrieve exactly the same information you get with the "top" command.

I use it frequently for more than a year on 64Bit amd64 systems with
kernels 3.2 - 5.5 and have tested a little bit on i386 32Bit systems,
too (Debian & Ubuntu).

But I am not sure if it will work in every situation on 32Bit systems as
memory management and limitations are much more complicated there. Would
it be better to provide it for 64Bit only?

Should a warning or an error be thrown on other architectures?

And there are some compatibility issues with the matlab function.

E.g. naming of fields: Should "MemUsedMATLAB" become "MemUsedOctave" or
should both names be provided?

Matlab's MemAvailableAllArrays and MaxPossibleArrayBytes return the
available memory amount *including swap*. Which renders the system
almost unusable when fully allocated.

So I introduced few new, octave-only fields, e.g.
"RamAvailableAllArrays" which returns the available *ram*, which is what
you really want to know for efficient work. Is this ok in regard to
usability / compatibility?

If you think it is worth the effort, I will start to make the code
suitable for publication and submit it to Savanahs patches. But I will
need some guidance and help for all the work necessary to introduce a
new function to octave: Documentation, compatibility issues, tests etc.

Any comments? Lars


Reply | Threaded
Open this post in threaded view
|

Re: memory() function for Octave

Pantxo
Lars Kindermann wrote

> E.g. naming of fields: Should "MemUsedMATLAB" become "MemUsedOctave" or
> should both names be provided?
>
> Matlab's MemAvailableAllArrays and MaxPossibleArrayBytes return the
> available memory amount *including swap*. Which renders the system
> almost unusable when fully allocated.
>
> So I introduced few new, octave-only fields, e.g.
> "RamAvailableAllArrays" which returns the available *ram*, which is what
> you really want to know for efficient work. Is this ok in regard to
> usability / compatibility?
>
> If you think it is worth the effort, I will start to make the code
> suitable for publication and submit it to Savanahs patches. But I will
> need some guidance and help for all the work necessary to introduce a
> new function to octave: Documentation, compatibility issues, tests etc.
>
> Any comments? Lars

Hi Lars,

I'd be very happy to see this function intyegrated into Octave core at some
point. I have happily used it in many situation and found it very useful.

My answer to you questions:
* Not all ML fields have to be implemented at first.
* If you implement MemUsedOctave then MemUsedMATLAB should exist even if
it's an alias: this will ensure a code that uses this field in ML will work
in Octave
* You are free to propose Octave only features. Their usefulness will
probably be discussed before integration in Octave.

At some point I can help to integrate the function into Octave code base but
I think the very first effort is to see how this function can be made
cross-platform:
* your version is linux only (BSD? Mac?) and partly ML compatible
* AFAICS the one in macgyver_utils is cross-platform but is not ML
compatible and returns different data types depending on the platform.

You should eventually create a patch on Savannah to centralize this effort
and invite Phillip, Andy and whoever is interested to come and help.

Pantxo



--
Sent from: https://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: memory() function for Octave

Lars Kindermann
I submittet memory.m in patch #9942
https://savannah.gnu.org/patch/?9924
I think the discussion should continue there.
Lars

Am 21.04.20 um 11:51 schrieb Pantxo:

> Lars Kindermann wrote
>> E.g. naming of fields: Should "MemUsedMATLAB" become "MemUsedOctave" or
>> should both names be provided?
>>
>> Matlab's MemAvailableAllArrays and MaxPossibleArrayBytes return the
>> available memory amount *including swap*. Which renders the system
>> almost unusable when fully allocated.
>>
>> So I introduced few new, octave-only fields, e.g.
>> "RamAvailableAllArrays" which returns the available *ram*, which is what
>> you really want to know for efficient work. Is this ok in regard to
>> usability / compatibility?
>>
>> If you think it is worth the effort, I will start to make the code
>> suitable for publication and submit it to Savanahs patches. But I will
>> need some guidance and help for all the work necessary to introduce a
>> new function to octave: Documentation, compatibility issues, tests etc.
>>
>> Any comments? Lars
>
> Hi Lars,
>
> I'd be very happy to see this function intyegrated into Octave core at some
> point. I have happily used it in many situation and found it very useful.
>
> My answer to you questions:
> * Not all ML fields have to be implemented at first.
> * If you implement MemUsedOctave then MemUsedMATLAB should exist even if
> it's an alias: this will ensure a code that uses this field in ML will work
> in Octave
> * You are free to propose Octave only features. Their usefulness will
> probably be discussed before integration in Octave.
>
> At some point I can help to integrate the function into Octave code base but
> I think the very first effort is to see how this function can be made
> cross-platform:
> * your version is linux only (BSD? Mac?) and partly ML compatible
> * AFAICS the one in macgyver_utils is cross-platform but is not ML
> compatible and returns different data types depending on the platform.
>
> You should eventually create a patch on Savannah to centralize this effort
> and invite Phillip, Andy and whoever is interested to come and help.
>
> Pantxo
>
>
>
> --
> Sent from: https://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html
>