octave hangs when starting from a directory with too many files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

octave hangs when starting from a directory with too many files

Terikin
I do some work with computational clusters (currently running a mix of Ubuntu 14.04 / Octave 4.0.0 and Ubuntu 16.04 / Octave 4.2.1), and I made a mistake (with a bash script) that produced around 150000 small files in a particular directory "BADDIR" while running over the course of a couple days.  If I called 'ls' on BADDIR, it took somewhere between 15 and 60 minutes to run (the folder is accessed over a 10Gbps network, but there's still some latency).  If I tried to start octave with BADDIR as the current directory, then octave would hang - at least for 10 minutes; I don't think I tried waiting any longer than that when running it interactively.  However, I could start octave from a different directory and then run files from the problematic directory without any delays or problems.  I've since removed all the extraneous files and can work normally again.

Is what I describe expected behavior?  I'm guessing that octave runs some operation similar to 'ls' when starting, perhaps to check for startup.m or .octaverc?  Can that be done more efficiently?

I came here for comments and to report what I found (in case someone else has the same problem), since I wasn't sure whether to really consider it to be a bug.


Reply | Threaded
Open this post in threaded view
|

Re: octave hangs when starting from a directory with too many files

Mike Miller-4
On Fri, Jul 20, 2018 at 11:51:33 -0400, Clinton S wrote:

> I do some work with computational clusters (currently running a mix of
> Ubuntu 14.04 / Octave 4.0.0 and Ubuntu 16.04 / Octave 4.2.1), and I made a
> mistake (with a bash script) that produced around 150000 small files in a
> particular directory "BADDIR" while running over the course of a couple
> days.  If I called 'ls' on BADDIR, it took somewhere between 15 and 60
> minutes to run (the folder is accessed over a 10Gbps network, but there's
> still some latency).  If I tried to start octave with BADDIR as the current
> directory, then octave would hang - at least for 10 minutes; I don't think
> I tried waiting any longer than that when running it interactively.
> However, I could start octave from a different directory and then run files
> from the problematic directory without any delays or problems.  I've since
> removed all the extraneous files and can work normally again.
>
> Is what I describe expected behavior?  I'm guessing that octave runs some
> operation similar to 'ls' when starting, perhaps to check for startup.m or
> .octaverc?  Can that be done more efficiently?
Yes, you're close. The current directory is always in Octave's load
path, meaning any .m files in the current directory can be called as
scripts or functions. All callable files in the load path are scanned by
Octave and cached internally.

I approximated your setup and got the same behavior. When I inspect
Octave with gdb, sure enough it is in the routine that initializes the
load path, which iterates over a list of the files in the current
directory.

With my setup, it took about 9 minutes for 'ls' to complete, and it took
about the same amount of time for 'octave-cli' to start and show the
prompt.

It may be possible to optimize this better for directories with
thousands of files, none of which are .m files that Octave needs to know
about. For example, in the same directory where 'ls' takes 9 minutes,
the command 'ls *.m' takes only 25 seconds to return an error.

--
mike



signature.asc (849 bytes) Download Attachment