dynamic loading in static exe

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

dynamic loading in static exe

Paul Kienzle-2
All,

I've been playing with dynamic loading into statically
linked octave.  The attached tarball demonstrates the
case of a dynamically loaded function calling back into
the main exe under mingw.  I know similar things are
possible on IRIX, and I believe they are under Linux.
I don't know about OS/X.

The question is, do we want to support it?  Is there
any advantage to static linking under Windows, other
than fewer files to work with?  

Andy and I have noticed that oct-files are large even
when stripped, but we can get that benefit back for
pre-packaged oct-files by reintroducing  
        --disable-lite-kernel
or at least on windows, by clearing ENABLE_DYNAMIC_LINKING
even if --enable-shared.

Paul Kienzle
[hidden email]

For archiving convenience, here is the Makefile:

%.o: %.c ; $(CC) $(CFLAGS) -c -o $@ $<

# objects included in the exe that
# dl objects can refer to.
OBJS=dynmain.o

# objects included in the exe that
# dl objects cannot refer to.
MAINOBJ=main.o

# dl objects
SO=dynsub.dll

# main program name
PROG=testdyn

all: $(SO) $(PROG).exe

dynsub.dll: dynsub.o lib$(PROG).a
        gcc -shared -Wl,--export-all-symbols -Wl,--enable-auto-import dynsub.o -o dynsub.dll -L. -l$(PROG)

$(PROG).def: $(OBJS)
        # guess the export definitions from the objects
        dlltool --export-all --output-def $(PROG).def $(OBJS)

lib$(PROG).a: $(PROG).def
        # create the import symbols from the export definitions
        dlltool --dllname $(PROG).exe --def $(PROG).def --output-lib lib$(PROG).a

$(PROG).exe: $(MAINOBJ) lib$(PROG).a
        # create the export symbols from the export definitions
        dlltool --dllname $(PROG).exe --output-exp $(PROG).exp --def $(PROG).def
        # create the exe, including the export symbols
        gcc -o $(PROG).exe $(PROG).exp $(MAINOBJ) $(OBJS) -L. -l$(PROG)

clean:
        rm -f *.{o,a,exe,dll,exp,def}


testdyn.tar.gz (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: dynamic loading in static exe

Andy Adler
On Thu, 23 Jan 2003, Paul Kienzle wrote:
> I've been playing with dynamic loading into statically
> linked octave.  The attached tarball demonstrates the
> case of a dynamically loaded function calling back into
> the main exe under mingw.  I know similar things are
> possible on IRIX, and I believe they are under Linux.
> I don't know about OS/X.

This is cool. I knew this was possible under Linux, but
I thought that the broken implementation of dll's under
windows disallowed it.

The advantage of this idea would then be:
    You could build *oct files on a development system,
    and then distribute them to people having a static
    octave build.

I am in favour of this idea. Static octave builds
are very easy to package and distribute for batch
processing tasks.

My only question is: does this process work with the
exact same *oct files as for --enable-shared?

Andy


Reply | Threaded
Open this post in threaded view
|

Re: dynamic loading in static exe

Paul Kienzle-2
On Fri, Jan 24, 2003 at 11:41:33AM -0500, Andy Adler wrote:
> My only question is: does this process work with the
> exact same *oct files as for --enable-shared?

Grrr.. Probably not.  Windows needs to know in advance which
file the symbol is coming from.

This may not be a big problem though if we only had one
octave for windows bundle rather than what we have now.  Is
it 3? 4? 5? different packages?  Could we decide to all work
together on a single distribution hosted at octave-forge?

- Paul


Reply | Threaded
Open this post in threaded view
|

Re: dynamic loading in static exe

Mumit Khan-4
In reply to this post by Paul Kienzle-2
On Thu, 23 Jan 2003, Paul Kienzle wrote:

> I've been playing with dynamic loading into statically
> linked octave.  The attached tarball demonstrates the
> case of a dynamically loaded function calling back into
> the main exe under mingw.  I know similar things are
> possible on IRIX, and I believe they are under Linux.
> I don't know about OS/X.

This is quite a useful tool, expecially when you have "fat"
executables that export lots of useful stuff for other
loadable module style shared libraries to use. I haven't used
it in a while in new code -- I've found easier to just factor
out common code than have to worry about portability. I don't
see an obvious way to do this under OSX. The static linker
does have a way to export selected symbols, but I don't know
what it really does yet.

IMO, this technique is really useful for closed source apps
that deliver as a large executable, and allows the user to
extend it using shared libraries. I used one of these years
ago (it was an EM propagation code, now defunct).

> The question is, do we want to support it?  Is there
> any advantage to static linking under Windows, other
> than fewer files to work with?

Doesn't Octave already support it? At least on most Unix'y
platforms, it uses -rdynamic, the portable name for which
is -E or --export-dynamic, and does what you expect.

I don't think there's much advantage in Octave as the
loadable modules use symbols out of liboctave, libcruft
and not really from Octave executable itself. One reason
that the .oct files are so big under Windows is due to
the lack of shared libstdc++, which is always linked in
each .oct file.

Regards,
Mumit



Reply | Threaded
Open this post in threaded view
|

Re: dynamic loading in static exe

Paul Kienzle
Mumit Khan wrote:

>On Thu, 23 Jan 2003, Paul Kienzle wrote:
>
>  
>
>>The question is, do we want to support it?  Is there
>>any advantage to static linking under Windows, other
>>than fewer files to work with?
>>    
>>
>
>Doesn't Octave already support it? At least on most Unix'y
>platforms, it uses -rdynamic, the portable name for which
>is -E or --export-dynamic, and does what you expect.
>
I haven't tried.  Maybe "./configure --disable-shared --enable-dl " will
do it.

>I don't think there's much advantage in Octave as the
>loadable modules use symbols out of liboctave, libcruft
>and not really from Octave executable itself. One reason
>that the .oct files are so big under Windows is due to
>the lack of shared libstdc++, which is always linked in
>each .oct file.
>
Is there any reason not to make a shared version of libstdc++?  I see that
the symbols are available in the .a file [of course!] --- I will try
today to create
the dll with dlltool.

Paul Kienzle
[hidden email]



Reply | Threaded
Open this post in threaded view
|

Re: dynamic loading in static exe

Paul Kienzle
Paul Kienzle wrote:

> Is there any reason not to make a shared version of libstdc++?  I see
> that
> the symbols are available in the .a file [of course!] --- I will try
> today to create
> the dll with dlltool.

Okay, it didn't work.  The dll was easy to create:

    dlltool --export-all --output-exp libstdc++.exp --output-lib
libstdc++.dll.a /usr/lib/libstdc++.a
    gcc -shared libstdc++.exp /usr/lib/libstdc++.a -o libstdc++.dll
    rm libstdc++.exp

But there were a couple of link errors from the following main:

#include <iostream>
int main(int argc, char *argv[])
{
  std::cout << "hello world" << std::endl;
  return 0;
}

$ gcc hello.o -L. -lstdc++.dll -o direct
fu000001.o: undefined reference to `_libstdc__dll_a_iname'
nmth000000.o: undefined reference to `__nm___ZSt4cout'
Info: resolving std::cout  by linking to __imp___ZSt4cout (auto-import)
collect2: ld returned 1 exit status

Any suggestions?

Paul Kienzle
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: dynamic loading in static exe

Mumit Khan-4
On Mon, 27 Jan 2003, Paul Kienzle wrote:

> Okay, it didn't work.  The dll was easy to create:
>
>     dlltool --export-all --output-exp libstdc++.exp --output-lib
> libstdc++.dll.a /usr/lib/libstdc++.a
>     gcc -shared libstdc++.exp /usr/lib/libstdc++.a -o libstdc++.dll
>     rm libstdc++.exp

That doesn't really do anything (``objdump -p libstdc++.dll'' will show
what I mean), you'll need to either use -Wl,--whole-archive, which will
cause you other problems, or extract all the objects from static libstdc++
and build a DLL from those.

  $ mkdir tmp
  $ (cd tmp; ar x /usr/lib/libstdc++.a)
  $ gcc -shared -o libstdc++.dll tmp/*.o

  $ cat testdll.cc
  #include <iostream>
  int main() { std::cout << "Hello world" << std::endl; }

  $ gcc -o testdll testdll.cc libstdc++.dll
  (or)
  $ gcc -o testdll testdll.cc -L. -lstdc++

Chances are the that ./testdll will crash as some of the static objects
needed by libstdc++ won't be created correctly. You'd also need a "init"
routine for the DLL (typically called DllMain) that will initialize all
the static globals such as std::ios_base::ioinit. It's been too long since
I'd looked at these issues, so this is a good to stop before spreading
misinformation. There were people in Mingw list who have looked at it,
and they might help things along (Danny Smith has been there, done that,
etc).

Regards,
Mumit



Reply | Threaded
Open this post in threaded view
|

Re: dynamic loading in static exe

Paul Kienzle
Mumit Khan wrote:

>On Mon, 27 Jan 2003, Paul Kienzle wrote:
>
>  
>
>>Okay, it didn't work.  The dll was easy to create:
>>
>>    dlltool --export-all --output-exp libstdc++.exp --output-lib
>>libstdc++.dll.a /usr/lib/libstdc++.a
>>    gcc -shared libstdc++.exp /usr/lib/libstdc++.a -o libstdc++.dll
>>    rm libstdc++.exp
>>    
>>
>
>That doesn't really do anything
>
I haven't run objdump yet, but I was able to compile and run the
following hello world program
(only 27k, rather than the 450k I get from static linking):

#include <iostream>
namespace std { __declspec(dllimport) ostream cout; } ;
int main(int argc, char *argv[])
{
    std::cout << "Hello, world!" << std::endl;
    return 0;
}

I used the following compile/link:

    g++ -c hello.cc
    gcc hello.o -Wl,--enable-auto-import -L. -lstdc++.dll

There are still problems since __declspec(dllimport) in multiple C++ modules
causes link problems (even if they are referring to different symbols).  
I sent
a message to the gcc list about it.

I haven't tried compiling oct files with it yet --- I need a faster
compiler or
more time before I can do that.

>(``objdump -p libstdc++.dll'' will show
>what I mean), you'll need to either use -Wl,--whole-archive, which will
>cause you other problems, or extract all the objects from static libstdc++
>and build a DLL from those.
>
It hasn't been all problem yet, but my test examples have been very small.

Thanks for the heads up though.  I will send the gcc message to the
mingw list.

Paul Kienzle
[hidden email]