[app porting to cygwin] Invoking DLL exported routine problem

[app porting to cygwin] Invoking DLL exported routine problem

[Peter Novak]

[-- Attachment #1: Type: text/plain, Size: 6598 bytes --]

Hello,
I am struggling with porting my application from Linux to Cygwin and
as I am not 100% at home in Cygwin regarding DLLs and friends, I hope to
get some advice, or hints here. I apologize in advance for a question
looking like a newbie cry for help with "everything", but I tried my
best to study the scarce resources on magic of building DLLs and loading
them in Cygwin without a result.

In short, I have a program using plug-ins implemented as DLLs providing
a fixed interface in a form of a C++ class and C exported routines. The
problem is that loading the DLL and invoking the exported C functions
works perfectly well when loaded from a simple testing code, however
fails when the same code is embedded within a more complex application
setting. I suspect a corrupted stack, or some improper pointer
conversion which I do not understand. Obviously a return type of the
routine seems to be important somehow...

As I said, my application loads DLL plug-ins. However, as I need several
independent instances of the same DLL to be loaded at the same time
(plug-ins can be loaded several times and serve completely independently
with different data fed to them), for each I fork a separate process
which loads a single plug-in DLL and communicates with the main process
via shared memory and mutex signaling. This all works well on Linux and
I managed to compile my application successfully under cygwin as well
(everything including the shared memory communication works and is
ported correctly). What does not work is invocation of certain functions
exported from the DLL plug-ins. That means that dl_openext and dl_sym
work properly, however invocation of routing inside the DLL makes the
subprocess crash. The same code, when put to a standalone testing
program without subprocesses works however correctly.

The interface of the DLL includes code similar to the following:

extern "C" 
{
	unsigned int get_api_version()
	{
		return 123;
	}

	CInterface* register_api()
	{
		CMyInterface pModule = new CMyInterface;
		return pModule;
	}
}

where CMyInterface is a subclass of a purely abstract (all methods are
virtual) root class CMyInterface. All the following operation the
plug-in is to perform are invoked as method calls on the instance of
CInterface. The pattern is modeled according to DLL C++ API from
http://aegisknight.org/cppinterface.html.

The code which loads the DLL uses libtool's ltdl library and invokes the
two functions looks similar to the following (error checking omitted):

	// initialize ltdl
        lt_dlinit();

        // open the library (jztemplate is the name of my library above)
        lt_dlhandle m_hLibrary = lt_dlopenext("cygjztemplate-0");

        // find the KR module API version routine <-- WORKS WELL
        lt_ptr hVersionFunc = lt_dlsym (m_hLibrary, "get_api_version");

	// Invocation of the 
        std::cout << ((TPtrVerFunc)hVersionFunc)() << std::endl;

	// find the interface loading routine <-- WORKS WELL
        lt_ptr hLoadFunc = lt_dlsym (m_hLibrary, "regsiter_api");

	// get the instance of the interface <-- PROBLEM HERE
	CInterface* pModule = ((TPtrLoadFunc) hLoadFunc)();
	
	//...

Signatures of the DLL exported functions are defined as:

	typedef unsigned int (*TPtrVerFunc)();
	typedef CInterface* (*TPtrLoadFunc)();


PROBLEM:
When the code above is invoked from a simple testing program (that code
is wrapped into a main() function) it works without a problem. If
however, it is invoked from a subprocess of my complex application, it
crashes on invocation of the hLoadFunc. However hVersionFunc is invoked
correctly as many times as I call it.

Finally, let me also list the linking options for the main program,
testing program and the library itself.

*** Main program
g++ -DDEBUG -DLOGGING_ON -Wall -DPACKAGELIBDIR=\"/usr/local/lib/jazzyk\"
-g -O2 -L/usr/lib -o app.exe app.o (other *.o files omitted)
-lboost_program_options-gcc-mt -lltdl


*** Testing program:
g++ Test.cpp -o test.exe -I/usr/include/boost-1_33_1/ -lltdl


*** DLL library (as produced by automake/libtool):
/bin/sh ./libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.  -g
-DJZMODULEDLL   -g -O2 -MT libjztemplate_la-module.lo -MD -MP -MF
.deps/libjztemplate_la-module.Tpo -c -o libjztemplate_la-module.lo `test
-f 'module.cpp' || echo './'`module.cpp

mkdir .libs

g++ -DHAVE_CONFIG_H -I. -g -DJZMODULEDLL -g -O2 -MT
libjztemplate_la-module.lo -MD -MP -MF .deps/libjztemplate_la-module.Tpo
-c module.cpp  -DDLL_EXPORT -DPIC -o .libs/libjztemplate_la-module.o

g++ -DHAVE_CONFIG_H -I. -g -DJZMODULEDLL -g -O2 -MT
libjztemplate_la-module.lo -MD -MP -MF .deps/libjztemplate_la-module.Tpo
-c module.cpp -o libjztemplate_la-module.o >/dev/null 2>&1

mv -f .deps/libjztemplate_la-module.Tpo
.deps/libjztemplate_la-module.Plo

/bin/sh ./libtool --tag=CXX   --mode=link g++  -g -O2 -no-undefined
-version-info 0:0:0  -o libjztemplate.la -rpath /usr/local/lib
libjztemplate_la-module.lo  

g++ -shared -nostdlib   .libs/libjztemplate_la-module.o
-L/usr/lib/gcc/i686-pc-cygwin/3.4.4
-L/usr/lib/gcc/i686-pc-cygwin/3.4.4/../../.. -lstdc++ -lgcc -lcygwin
-luser32 -lkernel32 -ladvapi32 -lshell32 -lgcc   -o
.libs/cygjztemplate-0.dll -Wl,--enable-auto-image-base -Xlinker
--out-implib -Xlinker .libs/libjztemplate.dll.a


For completeness here is the output of nm on the DLL:
65881000 T _get_api_version
65881030 T _register_api

65881380 t __GLOBAL__D_get_api_version
65881370 t __GLOBAL__I_get_api_version

Notice, that for get_api_version, there are also two entries with prefix
__GLOBAL, while for the register_api there's nothing like that. Does
that mean anything? I tried to add other functions to the interface and
invoking such which had only a one entry was also OK. It seems to be
rather the return type which causes problems.

Also note that pointers returned by dl_sym correspond exactly to those
in the first two entries of type "T".

Is there perhaps some issue hidden in the interplay of forked
subprocesses which load DLLs I should know about? Obvioulsy fork on
cygwin is a non-trivial issue itself. And why does libtool generate DLL
which contains for some routines a single entry, but for some several?
Is it significant?

I would be extremely thankful if somebody could direct me to something
suspicious in what I described above, or possibly point out some facts
regarding DLLs in Cygwin which might help me resolve the problem.

Thanks in advance,

Peter Novak.


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Ad: [app porting to cygwin] Invoking DLL exported routine problem - resolution(?)

[Peter Novak]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

Hello,
this is a follow-up to my previous mail. It seems that after all I found
a solution to my problem described in the previous e-mail. However, I do
not fully understand what's going on as it seems to have something to do
with the specification of C++ language and it seems to be a kind of g++
issue.  Therefore, I would be glad if there's somebody who is able to
enlighten me on the details of the issue together with why it works on
Linux with gcc 4.1.3 and not on Cygwin with gcc 3.4.4. I would like to
learn something from this.

What I forgot to mention in my previous e-mail and what turned out to be
crucial, is that my DLL uses std::cout for writing output to the
console. I found, that whenever I have a print to cout in a routine, the
subprocess crashes even *before* entering the routine(!). Struggling
with gdb and multi-threaded application showed that the crash occurred
in an esoteric place of:

#0  0x6526f575 in std::ostream::sentry::sentry ()
    at /usr/lib/gcc/i686-pc-cygwin/3.4.4/include/c++/bits/ostream.tcc:63

The code there lists a remark about supposedly an issue of GLIBC (???):
59   template<typename _CharT, typename _Traits>
60     basic_ostream<_CharT, _Traits>&
61     basic_ostream<_CharT, _Traits>::
62     operator<<(__ostream_type& (*__pf)(__ostream_type&))
63     {
64       // _GLIBCXX_RESOLVE_LIB_DEFECTS
65       // DR 60. What is a formatted input function?
66       // The inserters for manipulators are *not* formatted output functions.
67       return __pf(*this);
68     }

Further research proved fruitful and I found an issue in the GCC
bugzilla speaking about a similar problem:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26123

According to Comment #9
(http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26123#c9) the problem is
with creation of the cout object and can be healed by a creation of a
static instance of std::ios_base::Init in the routine. This indeed
resolved my problem.


Can somebody explain me why this worked the way it did? My guess is that
it has something to do with Linux loading a shared library into the
memory space of the running process, while on Win32 the DLL has it's own
memory space (?)... Anyway, it does not explain why loading DLL from the
main thread was OK, while loading it from a subprocess wasn't (all
components, i.e. main process, subprocess and DLL use cout to print
stuff to console).

Thanks for any explanations.

Best,

Peter.

-- 
peter.novak [at] tu-clausthal.de | http://cig.in.tu-clausthal.de/
Department of Informatics | Clausthal University of Technology
Julius-Albert-Str. 4 | D-38678 Clausthal-Zellerfeld | Germany
Tel +49 5323 72 71 90 | Fax +49 5323 72 71 39 


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Ad: [app porting to cygwin] Invoking DLL exported routine problem - resolution(?)

[Brian Dessent]

Peter Novak wrote:

> What I forgot to mention in my previous e-mail and what turned out to be
> crucial, is that my DLL uses std::cout for writing output to the
> console. I found, that whenever I have a print to cout in a routine, the
> subprocess crashes even *before* entering the routine(!). Struggling
> with gdb and multi-threaded application showed that the crash occurred
> in an esoteric place of:

The reason it crashes before your routine is called is that (assuming
PR26123 is in fact the cause) the problem has to do with the order of
static constructors.  This happens at program startup (for objects in
the .exe) or at DLL load time (for objects in the .dll).

> The code there lists a remark about supposedly an issue of GLIBC (???):
> 59   template<typename _CharT, typename _Traits>
> 60     basic_ostream<_CharT, _Traits>&
> 61     basic_ostream<_CharT, _Traits>::
> 62     operator<<(__ostream_type& (*__pf)(__ostream_type&))
> 63     {
> 64       // _GLIBCXX_RESOLVE_LIB_DEFECTS
> 65       // DR 60. What is a formatted input function?
> 66       // The inserters for manipulators are *not* formatted output functions.
> 67       return __pf(*this);
> 68     }

The symbol/namespace _GLIBCXX in this context refers to gcc's libstdc++,
not glibc, so that's kind of a red herring.  I don't think the above
comment has anything to do with your problem per se, as DR 60 is about
formatted input:
<http://gcc.gnu.org/onlinedocs/libstdc++/ext/lwg-defects.html#60>.

> Can somebody explain me why this worked the way it did? My guess is that
> it has something to do with Linux loading a shared library into the
> memory space of the running process, while on Win32 the DLL has it's own
> memory space (?)...

On both Win32 and Linux, shared libraries are in the same memory address
space as the executable; there's no difference in that respect.

> Anyway, it does not explain why loading DLL from the
> main thread was OK, while loading it from a subprocess wasn't (all
> components, i.e. main process, subprocess and DLL use cout to print
> stuff to console).

Well again, if the bug is in fact related to the order that static
constructors are run, then the difference in behavior could simply be
due to the fact that under the hood PE and ELF use different methods of
dispatching static ctors, so it follows that the order (which is
unspecified) could be volatile.  By explicitly adding a
std::ios_base::Init static object I suppose you are somehow affecting
this order to cause it to be correct.  I can't say for sure why this is,
but it's in general a very tricky aspect of C++, see also:
<http://www.parashift.com/c++-faq-lite/ctors.html#faq-10.12>.

Brian

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/