Re: Problem with differences with DLOPEN / DLSYM compared to ubuntu (16.04) / debian (stretch).

Re: Problem with differences with DLOPEN / DLSYM compared to ubuntu (16.04) / debian (stretch).

[Kaz Kylheku]

On 14.09.2017 08:44, Gary Schneir wrote:
> Can anyone provide some assistance to understanding the differences and
> a way get a single code base to work in all three environments without
> naming the library file?

On Cygwin, libraries are Windows DLL's. That appears to be a
conscious project decision.

The TL;DR is that Windows DLL's are not going to give you the behaviors 
of ELF
shared libraries on GNU/Linux. Cygwin would probably have to whip up its 
own
shared library format for that, or port one.

The complication doesn't seem worth it.

You can think of dlopen as a wrapper around LoadLibrary, and dlsym doing 
GetProcAddress.

In Windows programming, doing GetProcAddress on symbols within an .exe 
is something
that isn't normally done; it's not clear that it is supposed to work at 
all.

For it to even have any hope of working, the function would have to be 
somehow
declared "dllexport" (in Visual C terms).

I think the shared lib building mechanism under Cygwin hides that; there 
is no
Visual C __declspec(dllexport) stuff:

See: https://cygwin.com/cygwin-ug-net/dll.html

Still, the toolchain has to be arranging for the equivalent of 
__declspec(dllexport)
for the external symbols.

It's probably not configured to do that for the symbols in a regular 
executable.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with differences with DLOPEN / DLSYM compared to ubuntu (16.04) / debian (stretch).

[Gary Schneir]

Thanks for the response but I am a little confused by it.  If Cygwin is
supposed to provide POSIX API functionality and DLOPEN / DLSYM are
supported in CYGWIN, then I shouldn't care about the underlying
complexity or restrictions of running within the Windows environment and
using DLLs.  The behavior should be the same as in other POSIX environments.


If you are saying that I did not include some sort of
__declspec(dllexport) directive in my code so that it can find my
symbols, that is something else but you indicate that you think cygwin
hides that complexity in shared libraries.


Gary


On 09/14/2017 06:15 PM, Kaz Kylheku wrote:
> On 14.09.2017 08:44, Gary Schneir wrote:
>> Can anyone provide some assistance to understanding the differences and
>> a way get a single code base to work in all three environments without
>> naming the library file?
>
> On Cygwin, libraries are Windows DLL's. That appears to be a
> conscious project decision.
>
> The TL;DR is that Windows DLL's are not going to give you the
> behaviors of ELF
> shared libraries on GNU/Linux. Cygwin would probably have to whip up
> its own
> shared library format for that, or port one.
>
> The complication doesn't seem worth it.
>
> You can think of dlopen as a wrapper around LoadLibrary, and dlsym
> doing GetProcAddress.
>
> In Windows programming, doing GetProcAddress on symbols within an .exe
> is something
> that isn't normally done; it's not clear that it is supposed to work
> at all.
>
> For it to even have any hope of working, the function would have to be
> somehow
> declared "dllexport" (in Visual C terms).
>
> I think the shared lib building mechanism under Cygwin hides that;
> there is no
> Visual C __declspec(dllexport) stuff:
>
> See: https://cygwin.com/cygwin-ug-net/dll.html
>
> Still, the toolchain has to be arranging for the equivalent of
> __declspec(dllexport)
> for the external symbols.
>
> It's probably not configured to do that for the symbols in a regular
> executable.
>
>
>
> -- 
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with differences with DLOPEN / DLSYM compared to ubuntu (16.04) / debian (stretch).

[cyg Simple]

Please consider using an interleaving method of posting on this list.
Top posting is considered rude.

On 9/15/2017 9:51 AM, Gary Schneir wrote:
> Thanks for the response but I am a little confused by it.  If Cygwin is
> supposed to provide POSIX API functionality and DLOPEN / DLSYM are
> supported in CYGWIN, then I shouldn't care about the underlying
> complexity or restrictions of running within the Windows environment and
> using DLLs.  The behavior should be the same as in other POSIX environments.
> 

You presented your case well and I was waiting on someone familiar with
the code to respond.  I'm not sure that would be Kaz, he was just trying
to be helpful from his experiences.  I agree with your surmise that
Cygwin should perform similar results as Linux in this case.

> 
> If you are saying that I did not include some sort of
> __declspec(dllexport) directive in my code so that it can find my
> symbols, that is something else but you indicate that you think cygwin
> hides that complexity in shared libraries.
> 

Actually it would be binutils, regardless of Cygwin or MinGW, that is
trying to hide the complexity of needing to supply the
__declspec([export|import]) declarations.  The logic for that is a bit
confusing but if none is given then all symbols are exported.

-- 
cyg Simple

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with differences with DLOPEN / DLSYM compared to ubuntu (16.04) / debian (stretch).

[Jon Turney]

On 15/09/2017 17:07, cyg Simple wrote:
> Please consider using an interleaving method of posting on this list.
> Top posting is considered rude.
> 
> On 9/15/2017 9:51 AM, Gary Schneir wrote:
>> Thanks for the response but I am a little confused by it.  If Cygwin is
>> supposed to provide POSIX API functionality and DLOPEN / DLSYM are
>> supported in CYGWIN, then I shouldn't care about the underlying
>> complexity or restrictions of running within the Windows environment and
>> using DLLs.  The behavior should be the same as in other POSIX environments.
> 
> You presented your case well and I was waiting on someone familiar with
> the code to respond.  I'm not sure that would be Kaz, he was just trying
> to be helpful from his experiences.  I agree with your surmise that
> Cygwin should perform similar results as Linux in this case.

...

>> If you are saying that I did not include some sort of
>> __declspec(dllexport) directive in my code so that it can find my
>> symbols, that is something else but you indicate that you think cygwin
>> hides that complexity in shared libraries.
> 
> Actually it would be binutils, regardless of Cygwin or MinGW, that is
> trying to hide the complexity of needing to supply the
> __declspec([export|import]) declarations.  The logic for that is a bit
> confusing but if none is given then all symbols are exported.

You need to decorate the symbols you wish to be visible with 
'__attribute__ ((dllexport))' or '__declspec(dllexport)' (MSVC syntax 
which is also supported by gcc)

See [1] for an example of this done portably

[1] https://gcc.gnu.org/wiki/Visibility

Alternatively, you can use the ld flag --export-all-symbols (cf. with 
the ELF option --export-dynamic, which I think you must be using to get 
the observed behaviour on linux) to make all symbols visible.

Taking your example, and making it compilable:

$ cat dlopen.cc
#include <iostream>
#include <memory>
#include <dlfcn.h>

void * handle, * symbol;
const char * errorStr;

int main()
{
   /* get access to the executable's symbol table */
   handle = dlopen(NULL, RTLD_LAZY);
   errorStr = dlerror();

   if (errorStr)
     {
       std::clog << "dlopen error '" << errorStr << "'" << std::endl;
     }
   if (handle)
     {
       std::clog << "handle ok " << std::endl;
     }
   else
     {
       std::clog << "handle NULL " << std::endl;
     }

   /* look up the from_string function */
   symbol = dlsym(handle, "functionname");
   errorStr = dlerror();

   if (symbol)
     {
       std::clog << "dlsym symbol ok " << std::endl;
     }
   else
     {
       std::clog << "dlsym symbol NULL " << std::endl;
     }
   if (errorStr)
     {
       std::clog << "dlsym error '" << errorStr << "'" << std::endl;
     }
}

extern "C" __attribute__ ((dllexport))
void functionname()
{
}

$ g++ dlopen.cc -o dlopen

$ ./dlopen
handle ok
dlsym symbol ok


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with differences with DLOPEN / DLSYM compared to ubuntu (16.04) / debian (stretch).

[Kaz Kylheku]

On 15.09.2017 06:51, Gary Schneir wrote:
> Thanks for the response but I am a little confused by it.  If Cygwin is
> supposed to provide POSIX API functionality and DLOPEN / DLSYM are
> supported in CYGWIN, then I shouldn't care about the underlying
> complexity or restrictions of running within the Windows environment 
> and
> using DLLs.  The behavior should be the same as in other POSIX 
> environments.

I don't see this behavior on Linux.

This is a transcript from an Ubuntu 16.04 system:

$ txr
This is the TXR Lisp interactive listener of TXR 186.
Quit with :quit or Ctrl-D on empty line. Ctrl-X ? for cheatsheet.
1> (dlopen nil)
#<cptr dlhandle: 480918>
2> (dlsym *1 "cons")
#<cptr dlsym: 0>
3> (dlsym *1 "car")
#<cptr dlsym: 0>
4> (dlsym *1 "cdr")
#<cptr dlsym: 0>
5> (dlsym *1 "malloc")
#<cptr dlsym: b3fd70>
6> (dlsym *1 "printf")
#<cptr dlsym: b12ed0>

The external functions cons, car and cdr in the txr executable
(I assure everyone they are there) cannot be found by dlsym,
but malloc and printf in the linked C library are found.

I.e. the same issue you're having on Cygwin.

Cygwin aims for Linux compatibility more than POSIX. The motto
is "get that Linux feeling on Windows", after all.
It's splashed right below the Cygwin banner here:
https://cygwin.com/

Not finding the executable's own symbol with dlsym (unless
some special arrangements are made to dynamically export
the symbols) looks very much like a "Linux feeling"!

If we run "nm -D" on the executable to see what dynamic symbols
are provided, we find only "_init" and "_fini":

$ nm -D /usr/local/bin/txr | grep T
080fa3cc T _fini
0804a7a4 T _init

and, by golly, these *can* be found with dlsym on the
dlopen(NULL, ...) handle!

7> (dlsym *1 "_fini")
#<cptr dlsym: 80fa3cc>
8> (dlsym *1 "_init")
#<cptr dlsym: 804a7a4>


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with differences with DLOPEN / DLSYM compared to ubuntu (16.04) / debian (stretch).

[Brian Inglis]

On 2017-09-15 10:59, Jon Turney wrote:
> On 15/09/2017 17:07, cyg Simple wrote:
>> Please consider using an interleaving method of posting on this list.
>> Top posting is considered rude.
>>
>> On 9/15/2017 9:51 AM, Gary Schneir wrote:
>>> Thanks for the response but I am a little confused by it.  If Cygwin is
>>> supposed to provide POSIX API functionality and DLOPEN / DLSYM are
>>> supported in CYGWIN, then I shouldn't care about the underlying
>>> complexity or restrictions of running within the Windows environment and
>>> using DLLs.  The behavior should be the same as in other POSIX environments.
>>
>> You presented your case well and I was waiting on someone familiar with
>> the code to respond.  I'm not sure that would be Kaz, he was just trying
>> to be helpful from his experiences.  I agree with your surmise that
>> Cygwin should perform similar results as Linux in this case.
> 
> ...
> 
>>> If you are saying that I did not include some sort of
>>> __declspec(dllexport) directive in my code so that it can find my
>>> symbols, that is something else but you indicate that you think cygwin
>>> hides that complexity in shared libraries.
>>
>> Actually it would be binutils, regardless of Cygwin or MinGW, that is
>> trying to hide the complexity of needing to supply the
>> __declspec([export|import]) declarations.  The logic for that is a bit
>> confusing but if none is given then all symbols are exported.
> 
> You need to decorate the symbols you wish to be visible with '__attribute__
> ((dllexport))' or '__declspec(dllexport)' (MSVC syntax which is also supported
> by gcc)
> 
> See [1] for an example of this done portably
> 
> [1] https://gcc.gnu.org/wiki/Visibility
> 
> Alternatively, you can use the ld flag --export-all-symbols (cf. with the ELF
> option --export-dynamic, which I think you must be using to get the observed
> behaviour on linux) to make all symbols visible.
> 
> Taking your example, and making it compilable:
> 
> $ cat dlopen.cc
> #include <iostream>
> #include <memory>
> #include <dlfcn.h>
> 
> void * handle, * symbol;
> const char * errorStr;
> 
> int main()
> {
>   /* get access to the executable's symbol table */
>   handle = dlopen(NULL, RTLD_LAZY);
>   errorStr = dlerror();
> 
>   if (errorStr)
>     {
>       std::clog << "dlopen error '" << errorStr << "'" << std::endl;
>     }
>   if (handle)
>     {
>       std::clog << "handle ok " << std::endl;
>     }
>   else
>     {
>       std::clog << "handle NULL " << std::endl;
>     }
> 
>   /* look up the from_string function */
>   symbol = dlsym(handle, "functionname");
>   errorStr = dlerror();
> 
>   if (symbol)
>     {
>       std::clog << "dlsym symbol ok " << std::endl;
>     }
>   else
>     {
>       std::clog << "dlsym symbol NULL " << std::endl;
>     }
>   if (errorStr)
>     {
>       std::clog << "dlsym error '" << errorStr << "'" << std::endl;
>     }
> }
> 
> extern "C" __attribute__ ((dllexport))
> void functionname()
> {
> }
> 
> $ g++ dlopen.cc -o dlopen
> 
> $ ./dlopen
> handle ok
> dlsym symbol ok

No really comparable as the OP example was in a .dll/.so.
You would have to make your main e.g test, build into a dll, and call from a
separate main program.

The issue appears to be that dlopen( NULL, ...) should work as if it was a
reference to the dll containing the call, not the main program.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Problem with differences with DLOPEN / DLSYM compared to ubuntu (16.04) / debian (stretch).

[Gary Schneir]

I am finding a behavior difference with DLOPEN / DLSYM compared to
ubuntu (16.04) and debian (stretch), specifically when the DLOPEN is
passed NULL for the filename.

I have a shared library (.so) file that contains some functions that I
need to location by name.
The code executing this is within the same .so file.  I am trying to
avoid naming the file in the DLOPEN statement.
According to the documentation, the NULL should indicate the "main
program".  It is unclear if this is just the executable, the executable
and its associated libraries, or the library that is running this
function.  Ubuntu and Debian, both seem to treat it as able to find
within the shared library  Regardless of this, there are still
differences / issues, if the filename happens to be invalid.



ubuntu
   $ uname -a
   Linux vm 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC
2017 x86_64 x86_64 x86_64 GNU/Linux

   $ g++ --version
   g++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609

cygwin
   $ uname -a
   CYGWIN_NT-10.0 DESKTOP-FCHFNAG 2.9.0(0.318/5/3) 2017-09-08 10:19
x86_64 Cygwin

   $ g++ --version
   g++ (GCC) 6.4.0

debian
   $ uname -a
   Linux debian 4.9.0-2-amd64 #1 SMP Debian 4.9.18-1 (2017-03-30) x86_64
GNU/Linux

   $ g++ --version
   g++ (Debian 6.3.0-18) 6.3.0 20170516



LD_LIBRARY_PATH is set the same in all three environments.

===========================

I have the following code in both ubuntu

    std::unique_ptr<const myobject> myobject;
    void * handle, * symbol;
    const char * errorStr;

    /* get access to the executable's symbol table */
    handle = dlopen(NULL, RTLD_LAZY);
    errorStr = dlerror();

if (errorStr)
{
    std::clog << " dlopen error '" << errorStr << "'" << std::endl;
}
if (handle)
{
    std::clog << " handle ok " << std::endl;
}
else
{
    std::clog << " handle NULL " << std::endl;
}

    /* look up the from_string function */
    symbol = dlsym(handle, "functionname");
    errorStr = dlerror();

    if (symbol)
    {
        std::clog << "dlsym symbol ok " << std::endl;
    }
    else
    {
        std::clog << "dlsym symbol NULL " << std::endl;
    }
    if (errorStr)
    {
        std::clog << "dlsym error '" << errorStr << "'" << std::endl;
    }
    /*
     * if the associated function is found, the string is parsed.
     * if not, then a nullptr is returned to the caller.
     */
    if (symbol)
    {
        std::clog << " calling symbol " << std::endl;
        myobject = (*(from_string_type *)symbol)(str);
    }
.....

================================


If I pass a NULL parameter to DLOPEN - different behavior

handle = dlopen(NULL, RTLD_LAZY)

under ubuntu
 handle ok
 dlsym symbol ok
 calling symbol

>> Correctly finds symbol and works correctly.

under cygwin
 handle ok
 dlsym symbol NULL
 dlsym error 'No such process'

>> Cannot find symbol and therefore fails


under debian
 handle ok
 dlsym symbol ok
 calling symbol

>> Correctly finds symbol and works correctly.

================================

If I pass an invalid library name parameter to DLOPEN - different behvior

handle = dlopen("invalidname", RTLD_LAZY)


On ubuntu,
 dlopen error 'invalidname: cannot open shared object file: No such file
or directory'
 handle NULL
 dlsym symbol ok
 calling symbol
 Segmentation fault (core dumped)

>> Symbol was not correct, even though it was not null an no error was
returned

on cygwin,
 dlopen error 'No such file or directory'
 handle NULL
 dlsym symbol ok
 calling symbol

>> Correctly finds symbol and works correctly.

On debian,
 dlopen error 'invalidname: cannot open shared object file: No such file
or directory'
 handle NULL
 dlsym symbol ok
 calling symbol
 Segmentation fault

>> Symbol was not correct, even though it was not null an no error was
returned

================================

If I pass an correct library name parameter to DLOPEN - same behvior

handle = dlopen("libmine.so", RTLD_LAZY)
On ubuntu,
 handle ok
 dlsym symbol ok
 calling symbol

>> Correctly finds symbol and works correctly.

on cygwin,
 handle ok
 dlsym symbol ok
 calling symbol

>> Correctly finds symbol and works correctly.

On debian,
 handle ok
 dlsym symbol ok
 calling symbol

>> Correctly finds symbol and works correctly.


Can anyone provide some assistance to understanding the differences and
a way get a single code base to work in all three environments without
naming the library file?


Thanks


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple