From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 44652 invoked by alias); 1 Jun 2016 20:18:06 -0000 Mailing-List: contact cygwin-developers-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-developers-owner@cygwin.com Mail-Followup-To: cygwin-developers@cygwin.com Received: (qmail 44325 invoked by uid 89); 1 Jun 2016 20:18:05 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-96.2 required=5.0 tests=AWL,BAYES_00,GOOD_FROM_CORINNA_CYGWIN,KAM_LAZY_DOMAIN_SECURITY,RCVD_IN_PBL,RCVD_IN_SORBS_DUL,RDNS_DYNAMIC autolearn=ham version=3.3.2 spammy=loader, UD:7.dll, H*Ad:U*cygwin-developers, management X-HELO: calimero.vinschen.de Received: from ipbcc0227e.dynamic.kabel-deutschland.de (HELO calimero.vinschen.de) (188.192.34.126) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 01 Jun 2016 20:17:50 +0000 Received: by calimero.vinschen.de (Postfix, from userid 500) id 7772DA805D3; Wed, 1 Jun 2016 22:17:48 +0200 (CEST) Date: Wed, 01 Jun 2016 20:18:00 -0000 From: Corinna Vinschen To: cygwin-developers@cygwin.com Subject: Re: About the dll search algorithm of dlopen Message-ID: <20160601201748.GI11431@calimero.vinschen.de> Reply-To: cygwin-developers@cygwin.com Mail-Followup-To: cygwin-developers@cygwin.com References: <574E835E.7090109@ssi-schaefer.com> <20160601110947.GE11431@calimero.vinschen.de> <574EF07B.1060806@ssi-schaefer.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="7uYPyRQQ5N0D02nI" Content-Disposition: inline In-Reply-To: <574EF07B.1060806@ssi-schaefer.com> User-Agent: Mutt/1.6.1 (2016-04-27) X-SW-Source: 2016-06/txt/msg00003.txt.bz2 --7uYPyRQQ5N0D02nI Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-length: 5509 On Jun 1 16:26, Michael Haubenwallner wrote: > On 06/01/2016 01:09 PM, Corinna Vinschen wrote: > > On Jun 1 08:40, Michael Haubenwallner wrote: > >> Hi, > >> > >> two issues with dlopen here (I'm about to prepare patches): > >> > >> *) The algorithm to combine dll file name variants with the search path > >> entries needs to be reordered, as in: > >> - for each dll file name variant: > >> - for each search path: > >> + for each search path entry: > >> + for each dll file name variant: > >> check if useable > >=20 > > Rationale? We only need to find one version of the file and there > > usually only is one. This is mainly for moduled systems like perl, > > python, etc. >=20 > While I indeed didn't face a problem here yet, the rationale behind is > that I need to install a large application that provides its own (portabl= e) > package build & management system, including lots of packages probably in= stalled > in the host (Cygwin) system as well, most likely in (slightly) different = versions. >=20 > When these package management system now does provide a different dll nam= ing > scheme than the host system, like stick to "liblib.dll" rather than "cygl= ib.dll", > and the application wants to dlopen its own "liblib.dll", currently the h= ost's > "cyglib.dll" is loaded. Sounds a teeny bit artificial, but basically that's possible, yes. > > However, if you can speed up the search process ignore the > > question... >=20 > This also depends on whether find_exec really is necessary here. Not as such necessary, it's just the function used to search in a search path. If you want to change that you have to rewrite the same logic again, just reversed. One way around YA code duplication could be some kind of path iterator class which could be used from find_exec as well as from get_full_path_of_dll. > >> *) The directory of the current main executable should be searched > >> after LD_LIBRARY_PATH and before /usr/bin:/usr/lib. > >> And PATH should be searched before /usr/bin:/usr/lib as well. > >=20 > > Checking the executable path and $PATH are Windows concepts. dlopen > > doesn't do that on POSIX systems and we're not doing that either. >=20 > Agreed, but POSIX also does have the concept of embedded RUNPATH, > which is completely missing in Cygwin as far as I can see. RPATH and RUNPATH are ELF dynamic loader features, not supported by PE/COFF. > However, there is one path name that can easily serve as minimal > "embedded RUNPATH" - the executable's directory. >=20 > This is where I do have a problem right now: >=20 > My own /application/bin/python2.7.exe is linked to libpython2.7.dll, > located in /application/bin. Now there is some python script that does > have some - strange enough - cygwin-conditional code that reads: >=20 > import _ctypes > _ctypes.dlopen("libpython%d.%d.dll" % sys.version_info[:2]) >=20 > While this is questionable by itself, it really shouldn't load another > libpython2.7.dll than /application/bin/python2.7.exe has already loaded > just because dlopen using a different search algorithm than CreateProcess= (). >=20 > However, when dlopen is about to search some path list, I can imagine to > search the list of already loaded dlls first as well, but I'd prefer to > leave this up to LoadLibrary... This problem would only occur if dlopen is not called with a path. If the given pathname is a plain filename, we could simply add a call to GetModuleHandle and if it succeeds, return the handle (after checking for RTLD_NODELETE). > > Having said that, LoadLibrary will search the usual paths. After 2.5.2, > > we're leaving XP/2003 behind, and then we probably should tighten the > > search algorithm along the lines of > >=20 > > AddDllDirectory ("/usr/bin"); > > AddDllDirectory ("/usr/lib"); > > [...] > > LoadLibraryEx (path, NULL, LOAD_LIBRARY_SEARCH_USER_DIRS > > | LOAD_LIBRARY_SEARCH_SYSTEM32); >=20 > /me fails to see how this does help with the missing embedded RUNPATH. It doesn't. It just tightens the search path to not load from the cwd or the application path. If you want that, add it to LD_LIBRARY_PATH explicitely. > >> For consistency, IMO, when any searched path ends in either > >> x/bin or x/lib, we should search x/bin:x/lib. > >=20 > > This might make sense, at least in the direction lib->bin. >=20 > Fine with me too. >=20 > Side note: > We also use some cl.exe/link.exe wrapper that supports LD_PRELOAD, > LD_LIBRARY_PATH, embedded RUNPATH, as well as lazy loading for both > LoadLibrary and CreateProcess: https://github.com/haubi/parity > Basically I'm wondering why Cygwin doesn't provide that (yet?)... We discussed implementing our own dynamic loader once, but gave up due to workload. Parity is LGPLed and thus can't be included into Cygwin itself. DT_RUNTIME should be possible with not too much effort, but would require gcc/binutils support. Some tricking with a symbol in the linker script, setting a pointer to it in _cygwin_crt0_common, and a matching call to AddDllDirectory comes to mind... LD_PRELOAD is (kind of) implemented but I think doesn't work as intended. Importing symbols is bound to the name of the DLL they came from in a PE/COFF file. To implement the full set of ELF dynloader features would require some major effort, like what parity does. Corinna --=20 Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat --7uYPyRQQ5N0D02nI Content-Type: application/pgp-signature; name="signature.asc" Content-length: 819 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJXT0LsAAoJEPU2Bp2uRE+gWR4QAKYmQQRJ9JfuVq8jDz4bcJrk yqSnoebBhGcjYx2r3B5NLZxWPYojgkS2pR+chszdx/H43fCXvcCsF0JDhkXi0fa5 hj9hdwrM6knOWbooLMUHcWkD3bGzYrDl1jXiX/lr1UK2PErImfbIcNS7fyA0ntuQ QcWYbGs3ubuRsFLPstl0PUZESTrFvmqMVRecWWODTCCdrdmvx5AgjEK/rVknYWej 67DcgjQHpBKZHkdwUmSQkpmWYx5qHfBS7czFg7ohhdFuYpAlon7R4tfTHxwmuFl+ JxlnlBenC/Z/WeCTvxU5X6snBkmxbKcZWvNKS8n/Vd+KUkxzYQRUOUQmvN0nbBd7 QttKcvCBgoVSuvem3XcHJQlGkyd3E90/JImyB8TZhvyHBMKpsx0Qax0hs+nrn/jE FzK8bzAhaj3OS/fzJOnRlZ96oXgsBHHAEEw7WV4ImU0DyJ+vWyJRlFmOK2rBjPJa 3y8RXo7kyne9h+Dc+eem3dKueDFpZqCdEDHu5fItzf0VfpDVikAuMCBuvs14DGyA RhHh8qIkdKIPauzmMn4FW3TtjkHd9A07RKybQ/jvvm/eJDYQ6WU4hh6vdJpc3lip Kzpf8u0+6SG2vyHsjqGzESC+zqrD0XP1gZg6umIpywzt+HcuT0hu1o3W5yIJ3VI7 vT1whg14kEgKHsLWSrK+ =SSQa -----END PGP SIGNATURE----- --7uYPyRQQ5N0D02nI--