Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

[Kaz Kylheku]

Hi all,

I am having trouble linking a "hello world" program cross-compiled
for a mips-linux target. GCC can find the headers properly, but
there is some --sysroot silliness going on: mips-linux-ld ends
up catenating two absolute paths, and concludes that it cannot
find libc.so.6, which is there.

GCC is version 4.1.1
Binutils 2.17
Glibc is CVS head.

When configuring gcc, I used --with-sysroot=/path/to/sysroot,
where glibc is installed under usr/. The header files are
found correctly.

When the linker is invoked, it is given

  --sysroot=/path/to/sysroot.

The linker resolves  -lc  to the absolute name
/path/to/sysroot/usr/lib/libc.so.6. But then it catenates that
to the sysroot, resulting in

  /path/to/sysroot/path/to/sysroot/usr/lib/libc.so.6 which is nonsense.

I can see using strace that it's actually trying to access that path.
And so it dies with "cannot find /path/to/sysroot/lib/libc.so.6
inside /path/to/sysroot".

So, what's the /right/ way to coordinate the location of the
target headers and libraries between gcc and binutils when
cross-compiling?

Thanks.

Re: Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

[Kai Ruottu]

Kaz Kylheku wrote:
> GCC is version 4.1.1
> Binutils 2.17
> Glibc is CVS head.
>
> When configuring gcc, I used --with-sysroot=/path/to/sysroot,
> where glibc is installed under usr/. The header files are
> found correctly.
>   
 What about the binutils, was the same '--with-sysroot='
given in their configure?

 Finding the headers happens with the built-in 'cpp' in the
'cc1', 'cc1plus' etc. real compilers, so only the GCC config
options will have any influence to this.

> When the linker is invoked, it is given
>
>   --sysroot=/path/to/sysroot.
>   
 Maybe this option is aimed for a 'ld' which was NOT configured
using the '--with-sysroot='...
> The linker resolves  -lc  to the absolute name
> /path/to/sysroot/usr/lib/libc.so.6. But then it catenates that
> to the sysroot, resulting in
>
>   /path/to/sysroot/path/to/sysroot/usr/lib/libc.so.6 which is nonsense.
>
> I can see using strace that it's actually trying to access that path.
> And so it dies with "cannot find /path/to/sysroot/lib/libc.so.6
> inside /path/to/sysroot".
>
> So, what's the /right/ way to coordinate the location of the
> target headers and libraries between gcc and binutils when
> cross-compiling?
>   
 My guess after seeing this "behaviour" would be to NOT use the
'--with-sysroot=' in the binutils configure with the newer binutils like
the '2.17'....  The GCC configured with it then giving that necessary
'--sysroot=/path/to/sysroot' to the linker. And the older GCCs which
didn't have the '--with-sysroot=' of course not giving that...

 Meanwhile any binutils configured using '--with-sysroot=', would add
the necessary 'offsets' automatically, so also the older GCCs could be
capable to use the 'sysroot'ed C library...

 I myself haven't used the '--with-sysroot=', only made symlinks like :

     $prefix/$target/include  ->  $sysroot/usr/include
     $prefix/$target/lib  ->  $sysroot/usr/lib

because the 'usr/lib' already has symlinks to '../../lib' for the majority
of its shared '.so' files, only symlinks to the '../../lib/libc.so.6' 
and to the
'../../ld-linux.so.2' (or '../../lib/ld.so.1' or something) must be manually
added just as editing the 'libc.so' script to not have any absolute paths
in its 'libc.so.6 libc_nonshared.a' or something GROUP row...  This has
been the "right" way for me because of using GCCs BEFORE the new
'--with-sysroot=' appeared, and still having those pre-gcc-3.3.3 or what
was the GCC version where this option appeared...  The Robert Heinlein's
"By-Law of Time": "If At Last You Do Succeed, Never Try Again", which
is so common among the most crosscompiler builders - they really seem
to build their crosscompilers only once, or if being obliged to do that
again, then starting everything from scratch again -  is not understood
by me :-(

 Using 5 or so GCC versions with the same $target binutils in $prefix/bin
and in $prefix/$target/bin, would cause me problems if trying to use the
'--with-sysroot'...  So things are easier with using only the idea, not 
really
using the full implementation...

Re: Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

[Kai Ruottu]

Kaz Kylheku kirjoitti:
>> Kai Ruottu wrote: 
>>  What I have thought being the aim in producing a "normal" glibc,
>> one which should work both as a "native" glibc, and as a "cross"
>> glibc, is using the '--prefix=/usr'.
>>     
>
> In my case, my native system is x86, and the cross one is MIPS.
>   
 The "native" platform I meaned of course was the "target"
Linux/MIPS !  How you could think I maybe meaning anything
else?

 When you produce a totally normal glibc for any Linux, it is
always configured using '--prefix=/usr',  whether you build
it natively or cross-building it!  The aim for the build is to
install it later onto the "native" target platform, or otherwise
the build is vain... If the target system already has its original
glibc, then of course that should be used, not trying to "reinvent
the wheel" !  People build only things which are still missing,
and not try to make any duplicates (= fully identical being the
requirement here because of possible remote debugging - all
the memory addresses in the libs on the cross host and on the
target system must be identical).

 The main goal with the '--with-sysroot=' therefore should be
to make this use of identical library stuff on both the cross host
and on the target system, as easy as possible !  That "alien"
stuff cannot be put into any "host" places, but a '$sysroot' like
'/opt/host-mips-linux-gnu' can be used always...

 

RE: Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

[Kaz Kylheku]

> Kai Ruottu wrote: 
>  What I have thought being the aim in producing a "normal" glibc,
> one which should work both as a "native" glibc, and as a "cross"
> glibc, is using the '--prefix=/usr'.

In my case, my native system is x86, and the cross one is MIPS.

Even if the architectures are the same, I don't believe in
mixing target code with the development platform. There is
the potential to screw things up.

If you're cross-compiling, you should run the code on the
target system, or in some isolated environment on the development
system (emulator with support for a different CPU if necessary,
chrooted tree, ...).  With chroot, you can have the library
installed in /lib, exactly as on the target.

There is the problem of sharing the same kernel as the host
system too. On my development system, I might be running some
old kernel, but want a newer one underneath the target glibc.

> 
>      make install_root=$sysroot

I patched glibc's configure.in and config.make to support
--with-sysroot, which sets up that variable at configuration time.

It's surprising that such an important variable is left out from
the configuration.

Re: Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

[Kai Ruottu]

Kai Ruottu wrote :
> If you use the '--prefix=/usr' in your glibc configure and use the 
> suggested
>
>     make install_root=$sysroot
>
> to install it below the $sysroot,

 OOPS, the 'install' was forgotten, the install command of course should 
be :

    make install_root=$sysroot install

Re: Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

[Kai Ruottu]

Kaz Kylheku wrote :
> Solved!
>
> This problem is caused by a miscconfigured glibc build with an incorrect
> --prefix. The absolute paths are picked up from libc.so, which is just a
> linker script generated by glibc.  This linker script must contain paths
> that are correct for the target system system. Of course, those paths
> don't work on the cross-compiling system, which is where the catenation
> with --sysroot comes in.
 What I have thought being the aim in producing a "normal" glibc,
one which should work both as a "native" glibc, and as a "cross"
glibc, is using the '--prefix=/usr'.  Then the install scheme includes
the '.../lib' and the '.../usr/lib'.  In a native install the '.../' is 
the '/'
and in a cross install it is the '$sysroot/'.

 So the resulted 'libc.so' has just the same 'native' like pathnames :

/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-i386)
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a )

and the 'libpthread.so' script too :

/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-i386)
GROUP ( /lib/libpthread.so.0 /usr/lib/libpthread_nonshared.a )

 And because of using the '--with-sysroot=$sysroot', the 'ld' interprets
these absolute paths to mean '$sysroot/lib/...' and '$sysroot/usr/lib/...' !

 Otherwise the '--with-sysroot=' simply doesn't work "as expected"...

 Being able to copy a glibc (plus the other target libraries) between the
native and the cross host(s) in 1-to-1 scheme (just as any other target C
libraries, the proprietary ones don't even have any sources so copying
them is the only possibility...  And of course rebuilding already built libs
for the Linux distros is quite stupid!) should be the motive for using the
'--with-sysroot=' !   Traditionally this hasn't succeeded, fixing some 
things
has been obligatory because the cross GCC scheme knows only one 'lib'.

 If you use the '--prefix=/usr' in your glibc configure and use the 
suggested

     make install_root=$sysroot

to install it below the $sysroot, the glibc should be there waiting that
1-to-1 copying onto the native target system.  And should work in your
crosstoolchain, that configured using the '--with-sysroot=$sysroot' !

RE: Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

[Kaz Kylheku]

Solved!

This problem is caused by a miscconfigured glibc build with an incorrect
--prefix. The absolute paths are picked up from libc.so, which is just a
linker script generated by glibc.  This linker script must contain paths
that are correct for the target system system. Of course, those paths
don't work on the cross-compiling system, which is where the catenation
with --sysroot comes in. And of course install_root must be used in
glibc to avoid actually trying to place files at the fictitious --prefix
location. So install_root must match your --sysroot location, and
--prefix is just / . 

Yesterday, I got a mini cross-compiled Linux system to run on real MIPS
hardware, starting from scratch, with very little hints from anywhere. I
just kept doing the "obvious" steps until I got a shell prompt on the
hardware.

How did I debug the problem? Well, after my silly patch to binutils,
which I can now back out, I got everything to link surely enough. But
with the wrong paths. So when I tried to boot the system, my shiny new
/sbin/init was able to fetch the dynamic linker /lib/ld.so.1, but the
dynamic linker quit, with a message about not being able to find
libc.so.6! What? It's right there!

So then I went into the kernel, and compiled an extra environment
variable into init's env[] array, so that I could set it from the kernel
command line. I was then able to "LD_DEBUG=all" on the kernel command
line! I rebooted the MIPS board and ld.so nicely dumped out to the
serial console that it was trying to use build-system paths to look for
the library rather than the correct target-system paths. I band-aided
that with a symbolic link, and got the system to boot, then worked out
the real solution.

RE: Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

[Kaz Kylheku]

> I wrote previously
> I think I just have to figure out the way to configure gcc to do all
> that exactly as it is doing now, but not generate the 
> --sysroot option.
> I really want to avoid patching it.

I hacked up a silly patch in binutils after which it works. I'm now
happily cross-compling core-utils against glibc for MIPS, on an x86 box.
I've looked at older versions of this function and it didn't have the
else. The idea is to try the catenated name with sysroot, and if that
doesn't work, try the uncatenated name.

I really hate this "solution", but it gets me past a hurdle. I already
have a patched kernel and a patched glibc, so what the heck, right?

I think ld expects gcc to feed it absolute paths which are to be
interpreted relative to the sysroot. But that isn't what gcc is doing:
it's feeding absolute paths from the real system root. It would be nice
if I could figure out how to get gcc to feed absolute paths to collect
which /require/ the root to be tacked on. E.g. /usr/lib/crt<whatever>.o
which would be combined with /path/to/sysroot to create
/path/to/sysroot/usr/lib/crt<whatever>.o.

Ah well.

--- binutils-2.17.orig/ld/ldfile.c      2005-05-12 03:32:02.000000000
-0400
+++ binutils-2.17/ld/ldfile.c   2006-08-24 20:38:38.335091736 -0400
@@ -317,7 +317,8 @@
            }
          free (name);
        }
-      else if (ldfile_try_open_bfd (entry->filename, entry))
+
+      if (ldfile_try_open_bfd (entry->filename, entry))
        {
          entry->sysrooted = IS_ABSOLUTE_PATH (entry->filename)
            && is_sysrooted_pathname (entry->filename, TRUE);

 

RE: Gcc+binutils+libc cross-compiling: path to libc.so.6 malformed in "ld".

[Kaz Kylheku]

Kai Ruottu wrote:
> Kaz Kylheku wrote:
> > GCC is version 4.1.1
> > Binutils 2.17
> > Glibc is CVS head.
> >
> > When configuring gcc, I used --with-sysroot=/path/to/sysroot,
> > where glibc is installed under usr/. The header files are
> > found correctly.
> >   
>  What about the binutils, was the same '--with-sysroot='
> given in their configure?

I did rebuild binutils using --with-sysroot, only because the they
complain that they were not built with sysroot support and bail out.

It doesn't matter at all what the --with-sysroot argument is for
configuring binutils. You can even omit it.

If I take the non-working collect2 command line (captured from gcc -v),
and edit out the --sysroot, it links! It's the --sysroot being passed
down to ld that is screwing things up.

The gcc front end is already passing nice absolute path names down for
everything: the crtN modules, the -L option to find libs, etc. 

I think I just have to figure out the way to configure gcc to do all
that exactly as it is doing now, but not generate the --sysroot option.
I really want to avoid patching it.

There is an Autoconf check in gcc to test whether the linker understands
--sysroot. That should have worked the first time when I built gcc using
--with-sysroot with binutils compiled without sysroot. Ah, but the
problem is that the configure script only runs "ld --help" and greps
that for --sysroot. If you build ld without sysroot, it still has that
option listed in its --help output. Oops!

>  Finding the headers happens with the built-in 'cpp' in the
> 'cc1', 'cc1plus' etc. real compilers, so only the GCC config
> options will have any influence to this.
> 
> > When the linker is invoked, it is given
> >
> >   --sysroot=/path/to/sysroot.
> >   
>  Maybe this option is aimed for a 'ld' which was NOT configured
> using the '--with-sysroot='...

Shrug. I don't see the point of giving a sysroot to a tool, and at the
same time, giving it absolute path names to all of the objects and
directories. It would perhaps make sense if the paths were relative to
the root.

>  I myself haven't used the '--with-sysroot=', only made 
> symlinks like :
> 
>      $prefix/$target/include  ->  $sysroot/usr/include
>      $prefix/$target/lib  ->  $sysroot/usr/lib

I thought about symlinks, but the problem is they are too much of a
bandaid.

This stuff is supposed to work, right?

If we can't get gcc to cleanly play along with binutils, where are we?
:)