Re: Binutils + OpenSSL issue

Re: Binutils + OpenSSL issue

[H. J. Lu]

Alan,

I was told your patches

http://sources.redhat.com/ml/binutils/2003-02/msg00026.html
http://sources.redhat.com/ml/binutils/2003-02/msg00027.html

brake Linux/mips. Kumba, can you verify exactly which patch caused
the problem?


H.J.
On Mon, May 19, 2003 at 05:16:59AM -0400, Kumba wrote:
> 
> 	Greetings,
> 
> 	Been trying to chase down a pretty obscure Binutils bug  on the 
> linux/mips platform, thought you might be interested about it.  To 
> summarize, building OpenSSL on Linux/MIPS with any binutils after around 
> Feb 6th, 2003 seems to start "breaking" the library, specifically 
> libcrypto and libssl.
> 
> 	Running the command:
> 
> objdump -R /usr/lib/libcrypto.so.0.9.6 |grep R_MIPS |grep -v 
> \(R_MIPS_32\|R_MIPS_NONE\)
> 
> 	Will produce error messages similar to the following:
> 
> BFD: BFD 2.13.90.0.18 20030121 assertion fail elf32-mips.c:1498
> BFD: usr/lib/libcrypto.so.0.9.6(.rel.dyn): relocation 3755 has invalid 
> symbol index 0
> 
> 	Only replicated many times.  With regards to Gentoo linux, we've seen 
> this behavior since patching a 2.13.90.0.18 binutils with a CVS patch 
> from Fed 6th, 2003.  It's been confirmed in every binutils version there 
> after, even in 2.14.90.0.2.
> 
> 	To actually trigger it, OpenSSL needs to be compiled using -mips3 or 
> higher optimizations (assuming gcc 3.2.x, haven't tested with gcc-3.3 
> yet).  That's when these failed assertions actually start causing 
> problems.  They are still produced in libcrypto and libssl even with the 
> default -mips1 optimizations, just they don't seem to get in the way of 
> things until -mips3 or -mips4.
> 
> 	I've consulted with Thiemo Seufer on this matter, and he's got it 
> pinned down to the patch attached in bzip2 format to this mail.  He 
> thinks the offending code is within it, and it relates to the GOT Table 
> not being initialized under certain circumstances.  I'm not sure if he's 
> working on some fix for it, he seems a busy enough person as is.
> 
> 	If you need any additional information, let me know.  Several gentoo 
> users have built entire systems optimized to -mips4 on Indy machines 
> only to discover OpenSSL becomes non-functional, rendering programs like 
> SSH non-functional as well.  That's when this glitch was first noticed.
> 
> --Kumba

Re: Binutils + OpenSSL issue

[Thiemo Seufer]

H. J. Lu wrote:
[snip]
> > 	I've consulted with Thiemo Seufer on this matter, and he's got it 
> > pinned down to the patch attached in bzip2 format to this mail.

Which was wrong, sorry.

> > He 
> > thinks the offending code is within it, and it relates to the GOT Table 
> > not being initialized under certain circumstances.  I'm not sure if he's 
> > working on some fix for it, he seems a busy enough person as is.

It was Alexandre Oliva's patch to introduce multi-GOT which triggers
the problem, specifically elfxx-mips.c, CVS version 1.38 -> 1.39.


Thiemo

Re: Binutils + OpenSSL issue

[Kumba]

Thiemo Seufer wrote:

> It was Alexandre Oliva's patch to introduce multi-GOT which triggers
> the problem, specifically elfxx-mips.c, CVS version 1.38 -> 1.39.
> 
> 
> Thiemo
> 

	Hrmm, quite a large patch.  I've tried "unpatching" it to test, but I 
admit I am no expert at patching/unpatching (yet) and still wind up with 
a minimum of two failures.  It seems other patches added afterwards rely 
on elements this specific patch added in.

	Aside from testing any fixes someone can come up with, I doubt I can 
offer much else regarding this glitch/bug.

--Kumba

Re: Binutils + OpenSSL issue

[Eric Christopher]

On Mon, 2003-05-19 at 13:33, Kumba wrote:
> Thiemo Seufer wrote:
> 
> > It was Alexandre Oliva's patch to introduce multi-GOT which triggers
> > the problem, specifically elfxx-mips.c, CVS version 1.38 -> 1.39.
> > 
> > 
> > Thiemo
> > 
> 
> 	Hrmm, quite a large patch.  I've tried "unpatching" it to test, but I 
> admit I am no expert at patching/unpatching (yet) and still wind up with 
> a minimum of two failures.  It seems other patches added afterwards rely 
> on elements this specific patch added in.
> 
> 	Aside from testing any fixes someone can come up with, I doubt I can 
> offer much else regarding this glitch/bug.

What was the bug again?

-eric

-- 
Eric Christopher <echristo@redhat.com>

Re: Binutils + OpenSSL issue

[Thiemo Seufer]

Eric Christopher wrote:
> On Mon, 2003-05-19 at 13:33, Kumba wrote:
> > Thiemo Seufer wrote:
> > 
> > > It was Alexandre Oliva's patch to introduce multi-GOT which triggers
> > > the problem, specifically elfxx-mips.c, CVS version 1.38 -> 1.39.
> > > 
> > > 
> > > Thiemo
> > > 
> > 
> > 	Hrmm, quite a large patch.  I've tried "unpatching" it to test, but I 
> > admit I am no expert at patching/unpatching (yet) and still wind up with 
> > a minimum of two failures.  It seems other patches added afterwards rely 
> > on elements this specific patch added in.
> > 
> > 	Aside from testing any fixes someone can come up with, I doubt I can 
> > offer much else regarding this glitch/bug.
> 
> What was the bug again?

Broken dynamic relocation table in libcrypto.so.0.9.7 because the global
GOT entry count is to low. This probably happens for all -Bsymbolic
--whole-archive links. The appended patch works around the special case
for libcrypto, but will likely break other things. Just a hint wher to
look.


Thiemo


--- src-orig/bfd/elfxx-mips.c	9 May 2003 02:27:09 -0000
+++ src/bfd/elfxx-mips.c	19 May 2003 22:23:50 -0000
@@ -7851,8 +7851,8 @@ _bfd_mips_elf_hide_symbol (info, entry, 
 	     been marked for a global got entry, give it a local entry
 	     instead.  */
 	  BFD_ASSERT (g->global_gotno > 0);
-	  g->local_gotno++;
-	  g->global_gotno--;
+//	  g->local_gotno++;
+//	  g->global_gotno--;
 	}
     }
 

Re: Binutils + OpenSSL issue

[Kumba]

	Building OpenSSL with this now.  If this works, I'll build a basic 
system and run objdump on all the libraries to see if anything else 
breaks. (Just in the hypothetical possibilitiy such a small fix really 
does fix the problem completely.)

--Kumba


Thiemo Seufer wrote:
> Broken dynamic relocation table in libcrypto.so.0.9.7 because the global
> GOT entry count is to low. This probably happens for all -Bsymbolic
> --whole-archive links. The appended patch works around the special case
> for libcrypto, but will likely break other things. Just a hint wher to
> look.
> 
> 
> Thiemo
> 
> 
> --- src-orig/bfd/elfxx-mips.c	9 May 2003 02:27:09 -0000
> +++ src/bfd/elfxx-mips.c	19 May 2003 22:23:50 -0000
> @@ -7851,8 +7851,8 @@ _bfd_mips_elf_hide_symbol (info, entry, 
>  	     been marked for a global got entry, give it a local entry
>  	     instead.  */
>  	  BFD_ASSERT (g->global_gotno > 0);
> -	  g->local_gotno++;
> -	  g->global_gotno--;
> +//	  g->local_gotno++;
> +//	  g->global_gotno--;
>  	}
>      }
>  
> 

Re: Binutils + OpenSSL issue

[Kumba]

Eric Christopher wrote:

> What was the bug again?
> 
> -eric
> 


	(Guess you could say this is the "long" explanation)

	Using binutils anytime after oh, Feb 6th, 2003 (That's a rough guess) 
caused it to start generating a broken OpenSSL package if compiled from 
source.  Specifically, libcrypto and libssl start breaking.  If you run 
the command below:

objdump -R /usr/lib/libcrypto.so.0.9.6 |grep R_MIPS |grep -v 
(R_MIPS_32\|R_MIPS_NONE\)

	You will start getting error messages similar to the ones below:

BFD: BFD 2.13.90.0.18 20030121 assertion fail elf32-mips.c:1498 BFD: 
usr/lib/libcrypto.so.0.9.6(.rel.dyn): relocation 3755 has invalid symbol 
index 0


	The brokenness is introduced even using -mips1 optimisations (gcc-3.2.x 
compiler), however, it appears that the amount of brokeness is small, 
and doesn't interfere with the operation of OpenSSL (although the 
possibility is there).  It's not until you use -mips3 or -mips4 
optimisations that the brokeness starts to to become apparent, and 
programs that link to OpenSSL, or call upon its various functions, for 
example, wget's configure script, get an error like the one below:

./conftest: error while loading shared libraries: 
usr/lib/libcrypto.so.0.9.6: unexpected reloc type 0x68


	At first, it was believed it was an error specific to Gentoo, due to 
the sometimes odd/wacky optimisations people use or to some rogue/bad 
patch called in by the portage system during compile.  However, as 
Thiemo put it, "the unlikely happened", and he was able to reproduce the 
error as well by building OpenSSL with binutils CVS.

	I've nagged a few people about the bug as I've considered it a show 
stopper for myself and a few others running gentoo on their Indigo2/Indy 
machines due to having to live with what one might say is an outdated 
binutils.  Currently, we consider binutils-2.13.90.0.16 to be the 
"stable" version, since OpenSSL is such a critical package in the system.

	It seems to be a pretty isolated bug, having gone undetected for quite 
some time by anyone outside of gentoo really.  Hopefully now it can be 
eliminated.

--Kumba

Re: Binutils + OpenSSL issue

[Kumba]

	Okay, this probably relates to the libcrypto/libssl workaround Thiemo 
developed earlier, and it did fix binutils (which built a correct 
OpenSSL this time), and didn't seem to cause anymore issues until 
building procps-3.1.8.  The actual error is below.

gcc -shared -Wl,-soname,libproc.so.3.1.8 
-Wl,--version-script=proc/library.map -o proc/libproc.so.3.1.8 
proc/alloc.o proc/devname.o proc/escape.o proc/ksym.o proc/prof.o 
proc/pwcache.o proc/readproc.o proc/sig.o proc/status.o proc/sysinfo.o 
proc/version.o proc/whattime.o -lc
/usr/lib/gcc-lib/mips-unknown-linux-gnu/3.2.3/../../../../mips-unknown-linux-gnu/bin/ld: 
not enough GOT space for local GOT entries
/usr/lib/gcc-lib/mips-unknown-linux-gnu/3.2.3/../../../../mips-unknown-linux-gnu/bin/ld: 
BFD 2.14.90.0.2 20030515 internal error, aborting at elfxx-mips.c line 
6558 in _bfd_mips_elf_relocate_section

/usr/lib/gcc-lib/mips-unknown-linux-gnu/3.2.3/../../../../mips-unknown-linux-gnu/bin/ld: 
Please report this bug.

collect2: ld returned 1 exit status
make: *** [proc/libproc.so.3.1.8] Error 1



	Of note, is this error seems specific to procps-3.1.8 only.  I 
successfully compiled procps-2.0.11 w/o issues.


--Kumba



Thiemo Seufer wrote:

> Eric Christopher wrote:
>>
>>What was the bug again?
> 
> 
> Broken dynamic relocation table in libcrypto.so.0.9.7 because the global
> GOT entry count is to low. This probably happens for all -Bsymbolic
> --whole-archive links. The appended patch works around the special case
> for libcrypto, but will likely break other things. Just a hint wher to
> look.
> 
> 
> Thiemo
> 
> 
> --- src-orig/bfd/elfxx-mips.c	9 May 2003 02:27:09 -0000
> +++ src/bfd/elfxx-mips.c	19 May 2003 22:23:50 -0000
> @@ -7851,8 +7851,8 @@ _bfd_mips_elf_hide_symbol (info, entry, 
>  	     been marked for a global got entry, give it a local entry
>  	     instead.  */
>  	  BFD_ASSERT (g->global_gotno > 0);
> -	  g->local_gotno++;
> -	  g->global_gotno--;
> +//	  g->local_gotno++;
> +//	  g->global_gotno--;
>  	}
>      }
>  
> 

Re: Binutils + OpenSSL issue

[Kumba]

	Well, gcc and glibc also fail from the same error as I reported with 
procps.  Not enough GOT space for local GOT entries.  Most other 
libraries seems to be fine, though, so I'm guessing the patch Thiemo 
rigged was on the right track.  My own, binutils-inexperienced guess has 
me thinking some kind of conditionals need to be added to check the GOT 
entry count before decrementing global and/or incrementing local.  Just 
a guess, though.

--Kumba


Thiemo Seufer wrote:

> Eric Christopher wrote:
> 
>What was the bug again?
> 
> 
> Broken dynamic relocation table in libcrypto.so.0.9.7 because the global
> GOT entry count is to low. This probably happens for all -Bsymbolic
> --whole-archive links. The appended patch works around the special case
> for libcrypto, but will likely break other things. Just a hint wher to
> look.
> 
> 
> Thiemo
> 
> 
> --- src-orig/bfd/elfxx-mips.c	9 May 2003 02:27:09 -0000
> +++ src/bfd/elfxx-mips.c	19 May 2003 22:23:50 -0000
> @@ -7851,8 +7851,8 @@ _bfd_mips_elf_hide_symbol (info, entry, 
>  	     been marked for a global got entry, give it a local entry
>  	     instead.  */
>  	  BFD_ASSERT (g->global_gotno > 0);
> -	  g->local_gotno++;
> -	  g->global_gotno--;
> +//	  g->local_gotno++;
> +//	  g->global_gotno--;
>  	}
>      }
>  
> 

Re: Binutils + OpenSSL issue

[Eric Christopher]

On Wed, 2003-05-21 at 09:09, Kumba wrote:
> 	Well, gcc and glibc also fail from the same error as I reported with 
> procps.  Not enough GOT space for local GOT entries.  Most other 
> libraries seems to be fine, though, so I'm guessing the patch Thiemo 
> rigged was on the right track.  My own, binutils-inexperienced guess has 
> me thinking some kind of conditionals need to be added to check the GOT 
> entry count before decrementing global and/or incrementing local.  Just 
> a guess, though.

This is on a gcc bootstrap? We're currently able to do that on our end,
do you have any local patches?

-eric

-- 
Eric Christopher <echristo@redhat.com>

Re: Binutils + OpenSSL issue

[Kumba]

	The gcc I compile does include some patches, but I figured it'd break 
on a package this large.  Thiemo said the patch, while fixing openssl, 
would probably break on other things.  For a minimal system, it seems 
that breakage is just linked to procps, gcc and glibc.  I'm sure 
building addition packages would probably reveal more breakage.

	It fixes the openssl issue, though, Now I guess it's just a matter of 
figuring out how to fix openssl issue while not breaking everything else.

--Kumba


Eric Christopher wrote:

> On Wed, 2003-05-21 at 09:09, Kumba wrote:
> 
>>	Well, gcc and glibc also fail from the same error as I reported with 
>>procps.  Not enough GOT space for local GOT entries.  Most other 
>>libraries seems to be fine, though, so I'm guessing the patch Thiemo 
>>rigged was on the right track.  My own, binutils-inexperienced guess has 
>>me thinking some kind of conditionals need to be added to check the GOT 
>>entry count before decrementing global and/or incrementing local.  Just 
>>a guess, though.
> 
> 
> This is on a gcc bootstrap? We're currently able to do that on our end,
> do you have any local patches?
> 
> -eric
>