RFC: Add support for SHF_COMPRESSED

RFC: Add support for SHF_COMPRESSED

[H.J. Lu]

I am working on SHF_COMPRESSED support:

http://www.sco.com/developers/gabi/latest/ch4.sheader.html

We already have .zdebug* sections, which aren't compatible
with SHF_COMPRESSED scheme due to missing compression
header.   What section names should we use for SHF_COMPRESSED
debug sections.  Using .debug* or .zdebug*  will break existing tools.
I am thinking .zlibdebug*  or ..Zdebug*.  Any suggestions?

Thanks.

-- 
H.J.

Re: RFC: Add support for SHF_COMPRESSED

[Jan Beulich]

>>> On 13.03.15 at 16:31, <hjl.tools@gmail.com> wrote:
> I am working on SHF_COMPRESSED support:
> 
> http://www.sco.com/developers/gabi/latest/ch4.sheader.html 
> 
> We already have .zdebug* sections, which aren't compatible
> with SHF_COMPRESSED scheme due to missing compression
> header.   What section names should we use for SHF_COMPRESSED
> debug sections.  Using .debug* or .zdebug*  will break existing tools.
> I am thinking .zlibdebug*  or ..Zdebug*.  Any suggestions?

Why would they need to have different names anyway? They're
distinguishable by the flag.

Jan

Re: RFC: Add support for SHF_COMPRESSED

[H.J. Lu]

On Fri, Mar 13, 2015 at 9:05 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 13.03.15 at 16:31, <hjl.tools@gmail.com> wrote:
>> I am working on SHF_COMPRESSED support:
>>
>> http://www.sco.com/developers/gabi/latest/ch4.sheader.html
>>
>> We already have .zdebug* sections, which aren't compatible
>> with SHF_COMPRESSED scheme due to missing compression
>> header.   What section names should we use for SHF_COMPRESSED
>> debug sections.  Using .debug* or .zdebug*  will break existing tools.
>> I am thinking .zlibdebug*  or ..Zdebug*.  Any suggestions?
>
> Why would they need to have different names anyway? They're
> distinguishable by the flag.

Won't the exiting tools, like GDB, choke on compressed .debug* sections?


-- 
H.J.

Re: RFC: Add support for SHF_COMPRESSED

[Jan Beulich]

>>> On 13.03.15 at 17:13, <hjl.tools@gmail.com> wrote:
> On Fri, Mar 13, 2015 at 9:05 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 13.03.15 at 16:31, <hjl.tools@gmail.com> wrote:
>>> I am working on SHF_COMPRESSED support:
>>>
>>> http://www.sco.com/developers/gabi/latest/ch4.sheader.html 
>>>
>>> We already have .zdebug* sections, which aren't compatible
>>> with SHF_COMPRESSED scheme due to missing compression
>>> header.   What section names should we use for SHF_COMPRESSED
>>> debug sections.  Using .debug* or .zdebug*  will break existing tools.
>>> I am thinking .zlibdebug*  or ..Zdebug*.  Any suggestions?
>>
>> Why would they need to have different names anyway? They're
>> distinguishable by the flag.
> 
> Won't the exiting tools, like GDB, choke on compressed .debug* sections?

If they're well behaved, they shouldn't (they should just ignore
them).

Jan

Re: RFC: Add support for SHF_COMPRESSED

[H.J. Lu]

On Fri, Mar 13, 2015 at 9:19 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 13.03.15 at 17:13, <hjl.tools@gmail.com> wrote:
>> On Fri, Mar 13, 2015 at 9:05 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 13.03.15 at 16:31, <hjl.tools@gmail.com> wrote:
>>>> I am working on SHF_COMPRESSED support:
>>>>
>>>> http://www.sco.com/developers/gabi/latest/ch4.sheader.html
>>>>
>>>> We already have .zdebug* sections, which aren't compatible
>>>> with SHF_COMPRESSED scheme due to missing compression
>>>> header.   What section names should we use for SHF_COMPRESSED
>>>> debug sections.  Using .debug* or .zdebug*  will break existing tools.
>>>> I am thinking .zlibdebug*  or ..Zdebug*.  Any suggestions?
>>>
>>> Why would they need to have different names anyway? They're
>>> distinguishable by the flag.
>>
>> Won't the exiting tools, like GDB, choke on compressed .debug* sections?
>
> If they're well behaved, they shouldn't (they should just ignore
> them).
>

I implemented SHF_COMPRESSED on users/hjl/compressed
branch. The old gdb doesn't understand SHF_COMPRESSED:

Reading symbols from bfd.gz...Dwarf Error: wrong version in
compilation unit header (is 136, should be 2, 3, or 4) [in module
/export/home/hjl/bugs/binutils/compressed-2/bfd.gz]
(no debugging symbols found)...done.
(gdb)

and the new gdb works fine:

Reading symbols from gold.gz...done.
(gdb) b main
Breakpoint 1 at 0x400450: file x.c, line 6.
(gdb) r
Starting program: /export/home/hjl/bugs/binutils/compressed-2/gold.gz

Breakpoint 1, main () at x.c:6
6  printf ("hello world\n");
(gdb)

I added --compressed-debug-sections=[none|zlib] to as and
objcopy.  I updated gold to decompress SHF_COMPRESSED
section.  I didn't add --compressed-debug-sections=[none|zlib]
to gold since I don't know if this name is a final choice nor
gold maintainers want such an option.

I wrote some as/objcopy tests:

[hjl@gnu-6 compressed-1]$ make
./as --x32 --nocompress-debug-sections -o x.o x.s
./as --x32 --compressed-debug-sections=zlib -o compressed0.o x.s
./objcopy --compressed-debug-sections=zlib x.o compressed.o
cmp compressed.o compressed0.o
./as --x32 --compress-debug-sections -o compress0.o x.s
./objcopy --compress-debug-sections x.o compress.o
cmp compress.o compress0.o
./objcopy --compressed-debug-sections=none x.o decompressed.o
cmp x.o decompressed.o
./objcopy --compressed-debug-sections=none compressed.o decompressed.o
cmp compressed.o decompressed.o
./objcopy --compressed-debug-sections=none compress.o decompressed.o
cmp compress.o decompressed.o
./objcopy --decompress-debug-sections compressed.o decompressed.o
cmp x.o decompressed.o
./objcopy --decompress-debug-sections compress.o decompressed.o
cmp x.o decompressed.o
cp x.o y.o
ar rv libx.a y.o
ar: creating libx.a
a - y.o
ar rc libx.a y.o
./objcopy --compressed-debug-sections=zlib x.o compressed1.o
cmp compressed1.o compressed.o
./objcopy --compressed-debug-sections=zlib compress.o compressed2.o
cmp compressed2.o compressed.o
./objcopy --compress-debug-sections compressed.o compresse1.o
cmp compresse1.o compress.o
./objcopy --compressed-debug-sections=zlib libx.a liby.a
ar xv liby.a y.o
x - y.o
cmp y.o compressed.o
./objcopy --compress-debug-sections liby.a
ar xv liby.a y.o
x - y.o
cmp y.o compress.o
./objcopy --compressed-debug-sections=zlib liby.a
ar xv liby.a y.o
x - y.o
cmp y.o compressed.o
./objcopy --decompress-debug-sections liby.a
ar xv liby.a y.o
x - y.o
cmp y.o decompressed.o
[hjl@gnu-6 compressed-1]$

and ld/gold/gdb tests:

gcc -mx32 -B./ -g -Wa,--compressed-debug-sections=zlib -c x.c -o x.o
gcc -mx32 -B./ -o bfd x.o -fuse-ld=bfd
gcc -mx32 -B./ -o gold x.o -fuse-ld=gold
./objcopy --compressed-debug-sections=zlib gold gold.gz
./objcopy --compressed-debug-sections=zlib bfd bfd.gz
./gdb -batch -x gdb.cmd bfd.gz
Breakpoint 1 at 0x4003b0: file x.c, line 6.

Breakpoint 1, main () at x.c:6
6  printf ("hello world\n");
A debugging session is active.

Inferior 1 [process 5951] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]
./gdb -batch -x gdb.cmd gold.gz
Breakpoint 1 at 0x400450: file x.c, line 6.

Breakpoint 1, main () at x.c:6
6  printf ("hello world\n");
A debugging session is active.

Inferior 1 [process 5962] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]
[hjl@gnu-6 compressed-2]$

Any feedbacks?

Thanks.

-- 
H.J.

Re: RFC: Add support for SHF_COMPRESSED

[Cary Coutant]

> I added --compressed-debug-sections=[none|zlib] to as and
> objcopy.  I updated gold to decompress SHF_COMPRESSED
> section.  I didn't add --compressed-debug-sections=[none|zlib]
> to gold since I don't know if this name is a final choice nor
> gold maintainers want such an option.

objcopy, gas, and gold already have the --compress-debug-sections
option (though only gold takes "=[none|zlib]" as a parameter; objcopy
and gas always use zlib, and spell the negative as
--nocompress-debug-sections).

I don't see any point in adding another option, especially one spelled
so similarly -- just use the existing option and make it generate the
new form, after all the consumers understand both formats. (And I
prefer a verb to an adjective.)

If you want the option to generate either format, how about something
like "--compress-debug-sections=[none|zlib|zlib-old|zlib-new]"? (Where
"zlib" would generate the old for a while, then we'd switch it to the
new after some time.

-cary

Re: RFC: Add support for SHF_COMPRESSED

[H.J. Lu]

On Thu, Mar 19, 2015 at 11:18 AM, Cary Coutant <ccoutant@google.com> wrote:
>> I added --compressed-debug-sections=[none|zlib] to as and
>> objcopy.  I updated gold to decompress SHF_COMPRESSED
>> section.  I didn't add --compressed-debug-sections=[none|zlib]
>> to gold since I don't know if this name is a final choice nor
>> gold maintainers want such an option.
>
> objcopy, gas, and gold already have the --compress-debug-sections
> option (though only gold takes "=[none|zlib]" as a parameter; objcopy
> and gas always use zlib, and spell the negative as
> --nocompress-debug-sections).
>
> I don't see any point in adding another option, especially one spelled
> so similarly -- just use the existing option and make it generate the
> new form, after all the consumers understand both formats. (And I
> prefer a verb to an adjective.)

Which consumers do we care?

> If you want the option to generate either format, how about something
> like "--compress-debug-sections=[none|zlib|zlib-old|zlib-new]"? (Where
> "zlib" would generate the old for a while, then we'd switch it to the
> new after some time.
>

I will do it.

Thanks.


-- 
H.J.

Re: RFC: Add support for SHF_COMPRESSED

[Pedro Alves]

On 03/19/2015 06:18 PM, Cary Coutant wrote:
> 
> If you want the option to generate either format, how about something
> like "--compress-debug-sections=[none|zlib|zlib-old|zlib-new]"? (Where
> "zlib" would generate the old for a while, then we'd switch it to the
> new after some time.

Can we please find terms that describe that "old" and "new" mean and
use those instead?  "new" will eventually be old news too.
We did that mistake with "--enable-new-dtags", let's not repeat it.

For example:

 --compress-debug-sections=[none|zlib|zlib-gnu|zlib-shf-compressed]
 --compress-debug-sections=[none|zlib|zlib-gnu|zlib-shf]
 --compress-debug-sections=[none|zlib|zlib-gnu|zlib-gabi]

or even:

 --compress-debug-sections=[none|zlib|zlib-v1|zlib-v2]

Thanks,
Pedro Alves

Re: RFC: Add support for SHF_COMPRESSED

[H.J. Lu]

On Fri, Mar 20, 2015 at 2:01 AM, Pedro Alves <palves@redhat.com> wrote:
> On 03/19/2015 06:18 PM, Cary Coutant wrote:
>>
>> If you want the option to generate either format, how about something
>> like "--compress-debug-sections=[none|zlib|zlib-old|zlib-new]"? (Where
>> "zlib" would generate the old for a while, then we'd switch it to the
>> new after some time.
>
> Can we please find terms that describe that "old" and "new" mean and
> use those instead?  "new" will eventually be old news too.
> We did that mistake with "--enable-new-dtags", let's not repeat it.
>
> For example:
>  --compress-debug-sections=[none|zlib|zlib-gnu|zlib-gabi]
>

I will go with this one and  "zlib" will be the same as "zlib-gnu" for now.


-- 
H.J.

Re: RFC: Add support for SHF_COMPRESSED

[H.J. Lu]

On Fri, Mar 20, 2015 at 5:07 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Fri, Mar 20, 2015 at 2:01 AM, Pedro Alves <palves@redhat.com> wrote:
>> On 03/19/2015 06:18 PM, Cary Coutant wrote:
>>>
>>> If you want the option to generate either format, how about something
>>> like "--compress-debug-sections=[none|zlib|zlib-old|zlib-new]"? (Where
>>> "zlib" would generate the old for a while, then we'd switch it to the
>>> new after some time.
>>
>> Can we please find terms that describe that "old" and "new" mean and
>> use those instead?  "new" will eventually be old news too.
>> We did that mistake with "--enable-new-dtags", let's not repeat it.
>>
>> For example:
>>  --compress-debug-sections=[none|zlib|zlib-gnu|zlib-gabi]
>>
>
> I will go with this one and  "zlib" will be the same as "zlib-gnu" for now.
>

I implemented it on users/hjl/compressed branch.  Any feedbacks?

Thanks.


-- 
H.J.

Re: RFC: Add support for SHF_COMPRESSED

[H.J. Lu]

On Fri, Mar 20, 2015 at 6:36 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Fri, Mar 20, 2015 at 5:07 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>> On Fri, Mar 20, 2015 at 2:01 AM, Pedro Alves <palves@redhat.com> wrote:
>>> On 03/19/2015 06:18 PM, Cary Coutant wrote:
>>>>
>>>> If you want the option to generate either format, how about something
>>>> like "--compress-debug-sections=[none|zlib|zlib-old|zlib-new]"? (Where
>>>> "zlib" would generate the old for a while, then we'd switch it to the
>>>> new after some time.
>>>
>>> Can we please find terms that describe that "old" and "new" mean and
>>> use those instead?  "new" will eventually be old news too.
>>> We did that mistake with "--enable-new-dtags", let's not repeat it.
>>>
>>> For example:
>>>  --compress-debug-sections=[none|zlib|zlib-gnu|zlib-gabi]
>>>
>>
>> I will go with this one and  "zlib" will be the same as "zlib-gnu" for now.
>>
>
> I implemented it on users/hjl/compressed branch.  Any feedbacks?
>

I updated users/hjl/compressed branch with gold support.  Does it look
OK?


-- 
H.J.