Re: cygport may not create debug info if top directory contains a symlink

Re: cygport may not create debug info if top directory contains a symlink

[Jon Turney]

On 16/09/2023 15:17, Christian Franke via Cygwin wrote:
> Found during tests of busybox package:
> If the path of the top build directory contains a symlink and the 
> project's build scripts normalize pathnames, no debug info is created by 
> cygport.
> 
> This is because options like
>   -fdebug-prefix-map=${B}=/usr/src/debug/${PF}
> have no effect because ${B} contains a symlink but the compiler is run 
> with the real source path.

I think that there was some historical bug with gcc where a relative 
path for the old path in this mapping wasn't correctly handled, which is 
why were using an absolute path here at all.

So changing it to something like [1] (if that works), might be better.

[1] 
https://github.com/jon-turney/cygport/commit/4175d456a9184c5cdebd8bfb4b5ba30583cedd66

Sidenote: we should probably also be using file-prefix-map, now we're on 
a gcc which supports it.

> The postinstall code then does not find any line number info with source 
> path /usr/src/debug/${PF}/...
> 
> Could be fixed easily in line 414 of /bin/cygport:
> 
> -declare -r top=$(cd ${_topdir}; pwd);
> +declare -r top=$(cd ${_topdir}; /bin/pwd);

Can you explain why this makes a difference?

> No patch provided because I'm not sure whether this has other negative 
> side effects.
> 
> If this is the case, it possibly makes sense to print a warning if 
> "$(pwd)" != "$(/bin/pwd)".

This is not unreasonable, and I would take a patch doing this, as there 
have been places in cygport where there are bugs handling that in the 
past (and probably still are some, since it's not something that gets 
tested often).

Re: cygport may not create debug info if top directory contains a symlink

[Brian Inglis]

On 2023-09-17 08:01, Jon Turney via Cygwin-apps wrote:
> On 16/09/2023 15:17, Christian Franke via Cygwin wrote:
>> Found during tests of busybox package:
>> If the path of the top build directory contains a symlink and the project's 
>> build scripts normalize pathnames, no debug info is created by cygport.
>>
>> This is because options like
>>   -fdebug-prefix-map=${B}=/usr/src/debug/${PF}
>> have no effect because ${B} contains a symlink but the compiler is run with 
>> the real source path.
> 
> I think that there was some historical bug with gcc where a relative path for 
> the old path in this mapping wasn't correctly handled, which is why were using 
> an absolute path here at all.
> 
> So changing it to something like [1] (if that works), might be better.
> 
> [1] 
> https://github.com/jon-turney/cygport/commit/4175d456a9184c5cdebd8bfb4b5ba30583cedd66
> 
> Sidenote: we should probably also be using file-prefix-map, now we're on a gcc 
> which supports it.
> 
>> The postinstall code then does not find any line number info with source path 
>> /usr/src/debug/${PF}/...
>>
>> Could be fixed easily in line 414 of /bin/cygport:
>>
>> -declare -r top=$(cd ${_topdir}; pwd);
>> +declare -r top=$(cd ${_topdir}; /bin/pwd);
> 
> Can you explain why this makes a difference?

In cygport, pwd is a bash builtin defaulting to -L; /bin/pwd defaults to -P.
Both commands support both options and we might expect the same output.
It would be better to use builtin `pwd -P` if that produces the correct result.
An STC script which creates test dirs to demonstrate the issue and show the 
alternative outputs would be nice so anyone can see.

>> No patch provided because I'm not sure whether this has other negative side 
>> effects.
>>
>> If this is the case, it possibly makes sense to print a warning if "$(pwd)" != 
>> "$(/bin/pwd)".
> 
> This is not unreasonable, and I would take a patch doing this, as there have 
> been places in cygport where there are bugs handling that in the past (and 
> probably still are some, since it's not something that gets tested often).

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

Re: cygport may not create debug info if top directory contains a symlink

[Christian Franke]

Brian Inglis wrote:
> On 2023-09-17 08:01, Jon Turney via Cygwin-apps wrote:
>> On 16/09/2023 15:17, Christian Franke via Cygwin wrote:
>>> Found during tests of busybox package:
>>> If the path of the top build directory contains a symlink and the 
>>> project's build scripts normalize pathnames, no debug info is 
>>> created by cygport.
>>>
>>> This is because options like
>>>   -fdebug-prefix-map=${B}=/usr/src/debug/${PF}
>>> have no effect because ${B} contains a symlink but the compiler is 
>>> run with the real source path.
>>
>> I think that there was some historical bug with gcc where a relative 
>> path for the old path in this mapping wasn't correctly handled, which 
>> is why were using an absolute path here at all.
>>
>> So changing it to something like [1] (if that works), might be better.
>>
>> [1] 
>> https://github.com/jon-turney/cygport/commit/4175d456a9184c5cdebd8bfb4b5ba30583cedd66
>>
>> Sidenote: we should probably also be using file-prefix-map, now we're 
>> on a gcc which supports it.

Definitely. in particular useful in conjunction with reproducible builds 
and this cygport patch:
https://sourceware.org/pipermail/cygwin-apps/2023-August/043108.html
The related newlib-cygwin patch has been pushed already:
https://cygwin.com/git/?p=newlib-cygwin.git;a=commit;h=f5e37b93


>>
>>
>>> The postinstall code then does not find any line number info with 
>>> source path /usr/src/debug/${PF}/...
>>>
>>> Could be fixed easily in line 414 of /bin/cygport:
>>>
>>> -declare -r top=$(cd ${_topdir}; pwd);
>>> +declare -r top=$(cd ${_topdir}; /bin/pwd);
>>
>> Can you explain why this makes a difference?
>
> In cygport, pwd is a bash builtin defaulting to -L; /bin/pwd defaults 
> to -P.
> Both commands support both options and we might expect the same output.
> It would be better to use builtin `pwd -P` if that produces the 
> correct result.

It does.


>
> An STC script which creates test dirs to demonstrate the issue and 
> show the alternative outputs would be nice so anyone can see.

$ ln -s /usr/src /tmp/source

$ cd /tmp/source

$ pwd
/tmp/source

$ /bin/pwd
/usr/src

$ pwd -P
/usr/src

$ /bin/pwd -L
/tmp/source

-- 
Regards,
Christian

Re: cygport may not create debug info if top directory contains a symlink

[Achim Gratz]

Am 18.09.2023 um 12:41 schrieb Christian Franke via Cygwin-apps:
> $ pwd -P
> /usr/src
> 
> $ /bin/pwd -L
> /tmp/source

Generally speaking, if you really need to mess with /usr/src (which is a 
packaged directory, so any Cygwin application can assume it's existence 
as a directory) you should do that via a (bind) mount.


-- 
Achim.

(on the road :-)

Re: cygport may not create debug info if top directory contains a symlink

[Brian Inglis]

On 2023-09-18 04:41, Christian Franke via Cygwin-apps wrote:
> Brian Inglis wrote:
>> On 2023-09-17 08:01, Jon Turney via Cygwin-apps wrote:
>>> On 16/09/2023 15:17, Christian Franke via Cygwin wrote:
>>>> Found during tests of busybox package:
>>>> If the path of the top build directory contains a symlink and the project's 
>>>> build scripts normalize pathnames, no debug info is created by cygport.
>>>>
>>>> This is because options like
>>>>   -fdebug-prefix-map=${B}=/usr/src/debug/${PF}
>>>> have no effect because ${B} contains a symlink but the compiler is run with 
>>>> the real source path.
>>>
>>> I think that there was some historical bug with gcc where a relative path for 
>>> the old path in this mapping wasn't correctly handled, which is why were 
>>> using an absolute path here at all.
>>>
>>> So changing it to something like [1] (if that works), might be better.
>>>
>>> [1] 
>>> https://github.com/jon-turney/cygport/commit/4175d456a9184c5cdebd8bfb4b5ba30583cedd66

Should bin/cygport.in:534: not have $B between the == as in line 531:

	declare ${flags}+=" -fdebug-prefix-map=${B}=/usr/src/debug/${PF}"
	...
	declare ${flags}+=" -fdebug-prefix-map==/usr/src/debug/${PF}"

or be hoist above the condition if identical, unless that is some undocumented 
default for cwd?

>>> Sidenote: we should probably also be using file-prefix-map, now we're on a 
>>> gcc which supports it.

... also macro-prefix-map, although it looks like changing to -ffile-prefix-map 
is equivalent to -f*-prefix-map which future proofs the options!

>>>> The postinstall code then does not find any line number info with source 
>>>> path /usr/src/debug/${PF}/...
>>>>
>>>> Could be fixed easily in line 414 of /bin/cygport:
>>>>
>>>> -declare -r top=$(cd ${_topdir}; pwd);
>>>> +declare -r top=$(cd ${_topdir}; /bin/pwd);
>>>
>>> Can you explain why this makes a difference?
>>
>> In cygport, pwd is a bash builtin defaulting to -L; /bin/pwd defaults to -P.
>> Both commands support both options and we might expect the same output.
>> It would be better to use builtin `pwd -P` if that produces the correct result.

> It does.

>> An STC script which creates test dirs to demonstrate the issue and show the 
>> alternative outputs would be nice so anyone can see.

> $ ln -s /usr/src /tmp/source
> 
> $ cd /tmp/source
> 
> $ pwd
> /tmp/source
> 
> $ /bin/pwd
> /usr/src
> 
> $ pwd -P
> /usr/src
> 
> $ /bin/pwd -L
> /tmp/source

Thanks, looks good - care to submit a patch, including above suggestions, to 
cover all bases?

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

Re: cygport may not create debug info if top directory contains a symlink

[Christian Franke]

Brian Inglis wrote:
> On 2023-09-18 04:41, Christian Franke via Cygwin-apps wrote:
>> Brian Inglis wrote:
>>> On 2023-09-17 08:01, Jon Turney via Cygwin-apps wrote:
>>>> On 16/09/2023 15:17, Christian Franke via Cygwin wrote:
>>>>> Found during tests of busybox package:
>>>>> If the path of the top build directory contains a symlink and the 
>>>>> project's build scripts normalize pathnames, no debug info is 
>>>>> created by cygport.
>>>>>
>>>>> This is because options like
>>>>>   -fdebug-prefix-map=${B}=/usr/src/debug/${PF}
>>>>> have no effect because ${B} contains a symlink but the compiler is 
>>>>> run with the real source path.
>>>>
>>>> I think that there was some historical bug with gcc where a 
>>>> relative path for the old path in this mapping wasn't correctly 
>>>> handled, which is why were using an absolute path here at all.
>>>>
>>>> So changing it to something like [1] (if that works), might be better.
>>>>
>>>> [1] 
>>>> https://github.com/jon-turney/cygport/commit/4175d456a9184c5cdebd8bfb4b5ba30583cedd66
>
> Should bin/cygport.in:534: not have $B between the == as in line 531:
>
>     declare ${flags}+=" -fdebug-prefix-map=${B}=/usr/src/debug/${PF}"
>     ...
>     declare ${flags}+=" -fdebug-prefix-map==/usr/src/debug/${PF}"
>
> or be hoist above the condition if identical, unless that is some 
> undocumented default for cwd?

I guess the == without ${B} is intentional because it makes the debug 
source path relative to ${B} as lines 535-536 also do.


> ...
>>> An STC script which creates test dirs to demonstrate the issue and 
>>> show the alternative outputs would be nice so anyone can see.
>
>> $ ln -s /usr/src /tmp/source
>>
>> $ cd /tmp/source
>>
>> $ pwd
>> /tmp/source
>>
>> $ /bin/pwd
>> /usr/src
>>
>> $ pwd -P
>> /usr/src
>>
>> $ /bin/pwd -L
>> /tmp/source
>
> Thanks, looks good - care to submit a patch, including above 
> suggestions, to cover all bases?
>

Users may have a good reason to use a symlinked directory, e.g. fake the 
original build path during a rebuild. So I'm still not sure how to 
handle this.

- Simply warn the user:

   declare -r top=$(cd ${_topdir}; pwd);
  +if [ "${top}" != "$(cd ${_topdir}; pwd -P)" ]
  +then
  +       warning "symlinks in ${top} do not work with some build systems."
  +fi
   unset _topdir;

- or enforce the real path:

  -declare -r top=$(cd ${_topdir}; pwd);
  +declare -r top=$(cd ${_topdir}; pwd -P);
  +if [ "${top}" != "$(cd ${_topdir}; pwd -L)" ]
  +then
  +       inform "using real path ${top} as top level directory."
  +fi
   unset _topdir;

Projects using GNU autotools and cyg{conf,make,install} in cygport are 
usually not affected by symlinks in ${top}.

-- 
Regards,
Christian

Re: cygport may not create debug info if top directory contains a symlink

[Jon Turney]

On 20/09/2023 11:58, Christian Franke via Cygwin-apps wrote:
> Brian Inglis wrote:
>> On 2023-09-18 04:41, Christian Franke via Cygwin-apps wrote:
>>> Brian Inglis wrote:
>>>> On 2023-09-17 08:01, Jon Turney via Cygwin-apps wrote:
>>>>> On 16/09/2023 15:17, Christian Franke via Cygwin wrote:
>>>>>> Found during tests of busybox package:
>>>>>> If the path of the top build directory contains a symlink and the 
>>>>>> project's build scripts normalize pathnames, no debug info is 
>>>>>> created by cygport.
>>>>>>
>>>>>> This is because options like
>>>>>>   -fdebug-prefix-map=${B}=/usr/src/debug/${PF}
>>>>>> have no effect because ${B} contains a symlink but the compiler is 
>>>>>> run with the real source path.
>>>>>
>>>>> I think that there was some historical bug with gcc where a 
>>>>> relative path for the old path in this mapping wasn't correctly 
>>>>> handled, which is why were using an absolute path here at all.
>>>>>
>>>>> So changing it to something like [1] (if that works), might be better.
>>>>>
>>>>> [1] 
>>>>> https://github.com/jon-turney/cygport/commit/4175d456a9184c5cdebd8bfb4b5ba30583cedd66
>>
>> Should bin/cygport.in:534: not have $B between the == as in line 531:
>>
>>     declare ${flags}+=" -fdebug-prefix-map=${B}=/usr/src/debug/${PF}"
>>     ...
>>     declare ${flags}+=" -fdebug-prefix-map==/usr/src/debug/${PF}"
>>
>> or be hoist above the condition if identical, unless that is some 
>> undocumented default for cwd?
> 
> I guess the == without ${B} is intentional because it makes the debug 
> source path relative to ${B} as lines 535-536 also do.

Yeah, I think that "==" is shorthand for "=.=", i.e. relative to cwd.

Does using relative paths in the prefix-map resolve the original issue 
seen with busybox?

(It's unclear to me how gcc compares paths to apply this mapping. If 
it's a literal string prefix, rather than on some (semi-)canonicalized 
path, then we're maybe going to lose here sometimes, depending on the 
vagaries of the build-system, unless we list all of relative, absolute, 
and canonical absolute paths?)

(But then maybe we can push dealing with or indicating which of those is 
correct off onto the individual cygport?)

Re: cygport may not create debug info if top directory contains a symlink

[Christian Franke]

Jon Turney wrote:
> On 20/09/2023 11:58, Christian Franke via Cygwin-apps wrote:
>> Brian Inglis wrote:
>>> On 2023-09-18 04:41, Christian Franke via Cygwin-apps wrote:
>>>> Brian Inglis wrote:
>>>>> On 2023-09-17 08:01, Jon Turney via Cygwin-apps wrote:
>>>>>> On 16/09/2023 15:17, Christian Franke via Cygwin wrote:
>>>>>>> Found during tests of busybox package:
>>>>>>> If the path of the top build directory contains a symlink and 
>>>>>>> the project's build scripts normalize pathnames, no debug info 
>>>>>>> is created by cygport.
>>>>>>>
>>>>>>> This is because options like
>>>>>>>   -fdebug-prefix-map=${B}=/usr/src/debug/${PF}
>>>>>>> have no effect because ${B} contains a symlink but the compiler 
>>>>>>> is run with the real source path.
>>>>>>
>>>>>> I think that there was some historical bug with gcc where a 
>>>>>> relative path for the old path in this mapping wasn't correctly 
>>>>>> handled, which is why were using an absolute path here at all.
>>>>>>
>>>>>> So changing it to something like [1] (if that works), might be 
>>>>>> better.
>>>>>>
>>>>>> [1] 
>>>>>> https://github.com/jon-turney/cygport/commit/4175d456a9184c5cdebd8bfb4b5ba30583cedd66
>>>
>>> Should bin/cygport.in:534: not have $B between the == as in line 531:
>>>
>>>     declare ${flags}+=" -fdebug-prefix-map=${B}=/usr/src/debug/${PF}"
>>>     ...
>>>     declare ${flags}+=" -fdebug-prefix-map==/usr/src/debug/${PF}"
>>>
>>> or be hoist above the condition if identical, unless that is some 
>>> undocumented default for cwd?
>>
>> I guess the == without ${B} is intentional because it makes the debug 
>> source path relative to ${B} as lines 535-536 also do.
>
> Yeah, I think that "==" is shorthand for "=.=", i.e. relative to cwd.
>

Further tests show that "==" behaves strange and is not useful at all 
(no problem as it doesn't actually appear in cygport.in).

The "=.=" only does a simple replacement of first "." in source name.

Testcases with file.c in /var/tmp/test:

$ gcc -g file.c && objdump -dl a.exe | grep -F file.c
/var/tmp/test/file.c:1

$ gcc -g ./file.c && objdump -dl a.exe | grep -F file.c
/var/tmp/test/./file.c:1

$ gcc -g ../test/file.c && objdump -dl a.exe | grep -F file.c
/var/tmp/test/../test/file.c:1

$ gcc -g -fdebug-prefix-map==/foo/bar file.c && ...
/foo/bar/foo/barfile.c:1

$ gcc -g -fdebug-prefix-map=.=/foo/bar file.c && ...
/var/tmp/test/file.c:1

$ gcc -g -fdebug-prefix-map=.=/foo/bar ./file.c && ...
/foo/bar/file.c:1

$ gcc -g -fdebug-prefix-map=/var/tmp/test=/foo/bar file.c && ...
/foo/bar/file.c:1

Source directories with symlinks are preserved in line number info like 
with 'pwd -L'.


> Does using relative paths in the prefix-map resolve the original issue 
> seen with busybox?

No, relative paths do not work. Adding this to top of src_compile() works:

    CFLAGS+=" -fdebug-prefix-map=$(cd ${S} && pwd -P)=/usr/src/debug/${PF}"

Same for ${B} is not required for this project. This also works (B and S 
are not set yet):

  DEBUG_PREFIX_MAPS=( "$(pwd -P)/${PF}.${ARCH}/src/${PN}-${PV}" )


> (It's unclear to me how gcc compares paths to apply this mapping. If 
> it's a literal string prefix, rather than on some (semi-)canonicalized 
> path, then we're maybe going to lose here sometimes, depending on the 
> vagaries of the build-system, unless we list all of relative, 
> absolute, and canonical absolute paths?)
>
> (But then maybe we can push dealing with or indicating which of those 
> is correct off onto the individual cygport?)
>

Adding this if "$(cd ${S} && pwd -P)" != "${S}" should IMO be safe:
   -fdebug-prefix-map=$(cd ${B} && pwd -P)=/usr/src/debug/${PF}
   -fdebug-prefix-map=$(cd ${S} && pwd -P)=/usr/src/debug/${PF}
(or use realpath)

Re: cygport may not create debug info if top directory contains a symlink

[Jon Turney]

On 30/10/2023 16:37, Christian Franke via Cygwin-apps wrote:
> Jon Turney wrote:
>> On 20/09/2023 11:58, Christian Franke via Cygwin-apps wrote:
>>> Brian Inglis wrote:
>>>> On 2023-09-18 04:41, Christian Franke via Cygwin-apps wrote:
>>>>> Brian Inglis wrote:
>>>>>> On 2023-09-17 08:01, Jon Turney via Cygwin-apps wrote:
>>>>>>> On 16/09/2023 15:17, Christian Franke via Cygwin wrote:
>>>>>>>> Found during tests of busybox package:
>>>>>>>> If the path of the top build directory contains a symlink and 
>>>>>>>> the project's build scripts normalize pathnames, no debug info 
>>>>>>>> is created by cygport.
>>>>>>>>
>>>>>>>> This is because options like
>>>>>>>>   -fdebug-prefix-map=${B}=/usr/src/debug/${PF}
>>>>>>>> have no effect because ${B} contains a symlink but the compiler 
>>>>>>>> is run with the real source path.
>>>>>>>
>>>>>>> I think that there was some historical bug with gcc where a 
>>>>>>> relative path for the old path in this mapping wasn't correctly 
>>>>>>> handled, which is why were using an absolute path here at all.
>>>>>>>
>>>>>>> So changing it to something like [1] (if that works), might be 
>>>>>>> better.
>>>>>>>
>>>>>>> [1] 
>>>>>>> https://github.com/jon-turney/cygport/commit/4175d456a9184c5cdebd8bfb4b5ba30583cedd66
>>>>
>>>> Should bin/cygport.in:534: not have $B between the == as in line 531:
>>>>
>>>>     declare ${flags}+=" -fdebug-prefix-map=${B}=/usr/src/debug/${PF}"
>>>>     ...
>>>>     declare ${flags}+=" -fdebug-prefix-map==/usr/src/debug/${PF}"
>>>>
>>>> or be hoist above the condition if identical, unless that is some 
>>>> undocumented default for cwd?
>>>
>>> I guess the == without ${B} is intentional because it makes the debug 
>>> source path relative to ${B} as lines 535-536 also do.
>>
[...]>
>> (It's unclear to me how gcc compares paths to apply this mapping. If 
>> it's a literal string prefix, rather than on some (semi-)canonicalized 
>> path, then we're maybe going to lose here sometimes, depending on the 
>> vagaries of the build-system, unless we list all of relative, 
>> absolute, and canonical absolute paths?)
>>
>> (But then maybe we can push dealing with or indicating which of those 
>> is correct off onto the individual cygport?)
>>
> 
> Adding this if "$(cd ${S} && pwd -P)" != "${S}" should IMO be safe:
>    -fdebug-prefix-map=$(cd ${B} && pwd -P)=/usr/src/debug/${PF}
>    -fdebug-prefix-map=$(cd ${S} && pwd -P)=/usr/src/debug/${PF}
> (or use realpath)
> 

So it seems there's a new flag '-fcanon-prefix-map' in GCC 13, which 
looks like it maybe solves this problem.

Re: cygport may not create debug info if top directory contains a symlink

[Jon Turney]

On 18/09/2023 18:24, Brian Inglis via Cygwin-apps wrote:
> On 2023-09-18 04:41, Christian Franke via Cygwin-apps wrote:
>> Brian Inglis wrote:
>>> On 2023-09-17 08:01, Jon Turney via Cygwin-apps wrote:
>>>> On 16/09/2023 15:17, Christian Franke via Cygwin wrote:
>>>>> Found during tests of busybox package:
>>>>> If the path of the top build directory contains a symlink and the 
>>>>> project's build scripts normalize pathnames, no debug info is 
>>>>> created by cygport.
>>>>>
>>>>> This is because options like
>>>>>   -fdebug-prefix-map=${B}=/usr/src/debug/${PF}
>>>>> have no effect because ${B} contains a symlink but the compiler is 
>>>>> run with the real source path.
>>>>
[...]
> 
>>>> Sidenote: we should probably also be using file-prefix-map, now 
>>>> we're on a gcc which supports it.
> 
> ... also macro-prefix-map, although it looks like changing to 
> -ffile-prefix-map is equivalent to -f*-prefix-map which future proofs 
> the options!

So I updated to using -ffile-prefix-map in cygport 0.36.8, since that 
seems like the "Right Thing(TM)"

I discovered today that, amazingly, this breaks compiling ruby, since in 
one place it does:

#include __FILE__

(yeah, that's pretty jaw dropping...)