[Bug other/55376] New: [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[Bug other/55376] New: [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

             Bug #: 55376
           Summary: [asan] libsanitizer/README.gcc must contain the exact
                    steps to do code changes and to port code from
                    upstream
    Classification: Unclassified
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: other
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: konstantin.s.serebryany@gmail.com
                CC: dseketel@redhat.com, dvyukov@google.com,
                    jakub@redhat.com, wmi@gcc.gnu.org


We need to document the libsanitizer update process in libsanitizer/README.gcc

Few things to cover:

- The changes need to be approved by one of the maintainers (or is it obvious)?
- Except for *really* trivial changes all patches should go through the
upstream tree first. 
- When updating from upstream, the comment header should be tampered with (oh,
my)
- Ideally, we need to have a fully automated script to grab the upstream
changes (including new/deleted/moved files, etc). Suggestions here? 
- What is the testing procedure when updating from upstream? (e.g. how do we
avoid regressions on the platforms to which the maintainers have no access?)

[Bug other/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[xinliangli at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

davidxl <xinliangli at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |xinliangli at gmail dot com

--- Comment #1 from davidxl <xinliangli at gmail dot com> 2012-11-18 05:18:11 UTC ---
(In reply to comment #0)
> We need to document the libsanitizer update process in libsanitizer/README.gcc
> 
> Few things to cover:
> 
> - The changes need to be approved by one of the maintainers (or is it obvious)?
> - Except for *really* trivial changes all patches should go through the
> upstream tree first. 
> - When updating from upstream, the comment header should be tampered with (oh,
> my)
> - Ideally, we need to have a fully automated script to grab the upstream
> changes (including new/deleted/moved files, etc). Suggestions here? 
> - What is the testing procedure when updating from upstream? (e.g. how do we
> avoid regressions on the platforms to which the maintainers have no access?)

Are all upstream changes considered reviewed and automatically approved for gcc
repo? If yes, then this should be automated (including gcc build and libasan
testing).

[Bug other/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[ebotcazou at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

Eric Botcazou <ebotcazou at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2012-11-18
                 CC|                            |ebotcazou at gcc dot
                   |                            |gnu.org
     Ever Confirmed|0                           |1

--- Comment #2 from Eric Botcazou <ebotcazou at gcc dot gnu.org> 2012-11-18 09:30:22 UTC ---
> Few things to cover:
> 
> - The changes need to be approved by one of the maintainers (or is it obvious)?

Yes, it's how GCC works.

> - Except for *really* trivial changes all patches should go through the
> upstream tree first. 

That's not acceptable.  We don't want to have to go through LLVM to fix issues
in GCC, especially for the platforms that LLVM doesn't support, i.e. most of
them.

> - What is the testing procedure when updating from upstream? (e.g. how do we
> avoid regressions on the platforms to which the maintainers have no access?)

Introducing such regressions is acceptable, provided that they can be quickly
fixed by the target maintainers.  Hence the "not acceptable" above.

[Bug other/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

--- Comment #3 from Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> 2012-11-18 18:47:19 UTC ---
>> Are all upstream changes considered reviewed and automatically approved for gcc repo? 

all upstream changes are pre- or post- reviewed, so my answer here is "yes".

>> That's not acceptable.  We don't want to have to go through LLVM to fix issues
in GCC, especially for the platforms that LLVM doesn't support, i.e. most of
them.

I've got your point, but please understand mine: if the trees go too much out
of sync we (the asan team) will lose control over one of the copies (gcc). 
It will mean that some one else (not us) will have to work on asan in gcc. 
Maybe that's not bad, but I don't want it. 

Syncing the trees in the presence of difference in the comment headers make the
syncing task a tiny bit more challenging. 
I hope that at some point when we get enough contributions to libsanitizer
from the GCC community, we will be able to unify the headers by saying 
"This is part of LLVM and GCC projects". WDYT?


As I understood from previous e-mails, there are libraries with similar
problems in the gcc tree. What are the solutions there? 

>> Introducing such regressions is acceptable, provided that they can be quickly
fixed by the target maintainers.

It's great that such regressions is acceptable, but if there is an
infrastructure 
that allows us to know about possible regressions before the commit (aka try
bots), I'd like to know.

[Bug other/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[ebotcazou at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

--- Comment #4 from Eric Botcazou <ebotcazou at gcc dot gnu.org> 2012-11-19 22:58:17 UTC ---
> I've got your point, but please understand mine: if the trees go too much out
> of sync we (the asan team) will lose control over one of the copies (gcc). 
> It will mean that some one else (not us) will have to work on asan in gcc. 
> Maybe that's not bad, but I don't want it. 

You're the maintainers, so you can merge in both directions.

> As I understood from previous e-mails, there are libraries with similar
> problems in the gcc tree. What are the solutions there? 

libffi is merged in both directions but is not very active. For the Go
compiler, Ian first applies the patches upstream and then merges in the GCC
tree.

> It's great that such regressions is acceptable, but if there is an
> infrastructure that allows us to know about possible regressions before the 
> commit (aka try bots), I'd like to know.

No, there is no automated command&control tool for the all the platforms,
you'll have to deal with humans for most of them.

[Bug other/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

--- Comment #5 from Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> 2012-11-20 05:40:34 UTC ---
Merging in both directions is possible, but painful, so I'd prefer to limit it. 
How about this phrasing? 

=== 
All changes in this directory should be pre-approved by one of the maintainers.
Trivial and urgent fixes (portability, build fixes, etc) may go directly to the
gcc tree. All non-trivial changes, functionality improvements, etc should go
through the upstream tree first and then be merged back to the gcc tree. 

=== 

I also want to have a semi-automated way to pull the updates from upstream. 
What is the preferred scripting language? Is bash (python, perl) ok? 

When pulling a new update, what text do we expect in the ChangeLog? 
Is the upstream SVN revision enough, or we want to copy all commit messages?

[Bug other/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[ebotcazou at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

--- Comment #6 from Eric Botcazou <ebotcazou at gcc dot gnu.org> 2012-11-20 07:23:26 UTC ---
> === 
> All changes in this directory should be pre-approved by one of the maintainers.
> Trivial and urgent fixes (portability, build fixes, etc) may go directly to the
> gcc tree. All non-trivial changes, functionality improvements, etc should go
> through the upstream tree first and then be merged back to the gcc tree. 
> 
> === 

The first sentence is odd, since all changes must already be approved as per
the GCC rules, but the rest is reasonable.  And it's GCC, not gcc, like LLVM.

> I also want to have a semi-automated way to pull the updates from upstream. 
> What is the preferred scripting language? Is bash (python, perl) ok? 

If it runs on your machine, why asking?

> When pulling a new update, what text do we expect in the ChangeLog? 
> Is the upstream SVN revision enough, or we want to copy all commit messages?

Ideally a script could parse the commit messages on the LLVM side and yield a
GNU-compatible ChangeLog; the granularity could be the file instead of the
function.  That being said, I don't know what LLVM commit messages look like,
so this might not really work.

[Bug other/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[manu at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

Manuel López-Ibáñez <manu at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |manu at gcc dot gnu.org

--- Comment #7 from Manuel López-Ibáñez <manu at gcc dot gnu.org> 2012-11-20 10:17:15 UTC ---
> When pulling a new update, what text do we expect in the ChangeLog? 
> Is the upstream SVN revision enough, or we want to copy all commit messages?

Why do you want to bother with a ChangeLog? libsanitizer is a upstream non-GNU
library, and for those the GNU coding conventions do not (need to) apply. The
go FE does not have a Changelog (only the glue code).

[Bug other/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

--- Comment #8 from Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> 2012-11-21 04:23:21 UTC ---
>> Why do you want to bother with a ChangeLog?
I don't. 
I would prefer to simply mention the upstream revision to which was the last
sync.

[Bug sanitizer/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

--- Comment #9 from Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> 2012-11-23 10:56:44 UTC ---
Not that today's upstream tsan sources don't link with -fPIC at all
because our assembly blobs are not PIC-friendly. 

/usr/bin/ld: .libs/tsan_rtl_thread.o: relocation R_X86_64_PC32 against
undefined symbol `__tsan_trace_switch_thunk' can not be used when making a
shared objec

Apparently, Wei hacked this around by enabling pure-C++ code, which is slower.

[Bug sanitizer/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

--- Comment #10 from Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> 2012-11-23 12:44:14 UTC ---
libsanitizer/README.gcc was updated, and libsanitizer/merge.sh was create to
help with merges.

[Bug sanitizer/55376] [asan] libsanitizer/README.gcc must contain the exact steps to do code changes and to port code from upstream

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55376

Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #11 from Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> 2012-11-23 12:56:57 UTC ---
fixed