Who broke options.h?

Who broke options.h?

[Steve Kargl]

Someone (other than Richard who seems to have fixed his
bootstrap issue) in the last 3 days has broken bootstrap
on FreeBSD.  The generated file gcc/options.h contains
code of the form

  OPT_C = 116,                               /* -C */
  OPT_CC = 117,                              /* -CC */
  OPT_c = 118,                               /* -c */
  OPT_C = 119,                               /* -C */
  OPT_coverage = 120,                        /* -coverage */
  OPT_cpp_ = 121,                            /* -cpp= */
  OPT_cpp = 122,                             /* -cpp */
  OPT_d = 123,                               /* -d */
  OPT_D = 124,                               /* -D */
  OPT_d = 125,                               /* -d */
  OPT_defsym_ = 126,                         /* -defsym= */
  OPT_defsym = 127,                          /* -defsym */
  OPT_d = 128,                               /* -d */
  OPT_D = 129,                               /* -D */

The sudden dumping ground of everyone's pet project into
trunk after the new branch has been created is making it 
impossible to bisect this issue.


-- 
Steve
20161221 https://www.youtube.com/watch?v=IbCHE-hONow

Re: Who broke options.h?

[David Malcolm]

On Tue, 2017-04-25 at 06:59 -0700, Steve Kargl wrote:
> Someone (other than Richard who seems to have fixed his
> bootstrap issue) in the last 3 days has broken bootstrap
> on FreeBSD.  The generated file gcc/options.h contains
> code of the form
> 
>   OPT_C = 116,                               /* -C */
>   OPT_CC = 117,                              /* -CC */
>   OPT_c = 118,                               /* -c */
>   OPT_C = 119,                               /* -C */
>   OPT_coverage = 120,                        /* -coverage */
>   OPT_cpp_ = 121,                            /* -cpp= */
>   OPT_cpp = 122,                             /* -cpp */
>   OPT_d = 123,                               /* -d */
>   OPT_D = 124,                               /* -D */
>   OPT_d = 125,                               /* -d */
>   OPT_defsym_ = 126,                         /* -defsym= */
>   OPT_defsym = 127,                          /* -defsym */
>   OPT_d = 128,                               /* -d */
>   OPT_D = 129,                               /* -D */
> 
> The sudden dumping ground of everyone's pet project into
> trunk after the new branch has been created is making it 
> impossible to bisect this issue.

Presumably the issue is the duplicate names within an enum.  Above,
OPT_C has values 116, 119
OPT_d has values 123, 125, 128
OPT_D has values 124, 129
etc.

Looking at the code that writes out that enum in opth-gen.awk, I see:

   443  enum_value = 0
   444  for (i = 0; i < n_opts; i++) {
   445          # Combine the flags of identical switches.  Switches
   446          # appear many times if they are handled by many front
   447          # ends, for example.
   448          while( i + 1 != n_opts && opts[i] == opts[i + 1] ) {
   449                  flags[i + 1] = flags[i] " " flags[i + 1];
   450                  i++;
   451          }

Lines 445-451 suggest that the options are meant to be in some kind of
sorted order, so that duplicates will be adjacent.

In the output you posted the duplicates are *not* adjacent.  So is
there some implicit assumption in the awk code about the sorting of
options, that somehow isn't satisfied on the machine you're seeing this
on?

From what I can tell, the n_opts and opts in that file come direct from
 opt-read.awk, which gets them from opt-gather.awk, which appears to
sort them (but my awk skills are weak).

Alternatively, maybe the collisions are caused by some names needing
opt_sanitized_name?  (you could try making that return its argumen
unmodified to see if it shows anything, I guess)  But I don't see any
new option in trunk in the last 3 days.

Hope this is helpful
Dave

Re: Who broke options.h?

[David Edelsohn]

On Tue, Apr 25, 2017 at 11:03 AM, David Malcolm <dmalcolm@redhat.com> wrote:
> On Tue, 2017-04-25 at 06:59 -0700, Steve Kargl wrote:
>> Someone (other than Richard who seems to have fixed his
>> bootstrap issue) in the last 3 days has broken bootstrap
>> on FreeBSD.  The generated file gcc/options.h contains
>> code of the form
>>
>>   OPT_C = 116,                               /* -C */
>>   OPT_CC = 117,                              /* -CC */
>>   OPT_c = 118,                               /* -c */
>>   OPT_C = 119,                               /* -C */
>>   OPT_coverage = 120,                        /* -coverage */
>>   OPT_cpp_ = 121,                            /* -cpp= */
>>   OPT_cpp = 122,                             /* -cpp */
>>   OPT_d = 123,                               /* -d */
>>   OPT_D = 124,                               /* -D */
>>   OPT_d = 125,                               /* -d */
>>   OPT_defsym_ = 126,                         /* -defsym= */
>>   OPT_defsym = 127,                          /* -defsym */
>>   OPT_d = 128,                               /* -d */
>>   OPT_D = 129,                               /* -D */
>>
>> The sudden dumping ground of everyone's pet project into
>> trunk after the new branch has been created is making it
>> impossible to bisect this issue.
>
> Presumably the issue is the duplicate names within an enum.  Above,
> OPT_C has values 116, 119
> OPT_d has values 123, 125, 128
> OPT_D has values 124, 129
> etc.
>
> Looking at the code that writes out that enum in opth-gen.awk, I see:
>
>    443  enum_value = 0
>    444  for (i = 0; i < n_opts; i++) {
>    445          # Combine the flags of identical switches.  Switches
>    446          # appear many times if they are handled by many front
>    447          # ends, for example.
>    448          while( i + 1 != n_opts && opts[i] == opts[i + 1] ) {
>    449                  flags[i + 1] = flags[i] " " flags[i + 1];
>    450                  i++;
>    451          }
>
> Lines 445-451 suggest that the options are meant to be in some kind of
> sorted order, so that duplicates will be adjacent.
>
> In the output you posted the duplicates are *not* adjacent.  So is
> there some implicit assumption in the awk code about the sorting of
> options, that somehow isn't satisfied on the machine you're seeing this
> on?
>
> From what I can tell, the n_opts and opts in that file come direct from
>  opt-read.awk, which gets them from opt-gather.awk, which appears to
> sort them (but my awk skills are weak).
>
> Alternatively, maybe the collisions are caused by some names needing
> opt_sanitized_name?  (you could try making that return its argumen
> unmodified to see if it shows anything, I guess)  But I don't see any
> new option in trunk in the last 3 days.

Maybe BSD awk versus GNU awk?

Did the bootstrap system change or GNU awk is not in the path?

- David

Re: Who broke options.h?

[Jakub Jelinek]

On Tue, Apr 25, 2017 at 11:03:40AM -0400, David Malcolm wrote:
> >From what I can tell, the n_opts and opts in that file come direct from
>  opt-read.awk, which gets them from opt-gather.awk, which appears to
> sort them (but my awk skills are weak).

Perhaps a bug in the FreeBSD awk?
The *.awk files haven't changed since February, and the last modification
of any *.opt file has been in r247028 5 days ago.

	Jakub

Re: Who broke options.h?

[Andreas Schwab]

On Apr 25 2017, Steve Kargl <sgk@troutmask.apl.washington.edu> wrote:

> Someone (other than Richard who seems to have fixed his
> bootstrap issue) in the last 3 days has broken bootstrap
> on FreeBSD.

Did you change your locale since then?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Who broke options.h?

[Joseph Myers]

On Tue, 25 Apr 2017, David Malcolm wrote:

> >From what I can tell, the n_opts and opts in that file come direct from
>  opt-read.awk, which gets them from opt-gather.awk, which appears to
> sort them (but my awk skills are weak).

Maybe the opt-gather.awk call in Makefile.in needs to set LC_ALL=C.  If 
setting LC_ALL=C on the $(AWK) -f $(srcdir)/opt-gather.awk call works to 
fix this then that change is preapproved.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Who broke options.h?

[Steve Kargl]

On Tue, Apr 25, 2017 at 11:09:05AM -0400, David Edelsohn wrote:
> >
> > From what I can tell, the n_opts and opts in that file come direct from
> >  opt-read.awk, which gets them from opt-gather.awk, which appears to
> > sort them (but my awk skills are weak).
> >
> > Alternatively, maybe the collisions are caused by some names needing
> > opt_sanitized_name?  (you could try making that return its argumen
> > unmodified to see if it shows anything, I guess)  But I don't see any
> > new option in trunk in the last 3 days.
> 
> Maybe BSD awk versus GNU awk?
> 
> Did the bootstrap system change or GNU awk is not in the path?
> 

This did occur with nawk.

But, I think that there must be something environmental setting
on the user account causing the problem.  If I login as the 
user that owns the gcc/ account, I see the problem.  If I login
to my normal account and then su to the gcc/ account, the system
builds as normal.  I'm investigating.

On the bright side, I can confirm that Richard's patch does
fix the original issue.  Thanks, Richard.

-- 
Steve
20161221 https://www.youtube.com/watch?v=IbCHE-hONow

Re: Who broke options.h?

[Steve Kargl]

On Tue, Apr 25, 2017 at 04:13:45PM +0000, Joseph Myers wrote:
> On Tue, 25 Apr 2017, David Malcolm wrote:
> 
> > >From what I can tell, the n_opts and opts in that file come direct from
> >  opt-read.awk, which gets them from opt-gather.awk, which appears to
> > sort them (but my awk skills are weak).
> 
> Maybe the opt-gather.awk call in Makefile.in needs to set LC_ALL=C.  If 
> setting LC_ALL=C on the $(AWK) -f $(srcdir)/opt-gather.awk call works to 
> fix this then that change is preapproved.
> 

I think you've found the problem.  If I login as the user
that owns the gcc/ directory, I see the problem.  That 
user has

troutmask:sgk[202] locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=

If I login into my normal account and then su to user sgk, the
problem does not occur.  My normal account has

troutmask:kargl[201] locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_COLLATE=C
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=

So, it appears that LC_COLLATE=C is needed.  I'll see if I can
come up with a patch later today.

-- 
Steve
20161221 https://www.youtube.com/watch?v=IbCHE-hONow

Re: Who broke options.h?

[Steve Kargl]

On Tue, Apr 25, 2017 at 05:46:15PM +0200, Andreas Schwab wrote:
> On Apr 25 2017, Steve Kargl <sgk@troutmask.apl.washington.edu> wrote:
> 
> > Someone (other than Richard who seems to have fixed his
> > bootstrap issue) in the last 3 days has broken bootstrap
> > on FreeBSD.
> 
> Did you change your locale since then?
> 

See my reply to Joseph.  It is locale related.  I'm surprised
I haven't seen this before, which suggests that I must always
login into my normal account and then su to the gcc/ account.

-- 
Steve
20161221 https://www.youtube.com/watch?v=IbCHE-hONow

Re: Who broke options.h?

[Jakub Jelinek]

On Tue, Apr 25, 2017 at 09:54:04AM -0700, Steve Kargl wrote:
> On Tue, Apr 25, 2017 at 05:46:15PM +0200, Andreas Schwab wrote:
> > On Apr 25 2017, Steve Kargl <sgk@troutmask.apl.washington.edu> wrote:
> > 
> > > Someone (other than Richard who seems to have fixed his
> > > bootstrap issue) in the last 3 days has broken bootstrap
> > > on FreeBSD.
> > 
> > Did you change your locale since then?
> > 
> 
> See my reply to Joseph.  It is locale related.  I'm surprised
> I haven't seen this before, which suggests that I must always
> login into my normal account and then su to the gcc/ account.

So like (to follow how we set env vars in other spots):

	* Makefile.in (s-options): Call opt-gather.awk with
	LC_ALL=C in the environment.

--- gcc/Makefile.in	2017-04-18 21:16:24.703775156 +0200
+++ gcc/Makefile.in	2017-04-25 18:56:58.304963926 +0200
@@ -2139,6 +2139,7 @@ s-specs : Makefile
 
 optionlist: s-options ; @true
 s-options: $(ALL_OPT_FILES) Makefile $(srcdir)/opt-gather.awk
+	LC_ALL=C ; export LC_ALL ; \
 	$(AWK) -f $(srcdir)/opt-gather.awk $(ALL_OPT_FILES) > tmp-optionlist
 	$(SHELL) $(srcdir)/../move-if-change tmp-optionlist optionlist
 	$(STAMP) s-options

?  Untested.

	Jakub

Re: Who broke options.h?

[Joseph Myers]

On Tue, 25 Apr 2017, Jakub Jelinek wrote:

> On Tue, Apr 25, 2017 at 09:54:04AM -0700, Steve Kargl wrote:
> > On Tue, Apr 25, 2017 at 05:46:15PM +0200, Andreas Schwab wrote:
> > > On Apr 25 2017, Steve Kargl <sgk@troutmask.apl.washington.edu> wrote:
> > > 
> > > > Someone (other than Richard who seems to have fixed his
> > > > bootstrap issue) in the last 3 days has broken bootstrap
> > > > on FreeBSD.
> > > 
> > > Did you change your locale since then?
> > > 
> > 
> > See my reply to Joseph.  It is locale related.  I'm surprised
> > I haven't seen this before, which suggests that I must always
> > login into my normal account and then su to the gcc/ account.
> 
> So like (to follow how we set env vars in other spots):
> 
> 	* Makefile.in (s-options): Call opt-gather.awk with
> 	LC_ALL=C in the environment.

Yes, assuming this fixes the problem.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Who broke options.h?

[Steve Kargl]

On Tue, Apr 25, 2017 at 06:58:56PM +0200, Jakub Jelinek wrote:
> On Tue, Apr 25, 2017 at 09:54:04AM -0700, Steve Kargl wrote:
> > On Tue, Apr 25, 2017 at 05:46:15PM +0200, Andreas Schwab wrote:
> > > On Apr 25 2017, Steve Kargl <sgk@troutmask.apl.washington.edu> wrote:
> > > 
> > > > Someone (other than Richard who seems to have fixed his
> > > > bootstrap issue) in the last 3 days has broken bootstrap
> > > > on FreeBSD.
> > > 
> > > Did you change your locale since then?
> > > 
> > 
> > See my reply to Joseph.  It is locale related.  I'm surprised
> > I haven't seen this before, which suggests that I must always
> > login into my normal account and then su to the gcc/ account.
> 
> So like (to follow how we set env vars in other spots):
> 
> 	* Makefile.in (s-options): Call opt-gather.awk with
> 	LC_ALL=C in the environment.
> 
> --- gcc/Makefile.in	2017-04-18 21:16:24.703775156 +0200
> +++ gcc/Makefile.in	2017-04-25 18:56:58.304963926 +0200
> @@ -2139,6 +2139,7 @@ s-specs : Makefile
>  
>  optionlist: s-options ; @true
>  s-options: $(ALL_OPT_FILES) Makefile $(srcdir)/opt-gather.awk
> +	LC_ALL=C ; export LC_ALL ; \
>  	$(AWK) -f $(srcdir)/opt-gather.awk $(ALL_OPT_FILES) > tmp-optionlist
>  	$(SHELL) $(srcdir)/../move-if-change tmp-optionlist optionlist
>  	$(STAMP) s-options
> 
> ?  Untested.

I can test this.  However, as this is a change in Makefile.in,
do I need to add --enable-maintainer-mode or some such flag
to configure?

-- 
Steve
20161221 https://www.youtube.com/watch?v=IbCHE-hONow

Re: Who broke options.h?

[Jakub Jelinek]

On Tue, Apr 25, 2017 at 10:18:28AM -0700, Steve Kargl wrote:
> > --- gcc/Makefile.in	2017-04-18 21:16:24.703775156 +0200
> > +++ gcc/Makefile.in	2017-04-25 18:56:58.304963926 +0200
> > @@ -2139,6 +2139,7 @@ s-specs : Makefile
> >  
> >  optionlist: s-options ; @true
> >  s-options: $(ALL_OPT_FILES) Makefile $(srcdir)/opt-gather.awk
> > +	LC_ALL=C ; export LC_ALL ; \
> >  	$(AWK) -f $(srcdir)/opt-gather.awk $(ALL_OPT_FILES) > tmp-optionlist
> >  	$(SHELL) $(srcdir)/../move-if-change tmp-optionlist optionlist
> >  	$(STAMP) s-options
> > 
> > ?  Untested.
> 
> I can test this.  However, as this is a change in Makefile.in,
> do I need to add --enable-maintainer-mode or some such flag
> to configure?

No.  Makefile is not included, so it is always (re)generated.

	Jakub

Re: Who broke options.h?

[Steve Kargl]

On Tue, Apr 25, 2017 at 06:58:56PM +0200, Jakub Jelinek wrote:
> On Tue, Apr 25, 2017 at 09:54:04AM -0700, Steve Kargl wrote:
> > On Tue, Apr 25, 2017 at 05:46:15PM +0200, Andreas Schwab wrote:
> > > On Apr 25 2017, Steve Kargl <sgk@troutmask.apl.washington.edu> wrote:
> > > 
> > > > Someone (other than Richard who seems to have fixed his
> > > > bootstrap issue) in the last 3 days has broken bootstrap
> > > > on FreeBSD.
> > > 
> > > Did you change your locale since then?
> > > 
> > 
> > See my reply to Joseph.  It is locale related.  I'm surprised
> > I haven't seen this before, which suggests that I must always
> > login into my normal account and then su to the gcc/ account.
> 
> So like (to follow how we set env vars in other spots):
> 
> 	* Makefile.in (s-options): Call opt-gather.awk with
> 	LC_ALL=C in the environment.
> 
> --- gcc/Makefile.in	2017-04-18 21:16:24.703775156 +0200
> +++ gcc/Makefile.in	2017-04-25 18:56:58.304963926 +0200
> @@ -2139,6 +2139,7 @@ s-specs : Makefile
>  
>  optionlist: s-options ; @true
>  s-options: $(ALL_OPT_FILES) Makefile $(srcdir)/opt-gather.awk
> +	LC_ALL=C ; export LC_ALL ; \
>  	$(AWK) -f $(srcdir)/opt-gather.awk $(ALL_OPT_FILES) > tmp-optionlist
>  	$(SHELL) $(srcdir)/../move-if-change tmp-optionlist optionlist
>  	$(STAMP) s-options
> 
> ?  Untested.
> 

This appears to fix my problem.  Do you want to commit the patch
or would you rather have me do it?  Note, the problem is present
in 7-branch.  I haven't checked the other branches.

-- 
Steve
20161221 https://www.youtube.com/watch?v=IbCHE-hONow

Re: Who broke options.h?

[Jakub Jelinek]

On Tue, Apr 25, 2017 at 12:43:00PM -0700, Steve Kargl wrote:
> This appears to fix my problem.  Do you want to commit the patch
> or would you rather have me do it?  Note, the problem is present
> in 7-branch.  I haven't checked the other branches.

Committed to trunk, 7.x will need to wait until 7.1 is released,
the rc1 is already in the works and this isn't anything new,
I see the same thing already in GCC 4.0.

	Jakub

Re: Who broke options.h?

[Gerald Pfeifer]

On Tue, 25 Apr 2017, Jakub Jelinek wrote:
> Committed to trunk, 7.x will need to wait until 7.1 is released, 
> the rc1 is already in the works and this isn't anything new, I 
> see the same thing already in GCC 4.0.

Thanks, Jakub!  Are you planning to apply this to GCC 7 after the
release of 7.1?

And would you mind backporting this to active release branches?
(If not, okay if I do?)

Gerald

Re: Who broke options.h?

[Jakub Jelinek]

On Sun, Apr 30, 2017 at 07:49:45PM +0200, Gerald Pfeifer wrote:
> On Tue, 25 Apr 2017, Jakub Jelinek wrote:
> > Committed to trunk, 7.x will need to wait until 7.1 is released, 
> > the rc1 is already in the works and this isn't anything new, I 
> > see the same thing already in GCC 4.0.
> 
> Thanks, Jakub!  Are you planning to apply this to GCC 7 after the
> release of 7.1?

Yes.

> And would you mind backporting this to active release branches?
> (If not, okay if I do?)

I'll handle it together with other 6.x and 5.x backports of my patches.

	Jakub