* Heads up: Problems with parallel make
@ 2023-02-13 23:05 Ken Brown
2023-02-13 23:44 ` Brian Inglis
2023-02-14 2:33 ` Ken Brown
0 siblings, 2 replies; 25+ messages in thread
From: Ken Brown @ 2023-02-13 23:05 UTC (permalink / raw)
To: cygwin-apps
Several of my packages that used to build fine with parallel make now
require 'make -j1'. Without this I either get strange errors or a hang.
I can't find any pattern to explain when the problem occurs and when it
doesn't, but it seems to be due to a change in make, starting with
version 4.4. By default, make now uses a FIFO rather than a pipe in its
jobserver implementation. You can override this with the make flag
--jobserver-style=pipe
In all cases where I've tried this, parallel make works again.
It wouldn't surprise me if there's a bug in Cygwin's FIFO implementation
that's responsible for this, but I haven't yet tried to track it down.
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-13 23:05 Heads up: Problems with parallel make Ken Brown
@ 2023-02-13 23:44 ` Brian Inglis
2023-02-14 0:38 ` Ken Brown
2023-02-14 2:33 ` Ken Brown
1 sibling, 1 reply; 25+ messages in thread
From: Brian Inglis @ 2023-02-13 23:44 UTC (permalink / raw)
To: cygwin-apps; +Cc: Ken Brown
On 2023-02-13 16:05, Ken Brown via Cygwin-apps wrote:
> Several of my packages that used to build fine with parallel make now require
> 'make -j1'. Without this I either get strange errors or a hang.
> I can't find any pattern to explain when the problem occurs and when it doesn't,
> but it seems to be due to a change in make, starting with version 4.4. By
> default, make now uses a FIFO rather than a pipe in its jobserver
> implementation. You can override this with the make flag
> --jobserver-style=pipe
> In all cases where I've tried this, parallel make works again.
> It wouldn't surprise me if there's a bug in Cygwin's FIFO implementation that's
> responsible for this, but I haven't yet tried to track it down.
There is some glibc? compatibility issue also with this recommended workaround.
--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada
La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-13 23:44 ` Brian Inglis
@ 2023-02-14 0:38 ` Ken Brown
2023-02-14 16:49 ` Brian Inglis
0 siblings, 1 reply; 25+ messages in thread
From: Ken Brown @ 2023-02-14 0:38 UTC (permalink / raw)
To: cygwin-apps
On 2/13/2023 6:44 PM, Brian Inglis wrote:
> On 2023-02-13 16:05, Ken Brown via Cygwin-apps wrote:
>> Several of my packages that used to build fine with parallel make now
>> require 'make -j1'. Without this I either get strange errors or a hang.
>> I can't find any pattern to explain when the problem occurs and when
>> it doesn't, but it seems to be due to a change in make, starting with
>> version 4.4. By default, make now uses a FIFO rather than a pipe in
>> its jobserver implementation. You can override this with the make flag
>> --jobserver-style=pipe
>> In all cases where I've tried this, parallel make works again.
>> It wouldn't surprise me if there's a bug in Cygwin's FIFO
>> implementation that's responsible for this, but I haven't yet tried to
>> track it down.
>
> There is some glibc? compatibility issue also with this recommended
> workaround.
Could you elaborate? I don't know what issue you're seeing.
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-13 23:05 Heads up: Problems with parallel make Ken Brown
2023-02-13 23:44 ` Brian Inglis
@ 2023-02-14 2:33 ` Ken Brown
1 sibling, 0 replies; 25+ messages in thread
From: Ken Brown @ 2023-02-14 2:33 UTC (permalink / raw)
To: cygwin-apps
On 2/13/2023 6:05 PM, Ken Brown via Cygwin-apps wrote:
> Several of my packages that used to build fine with parallel make now
> require 'make -j1'. Without this I either get strange errors or a hang.
>
> I can't find any pattern to explain when the problem occurs and when it
> doesn't, but it seems to be due to a change in make, starting with
> version 4.4. By default, make now uses a FIFO rather than a pipe in its
> jobserver implementation. You can override this with the make flag
>
> --jobserver-style=pipe
>
> In all cases where I've tried this, parallel make works again.
>
> It wouldn't surprise me if there's a bug in Cygwin's FIFO implementation
> that's responsible for this, but I haven't yet tried to track it down.
I've now reported this upstream:
https://lists.gnu.org/archive/html/bug-make/2023-02/msg00029.html
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-14 0:38 ` Ken Brown
@ 2023-02-14 16:49 ` Brian Inglis
2023-02-14 21:23 ` Marco Atzeri
0 siblings, 1 reply; 25+ messages in thread
From: Brian Inglis @ 2023-02-14 16:49 UTC (permalink / raw)
To: cygwin-apps
On 2023-02-13 17:38, Ken Brown via Cygwin-apps wrote:
> On 2/13/2023 6:44 PM, Brian Inglis wrote:
>> On 2023-02-13 16:05, Ken Brown via Cygwin-apps wrote:
>>> Several of my packages that used to build fine with parallel make now require
>>> 'make -j1'. Without this I either get strange errors or a hang.
>>> I can't find any pattern to explain when the problem occurs and when it
>>> doesn't, but it seems to be due to a change in make, starting with version
>>> 4.4. By default, make now uses a FIFO rather than a pipe in its jobserver
>>> implementation. You can override this with the make flag
>>> --jobserver-style=pipe
>>> In all cases where I've tried this, parallel make works again.
>>> It wouldn't surprise me if there's a bug in Cygwin's FIFO implementation
>>> that's responsible for this, but I haven't yet tried to track it down.
>>
>> There is some glibc? compatibility issue also with this recommended workaround.
>
> Could you elaborate? I don't know what issue you're seeing.
There have been reports from various projects mentioning problems in other
packages ending up at make 4.4 and FIFOs solved by reverting to pipes.
There is a make 4.4.1 release candidate 4.40.0.90 with patches that may help to
deal with this:
https://alpha.gnu.org/gnu/make/make-4.4.0.90.tar.lz{,.sig}
https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=110&set=custom
--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada
La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-14 16:49 ` Brian Inglis
@ 2023-02-14 21:23 ` Marco Atzeri
2023-02-15 18:09 ` Ken Brown
0 siblings, 1 reply; 25+ messages in thread
From: Marco Atzeri @ 2023-02-14 21:23 UTC (permalink / raw)
To: cygwin-apps
On 14.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
> On 2023-02-13 17:38, Ken Brown via Cygwin-apps wrote:
>> On 2/13/2023 6:44 PM, Brian Inglis wrote:
>>> On 2023-02-13 16:05, Ken Brown via Cygwin-apps wrote:
>>>> Several of my packages that used to build fine with parallel make
>>>> now require 'make -j1'. Without this I either get strange errors or
>>>> a hang.
>>>> I can't find any pattern to explain when the problem occurs and when
>>>> it doesn't, but it seems to be due to a change in make, starting
>>>> with version 4.4. By default, make now uses a FIFO rather than a
>>>> pipe in its jobserver implementation. You can override this with
>>>> the make flag
>>>> --jobserver-style=pipe
>>>> In all cases where I've tried this, parallel make works again.
>>>> It wouldn't surprise me if there's a bug in Cygwin's FIFO
>>>> implementation that's responsible for this, but I haven't yet tried
>>>> to track it down.
>>>
>>> There is some glibc? compatibility issue also with this recommended
>>> workaround.
>>
>> Could you elaborate? I don't know what issue you're seeing.
>
> There have been reports from various projects mentioning problems in
> other packages ending up at make 4.4 and FIFOs solved by reverting to
> pipes.
>
> There is a make 4.4.1 release candidate 4.40.0.90 with patches that may
> help to deal with this:
>
> https://alpha.gnu.org/gnu/make/make-4.4.0.90.tar.lz{,.sig}
>
> https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=110&set=custom
>
test version built and on the way
please check
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-14 21:23 ` Marco Atzeri
@ 2023-02-15 18:09 ` Ken Brown
2023-02-16 7:47 ` Marco Atzeri
0 siblings, 1 reply; 25+ messages in thread
From: Ken Brown @ 2023-02-15 18:09 UTC (permalink / raw)
To: cygwin-apps
On 2/14/2023 4:23 PM, Marco Atzeri via Cygwin-apps wrote:
> On 14.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
>> There is a make 4.4.1 release candidate 4.40.0.90 with patches that
>> may help to deal with this:
>>
>> https://alpha.gnu.org/gnu/make/make-4.4.0.90.tar.lz{,.sig}
>>
>> https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=110&set=custom
>>
>
> test version built and on the way
The problems I reported with make 4.4 still exist in 4.4.0.90.
Until someone can figure out why this happens, you might want to
consider building make so that it doesn't default to using a FIFO for
its jobserver. I suggested to the make developers that they provide a
configure option for this purpose. If they don't do this, I think you
could accomplish the same thing by configuring make with
ac_cv_func_mkfifo=no.
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-15 18:09 ` Ken Brown
@ 2023-02-16 7:47 ` Marco Atzeri
2023-02-16 16:49 ` Brian Inglis
2023-02-17 2:21 ` Ken Brown
0 siblings, 2 replies; 25+ messages in thread
From: Marco Atzeri @ 2023-02-16 7:47 UTC (permalink / raw)
To: cygwin-apps
On 15.02.2023 19:09, Ken Brown via Cygwin-apps wrote:
> On 2/14/2023 4:23 PM, Marco Atzeri via Cygwin-apps wrote:
>> On 14.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
>>> There is a make 4.4.1 release candidate 4.40.0.90 with patches that
>>> may help to deal with this:
>>>
>>> https://alpha.gnu.org/gnu/make/make-4.4.0.90.tar.lz{,.sig}
>>>
>>> https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=110&set=custom
>>>
>>
>> test version built and on the way
>
> The problems I reported with make 4.4 still exist in 4.4.0.90.
>
> Until someone can figure out why this happens, you might want to
> consider building make so that it doesn't default to using a FIFO for
> its jobserver. I suggested to the make developers that they provide a
> configure option for this purpose. If they don't do this, I think you
> could accomplish the same thing by configuring make with
> ac_cv_func_mkfifo=no.
>
> Ken
version 2 uploaded.
Regards
Marco
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-16 7:47 ` Marco Atzeri
@ 2023-02-16 16:49 ` Brian Inglis
2023-02-16 18:57 ` Marco Atzeri
2023-02-17 2:21 ` Ken Brown
1 sibling, 1 reply; 25+ messages in thread
From: Brian Inglis @ 2023-02-16 16:49 UTC (permalink / raw)
To: cygwin-apps
On 2023-02-16 00:47, Marco Atzeri via Cygwin-apps wrote:
> On 15.02.2023 19:09, Ken Brown via Cygwin-apps wrote:
>> On 2/14/2023 4:23 PM, Marco Atzeri via Cygwin-apps wrote:
>>> On 14.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
>>>> There is a make 4.4.1 release candidate 4.40.0.90 with patches that may help
>>>> to deal with this:
>>>> https://alpha.gnu.org/gnu/make/make-4.4.0.90.tar.lz{,.sig}
>>>> https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=110&set=custom
>>> test version built and on the way
>> The problems I reported with make 4.4 still exist in 4.4.0.90.
>> Until someone can figure out why this happens, you might want to consider
>> building make so that it doesn't default to using a FIFO for its jobserver. I
>> suggested to the make developers that they provide a configure option for this
>> purpose. If they don't do this, I think you could accomplish the same thing
>> by configuring make with ac_cv_func_mkfifo=no.
> version 2 uploaded.
Hi guys,
Might be good to follow up on bug-make at gnu dot org to let them know that
pre-release does not help us, and it has been rereleased building with
ac_cv_func_mkfifo=no which does.
I notice that posix_spawn is defaulted, any chance it could be implicated?
--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada
La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-16 16:49 ` Brian Inglis
@ 2023-02-16 18:57 ` Marco Atzeri
2023-02-16 22:48 ` Brian Inglis
0 siblings, 1 reply; 25+ messages in thread
From: Marco Atzeri @ 2023-02-16 18:57 UTC (permalink / raw)
To: cygwin-apps
On 16.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
> On 2023-02-16 00:47, Marco Atzeri via Cygwin-apps wrote:
>> On 15.02.2023 19:09, Ken Brown via Cygwin-apps wrote:
>>> On 2/14/2023 4:23 PM, Marco Atzeri via Cygwin-apps wrote:
>>>> On 14.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
>>>>> There is a make 4.4.1 release candidate 4.40.0.90 with patches that
>>>>> may help to deal with this:
>>>>> https://alpha.gnu.org/gnu/make/make-4.4.0.90.tar.lz{,.sig}
>>>>> https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=110&set=custom
>
>>>> test version built and on the way
>
>>> The problems I reported with make 4.4 still exist in 4.4.0.90.
>>> Until someone can figure out why this happens, you might want to
>>> consider building make so that it doesn't default to using a FIFO for
>>> its jobserver. I suggested to the make developers that they provide
>>> a configure option for this purpose. If they don't do this, I think
>>> you could accomplish the same thing by configuring make with
>>> ac_cv_func_mkfifo=no.
>
>> version 2 uploaded.
>
> Hi guys,
>
> Might be good to follow up on bug-make at gnu dot org to let them know
> that pre-release does not help us, and it has been rereleased building
> with ac_cv_func_mkfifo=no which does.
>
> I notice that posix_spawn is defaulted, any chance it could be implicated?
>
what are you using as test ?
Standard build tests are fine for both -1 and -2
Marco
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-16 18:57 ` Marco Atzeri
@ 2023-02-16 22:48 ` Brian Inglis
0 siblings, 0 replies; 25+ messages in thread
From: Brian Inglis @ 2023-02-16 22:48 UTC (permalink / raw)
To: cygwin-apps
On 2023-02-16 11:57, Marco Atzeri via Cygwin-apps wrote:
> On 16.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
>> On 2023-02-16 00:47, Marco Atzeri via Cygwin-apps wrote:
>>> On 15.02.2023 19:09, Ken Brown via Cygwin-apps wrote:
>>>> On 2/14/2023 4:23 PM, Marco Atzeri via Cygwin-apps wrote:
>>>>> On 14.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
>>>>>> There is a make 4.4.1 release candidate 4.40.0.90 with patches that may
>>>>>> help to deal with this:
>>>>>> https://alpha.gnu.org/gnu/make/make-4.4.0.90.tar.lz{,.sig}
>>>>>> https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=110&set=custom
>>>>> test version built and on the way
>>>> The problems I reported with make 4.4 still exist in 4.4.0.90.
>>>> Until someone can figure out why this happens, you might want to consider
>>>> building make so that it doesn't default to using a FIFO for its jobserver.
>>>> I suggested to the make developers that they provide a configure option for
>>>> this purpose. If they don't do this, I think you could accomplish the same
>>>> thing by configuring make with ac_cv_func_mkfifo=no.
>>> version 2 uploaded.
>> Might be good to follow up on bug-make at gnu dot org to let them know that
>> pre-release does not help us, and it has been rereleased building with
>> ac_cv_func_mkfifo=no which does.
>> I notice that posix_spawn is defaulted, any chance it could be implicated?
> what are you using as test ?
> Standard build tests are fine for both -1 and -2
I'm not seeing make issues so far! x fingers - touch wood - no jinxes! ;^>
Are there high parallel make tests that show the issues with FIFO?
Just aware that if there are issues, sometimes flipping a config option gives a
different result, and occasionally a fix, or a broken build. :-(
--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada
La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-16 7:47 ` Marco Atzeri
2023-02-16 16:49 ` Brian Inglis
@ 2023-02-17 2:21 ` Ken Brown
2023-02-19 22:50 ` Ken Brown
2023-02-20 17:00 ` Achim Gratz
1 sibling, 2 replies; 25+ messages in thread
From: Ken Brown @ 2023-02-17 2:21 UTC (permalink / raw)
To: cygwin-apps
On 2/16/2023 2:47 AM, Marco Atzeri via Cygwin-apps wrote:
> On 15.02.2023 19:09, Ken Brown via Cygwin-apps wrote:
>> On 2/14/2023 4:23 PM, Marco Atzeri via Cygwin-apps wrote:
>>> On 14.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
>>>> There is a make 4.4.1 release candidate 4.40.0.90 with patches that
>>>> may help to deal with this:
>>>>
>>>> https://alpha.gnu.org/gnu/make/make-4.4.0.90.tar.lz{,.sig}
>>>>
>>>> https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=110&set=custom
>>>>
>>>
>>> test version built and on the way
>>
>> The problems I reported with make 4.4 still exist in 4.4.0.90.
>>
>> Until someone can figure out why this happens, you might want to
>> consider building make so that it doesn't default to using a FIFO for
>> its jobserver. I suggested to the make developers that they provide a
>> configure option for this purpose. If they don't do this, I think you
>> could accomplish the same thing by configuring make with
>> ac_cv_func_mkfifo=no.
>>
>> Ken
>
> version 2 uploaded.
Thanks, Marco. As expected, that fixes the problem for my test case
(building TeX Live). Obviously it would be better if the make
developers would provide a configure option to use a pipe for the
jobserver, but this is a good workaround if they don't.
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-17 2:21 ` Ken Brown
@ 2023-02-19 22:50 ` Ken Brown
2023-02-20 17:00 ` Achim Gratz
1 sibling, 0 replies; 25+ messages in thread
From: Ken Brown @ 2023-02-19 22:50 UTC (permalink / raw)
To: cygwin-apps
On 2/16/2023 9:21 PM, Ken Brown via Cygwin-apps wrote:
> On 2/16/2023 2:47 AM, Marco Atzeri via Cygwin-apps wrote:
>> On 15.02.2023 19:09, Ken Brown via Cygwin-apps wrote:
>>> On 2/14/2023 4:23 PM, Marco Atzeri via Cygwin-apps wrote:
>>>> On 14.02.2023 17:49, Brian Inglis via Cygwin-apps wrote:
>>>>> There is a make 4.4.1 release candidate 4.40.0.90 with patches that
>>>>> may help to deal with this:
>>>>>
>>>>> https://alpha.gnu.org/gnu/make/make-4.4.0.90.tar.lz{,.sig}
>>>>>
>>>>> https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=110&set=custom
>>>>>
>>>>
>>>> test version built and on the way
>>>
>>> The problems I reported with make 4.4 still exist in 4.4.0.90.
>>>
>>> Until someone can figure out why this happens, you might want to
>>> consider building make so that it doesn't default to using a FIFO for
>>> its jobserver. I suggested to the make developers that they provide
>>> a configure option for this purpose. If they don't do this, I think
>>> you could accomplish the same thing by configuring make with
>>> ac_cv_func_mkfifo=no.
>>>
>>> Ken
>>
>> version 2 uploaded.
>
> Thanks, Marco. As expected, that fixes the problem for my test case
> (building TeX Live). Obviously it would be better if the make
> developers would provide a configure option to use a pipe for the
> jobserver, but this is a good workaround if they don't.
Just to finish this off, there's now a new release candidate 4.4.0.91
for make, which makes the jobserver use a pipe by default on Cygwin.
You can override this default when building make by defining
JOBSERVER_USE_FIFO to 1, and you can override it when running make by
using the make option --jobserver-style=fifo. But there's no longer a
need to build make with ac_cv_func_mkfifo=no, and there's no longer a
need to use --jobserver-style=pipe when running make.
I'm very impressed with how responsive the make developer (Paul Smith) was.
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-17 2:21 ` Ken Brown
2023-02-19 22:50 ` Ken Brown
@ 2023-02-20 17:00 ` Achim Gratz
2023-02-20 17:14 ` Ken Brown
1 sibling, 1 reply; 25+ messages in thread
From: Achim Gratz @ 2023-02-20 17:00 UTC (permalink / raw)
To: cygwin-apps
Ken Brown via Cygwin-apps writes:
> Thanks, Marco. As expected, that fixes the problem for my test case
> (building TeX Live). Obviously it would be better if the make
> developers would provide a configure option to use a pipe for the
> jobserver, but this is a good workaround if they don't.
Does it actually do a parallel build? I've quickly tested a cmake based
project with it and it completely serializes the build. Reverting to
4.4 resolves that problem.
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-20 17:00 ` Achim Gratz
@ 2023-02-20 17:14 ` Ken Brown
2023-02-20 17:34 ` Marco Atzeri
2023-02-20 17:35 ` Brian Inglis
0 siblings, 2 replies; 25+ messages in thread
From: Ken Brown @ 2023-02-20 17:14 UTC (permalink / raw)
To: cygwin-apps
On 2/20/2023 12:00 PM, Achim Gratz via Cygwin-apps wrote:
> Ken Brown via Cygwin-apps writes:
>> Thanks, Marco. As expected, that fixes the problem for my test case
>> (building TeX Live). Obviously it would be better if the make
>> developers would provide a configure option to use a pipe for the
>> jobserver, but this is a good workaround if they don't.
>
> Does it actually do a parallel build? I've quickly tested a cmake based
> project with it and it completely serializes the build. Reverting to
> 4.4 resolves that problem.
It does for my build of TeX Live. I just tried 'make -j13 check', and I
saw many make processes and many sh processes running at once. [But I'm
currently testing a self-built make 4.4.0.91.]
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-20 17:14 ` Ken Brown
@ 2023-02-20 17:34 ` Marco Atzeri
2023-02-20 19:12 ` Marco Atzeri
2023-02-20 17:35 ` Brian Inglis
1 sibling, 1 reply; 25+ messages in thread
From: Marco Atzeri @ 2023-02-20 17:34 UTC (permalink / raw)
To: cygwin-apps
On 20.02.2023 18:14, Ken Brown via Cygwin-apps wrote:
> On 2/20/2023 12:00 PM, Achim Gratz via Cygwin-apps wrote:
>> Ken Brown via Cygwin-apps writes:
>>> Thanks, Marco. As expected, that fixes the problem for my test case
>>> (building TeX Live). Obviously it would be better if the make
>>> developers would provide a configure option to use a pipe for the
>>> jobserver, but this is a good workaround if they don't.
>>
>> Does it actually do a parallel build? I've quickly tested a cmake based
>> project with it and it completely serializes the build. Reverting to
>> 4.4 resolves that problem.
I noticed the same with another cmake build
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent
make rule.
but I do not see the message on Autoconf build
>
> It does for my build of TeX Live. I just tried 'make -j13 check', and I
> saw many make processes and many sh processes running at once. [But I'm
> currently testing a self-built make 4.4.0.91.]
>
> Ken
I will build it and test again
Regards
Marco
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-20 17:14 ` Ken Brown
2023-02-20 17:34 ` Marco Atzeri
@ 2023-02-20 17:35 ` Brian Inglis
1 sibling, 0 replies; 25+ messages in thread
From: Brian Inglis @ 2023-02-20 17:35 UTC (permalink / raw)
To: cygwin-apps
On 2023-02-20 10:14, Ken Brown via Cygwin-apps wrote:
> On 2/20/2023 12:00 PM, Achim Gratz via Cygwin-apps wrote:
>> Ken Brown via Cygwin-apps writes:
>>> Thanks, Marco. As expected, that fixes the problem for my test case
>>> (building TeX Live). Obviously it would be better if the make
>>> developers would provide a configure option to use a pipe for the
>>> jobserver, but this is a good workaround if they don't.
>>
>> Does it actually do a parallel build? I've quickly tested a cmake based
>> project with it and it completely serializes the build. Reverting to
>> 4.4 resolves that problem.
>
> It does for my build of TeX Live. I just tried 'make -j13 check', and I saw
> many make processes and many sh processes running at once. [But I'm currently
> testing a self-built make 4.4.0.91.]
Definitely parallel jobs visible in top with make 4.4.0.90-1
--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada
La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-20 17:34 ` Marco Atzeri
@ 2023-02-20 19:12 ` Marco Atzeri
2023-02-20 19:45 ` Ken Brown
2023-02-20 22:00 ` Ken Brown
0 siblings, 2 replies; 25+ messages in thread
From: Marco Atzeri @ 2023-02-20 19:12 UTC (permalink / raw)
To: cygwin-apps
On 20.02.2023 18:34, Marco Atzeri wrote:
> On 20.02.2023 18:14, Ken Brown via Cygwin-apps wrote:
>> On 2/20/2023 12:00 PM, Achim Gratz via Cygwin-apps wrote:
>>> Ken Brown via Cygwin-apps writes:
>>>> Thanks, Marco. As expected, that fixes the problem for my test case
>>>> (building TeX Live). Obviously it would be better if the make
>>>> developers would provide a configure option to use a pipe for the
>>>> jobserver, but this is a good workaround if they don't.
>>>
>>> Does it actually do a parallel build? I've quickly tested a cmake based
>>> project with it and it completely serializes the build. Reverting to
>>> 4.4 resolves that problem.
>
uploaded as test 4.4.0.91-1
following message is still present while missing in 4.4-1
> make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent
> make rule.
>
and I do not see clear evidence of parallelism like in 4.4-1
>>
>> It does for my build of TeX Live. I just tried 'make -j13 check', and
>> I saw many make processes and many sh processes running at once. [But
>> I'm currently testing a self-built make 4.4.0.91.]
>>
>> Ken
>
How are you configuring your build ?
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-20 19:12 ` Marco Atzeri
@ 2023-02-20 19:45 ` Ken Brown
2023-02-20 22:00 ` Ken Brown
1 sibling, 0 replies; 25+ messages in thread
From: Ken Brown @ 2023-02-20 19:45 UTC (permalink / raw)
To: cygwin-apps
On 2/20/2023 2:12 PM, Marco Atzeri via Cygwin-apps wrote:
>
>
> On 20.02.2023 18:34, Marco Atzeri wrote:
>> On 20.02.2023 18:14, Ken Brown via Cygwin-apps wrote:
>>> On 2/20/2023 12:00 PM, Achim Gratz via Cygwin-apps wrote:
>>>> Ken Brown via Cygwin-apps writes:
>>>>> Thanks, Marco. As expected, that fixes the problem for my test case
>>>>> (building TeX Live). Obviously it would be better if the make
>>>>> developers would provide a configure option to use a pipe for the
>>>>> jobserver, but this is a good workaround if they don't.
>>>>
>>>> Does it actually do a parallel build? I've quickly tested a cmake
>>>> based
>>>> project with it and it completely serializes the build. Reverting to
>>>> 4.4 resolves that problem.
>>
>
> uploaded as test 4.4.0.91-1
>
>
> following message is still present while missing in 4.4-1
>
>> make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent
>> make rule.
>>
>
> and I do not see clear evidence of parallelism like in 4.4-1
>
>>>
>>> It does for my build of TeX Live. I just tried 'make -j13 check',
>>> and I saw many make processes and many sh processes running at once.
>>> [But I'm currently testing a self-built make 4.4.0.91.]
>>>
>>> Ken
>>
>
> How are you configuring your build ?
Just a plain vanilla build, no configure options.
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-20 19:12 ` Marco Atzeri
2023-02-20 19:45 ` Ken Brown
@ 2023-02-20 22:00 ` Ken Brown
2023-02-20 23:29 ` Ken Brown
1 sibling, 1 reply; 25+ messages in thread
From: Ken Brown @ 2023-02-20 22:00 UTC (permalink / raw)
To: cygwin-apps
On 2/20/2023 2:12 PM, Marco Atzeri via Cygwin-apps wrote:
> uploaded as test 4.4.0.91-1
>
>
> following message is still present while missing in 4.4-1
>
>> make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent
>> make rule.
>>
>
> and I do not see clear evidence of parallelism like in 4.4-1
I doubt if this is related to the pipe vs. fifo issue, but maybe you
should test 4.4.0.90-1 to be sure.
Do you have a simple way to reproduce the problem? For example, does it
happen if you just make a toy cmake project?
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-20 22:00 ` Ken Brown
@ 2023-02-20 23:29 ` Ken Brown
2023-02-21 0:48 ` Ken Brown
0 siblings, 1 reply; 25+ messages in thread
From: Ken Brown @ 2023-02-20 23:29 UTC (permalink / raw)
To: cygwin-apps
On 2/20/2023 5:00 PM, Ken Brown via Cygwin-apps wrote:
> On 2/20/2023 2:12 PM, Marco Atzeri via Cygwin-apps wrote:
>> uploaded as test 4.4.0.91-1
>>
>>
>> following message is still present while missing in 4.4-1
>>
>>> make[1]: warning: jobserver unavailable: using -j1. Add '+' to
>>> parent make rule.
>>>
>>
>> and I do not see clear evidence of parallelism like in 4.4-1
>
> I doubt if this is related to the pipe vs. fifo issue, but maybe you
> should test 4.4.0.90-1 to be sure.
>
> Do you have a simple way to reproduce the problem? For example, does it
> happen if you just make a toy cmake project?
Ignore what I said about 'make -j13 check' working right in the TeX Live
build. I just retried it and saw the "jobserver unavailable" message.
When I saw several makes running at once, I think it was because of
recursion, not parallelism. So there's a real problem that can be
reproduced without cmake.
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-20 23:29 ` Ken Brown
@ 2023-02-21 0:48 ` Ken Brown
2023-02-27 21:41 ` Brian Inglis
0 siblings, 1 reply; 25+ messages in thread
From: Ken Brown @ 2023-02-21 0:48 UTC (permalink / raw)
To: cygwin-apps
On 2/20/2023 6:29 PM, Ken Brown via Cygwin-apps wrote:
> On 2/20/2023 5:00 PM, Ken Brown via Cygwin-apps wrote:
>> On 2/20/2023 2:12 PM, Marco Atzeri via Cygwin-apps wrote:
>>> uploaded as test 4.4.0.91-1
>>>
>>>
>>> following message is still present while missing in 4.4-1
>>>
>>>> make[1]: warning: jobserver unavailable: using -j1. Add '+' to
>>>> parent make rule.
>>>>
>>>
>>> and I do not see clear evidence of parallelism like in 4.4-1
>>
>> I doubt if this is related to the pipe vs. fifo issue, but maybe you
>> should test 4.4.0.90-1 to be sure.
>>
>> Do you have a simple way to reproduce the problem? For example, does
>> it happen if you just make a toy cmake project?
>
> Ignore what I said about 'make -j13 check' working right in the TeX Live
> build. I just retried it and saw the "jobserver unavailable" message.
> When I saw several makes running at once, I think it was because of
> recursion, not parallelism. So there's a real problem that can be
> reproduced without cmake.
I'm going to report this upstream, as a continuation of the upstream bug
report that I filed earlier.
Ken
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-21 0:48 ` Ken Brown
@ 2023-02-27 21:41 ` Brian Inglis
2023-02-27 22:45 ` Ken Brown
0 siblings, 1 reply; 25+ messages in thread
From: Brian Inglis @ 2023-02-27 21:41 UTC (permalink / raw)
To: cygwin-apps
On 2023-02-20 17:48, Ken Brown via Cygwin-apps wrote:
> On 2/20/2023 6:29 PM, Ken Brown via Cygwin-apps wrote:
>> On 2/20/2023 5:00 PM, Ken Brown via Cygwin-apps wrote:
>>> On 2/20/2023 2:12 PM, Marco Atzeri via Cygwin-apps wrote:
>>>> uploaded as test 4.4.0.91-1
>>>> following message is still present while missing in 4.4-1
>>>>> make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make
>>>>> rule.
>>>> and I do not see clear evidence of parallelism like in 4.4-1
>>> I doubt if this is related to the pipe vs. fifo issue, but maybe you should
>>> test 4.4.0.90-1 to be sure.
>>> Do you have a simple way to reproduce the problem? For example, does it
>>> happen if you just make a toy cmake project?
>> Ignore what I said about 'make -j13 check' working right in the TeX Live
>> build. I just retried it and saw the "jobserver unavailable" message. When I
>> saw several makes running at once, I think it was because of recursion, not
>> parallelism. So there's a real problem that can be reproduced without cmake.
> I'm going to report this upstream, as a continuation of the upstream bug report
> that I filed earlier.
Looks like the 4.4.1 release disables FIFO on Hurd and Cygwin.
Will be interesting to see if it is based on config test, for clues to what it
does not like, or just system?
--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada
La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-27 21:41 ` Brian Inglis
@ 2023-02-27 22:45 ` Ken Brown
2023-03-02 6:32 ` Marco Atzeri
0 siblings, 1 reply; 25+ messages in thread
From: Ken Brown @ 2023-02-27 22:45 UTC (permalink / raw)
To: cygwin-apps
On 2/27/2023 4:41 PM, Brian Inglis via Cygwin-apps wrote:
> Looks like the 4.4.1 release disables FIFO on Hurd and Cygwin.
> Will be interesting to see if it is based on config test, for clues to
> what it does not like, or just system?
In the case of HURD, it's because of failing tests in the test suite.
In the case of Cygwin, it's because of the problem I reported, in which
parallel make using a FIFO sometimes hangs when building TeX Live.
Anyone who wants to test this with their own builds can build make with
CPPFLAGS=-DJOBSERVER_USE_FIFO=1
That will re-enable the use a FIFO for the jobserver. One can then
force make to use a pipe on a given project by using the make option
--jobserver-style=pipe
I plan to do this myself, since I hope to debug the FIFO problem the
next time I see it. It's not easy to catch, because it doesn't happen
with every build of TeX Live. And when it does happen, the build has
typically been going on for close to two hours before it hangs.
P.S. Marco, I don't know how much trouble you want to go to on this, but
when you release make-4.4.1, you might also make a test release with
CPPFLAGS set as above. That would make it easier for adventurous people
to try builds that use a FIFO.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Heads up: Problems with parallel make
2023-02-27 22:45 ` Ken Brown
@ 2023-03-02 6:32 ` Marco Atzeri
0 siblings, 0 replies; 25+ messages in thread
From: Marco Atzeri @ 2023-03-02 6:32 UTC (permalink / raw)
To: cygwin-apps
On 27.02.2023 23:45, Ken Brown via Cygwin-apps wrote:
> On 2/27/2023 4:41 PM, Brian Inglis via Cygwin-apps wrote:
>> Looks like the 4.4.1 release disables FIFO on Hurd and Cygwin.
>> Will be interesting to see if it is based on config test, for clues to
>> what it does not like, or just system?
>
> In the case of HURD, it's because of failing tests in the test suite. In
> the case of Cygwin, it's because of the problem I reported, in which
> parallel make using a FIFO sometimes hangs when building TeX Live.
>
> Anyone who wants to test this with their own builds can build make with
>
> CPPFLAGS=-DJOBSERVER_USE_FIFO=1
>
> That will re-enable the use a FIFO for the jobserver. One can then
> force make to use a pipe on a given project by using the make option
>
> --jobserver-style=pipe
>
> I plan to do this myself, since I hope to debug the FIFO problem the
> next time I see it. It's not easy to catch, because it doesn't happen
> with every build of TeX Live. And when it does happen, the build has
> typically been going on for close to two hours before it hangs.
> but it could be
> P.S. Marco, I don't know how much trouble you want to go to on this, but
> when you release make-4.4.1, you might also make a test release with
> CPPFLAGS set as above. That would make it easier for adventurous people
> to try builds that use a FIFO.
uploaded 2 versions
4.4.1-1 default is built with CPPFLAGS=-DJOBSERVER_USE_FIFO=1
so it should behave like 4.4-1 previous build
4.4.1-2 test is vanilla build, so it should be built with pipe
the build logs are not very clear, so I need to dig more on them.
From my tests building qpdf, both are paralleling and maybe the
-2 is slight faster, but it could be due on machine load
real 19m38.493s
user 106m36.519s
sys 7m58.323s
real 18m51.680s
user 105m14.277s
sys 7m12.061s
Regards
Marco
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2023-03-02 6:32 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-02-13 23:05 Heads up: Problems with parallel make Ken Brown
2023-02-13 23:44 ` Brian Inglis
2023-02-14 0:38 ` Ken Brown
2023-02-14 16:49 ` Brian Inglis
2023-02-14 21:23 ` Marco Atzeri
2023-02-15 18:09 ` Ken Brown
2023-02-16 7:47 ` Marco Atzeri
2023-02-16 16:49 ` Brian Inglis
2023-02-16 18:57 ` Marco Atzeri
2023-02-16 22:48 ` Brian Inglis
2023-02-17 2:21 ` Ken Brown
2023-02-19 22:50 ` Ken Brown
2023-02-20 17:00 ` Achim Gratz
2023-02-20 17:14 ` Ken Brown
2023-02-20 17:34 ` Marco Atzeri
2023-02-20 19:12 ` Marco Atzeri
2023-02-20 19:45 ` Ken Brown
2023-02-20 22:00 ` Ken Brown
2023-02-20 23:29 ` Ken Brown
2023-02-21 0:48 ` Ken Brown
2023-02-27 21:41 ` Brian Inglis
2023-02-27 22:45 ` Ken Brown
2023-03-02 6:32 ` Marco Atzeri
2023-02-20 17:35 ` Brian Inglis
2023-02-14 2:33 ` Ken Brown
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).