Re: [PATCH cygport] lib/src_postinst.cygpart: parallelize __prepstrip

[Jon Turney <jon.turney@dronecode.org.uk>]

On 30/03/2023 20:49, Achim Gratz via Cygwin-apps wrote:
> Jon Turney via Cygwin-apps writes:
>> On 26/03/2023 19:12, Jon Turney via Cygwin-apps wrote:
>>>> -        usr/lib/gcc/*/lib*|usr/lib/gcc/*/*.o)
>>>> +        usr/lib/gcc/*/*.o)
>>> Why this change?
> 
> It looks like a mistake that I didn't catch.
> 
>>>
>>>> +    local nproc=$(nproc)
>>> This limit should probably be taken from the --jobs command line
>>> parameter, if specified
> 
> Yes, although one could argue that it should actually oversubscribe
> since the CPU load per process is expected to be significantly less than
> 100% per process.
> 
>> Looking at this a bit more, a couple of perhaps more serious problems:
>>
>> * The parallel invocations of __prepstrip_one are all appending to
>>    ${T}/.dbgsrc.out
>>
>> I don't see what makes that safe against interleaving of the output.
> 
> Line-buffering and the line being shorter than the buffer should.

and what causes line-buffering be on?

>> It's probably possible to have each instance write to a separate file
>> and collect them together in __prepdebugsrc
> 
> Nah, if you insist on making it _really_ safe regardless of line lenght
> and buffer size vagaries I'll do file locking on the output file.

If you lock the file while objdump is running, you've basically 
serialized this again...

>> * This patch causes several failures in the testsuite, e.g. with
>>    autotools/c testcase.
> 
> Which?
The full list of failing tests is:

21/54 autotools/c             FAIL            63.16s   exit status 1
22/54 autotools/gnome         FAIL           170.49s   exit status 1
23/54 autotools/gtkmm         FAIL           177.54s   exit status 1
24/54 autotools/mate          FAIL           131.24s   exit status 1
25/54 autotools/xfce          FAIL            85.64s   exit status 1
26/54 cmake/c                 FAIL            13.75s   exit status 1
28/54 cmake/qt5               FAIL            94.44s   exit status 1
37/54 lua/all                 FAIL            10.92s   exit status 1
38/54 httpd/apxs              FAIL            31.98s   exit status 1
44/54 perl/Module-Build       FAIL            16.09s   exit status 1
46/54 php/pecl                FAIL            48.13s   exit status 1
49/54 qmake/qt5               FAIL            46.84s   exit status 1

> 
>> On a brief attempt at debugging, it this looks like it's due to not
>> waiting for all the __prepstrip_one to complete before moving on, but
>> I think the final wait should prevent that, so idk.
> 
> I've seen an indication that the final wait doesn't work, but that is
> fixable by a sleep apparently.  Did You see that the process number
> limiting doesn't work?

No, but I haven't been trying to test it.

>> I'm not clear that invoking 'jobs', is actually doing anything, if
>> job-control is turned off in a non-interactive shell?
> 
> No, "jobs" shouldn't do anything, but wait should still work I think
> (the manpage talks about jobs, but it really means "children").  But
> then again, a bit of googling tells me that the bashism "wait -n"
> actually needs job control to be enabled, natch.