parallel make

parallel make

[Rich Elberger]

Hi folks,
Currently our build environment uses parallel make (-j jobs option) on all
our unixes using gnu tools.  We use an older version of gnu tools on our
windows boxes.  The older make on the windows box does not do parallel
make (or at least correctly).  I want to upgrade to the latest cygwin to
see if parallel make works, but this will require significant changes to
our build engine, so I would like to confirm a few things if possible.

1. Does the -j jobs option work well on windows. (part b: does it work
with the MSVC (6/7) compiler (which probably doesn't make a difference
anyway)?
2. Has anyone done this in a very large project, and if so, do you have
any performance gain stats (which, I acknowledge, is tied to
processor-intensive makes and how many processors the machine has).

I realize that dos does not allow for threading so I don't know if this is
a cmd.exe-related issue or not (since cmd.exe is the parent shell, I
don't know if this affects the behavior).

thanks in advance --


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: parallel make

[Rolf Campbell]

make -j does work in cygwin, to an extent.  If you try to use too many 
processes, cygwin seems to flip out.  -j20 does seem to work fine though 
(it only starts acting strang around -j100).

On my large build system, we have a slow disk, and I find that when the 
disk cache is empty, -j4 speeds it up about 40%.  When the build system 
is cached, -j slows things down by a few percent.  This is dealing with 
1 processor.  Of course, if you have multiple processors, the speed 
should scale linearly (with 4 cpu's, -j4 is about 70% faster).

Rich Elberger wrote:

> Hi folks,
> Currently our build environment uses parallel make (-j jobs option) on all
> our unixes using gnu tools.  We use an older version of gnu tools on our
> windows boxes.  The older make on the windows box does not do parallel
> make (or at least correctly).  I want to upgrade to the latest cygwin to
> see if parallel make works, but this will require significant changes to
> our build engine, so I would like to confirm a few things if possible.
> 
> 1. Does the -j jobs option work well on windows. (part b: does it work
> with the MSVC (6/7) compiler (which probably doesn't make a difference
> anyway)?
> 2. Has anyone done this in a very large project, and if so, do you have
> any performance gain stats (which, I acknowledge, is tied to
> processor-intensive makes and how many processors the machine has).
> 
> I realize that dos does not allow for threading so I don't know if this is
> a cmd.exe-related issue or not (since cmd.exe is the parent shell, I
> don't know if this affects the behavior).
> 
> thanks in advance --
> 
> 



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: parallel make

[Christopher Faylor]

On Wed, Aug 27, 2003 at 08:15:19PM -0400, Rolf Campbell wrote:
>make -j does work in cygwin, to an extent.  If you try to use too many 
>processes, cygwin seems to flip out.  -j20 does seem to work fine though 
>(it only starts acting strang around -j100).

More likely it "flips out" around -j62 or so.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: parallel make

[Rich Elberger]

This is starting to sound promising.  I hope others have similar
experiences.

Unfortunately I cannot just take make and the cyg dll.  The problem comes
in with how different utils interpret drive paths:

our old gnu:  //c/...
perl: c:/...
cyg:  /cygdrive/c/... (we mount NFS exports, hence the drive notation)...

So, we basically need to move things from the inside out rather than start
patching things on the outside of the system, eventually getting to the
core.  This makes the migration pretty expensive, so I would like to see
more cases before making the resource justification case.  If it's a set
of production machines, no problem, but the change would have to occur on
all development workstations.

I highly appreciate everyone's help because I think this is the only forum
I could ever get such kind of feedback.

-- rich

On Wed, 27 Aug 2003, Rolf Campbell wrote:

> make -j does work in cygwin, to an extent.  If you try to use too many
> processes, cygwin seems to flip out.  -j20 does seem to work fine though
> (it only starts acting strang around -j100).
>
> On my large build system, we have a slow disk, and I find that when the
> disk cache is empty, -j4 speeds it up about 40%.  When the build system
> is cached, -j slows things down by a few percent.  This is dealing with
> 1 processor.  Of course, if you have multiple processors, the speed
> should scale linearly (with 4 cpu's, -j4 is about 70% faster).
>
> Rich Elberger wrote:
>
> > Hi folks,
> > Currently our build environment uses parallel make (-j jobs option) on all
> > our unixes using gnu tools.  We use an older version of gnu tools on our
> > windows boxes.  The older make on the windows box does not do parallel
> > make (or at least correctly).  I want to upgrade to the latest cygwin to
> > see if parallel make works, but this will require significant changes to
> > our build engine, so I would like to confirm a few things if possible.
> >
> > 1. Does the -j jobs option work well on windows. (part b: does it work
> > with the MSVC (6/7) compiler (which probably doesn't make a difference
> > anyway)?
> > 2. Has anyone done this in a very large project, and if so, do you have
> > any performance gain stats (which, I acknowledge, is tied to
> > processor-intensive makes and how many processors the machine has).
> >
> > I realize that dos does not allow for threading so I don't know if this is
> > a cmd.exe-related issue or not (since cmd.exe is the parent shell, I
> > don't know if this affects the behavior).
> >
> > thanks in advance --
> >
> >
>
>
>
> --
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
> Problem reports:       http://cygwin.com/problems.html
> Documentation:         http://cygwin.com/docs.html
> FAQ:                   http://cygwin.com/faq/
>
>

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: parallel make

[Rolf Campbell]

Christopher Faylor wrote:

> On Wed, Aug 27, 2003 at 08:15:19PM -0400, Rolf Campbell wrote:
> 
>>make -j does work in cygwin, to an extent.  If you try to use too many 
>>processes, cygwin seems to flip out.  -j20 does seem to work fine though 
>>(it only starts acting strang around -j100).
> 
> More likely it "flips out" around -j62 or so.
> 
> cgf

Is this a real responce?  Or is this some more dry humour?

-Rolf



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: parallel make

[Christopher Faylor]

On Wed, Aug 27, 2003 at 09:03:55PM -0400, Rolf Campbell wrote:
>Christopher Faylor wrote:
>>On Wed, Aug 27, 2003 at 08:15:19PM -0400, Rolf Campbell wrote:
>>>make -j does work in cygwin, to an extent.  If you try to use too many
>>>processes, cygwin seems to flip out.  -j20 does seem to work fine
>>>though (it only starts acting strang around -j100).
>>
>>More likely it "flips out" around -j62 or so.
>
>Is this a real responce?  Or is this some more dry humour?

It's a real response.
--
Please use the resources at cygwin.com rather than sending personal email.
Special for spam email harvesters: send email to aaaspam@sourceware.org
and be permanently blocked from mailing lists at sources.redhat.com

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: parallel make

[Igor Pechtchanski]

Rich,

First off, if you're willing to forgo the ability to access UNC paths from
Cygwin, you can set your Cygdrive prefix to '//' to address the drives in
the same way you did in your old GNU toolset (see
<http://cygwin.com/ml/cygwin/2003-05/msg00825.html>).

Secondly, why not use Cygwin perl?  This way you automatically get
the cygdrive prefix that Cygwin is configured with.
	Igor

On Wed, 27 Aug 2003, Rich Elberger wrote:

> This is starting to sound promising.  I hope others have similar
> experiences.
>
> Unfortunately I cannot just take make and the cyg dll.  The problem comes
> in with how different utils interpret drive paths:
>
> our old gnu:  //c/...
> perl: c:/...
> cyg:  /cygdrive/c/... (we mount NFS exports, hence the drive notation)...
>
> So, we basically need to move things from the inside out rather than start
> patching things on the outside of the system, eventually getting to the
> core.  This makes the migration pretty expensive, so I would like to see
> more cases before making the resource justification case.  If it's a set
> of production machines, no problem, but the change would have to occur on
> all development workstations.
>
> I highly appreciate everyone's help because I think this is the only forum
> I could ever get such kind of feedback.
>
> -- rich
>
> On Wed, 27 Aug 2003, Rolf Campbell wrote:
>
> > make -j does work in cygwin, to an extent.  If you try to use too many
> > processes, cygwin seems to flip out.  -j20 does seem to work fine though
> > (it only starts acting strang around -j100).
> >
> > On my large build system, we have a slow disk, and I find that when the
> > disk cache is empty, -j4 speeds it up about 40%.  When the build system
> > is cached, -j slows things down by a few percent.  This is dealing with
> > 1 processor.  Of course, if you have multiple processors, the speed
> > should scale linearly (with 4 cpu's, -j4 is about 70% faster).
> >
> > Rich Elberger wrote:
> >
> > > Hi folks,
> > > Currently our build environment uses parallel make (-j jobs option) on all
> > > our unixes using gnu tools.  We use an older version of gnu tools on our
> > > windows boxes.  The older make on the windows box does not do parallel
> > > make (or at least correctly).  I want to upgrade to the latest cygwin to
> > > see if parallel make works, but this will require significant changes to
> > > our build engine, so I would like to confirm a few things if possible.
> > >
> > > 1. Does the -j jobs option work well on windows. (part b: does it work
> > > with the MSVC (6/7) compiler (which probably doesn't make a difference
> > > anyway)?
> > > 2. Has anyone done this in a very large project, and if so, do you have
> > > any performance gain stats (which, I acknowledge, is tied to
> > > processor-intensive makes and how many processors the machine has).
> > >
> > > I realize that dos does not allow for threading so I don't know if this is
> > > a cmd.exe-related issue or not (since cmd.exe is the parent shell, I
> > > don't know if this affects the behavior).
> > >
> > > thanks in advance --

-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_		pechtcha@cs.nyu.edu
ZZZzz /,`.-'`'    -.  ;-;;,_		igor@watson.ibm.com
     |,4-  ) )-,_. ,\ (  `'-'		Igor Pechtchanski, Ph.D.
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

"I have since come to realize that being between your mentor and his route
to the bathroom is a major career booster."  -- Patrick Naughton

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: parallel make

[Christopher Faylor]

On Wed, Aug 27, 2003 at 09:35:05PM -0400, Igor Pechtchanski wrote:
>First off, if you're willing to forgo the ability to access UNC paths
>from Cygwin, you can set your Cygdrive prefix to '//' to address the
>drives in the same way you did in your old GNU toolset

But not for long...

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: parallel make

[Rich Elberger]

On Wed, 27 Aug 2003, Igor Pechtchanski wrote:

> Rich,
>
> First off, if you're willing to forgo the ability to access UNC paths from
> Cygwin, you can set your Cygdrive prefix to '//' to address the drives in
> the same way you did in your old GNU toolset (see
> <http://cygwin.com/ml/cygwin/2003-05/msg00825.html>).
>
> Secondly, why not use Cygwin perl?  This way you automatically get
> the cygdrive prefix that Cygwin is configured with.

This is the way we will go, but this is another level of change. To date,
there has been a level of expectation, or set of assumtions, about the
win32 environment. These assumptions will change given the new
environment, and of course all the integration tests for the system
ensues.  I wish it could all be so simple, but really I would be fooling
myself.

> 	Igor
>
> On Wed, 27 Aug 2003, Rich Elberger wrote:
>
> > This is starting to sound promising.  I hope others have similar
> > experiences.
> >
> > Unfortunately I cannot just take make and the cyg dll.  The problem comes
> > in with how different utils interpret drive paths:
> >
> > our old gnu:  //c/...
> > perl: c:/...
> > cyg:  /cygdrive/c/... (we mount NFS exports, hence the drive notation)...
> >
> > So, we basically need to move things from the inside out rather than start
> > patching things on the outside of the system, eventually getting to the
> > core.  This makes the migration pretty expensive, so I would like to see
> > more cases before making the resource justification case.  If it's a set
> > of production machines, no problem, but the change would have to occur on
> > all development workstations.
> >
> > I highly appreciate everyone's help because I think this is the only forum
> > I could ever get such kind of feedback.
> >
> > -- rich
> >
> > On Wed, 27 Aug 2003, Rolf Campbell wrote:
> >
> > > make -j does work in cygwin, to an extent.  If you try to use too many
> > > processes, cygwin seems to flip out.  -j20 does seem to work fine though
> > > (it only starts acting strang around -j100).
> > >
> > > On my large build system, we have a slow disk, and I find that when the
> > > disk cache is empty, -j4 speeds it up about 40%.  When the build system
> > > is cached, -j slows things down by a few percent.  This is dealing with
> > > 1 processor.  Of course, if you have multiple processors, the speed
> > > should scale linearly (with 4 cpu's, -j4 is about 70% faster).
> > >
> > > Rich Elberger wrote:
> > >
> > > > Hi folks,
> > > > Currently our build environment uses parallel make (-j jobs option) on all
> > > > our unixes using gnu tools.  We use an older version of gnu tools on our
> > > > windows boxes.  The older make on the windows box does not do parallel
> > > > make (or at least correctly).  I want to upgrade to the latest cygwin to
> > > > see if parallel make works, but this will require significant changes to
> > > > our build engine, so I would like to confirm a few things if possible.
> > > >
> > > > 1. Does the -j jobs option work well on windows. (part b: does it work
> > > > with the MSVC (6/7) compiler (which probably doesn't make a difference
> > > > anyway)?
> > > > 2. Has anyone done this in a very large project, and if so, do you have
> > > > any performance gain stats (which, I acknowledge, is tied to
> > > > processor-intensive makes and how many processors the machine has).
> > > >
> > > > I realize that dos does not allow for threading so I don't know if this is
> > > > a cmd.exe-related issue or not (since cmd.exe is the parent shell, I
> > > > don't know if this affects the behavior).
> > > >
> > > > thanks in advance --
>
> --
> 				http://cs.nyu.edu/~pechtcha/
>       |\      _,,,---,,_		pechtcha@cs.nyu.edu
> ZZZzz /,`.-'`'    -.  ;-;;,_		igor@watson.ibm.com
>      |,4-  ) )-,_. ,\ (  `'-'		Igor Pechtchanski, Ph.D.
>     '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!
>
> "I have since come to realize that being between your mentor and his route
> to the bathroom is a major career booster."  -- Patrick Naughton
>

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: parallel make

[Rolf Campbell]

Christopher Faylor wrote:
> On Wed, Aug 27, 2003 at 09:03:55PM -0400, Rolf Campbell wrote:
> 
>>Christopher Faylor wrote:
>>
>>>On Wed, Aug 27, 2003 at 08:15:19PM -0400, Rolf Campbell wrote:
>>>
>>>>make -j does work in cygwin, to an extent.  If you try to use too many
>>>>processes, cygwin seems to flip out.  -j20 does seem to work fine
>>>>though (it only starts acting strang around -j100).
>>>
>>>More likely it "flips out" around -j62 or so.
>>
>>Is this a real responce?  Or is this some more dry humour?
> 
> 
> It's a real response.

So, you are saying that there are known scalability issuse with cygwin? 
  Like with the number of processes?

Does cygwin only allow 64 processes at once?

-Rolf



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/