* parallel make @ 2003-08-27 21:23 Rich Elberger 2003-08-28 0:55 ` Rolf Campbell 0 siblings, 1 reply; 10+ messages in thread From: Rich Elberger @ 2003-08-27 21:23 UTC (permalink / raw) To: cygwin Hi folks, Currently our build environment uses parallel make (-j jobs option) on all our unixes using gnu tools. We use an older version of gnu tools on our windows boxes. The older make on the windows box does not do parallel make (or at least correctly). I want to upgrade to the latest cygwin to see if parallel make works, but this will require significant changes to our build engine, so I would like to confirm a few things if possible. 1. Does the -j jobs option work well on windows. (part b: does it work with the MSVC (6/7) compiler (which probably doesn't make a difference anyway)? 2. Has anyone done this in a very large project, and if so, do you have any performance gain stats (which, I acknowledge, is tied to processor-intensive makes and how many processors the machine has). I realize that dos does not allow for threading so I don't know if this is a cmd.exe-related issue or not (since cmd.exe is the parent shell, I don't know if this affects the behavior). thanks in advance -- -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parallel make 2003-08-27 21:23 parallel make Rich Elberger @ 2003-08-28 0:55 ` Rolf Campbell 2003-08-28 1:04 ` Christopher Faylor 2003-08-28 1:32 ` Rich Elberger 0 siblings, 2 replies; 10+ messages in thread From: Rolf Campbell @ 2003-08-28 0:55 UTC (permalink / raw) To: cygwin make -j does work in cygwin, to an extent. If you try to use too many processes, cygwin seems to flip out. -j20 does seem to work fine though (it only starts acting strang around -j100). On my large build system, we have a slow disk, and I find that when the disk cache is empty, -j4 speeds it up about 40%. When the build system is cached, -j slows things down by a few percent. This is dealing with 1 processor. Of course, if you have multiple processors, the speed should scale linearly (with 4 cpu's, -j4 is about 70% faster). Rich Elberger wrote: > Hi folks, > Currently our build environment uses parallel make (-j jobs option) on all > our unixes using gnu tools. We use an older version of gnu tools on our > windows boxes. The older make on the windows box does not do parallel > make (or at least correctly). I want to upgrade to the latest cygwin to > see if parallel make works, but this will require significant changes to > our build engine, so I would like to confirm a few things if possible. > > 1. Does the -j jobs option work well on windows. (part b: does it work > with the MSVC (6/7) compiler (which probably doesn't make a difference > anyway)? > 2. Has anyone done this in a very large project, and if so, do you have > any performance gain stats (which, I acknowledge, is tied to > processor-intensive makes and how many processors the machine has). > > I realize that dos does not allow for threading so I don't know if this is > a cmd.exe-related issue or not (since cmd.exe is the parent shell, I > don't know if this affects the behavior). > > thanks in advance -- > > -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parallel make 2003-08-28 0:55 ` Rolf Campbell @ 2003-08-28 1:04 ` Christopher Faylor 2003-08-28 1:35 ` Rolf Campbell 2003-08-28 1:32 ` Rich Elberger 1 sibling, 1 reply; 10+ messages in thread From: Christopher Faylor @ 2003-08-28 1:04 UTC (permalink / raw) To: cygwin On Wed, Aug 27, 2003 at 08:15:19PM -0400, Rolf Campbell wrote: >make -j does work in cygwin, to an extent. If you try to use too many >processes, cygwin seems to flip out. -j20 does seem to work fine though >(it only starts acting strang around -j100). More likely it "flips out" around -j62 or so. cgf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parallel make 2003-08-28 1:04 ` Christopher Faylor @ 2003-08-28 1:35 ` Rolf Campbell 2003-08-28 1:35 ` Christopher Faylor 0 siblings, 1 reply; 10+ messages in thread From: Rolf Campbell @ 2003-08-28 1:35 UTC (permalink / raw) To: cygwin Christopher Faylor wrote: > On Wed, Aug 27, 2003 at 08:15:19PM -0400, Rolf Campbell wrote: > >>make -j does work in cygwin, to an extent. If you try to use too many >>processes, cygwin seems to flip out. -j20 does seem to work fine though >>(it only starts acting strang around -j100). > > More likely it "flips out" around -j62 or so. > > cgf Is this a real responce? Or is this some more dry humour? -Rolf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parallel make 2003-08-28 1:35 ` Rolf Campbell @ 2003-08-28 1:35 ` Christopher Faylor 2003-08-28 21:08 ` Rolf Campbell 0 siblings, 1 reply; 10+ messages in thread From: Christopher Faylor @ 2003-08-28 1:35 UTC (permalink / raw) To: cygwin On Wed, Aug 27, 2003 at 09:03:55PM -0400, Rolf Campbell wrote: >Christopher Faylor wrote: >>On Wed, Aug 27, 2003 at 08:15:19PM -0400, Rolf Campbell wrote: >>>make -j does work in cygwin, to an extent. If you try to use too many >>>processes, cygwin seems to flip out. -j20 does seem to work fine >>>though (it only starts acting strang around -j100). >> >>More likely it "flips out" around -j62 or so. > >Is this a real responce? Or is this some more dry humour? It's a real response. -- Please use the resources at cygwin.com rather than sending personal email. Special for spam email harvesters: send email to aaaspam@sourceware.org and be permanently blocked from mailing lists at sources.redhat.com -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parallel make 2003-08-28 1:35 ` Christopher Faylor @ 2003-08-28 21:08 ` Rolf Campbell 0 siblings, 0 replies; 10+ messages in thread From: Rolf Campbell @ 2003-08-28 21:08 UTC (permalink / raw) To: cygwin Christopher Faylor wrote: > On Wed, Aug 27, 2003 at 09:03:55PM -0400, Rolf Campbell wrote: > >>Christopher Faylor wrote: >> >>>On Wed, Aug 27, 2003 at 08:15:19PM -0400, Rolf Campbell wrote: >>> >>>>make -j does work in cygwin, to an extent. If you try to use too many >>>>processes, cygwin seems to flip out. -j20 does seem to work fine >>>>though (it only starts acting strang around -j100). >>> >>>More likely it "flips out" around -j62 or so. >> >>Is this a real responce? Or is this some more dry humour? > > > It's a real response. So, you are saying that there are known scalability issuse with cygwin? Like with the number of processes? Does cygwin only allow 64 processes at once? -Rolf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parallel make 2003-08-28 0:55 ` Rolf Campbell 2003-08-28 1:04 ` Christopher Faylor @ 2003-08-28 1:32 ` Rich Elberger 2003-08-28 2:16 ` Igor Pechtchanski 1 sibling, 1 reply; 10+ messages in thread From: Rich Elberger @ 2003-08-28 1:32 UTC (permalink / raw) To: Rolf Campbell, cygwin This is starting to sound promising. I hope others have similar experiences. Unfortunately I cannot just take make and the cyg dll. The problem comes in with how different utils interpret drive paths: our old gnu: //c/... perl: c:/... cyg: /cygdrive/c/... (we mount NFS exports, hence the drive notation)... So, we basically need to move things from the inside out rather than start patching things on the outside of the system, eventually getting to the core. This makes the migration pretty expensive, so I would like to see more cases before making the resource justification case. If it's a set of production machines, no problem, but the change would have to occur on all development workstations. I highly appreciate everyone's help because I think this is the only forum I could ever get such kind of feedback. -- rich On Wed, 27 Aug 2003, Rolf Campbell wrote: > make -j does work in cygwin, to an extent. If you try to use too many > processes, cygwin seems to flip out. -j20 does seem to work fine though > (it only starts acting strang around -j100). > > On my large build system, we have a slow disk, and I find that when the > disk cache is empty, -j4 speeds it up about 40%. When the build system > is cached, -j slows things down by a few percent. This is dealing with > 1 processor. Of course, if you have multiple processors, the speed > should scale linearly (with 4 cpu's, -j4 is about 70% faster). > > Rich Elberger wrote: > > > Hi folks, > > Currently our build environment uses parallel make (-j jobs option) on all > > our unixes using gnu tools. We use an older version of gnu tools on our > > windows boxes. The older make on the windows box does not do parallel > > make (or at least correctly). I want to upgrade to the latest cygwin to > > see if parallel make works, but this will require significant changes to > > our build engine, so I would like to confirm a few things if possible. > > > > 1. Does the -j jobs option work well on windows. (part b: does it work > > with the MSVC (6/7) compiler (which probably doesn't make a difference > > anyway)? > > 2. Has anyone done this in a very large project, and if so, do you have > > any performance gain stats (which, I acknowledge, is tied to > > processor-intensive makes and how many processors the machine has). > > > > I realize that dos does not allow for threading so I don't know if this is > > a cmd.exe-related issue or not (since cmd.exe is the parent shell, I > > don't know if this affects the behavior). > > > > thanks in advance -- > > > > > > > > -- > Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple > Problem reports: http://cygwin.com/problems.html > Documentation: http://cygwin.com/docs.html > FAQ: http://cygwin.com/faq/ > > -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parallel make 2003-08-28 1:32 ` Rich Elberger @ 2003-08-28 2:16 ` Igor Pechtchanski 2003-08-28 5:25 ` Christopher Faylor 2003-08-28 5:32 ` Rich Elberger 0 siblings, 2 replies; 10+ messages in thread From: Igor Pechtchanski @ 2003-08-28 2:16 UTC (permalink / raw) To: Rich Elberger; +Cc: cygwin Rich, First off, if you're willing to forgo the ability to access UNC paths from Cygwin, you can set your Cygdrive prefix to '//' to address the drives in the same way you did in your old GNU toolset (see <http://cygwin.com/ml/cygwin/2003-05/msg00825.html>). Secondly, why not use Cygwin perl? This way you automatically get the cygdrive prefix that Cygwin is configured with. Igor On Wed, 27 Aug 2003, Rich Elberger wrote: > This is starting to sound promising. I hope others have similar > experiences. > > Unfortunately I cannot just take make and the cyg dll. The problem comes > in with how different utils interpret drive paths: > > our old gnu: //c/... > perl: c:/... > cyg: /cygdrive/c/... (we mount NFS exports, hence the drive notation)... > > So, we basically need to move things from the inside out rather than start > patching things on the outside of the system, eventually getting to the > core. This makes the migration pretty expensive, so I would like to see > more cases before making the resource justification case. If it's a set > of production machines, no problem, but the change would have to occur on > all development workstations. > > I highly appreciate everyone's help because I think this is the only forum > I could ever get such kind of feedback. > > -- rich > > On Wed, 27 Aug 2003, Rolf Campbell wrote: > > > make -j does work in cygwin, to an extent. If you try to use too many > > processes, cygwin seems to flip out. -j20 does seem to work fine though > > (it only starts acting strang around -j100). > > > > On my large build system, we have a slow disk, and I find that when the > > disk cache is empty, -j4 speeds it up about 40%. When the build system > > is cached, -j slows things down by a few percent. This is dealing with > > 1 processor. Of course, if you have multiple processors, the speed > > should scale linearly (with 4 cpu's, -j4 is about 70% faster). > > > > Rich Elberger wrote: > > > > > Hi folks, > > > Currently our build environment uses parallel make (-j jobs option) on all > > > our unixes using gnu tools. We use an older version of gnu tools on our > > > windows boxes. The older make on the windows box does not do parallel > > > make (or at least correctly). I want to upgrade to the latest cygwin to > > > see if parallel make works, but this will require significant changes to > > > our build engine, so I would like to confirm a few things if possible. > > > > > > 1. Does the -j jobs option work well on windows. (part b: does it work > > > with the MSVC (6/7) compiler (which probably doesn't make a difference > > > anyway)? > > > 2. Has anyone done this in a very large project, and if so, do you have > > > any performance gain stats (which, I acknowledge, is tied to > > > processor-intensive makes and how many processors the machine has). > > > > > > I realize that dos does not allow for threading so I don't know if this is > > > a cmd.exe-related issue or not (since cmd.exe is the parent shell, I > > > don't know if this affects the behavior). > > > > > > thanks in advance -- -- http://cs.nyu.edu/~pechtcha/ |\ _,,,---,,_ pechtcha@cs.nyu.edu ZZZzz /,`.-'`' -. ;-;;,_ igor@watson.ibm.com |,4- ) )-,_. ,\ ( `'-' Igor Pechtchanski, Ph.D. '---''(_/--' `-'\_) fL a.k.a JaguaR-R-R-r-r-r-.-.-. Meow! "I have since come to realize that being between your mentor and his route to the bathroom is a major career booster." -- Patrick Naughton -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parallel make 2003-08-28 2:16 ` Igor Pechtchanski @ 2003-08-28 5:25 ` Christopher Faylor 2003-08-28 5:32 ` Rich Elberger 1 sibling, 0 replies; 10+ messages in thread From: Christopher Faylor @ 2003-08-28 5:25 UTC (permalink / raw) To: cygwin On Wed, Aug 27, 2003 at 09:35:05PM -0400, Igor Pechtchanski wrote: >First off, if you're willing to forgo the ability to access UNC paths >from Cygwin, you can set your Cygdrive prefix to '//' to address the >drives in the same way you did in your old GNU toolset But not for long... cgf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: parallel make 2003-08-28 2:16 ` Igor Pechtchanski 2003-08-28 5:25 ` Christopher Faylor @ 2003-08-28 5:32 ` Rich Elberger 1 sibling, 0 replies; 10+ messages in thread From: Rich Elberger @ 2003-08-28 5:32 UTC (permalink / raw) To: cygwin On Wed, 27 Aug 2003, Igor Pechtchanski wrote: > Rich, > > First off, if you're willing to forgo the ability to access UNC paths from > Cygwin, you can set your Cygdrive prefix to '//' to address the drives in > the same way you did in your old GNU toolset (see > <http://cygwin.com/ml/cygwin/2003-05/msg00825.html>). > > Secondly, why not use Cygwin perl? This way you automatically get > the cygdrive prefix that Cygwin is configured with. This is the way we will go, but this is another level of change. To date, there has been a level of expectation, or set of assumtions, about the win32 environment. These assumptions will change given the new environment, and of course all the integration tests for the system ensues. I wish it could all be so simple, but really I would be fooling myself. > Igor > > On Wed, 27 Aug 2003, Rich Elberger wrote: > > > This is starting to sound promising. I hope others have similar > > experiences. > > > > Unfortunately I cannot just take make and the cyg dll. The problem comes > > in with how different utils interpret drive paths: > > > > our old gnu: //c/... > > perl: c:/... > > cyg: /cygdrive/c/... (we mount NFS exports, hence the drive notation)... > > > > So, we basically need to move things from the inside out rather than start > > patching things on the outside of the system, eventually getting to the > > core. This makes the migration pretty expensive, so I would like to see > > more cases before making the resource justification case. If it's a set > > of production machines, no problem, but the change would have to occur on > > all development workstations. > > > > I highly appreciate everyone's help because I think this is the only forum > > I could ever get such kind of feedback. > > > > -- rich > > > > On Wed, 27 Aug 2003, Rolf Campbell wrote: > > > > > make -j does work in cygwin, to an extent. If you try to use too many > > > processes, cygwin seems to flip out. -j20 does seem to work fine though > > > (it only starts acting strang around -j100). > > > > > > On my large build system, we have a slow disk, and I find that when the > > > disk cache is empty, -j4 speeds it up about 40%. When the build system > > > is cached, -j slows things down by a few percent. This is dealing with > > > 1 processor. Of course, if you have multiple processors, the speed > > > should scale linearly (with 4 cpu's, -j4 is about 70% faster). > > > > > > Rich Elberger wrote: > > > > > > > Hi folks, > > > > Currently our build environment uses parallel make (-j jobs option) on all > > > > our unixes using gnu tools. We use an older version of gnu tools on our > > > > windows boxes. The older make on the windows box does not do parallel > > > > make (or at least correctly). I want to upgrade to the latest cygwin to > > > > see if parallel make works, but this will require significant changes to > > > > our build engine, so I would like to confirm a few things if possible. > > > > > > > > 1. Does the -j jobs option work well on windows. (part b: does it work > > > > with the MSVC (6/7) compiler (which probably doesn't make a difference > > > > anyway)? > > > > 2. Has anyone done this in a very large project, and if so, do you have > > > > any performance gain stats (which, I acknowledge, is tied to > > > > processor-intensive makes and how many processors the machine has). > > > > > > > > I realize that dos does not allow for threading so I don't know if this is > > > > a cmd.exe-related issue or not (since cmd.exe is the parent shell, I > > > > don't know if this affects the behavior). > > > > > > > > thanks in advance -- > > -- > http://cs.nyu.edu/~pechtcha/ > |\ _,,,---,,_ pechtcha@cs.nyu.edu > ZZZzz /,`.-'`' -. ;-;;,_ igor@watson.ibm.com > |,4- ) )-,_. ,\ ( `'-' Igor Pechtchanski, Ph.D. > '---''(_/--' `-'\_) fL a.k.a JaguaR-R-R-r-r-r-.-.-. Meow! > > "I have since come to realize that being between your mentor and his route > to the bathroom is a major career booster." -- Patrick Naughton > -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2003-08-28 20:19 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2003-08-27 21:23 parallel make Rich Elberger 2003-08-28 0:55 ` Rolf Campbell 2003-08-28 1:04 ` Christopher Faylor 2003-08-28 1:35 ` Rolf Campbell 2003-08-28 1:35 ` Christopher Faylor 2003-08-28 21:08 ` Rolf Campbell 2003-08-28 1:32 ` Rich Elberger 2003-08-28 2:16 ` Igor Pechtchanski 2003-08-28 5:25 ` Christopher Faylor 2003-08-28 5:32 ` Rich Elberger
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).