* chattr makes cygport slow @ 2023-07-06 12:19 Andrew Schulman 2023-07-06 16:18 ` Brian Inglis 2023-07-06 17:36 ` Andrew Schulman 0 siblings, 2 replies; 13+ messages in thread From: Andrew Schulman @ 2023-07-06 12:19 UTC (permalink / raw) To: cygwin-apps Recently I noticed that `cygport finish` has become really slow on some of my package source trees. After I run for example cygport libargp.cygport finish it waits for about 5 minutes without any message to the console, before the first "Removing work directory" message appears. pstree shows that during this time cygport is waiting for chattr. In /usr/bin/cygport I see: if [ $OSTYPE = "cygwin" ] then chattr -fR +C ${workdir} >/dev/null 2>&1 || true fi which is trying to make the workdir case-sensitive. Whatever the advantages of that are, it can take a long time. Would it be possible to skip it at least in the case of "finish"? It seems silly to spend all that time fixing up a directory tree that we then turn around and remove with rm -rf. Thanks, Andrew ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-06 12:19 chattr makes cygport slow Andrew Schulman @ 2023-07-06 16:18 ` Brian Inglis 2023-07-08 12:19 ` Jon Turney 2023-07-06 17:36 ` Andrew Schulman 1 sibling, 1 reply; 13+ messages in thread From: Brian Inglis @ 2023-07-06 16:18 UTC (permalink / raw) To: cygwin-apps On 2023-07-06 06:19, Andrew Schulman via Cygwin-apps wrote: > Recently I noticed that `cygport finish` has become really slow on some of my > package source trees. After I run for example > > cygport libargp.cygport finish > > it waits for about 5 minutes without any message to the console, before the > first "Removing work directory" message appears. > > pstree shows that during this time cygport is waiting for chattr. In > /usr/bin/cygport I see: > > if [ $OSTYPE = "cygwin" ] > then > chattr -fR +C ${workdir} >/dev/null 2>&1 || true > fi > > which is trying to make the workdir case-sensitive. > > Whatever the advantages of that are, it can take a long time. Would it be > possible to skip it at least in the case of "finish"? It seems silly to spend > all that time fixing up a directory tree that we then turn around and remove > with rm -rf. The attribute does not appear to be inheritable, so will not be applied to subdirectories created by make, or in cygport xargs commands, unless supported in cygwin1.dll, perhaps why it is open coded in cygport? Perhaps it could be moved below case prep, into src_prep `__mkdirs`, or globally define `mkdir_p()` which applies it to each directory in those paths? Could this also be defined in MKDIR_P for make or are those limited to binaries? All cygport scripts appear to consistently use `mkdir -p` including after `xargs -r` (`--no-run-if-empty`) except: /usr/share/cygport/cygclass/fossil.cygclass: mkdir ${FOSSIL_MODULE} -- Take care. Thanks, Brian Inglis Calgary, Alberta, Canada La perfection est atteinte Perfection is achieved non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut -- Antoine de Saint-Exupéry ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-06 16:18 ` Brian Inglis @ 2023-07-08 12:19 ` Jon Turney 0 siblings, 0 replies; 13+ messages in thread From: Jon Turney @ 2023-07-08 12:19 UTC (permalink / raw) To: cygwin-apps On 06/07/2023 17:18, Brian Inglis via Cygwin-apps wrote: > On 2023-07-06 06:19, Andrew Schulman via Cygwin-apps wrote: >> Recently I noticed that `cygport finish` has become really slow on >> some of my >> package source trees. After I run for example >> >> cygport libargp.cygport finish >> >> it waits for about 5 minutes without any message to the console, >> before the >> first "Removing work directory" message appears. >> >> pstree shows that during this time cygport is waiting for chattr. In >> /usr/bin/cygport I see: >> >> if [ $OSTYPE = "cygwin" ] >> then >> chattr -fR +C ${workdir} >/dev/null 2>&1 || true >> fi >> >> which is trying to make the workdir case-sensitive. >> >> Whatever the advantages of that are, it can take a long time. Would it be >> possible to skip it at least in the case of "finish"? It seems silly >> to spend >> all that time fixing up a directory tree that we then turn around and >> remove >> with rm -rf. > > The attribute does not appear to be inheritable, so will not be applied > to subdirectories created by make, or in cygport xargs commands, unless > supported in cygwin1.dll, perhaps why it is open coded in cygport? I believe that might have been the case in some previews, but it does inherit now. Otherwise it's almost completely, unusably useless. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-06 12:19 chattr makes cygport slow Andrew Schulman 2023-07-06 16:18 ` Brian Inglis @ 2023-07-06 17:36 ` Andrew Schulman 2023-07-08 12:16 ` Jon Turney 1 sibling, 1 reply; 13+ messages in thread From: Andrew Schulman @ 2023-07-06 17:36 UTC (permalink / raw) To: cygwin-apps > Recently I noticed that `cygport finish` has become really slow on some of my > package source trees. After I run for example > > cygport libargp.cygport finish > > it waits for about 5 minutes without any message to the console, before the > first "Removing work directory" message appears. > > pstree shows that during this time cygport is waiting for chattr. In > /usr/bin/cygport I see: > > if [ $OSTYPE = "cygwin" ] > then > chattr -fR +C ${workdir} >/dev/null 2>&1 || true > fi > > which is trying to make the workdir case-sensitive. > > Whatever the advantages of that are, it can take a long time. Would it be > possible to skip it at least in the case of "finish"? It seems silly to spend > all that time fixing up a directory tree that we then turn around and remove > with rm -rf. And/or, maybe add a switch to suppress it. I've commented it out of my /usr/bin/cygport for now, because it makes me wait an extra 5 minutes before every cygport command, at least on this particular source tree with 12K files in it. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-06 17:36 ` Andrew Schulman @ 2023-07-08 12:16 ` Jon Turney 2023-07-08 13:23 ` Andrew Schulman 2023-07-08 14:22 ` ASSI 0 siblings, 2 replies; 13+ messages in thread From: Jon Turney @ 2023-07-08 12:16 UTC (permalink / raw) To: Schulman, Andrew, cygwin-apps On 06/07/2023 18:36, Andrew Schulman via Cygwin-apps wrote: >> Recently I noticed that `cygport finish` has become really slow on some of my >> package source trees. After I run for example >> >> cygport libargp.cygport finish >> >> it waits for about 5 minutes without any message to the console, before the >> first "Removing work directory" message appears. >> >> pstree shows that during this time cygport is waiting for chattr. In >> /usr/bin/cygport I see: >> >> if [ $OSTYPE = "cygwin" ] >> then >> chattr -fR +C ${workdir} >/dev/null 2>&1 || true >> fi >> >> which is trying to make the workdir case-sensitive. >> >> Whatever the advantages of that are, it can take a long time. Would it be >> possible to skip it at least in the case of "finish"? It seems silly to spend >> all that time fixing up a directory tree that we then turn around and remove >> with rm -rf. For a long time, we've just been assuming that anyone using cygport has case-sensitivity turned on somehow (as the build underling a cygport might assume that it's present, as is normal on a unix) So, the idea here is that we try to ensure it's on, at least for the working directory. Skipping it when 'finish' is used isn't right, because then 'finish all' wouldn't work as desired. But yeah, it seems that this is in the wrong place. I'll look into moving it. I think there's possibly something else going wrong if it's taking 5 minutes, as that seems excessive. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-08 12:16 ` Jon Turney @ 2023-07-08 13:23 ` Andrew Schulman 2023-07-08 14:22 ` ASSI 1 sibling, 0 replies; 13+ messages in thread From: Andrew Schulman @ 2023-07-08 13:23 UTC (permalink / raw) To: cygwin-apps > On 06/07/2023 18:36, Andrew Schulman via Cygwin-apps wrote: > >> Recently I noticed that `cygport finish` has become really slow on some of my > >> package source trees. After I run for example > >> > >> cygport libargp.cygport finish > >> > >> it waits for about 5 minutes without any message to the console, before the > >> first "Removing work directory" message appears. > >> > >> pstree shows that during this time cygport is waiting for chattr. In > >> /usr/bin/cygport I see: > >> > >> if [ $OSTYPE = "cygwin" ] > >> then > >> chattr -fR +C ${workdir} >/dev/null 2>&1 || true > >> fi > >> > >> which is trying to make the workdir case-sensitive. > >> > >> Whatever the advantages of that are, it can take a long time. Would it be > >> possible to skip it at least in the case of "finish"? It seems silly to spend > >> all that time fixing up a directory tree that we then turn around and remove > >> with rm -rf. > > For a long time, we've just been assuming that anyone using cygport has > case-sensitivity turned on somehow (as the build underling a cygport > might assume that it's present, as is normal on a unix) > > So, the idea here is that we try to ensure it's on, at least for the > working directory. > > Skipping it when 'finish' is used isn't right, because then 'finish all' > wouldn't work as desired. But yeah, it seems that this is in the wrong > place. I'll look into moving it. Thanks. Annoying, I realize. Unfortunately doing it separately for each command isn't right either, if it runs 3 times instead of once when I run 'cygport prep make inst'. Also, just fixing it for 'finish' would still leave the delay for the other commands. Maybe a switch that would allow turning it off, either host-wide in ~/.cygport.conf, or for a project that didn't need it in *.cygport. > I think there's possibly something else going wrong if it's taking 5 > minutes, as that seems excessive. Could be BLODA. My host has MS antivirus. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-08 12:16 ` Jon Turney 2023-07-08 13:23 ` Andrew Schulman @ 2023-07-08 14:22 ` ASSI 2023-07-16 16:15 ` Jon Turney 1 sibling, 1 reply; 13+ messages in thread From: ASSI @ 2023-07-08 14:22 UTC (permalink / raw) To: cygwin-apps Jon Turney via Cygwin-apps writes: > So, the idea here is that we try to ensure it's on, at least for the > working directory. That IMHO should only be done when the working directory is created, but not retroactively applied to an existing workdir. > Skipping it when 'finish' is used isn't right, because then 'finish > all' wouldn't work as desired. But yeah, it seems that this is in the > wrong place. I'll look into moving it. Meanwhile I've done this: --8<---------------cut here---------------start------------->8--- if [ $OSTYPE = "cygwin" ] then + [ -n "$(find ${workdir} -maxdepth 0 -empty)" ] && chattr -fR +C ${workdir} >/dev/null 2>&1 || true fi --8<---------------cut here---------------end--------------->8--- I think it should be moved into a function that can be called before the prep command, but I haven't actually tried it for a longer time or checked what tests need changing due to the extra output: --8<---------------cut here---------------start------------->8--- From 4be00357be8cb14493413b7c8c80777671c21e5d Mon Sep 17 00:00:00 2001 Subject: [PATCH] cygport.in: move case-sensitivity testing / enabling into functions --- bin/cygport.in | 46 ++++++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 16 deletions(-) diff --git a/bin/cygport.in b/bin/cygport.in index 02e9b7bf..a6dabc91 100755 --- a/bin/cygport.in +++ b/bin/cygport.in @@ -577,25 +577,36 @@ declare -r pkg_count=${#pkg_name[*]}; # this requires workdir to be already defined mkdir -p ${workdir} -if [ $OSTYPE = "cygwin" ] -then - chattr -fR +C ${workdir} >/dev/null 2>&1 || true -fi - -rm -f ${workdir}/.probe_case_sensitivity -touch ${workdir}/.PROBE_CASE_SENSITIVITY -probe=$( [ -f ${workdir}/.probe_case_sensitivity ] && echo "failed" || echo "" ) -rm -f ${workdir}/.PROBE_CASE_SENSITIVITY -if [ -n "$probe" ] -then +__enable_case_sensitivity() { + inform "Trying to enable case sensitivity on ${workdir}" + if [ $OSTYPE = "cygwin" ] + then + [ -n "$(find ${workdir} -maxdepth 0 -empty)" ] && + chattr -fR +C ${workdir} >/dev/null 2>&1 || true + __probe_case_sensitivity + fi +} + +__probe_case_sensitivity() { + mkdir -p ${workdir} + rm -f ${workdir}/.probe_case_sensitivity + touch ${workdir}/.PROBE_CASE_SENSITIVITY + local probe=$( [ -f ${workdir}/.probe_case_sensitivity ] && echo "failed" || echo "" ) + rm -f ${workdir}/.PROBE_CASE_SENSITIVITY + if [ -n "$probe" ] + then if defined _CYGPORT_RESTRICT_case_insensitive_ then - error "Package ${NAME} cannot be built on a case-insensitive filesystem"; + error "Package ${NAME} cannot be built on a case-insensitive filesystem"; else - warning "Building on a case-insensitive filesystem"; + warning "Building on a case-insensitive filesystem"; fi -fi -unset probe + false + else + inform "Building on a case-sensitive filesystem"; + true + fi +} # this requires S and B to be already defined if ! defined _CYGPORT_RESTRICT_debuginfo_ @@ -638,15 +649,18 @@ while (( arg_n < argc )) do case ${argv[${arg_n}]} in downloadall|fetchall|wgetall|getall) + __probe_case_sensitivity || __enable_case_sensitivity; __src_fetch; _status=$?; ;; download|fetch|wget|get) + __probe_case_sensitivity || __enable_case_sensitivity; __DL_ONLY_MISSING=1 __src_fetch; _status=$?; ;; prepare|unpack|prep) - __stage Preparing; + __probe_case_sensitivity || __enable_case_sensitivity; + __stage Preparing; __src_prep; _status=$?; ;; -- 2.41.0 --8<---------------cut here---------------end--------------->8--- > I think there's possibly something else going wrong if it's taking 5 > minutes, as that seems excessive. Try it on the gcc build dir… Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Q+, Q and microQ: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-08 14:22 ` ASSI @ 2023-07-16 16:15 ` Jon Turney 2023-07-16 19:32 ` ASSI 0 siblings, 1 reply; 13+ messages in thread From: Jon Turney @ 2023-07-16 16:15 UTC (permalink / raw) To: cygwin-apps [-- Attachment #1: Type: text/plain, Size: 1598 bytes --] On 08/07/2023 15:22, ASSI via Cygwin-apps wrote: > Jon Turney via Cygwin-apps writes: >> So, the idea here is that we try to ensure it's on, at least for the >> working directory. > > That IMHO should only be done when the working directory is created, but > not retroactively applied to an existing workdir. Yeah, that's just a side effect of where I put it. >> Skipping it when 'finish' is used isn't right, because then 'finish >> all' wouldn't work as desired. But yeah, it seems that this is in the >> wrong place. I'll look into moving it. > > Meanwhile I've done this: > > --8<---------------cut here---------------start------------->8--- > if [ $OSTYPE = "cygwin" ] > then > + [ -n "$(find ${workdir} -maxdepth 0 -empty)" ] && > chattr -fR +C ${workdir} >/dev/null 2>&1 || true This seems a bit nuts as we've just done the mkdir a few lines above, we could just check if the directory doesn't exist and apply case-sensitivity if we are creating it. > fi > --8<---------------cut here---------------end--------------->8--- > > I think it should be moved into a function that can be called before the > prep command, but I haven't actually tried it for a longer time or > checked what tests need changing due to the extra output: The warning (error if RESTRICT=case_insensitive) should occur for all commands, not just prep. How about the attached. >> I think there's possibly something else going wrong if it's taking 5 >> minutes, as that seems excessive. > > Try it on the gcc build dir… Even then, it should only be modifying every directory, not every file. [-- Attachment #2: 0001-cygport.in-only-try-to-turn-on-case-sensitivity-when.patch --] [-- Type: text/plain, Size: 2238 bytes --] From 0e4c4261122d3b2373734816c9620c5c44eb72a5 Mon Sep 17 00:00:00 2001 From: Jon Turney <jon.turney@dronecode.org.uk> Date: Sun, 9 Jul 2023 18:35:51 +0100 Subject: [PATCH cygport] cygport.in: only try to turn on case-sensitivity when we create workdir Only try to turn on case-sensitivity when we create workdir: this avoids grovelling over all the files on subsequent runs. Make sure to check for case-sensitivity required on subsequent runs, though. --- bin/cygport.in | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/bin/cygport.in b/bin/cygport.in index 1791e5b3..3f89ac67 100755 --- a/bin/cygport.in +++ b/bin/cygport.in @@ -573,18 +573,34 @@ declare -ar pkg_name=(${PKG_NAMES:-${PN}}); declare -r pkg_count=${#pkg_name[*]}; # this requires workdir to be already defined -mkdir -p ${workdir} -if [ $OSTYPE = "cygwin" ] +__probe_case_sensitivity() { + mkdir -p ${workdir} + rm -f ${workdir}/.probe_case_sensitivity + touch ${workdir}/.PROBE_CASE_SENSITIVITY + local probe=$( [ -f ${workdir}/.probe_case_sensitivity ] && echo 1 || echo 0 ) + rm -f ${workdir}/.PROBE_CASE_SENSITIVITY + return $probe +} + +__enable_case_sensitivity() { + if [ $OSTYPE = "cygwin" ] + then + inform "Trying to enable case sensitivity on ${workdir}" + chattr -fR +C ${workdir} >/dev/null 2>&1 || true + fi +} + +# create workdir if it doesn't already exist, and try to enable case-sensitivity +# if it appears to be absent +if [ ! -d ${workdir} ] then - chattr -fR +C ${workdir} >/dev/null 2>&1 || true + mkdir -p ${workdir} + __probe_case_sensitivity || __enable_case_sensitivity fi -rm -f ${workdir}/.probe_case_sensitivity -touch ${workdir}/.PROBE_CASE_SENSITIVITY -probe=$( [ -f ${workdir}/.probe_case_sensitivity ] && echo "failed" || echo "" ) -rm -f ${workdir}/.PROBE_CASE_SENSITIVITY -if [ -n "$probe" ] +# in any case, probe for case sensitivity, stop if required but missing +if ! __probe_case_sensitivity then if defined _CYGPORT_RESTRICT_case_insensitive_ then @@ -593,7 +609,6 @@ then warning "Building on a case-insensitive filesystem"; fi fi -unset probe # this requires S and B to be already defined if ! defined _CYGPORT_RESTRICT_debuginfo_ -- 2.39.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-16 16:15 ` Jon Turney @ 2023-07-16 19:32 ` ASSI 2023-07-23 15:33 ` Jon Turney 0 siblings, 1 reply; 13+ messages in thread From: ASSI @ 2023-07-16 19:32 UTC (permalink / raw) To: cygwin-apps Jon Turney via Cygwin-apps writes: > The warning (error if RESTRICT=case_insensitive) should occur for all > commands, not just prep. OK. > How about the attached. Looks promising. > Even then, it should only be modifying every directory, not every file. Run strace on "chattr -r +C ." and see that it isn't that smart… Anyway, since it won't succeed on anything but empty directories it doesn't make sense to use to attempt a recursive change even if it were just looking at directories (which would of course be way faster at least). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-16 19:32 ` ASSI @ 2023-07-23 15:33 ` Jon Turney 2023-07-23 19:13 ` ASSI 2023-07-28 11:35 ` Andrew Schulman 0 siblings, 2 replies; 13+ messages in thread From: Jon Turney @ 2023-07-23 15:33 UTC (permalink / raw) To: cygwin-apps; +Cc: Schulman, Andrew On 16/07/2023 20:32, ASSI via Cygwin-apps wrote: > Jon Turney via Cygwin-apps writes: >> The warning (error if RESTRICT=case_insensitive) should occur for all >> commands, not just prep. > > OK. > >> How about the attached. > > Looks promising. Ok. I made a cygport 0.36.6 release with this change. Please let me know if it makes things better (or worse!)? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-23 15:33 ` Jon Turney @ 2023-07-23 19:13 ` ASSI 2023-07-24 21:07 ` Jon Turney 2023-07-28 11:35 ` Andrew Schulman 1 sibling, 1 reply; 13+ messages in thread From: ASSI @ 2023-07-23 19:13 UTC (permalink / raw) To: cygwin-apps Jon Turney via Cygwin-apps writes: > Ok. I made a cygport 0.36.6 release with this change. Thanks. You haven't tagged that release on GitHub yet it seems. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Waldorf MIDI Implementation & additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-23 19:13 ` ASSI @ 2023-07-24 21:07 ` Jon Turney 0 siblings, 0 replies; 13+ messages in thread From: Jon Turney @ 2023-07-24 21:07 UTC (permalink / raw) To: cygwin-apps On 23/07/2023 20:13, ASSI via Cygwin-apps wrote: > Jon Turney via Cygwin-apps writes: >> Ok. I made a cygport 0.36.6 release with this change. > > Thanks. You haven't tagged that release on GitHub yet it seems. The authoritative upstream is https://cygwin.com/git/cygwin-apps/cygport.git, which that is just a mirror of. Thanks for pointing out this oversight, should be fixed now. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: chattr makes cygport slow 2023-07-23 15:33 ` Jon Turney 2023-07-23 19:13 ` ASSI @ 2023-07-28 11:35 ` Andrew Schulman 1 sibling, 0 replies; 13+ messages in thread From: Andrew Schulman @ 2023-07-28 11:35 UTC (permalink / raw) To: cygwin-apps > On 16/07/2023 20:32, ASSI via Cygwin-apps wrote: > > Jon Turney via Cygwin-apps writes: > >> The warning (error if RESTRICT=case_insensitive) should occur for all > >> commands, not just prep. > > > > OK. > > > >> How about the attached. > > > > Looks promising. > > Ok. I made a cygport 0.36.6 release with this change. > > Please let me know if it makes things better (or worse!)? So much better. Thank you. Andrew ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2023-07-28 11:35 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-07-06 12:19 chattr makes cygport slow Andrew Schulman 2023-07-06 16:18 ` Brian Inglis 2023-07-08 12:19 ` Jon Turney 2023-07-06 17:36 ` Andrew Schulman 2023-07-08 12:16 ` Jon Turney 2023-07-08 13:23 ` Andrew Schulman 2023-07-08 14:22 ` ASSI 2023-07-16 16:15 ` Jon Turney 2023-07-16 19:32 ` ASSI 2023-07-23 15:33 ` Jon Turney 2023-07-23 19:13 ` ASSI 2023-07-24 21:07 ` Jon Turney 2023-07-28 11:35 ` Andrew Schulman
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).