Mirroring and stage and tmp dirs

Mirroring and stage and tmp dirs

[Robert Pendell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Should we be ignoring these folders during mirroring?  I was mirroring
them but recently started getting permission denied errors during
transfers for those folders so they are now excluded from rsync
transfers (i.e. not on my public mirror).  What is the stage folder used
for?

rsync: send_files failed to open "/stage/tcp_wrappers-7.6-2.sh" (in
cygwin-ftp): Permission denied (13)
rsync: send_files failed to open "/stage/tcp_wrappers_7.6.tar.gz" (in
cygwin-ftp): Permission denied (13)

And from earlier on...
rsync: send_files failed to open "/stage/monotone-0.26-1.sh" (in
cygwin-ftp): Permission denied (13)

- --
Robert Pendell
shinji257@uplink.net

Thawte Web of Trust Notary
CAcert Notary
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (MingW32)

iD8DBQFEQlfbNqnRaBCOmrYRAsPPAJ9i+XepgtloPoTs5nwbNKyjQZhhuQCgrz4p
LGjiwhg0+nnfrEhe8HUGqPM=
=l9u6
-----END PGP SIGNATURE-----

Re: Mirroring and stage and tmp dirs

[Igor Peshansky]

On Sun, 16 Apr 2006, Robert Pendell wrote:

> Should we be ignoring these folders during mirroring?  I was mirroring
> them but recently started getting permission denied errors during
> transfers for those folders so they are now excluded from rsync
> transfers (i.e. not on my public mirror).  What is the stage folder used
> for?

The /stage folder is used to build up parts of the changed directory
structure and transfer them over in one swell foop (to avoid a race with
the upset cron job, where it could catch the uploader in mid-update and
result in a broken setup.ini).

In short, you should not mirror that directory, as nothing in there is
going to be useful on a mirror.

> rsync: send_files failed to open "/stage/tcp_wrappers-7.6-2.sh" (in
> cygwin-ftp): Permission denied (13)
> rsync: send_files failed to open "/stage/tcp_wrappers_7.6.tar.gz" (in
> cygwin-ftp): Permission denied (13)
>
> And from earlier on...
> rsync: send_files failed to open "/stage/monotone-0.26-1.sh" (in
> cygwin-ftp): Permission denied (13)

You probably caught the uploader in the act of moving files from /stage to
/release.
	Igor
--- OBLIGATORY HIPPO COMMENT HERE ---
-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_	    pechtcha@cs.nyu.edu | igor@watson.ibm.com
ZZZzz /,`.-'`'    -.  ;-;;,_		Igor Peshansky, Ph.D. (name changed!)
     |,4-  ) )-,_. ,\ (  `'-'		old name: Igor Pechtchanski
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

"Las! je suis sot... -Mais non, tu ne l'es pas, puisque tu t'en rends compte."
"But no -- you are no fool; you call yourself a fool, there's proof enough in
that!" -- Rostand, "Cyrano de Bergerac"

Re: Mirroring and stage and tmp dirs

[Robert Pendell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Igor Peshansky wrote:
> On Sun, 16 Apr 2006, Robert Pendell wrote:
> 
>> Should we be ignoring these folders during mirroring?  I was mirroring
>> them but recently started getting permission denied errors during
>> transfers for those folders so they are now excluded from rsync
>> transfers (i.e. not on my public mirror).  What is the stage folder used
>> for?
> 
> The /stage folder is used to build up parts of the changed directory
> structure and transfer them over in one swell foop (to avoid a race with
> the upset cron job, where it could catch the uploader in mid-update and
> result in a broken setup.ini).
> 
> In short, you should not mirror that directory, as nothing in there is
> going to be useful on a mirror.
> 
>> rsync: send_files failed to open "/stage/tcp_wrappers-7.6-2.sh" (in
>> cygwin-ftp): Permission denied (13)
>> rsync: send_files failed to open "/stage/tcp_wrappers_7.6.tar.gz" (in
>> cygwin-ftp): Permission denied (13)
>>
>> And from earlier on...
>> rsync: send_files failed to open "/stage/monotone-0.26-1.sh" (in
>> cygwin-ftp): Permission denied (13)
> 
> You probably caught the uploader in the act of moving files from /stage to
> /release.
> 	Igor

Ok.  I updated the cron job last night to exclude those folders from now
on.  I've been trying to optimized the command so it causes as few 404's
as possible.  It is setup to delete files after getting the new ones but
there are still a few occuring.

Oh and an interesting statistic.  So far this month I have had more hits
from people using Wget then the regular setup program.  _Alot_ more hits.

Wget (various versions) - 76841
Cygwin Setup - 26035

This is versus last month where Cygwin and Wget was very close.  I
wonder if people are starting to branch (2nd tier) off my mirror...

- --
Robert Pendell
shinji257@uplink.net

Thawte Web of Trust Notary
CAcert Notary
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (MingW32)

iD8DBQFEQnFPNqnRaBCOmrYRAuiUAKCDgT/nvrCHNMj+L0MwKL9H2p5RhQCfQJWa
gFsRfDumtGgVxI2MgekP8e8=
=uS80
-----END PGP SIGNATURE-----

Re: Mirroring and stage and tmp dirs

[Igor Peshansky]

On Sun, 16 Apr 2006, Robert Pendell wrote:

> Igor Peshansky wrote:
> > On Sun, 16 Apr 2006, Robert Pendell wrote:
> >
> >> Should we be ignoring these folders during mirroring?  I was mirroring
> >> them but recently started getting permission denied errors during
> >> transfers for those folders so they are now excluded from rsync
> >> transfers (i.e. not on my public mirror).  What is the stage folder used
> >> for?
> >
> > The /stage folder is used to build up parts of the changed directory
> > structure and transfer them over in one swell foop (to avoid a race
> > with the upset cron job, where it could catch the uploader in
> > mid-update and result in a broken setup.ini).
> >
> > In short, you should not mirror that directory, as nothing in there is
> > going to be useful on a mirror.
> >
> >> rsync: send_files failed to open "/stage/tcp_wrappers-7.6-2.sh" (in
> >> cygwin-ftp): Permission denied (13)
> >> rsync: send_files failed to open "/stage/tcp_wrappers_7.6.tar.gz" (in
> >> cygwin-ftp): Permission denied (13)
> >>
> >> And from earlier on...
> >> rsync: send_files failed to open "/stage/monotone-0.26-1.sh" (in
> >> cygwin-ftp): Permission denied (13)

Heh, I just looked closely at the names of the files...  Looks like
someone unpacked a source archive there -- not part of your normal "create
files/directories and move them to /release" sequence...  Don't know why
this happened, but my explanation below is probably not correct for this
particular case.

> > You probably caught the uploader in the act of moving files from
> > /stage to /release.
> > 	Igor
>
> Ok.  I updated the cron job last night to exclude those folders from now
> on.  I've been trying to optimized the command so it causes as few 404's
> as possible.  It is setup to delete files after getting the new ones but
> there are still a few occuring.
>
> Oh and an interesting statistic.  So far this month I have had more hits
> from people using Wget then the regular setup program.  _Alot_ more hits.
>
> Wget (various versions) - 76841
> Cygwin Setup - 26035
>
> This is versus last month where Cygwin and Wget was very close.  I
> wonder if people are starting to branch (2nd tier) off my mirror...

I doubt it.  Can you also check whether the hits are for the current
binary tarballs, or for the older source/binary ones?  If the former, it
might indicate the more prominent usage of clean_setup.pl (which builds
lists of wget commands).  If the latter, people may be doing what I
occasionally do -- grabbing the whole older tarball to extract an older
version of some file...  Also, are some tarballs hit more than others?
	Igor
-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_	    pechtcha@cs.nyu.edu | igor@watson.ibm.com
ZZZzz /,`.-'`'    -.  ;-;;,_		Igor Peshansky, Ph.D. (name changed!)
     |,4-  ) )-,_. ,\ (  `'-'		old name: Igor Pechtchanski
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

"Las! je suis sot... -Mais non, tu ne l'es pas, puisque tu t'en rends compte."
"But no -- you are no fool; you call yourself a fool, there's proof enough in
that!" -- Rostand, "Cyrano de Bergerac"

Re: Mirroring and stage and tmp dirs

[Robert Pendell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Igor Peshansky wrote:
> On Sun, 16 Apr 2006, Robert Pendell wrote:
> 
>> Igor Peshansky wrote:
>>> On Sun, 16 Apr 2006, Robert Pendell wrote:
>>>
>>>> Should we be ignoring these folders during mirroring?  I was mirroring
>>>> them but recently started getting permission denied errors during
>>>> transfers for those folders so they are now excluded from rsync
>>>> transfers (i.e. not on my public mirror).  What is the stage folder used
>>>> for?
>>> The /stage folder is used to build up parts of the changed directory
>>> structure and transfer them over in one swell foop (to avoid a race
>>> with the upset cron job, where it could catch the uploader in
>>> mid-update and result in a broken setup.ini).
>>>
>>> In short, you should not mirror that directory, as nothing in there is
>>> going to be useful on a mirror.
>>>
>>>> rsync: send_files failed to open "/stage/tcp_wrappers-7.6-2.sh" (in
>>>> cygwin-ftp): Permission denied (13)
>>>> rsync: send_files failed to open "/stage/tcp_wrappers_7.6.tar.gz" (in
>>>> cygwin-ftp): Permission denied (13)
>>>>
>>>> And from earlier on...
>>>> rsync: send_files failed to open "/stage/monotone-0.26-1.sh" (in
>>>> cygwin-ftp): Permission denied (13)
> 
> Heh, I just looked closely at the names of the files...  Looks like
> someone unpacked a source archive there -- not part of your normal "create
> files/directories and move them to /release" sequence...  Don't know why
> this happened, but my explanation below is probably not correct for this
> particular case.
> 
>>> You probably caught the uploader in the act of moving files from
>>> /stage to /release.
>>> 	Igor
>> Ok.  I updated the cron job last night to exclude those folders from now
>> on.  I've been trying to optimized the command so it causes as few 404's
>> as possible.  It is setup to delete files after getting the new ones but
>> there are still a few occuring.
>>
>> Oh and an interesting statistic.  So far this month I have had more hits
>> from people using Wget then the regular setup program.  _Alot_ more hits.
>>
>> Wget (various versions) - 76841
>> Cygwin Setup - 26035
>>
>> This is versus last month where Cygwin and Wget was very close.  I
>> wonder if people are starting to branch (2nd tier) off my mirror...
> 
> I doubt it.  Can you also check whether the hits are for the current
> binary tarballs, or for the older source/binary ones?  If the former, it
> might indicate the more prominent usage of clean_setup.pl (which builds
> lists of wget commands).  If the latter, people may be doing what I
> occasionally do -- grabbing the whole older tarball to extract an older
> version of some file...  Also, are some tarballs hit more than others?
> 	Igor

Unfortunately the host is only providing Analog stats at the moment
which is limited on what information it provides and the raw logs are
only for the past week.

On the 11th I got a crap load of hits from a specific ip running Wget
1.10.2 and it was grabbing alot of directory listings.  It also grabed a
great deal of files too though.  In fact there were over 2000 files
retrieved via Wget from this one ip address.  It also went through the
stage, mail-archives, and tmp folders.  Alot of the directory listings
had "?N=D", "?M=A", "?S=A", "?D=A" appended to them.  What are these for?

- --
Robert Pendell
shinji257@uplink.net

Thawte Web of Trust Notary
CAcert Notary
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (MingW32)

iD8DBQFEQoQ0NqnRaBCOmrYRAsDNAJ4k6LUiqOiR6/BMwmDTnNGFtCE9JACgsDjO
mKAopq0X3D68vH42Omgg/UE=
=Ygoa
-----END PGP SIGNATURE-----

Re: Mirroring and stage and tmp dirs

[Brian Dessent]

Robert Pendell wrote:

> retrieved via Wget from this one ip address.  It also went through the
> stage, mail-archives, and tmp folders.  Alot of the directory listings
> had "?N=D", "?M=A", "?S=A", "?D=A" appended to them.  What are these for?

That happens when a spider blindly mirrors a site.  Go look on the
generated index.html pages.  You'll find that you can click on the
column headings to change the sort order.  Those url parameters control
that.

Brian

Re: Mirroring and stage and tmp dirs

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Igor Peshansky on 4/16/2006 9:04 AM:
> The /stage folder is used to build up parts of the changed directory
> structure and transfer them over in one swell foop (to avoid a race with
> the upset cron job, where it could catch the uploader in mid-update and
> result in a broken setup.ini).

And somebody just changed the permissions on that directory so that it is
no longer writable by members of the cygwin group.  In other words, when I
tried to use it this morning to upload bash-3.1-5, I got permissions
errors, whereas previously I had no problems.

$ ls -ld stage  # mtime is near when I used it to upload coreutils-5.94-5
drwxrwsr-x  2 root root 4096 Apr 15 16:30 stage
$ ls -ld --time=ctime stage  # ctime reflects permissions change
drwxrwsr-x  2 root root 4096 Apr 17 09:00 stage

cgf - care to fix it please?

- --
Life is short - so eat dessert first!

Eric Blake             ebb9@byu.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEQ5ZE84KuGfSFAYARAjSqAKDQiadqVS1OcXWdXhaiBYfzExLzZwCdGtSc
4/D9ReqHPOgF41JPUqeTiec=
=zFlC
-----END PGP SIGNATURE-----