Major changes coming to procedure for uploading to sourceware (long)

Major changes coming to procedure for uploading to sourceware (long)

[Christopher Faylor]

I think I now have a system set up which will allow maintainers to
upload their own packages to sourceware.  This system means that package
maintainers won't have to find a public web server to make their
packages available for the cygwin release.  Every package maintainer
will be able to upload their own stuff.  General login access to
sourceware will no longer be required to update the cygwin release.

The system relies on your providing a ssh key.  Once we have that,
you'll have limited 'sftp' access to sourceware where you will be able
to upload packages.  Your home directory on sourceware will look like
this:

Christopher Faylor		# Directory is currently your full name (you won't actually see this)
    !packages			# read-only file containing valid packages
    x86        x86_64		# arch-specific dir
    release    release		# release dir
      pkg			# package directory
       pkg-debuginfo		# package subdirectories

So, to release, say, grep, I would do the following:

    % lftp sftp://cygwin
    cd ok, cwd=/                                    

    lftp cygwin.com:/> ls
    -rw-rw-r--   1 root     cygstage      179 Sep  1 05:12 !packages
    drwxrwsr-x   4 root     cygstage     4096 Sep  1 05:16 .
    drwxrwxr-x  61 cygwin   root         4096 Sep  1 05:12 ..
    drwxrwsr-x   3 cygwin   cygstage     4096 Oct 12 19:35 x86
    drwxrwsr-x   3 cygwin   cygstage     4096 Aug 10 19:04 x86_64

    lftp cygwin.com:/> cd x86/release
    cd ok, cwd=/x86/release                     

    lftp cygwin.com:/x86/release> ls
    drwxrwsr-x   3 cygwin   cygstage     4096 Oct 12 18:28 .
    drwxrwsr-x   3 cygwin   cygstage     4096 Oct 12 19:35 ..
    drwxrwsr-x   3 cygwin   cygstage     4096 Oct 12 19:36 grep

    lftp cygwin.com:/x86/release> mirror -eR grep
    Total: 2 directories, 3 files, 0 symlinks                                                  
    New: 3 files, 0 symlinks
    2311396 bytes transferred in 14 seconds (166.2K/s)

    lftp cygwin.com:/x86/release> ls grep
    drwxrwsr-x   3 cygwin   cygstage     4096 Oct 12 19:47 .
    drwxrwsr-x   3 cygwin   cygstage     4096 Oct 12 18:28 ..
    -rw-r--r--   1 cygwin   cygstage  1204708 Oct 12 19:46 grep-2.14-3-src.tar.xz
    -rw-r--r--   1 cygwin   cygstage   229284 Oct 12 19:47 grep-2.14-3.tar.xz
    drwxrwsr-x   2 cygwin   cygstage     4096 Oct 12 19:47 grep-debuginfo

    lftp cygwin.com:/x86/release> ls grep/grep-debuginfo
    drwxrwsr-x   2 cygwin   cygstage     4096 Oct 12 19:47 . 
    drwxrwsr-x   3 cygwin   cygstage     4096 Oct 12 19:47 ..
    -rw-r--r--   1 cygwin   cygstage   877404 Oct 12 19:47 grep-debuginfo-2.14-3.tar.xz

    lftp cygwin.com:/x86/release> put /dev/null -o !ready

The package directories should exactly mirror what is currently in the
release directory.  The !ready file is used to tell upset that any files
older than this file in or under the given directory should be
processed.  The intent is that you create !ready when you have finished
uploading everything that you plan on uploading.

upset will periodically scan our home directories for packages, and, if
there are no errors, move the packages into the real release area.  New
directories will be created as needed.  If there are errors then
setup.ini will not be updated and the release area will remain pristine.

Although not shown above, you can also create a !mail file which
contains a list of email address (one per line) where any upset errors
should be sent.  If that isn't specified then you won't get email when
upset finds issues with what was uploaded.  Currently, this is not a
"per-directory" thing since upset parsing doesn't work that way so, if
you add a !email tag you will get all errors from upset - not just ones
associated to your uploaded files.

(Eventually, I'll fix this so that you only get errors if you actually
did upload something rather than all of the time)

To start this system going, I'd like people to start sending their public
ssh keys to cygwin-apps using this format:

Subject: SSH key for upload access

Name: Your name
Package: The name of one of the packages that you are responsible for
SSHkey: Your *public* ssh key

When specifying your name, use your exact name as shown here:

http://cygwin.com/cygwin-pkg-maint

I'll have an automated way of pulling this info into sourceware (after
first verifying it for accuracy).

So, in my case I would send mail to cygwin-apps with:

Subject: SSH key for upload access

Name: Christopher Faylor
Package: grep
SSHkey: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEArEBaxIX8MpBhJioygtOziwXb19w0vLnAHQcOEVgjjoStjCNZ0iAaX7+Ae7Jv78p7XAXZyrtFbvJ22siKKc0FG5z1DhE28sIFP2RxyF4F4ENIrqtlz4R8sFbuv16r91ZyLzMwpk5LAQGb/iSF+fU9KDpAohvvC5B8ZZ8dywQCXEE= cgf

Note that the SSHkey stuff may have something which looks like an email
address and that may be flagged by the spam filter so it makes sense to
remove the <at>address part.  Make sure that you send your PUBLIC key,
not your private key.  The public keys are located in your .ssh
directory and have a .pub extension.  Only send one.

The Package: key is just there for validation.  It shouldn't contain the
complete list of all of the packages you maintain.  The cygwin-pkg-maint
file is the master list for that information.

Hopefully I haven't made this sound more complicated than it really is.
It should be really simple.  You provide an ssh key.  We add it to
sourceware.  You send your packages to sourceware via sftp or
(preferred) lftp.  Create a "!ready" file.  You're done.

I hope that this will go live when I return from a brief vacation by mid
next week.

cgf

The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Christopher Faylor]

On Sat, Oct 12, 2013 at 04:22:37PM -0400, Christopher Faylor wrote:
>I think I now have a system set up which will allow maintainers to
>upload their own packages to sourceware.  This system means that package
>maintainers won't have to find a public web server to make their
>packages available for the cygwin release.  Every package maintainer
>will be able to upload their own stuff.  General login access to
>sourceware will no longer be required to update the cygwin release.

This system is now live.  Anyone who sent in their ssh key should be
able to access sourceware via lftp/sftp.  Make sure that you use
the user "cygwin" with no password and that you are using the same
ssh key as the one that you specified previously.

Also, please create a !mail file at the top level of your area with the
email address where any upset errors should be sent.

Remember also that you need to create a !ready file somewhere above the
directory holding the packages that you want to end up in the release.
So, if you have uploaded x86 and x86_64 packages put the !ready at the
"root" level.  If you have uploaded x86 and are working on x86_64 you
can put the !ready in the x86 directory and then put another !ready in
the x86_64 directory when you're done.  Putting the !ready directly in
the package directory should work too.

Packages won't be moved unless upset thinks everything is ok.  If there
is something broken and your have an email address in the !email file
then you should get email telling you what's wrong.  You'll get the
email even if the problem isn't your fault so don't panic.

Refer to: http://cygwin.com/ml/cygwin-apps/2013-10/msg00117.html for more
detail about what's going on.  Note that the sftp command used there should
be something like:

lftp -u cygwin, sftp://cygwin.com/

I dropped the .com in the original example.

Please send any problem reports here.

cgf

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Christopher Faylor]

I just used this command line to upload the time packages:

cd wherever
lftp -c 'connect sftp://cygwin:@cygwin.com/; cd x86/release; mirror -eR time; put /dev/null -o !ready'

FYI.

cgf

Re: Major changes coming to procedure for uploading to sourceware (long)

[Achim Gratz]

Christopher Faylor writes:
> The package directories should exactly mirror what is currently in the
> release directory.  The !ready file is used to tell upset that any files
> older than this file in or under the given directory should be
> processed.  The intent is that you create !ready when you have finished
> uploading everything that you plan on uploading.

Are we supposed to copy the existing contents of the mirrors (curr,
previous etc. archives) into our release area first before adding
anything new?  My release area is non-empty but incomplete due to
previous tests, but I don't want to remove or add anything there unless
I know how this gets handled.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: Major changes coming to procedure for uploading to sourceware (long)

[Christopher Faylor]

On Thu, Oct 17, 2013 at 09:19:35PM +0200, Achim Gratz wrote:
>Christopher Faylor writes:
>> The package directories should exactly mirror what is currently in the
>> release directory.  The !ready file is used to tell upset that any files
>> older than this file in or under the given directory should be
>> processed.  The intent is that you create !ready when you have finished
>> uploading everything that you plan on uploading.
>
>Are we supposed to copy the existing contents of the mirrors (curr,
>previous etc. archives) into our release area first before adding
>anything new?

No.  This is for adding new/updated packages to the release.  Putting
old stuff in the staging area would certainly confuse things.

>My release area is non-empty but incomplete due to previous tests, but
>I don't want to remove or add anything there unless I know how this
>gets handled.

I just wiped out and recreated your home directory.  There are no old
packages there now.

The process is:

1) upload new tarballs/setup.hints to package directory
2) package gets syntax checked
3) package gets moved to release area

What is currently missing is a way to delete old tarballs from the
release area.  I'll probably have upset do that when it finds old
packages which are not covered by setup.hint.

cgf

Re: Major changes coming to procedure for uploading to sourceware (long)

[Achim Gratz]

Christopher Faylor writes:
> No.  This is for adding new/updated packages to the release.  Putting
> old stuff in the staging area would certainly confuse things.

Thanks for the clarification.

>>My release area is non-empty but incomplete due to previous tests, but
>>I don't want to remove or add anything there unless I know how this
>>gets handled.
>
> I just wiped out and recreated your home directory.  There are no old
> packages there now.

Confirmed, thank you.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Major changes coming to procedure for uploading to sourceware (long)

[Achim Gratz]

Christopher Faylor writes:
> I just wiped out and recreated your home directory.  There are no old
> packages there now.

I don't have access to the top level, so I can't create the !mail file.

Also, copying things over via sshfs results in zero-sized files and
error messages like

cp: cannot create regular file ‘cygwin.com/x86_64/release/...’: No such
file or directory

the first time around.  Once these zero sized files are there, I can
clobber them with a second round of copying.

The pre-existing directories are group 22796, while the one I create get
group 65521 (I don't know if that's relatedto the error).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Major changes coming to procedure for uploading to sourceware (long)

[Christopher Faylor]

On Fri, Oct 18, 2013 at 08:43:24PM +0200, Achim Gratz wrote:
>Christopher Faylor writes:
>> I just wiped out and recreated your home directory.  There are no old
>> packages there now.
>
>I don't have access to the top level, so I can't create the !mail file.
>
>Also, copying things over via sshfs results in zero-sized files and
>error messages like

I think I mentioned sshfs before but I haven't tested it in the current
setup.  You can consider it unsupported and nonworking.  Please just use
sftp or lftp.

I fixed the problem with not being able to create files at the top
level.  I wiped out and recreated your directory again so it is once
again in a pristine state.

cgf

Re: Major changes coming to procedure for uploading to sourceware (long)

[Achim Gratz]

Christopher Faylor writes:
> I think I mentioned sshfs before but I haven't tested it in the current
> setup.  You can consider it unsupported and nonworking.  Please just use
> sftp or lftp.

I will use this next time, at least for small files lftp works.

Sshfs does use sftp to do the actual communication with the server,
however.  Based on the debug output from sshfs it appears that the newly
created file becomes visible on the server only after the existence of
the new file gets checked, so the check fails.  On the second attempt to
do the copy, that check succeeds and the truncation and re-write
succeeds as well.  I've switched off all caching for sftp/sshfs that I
could find to no avail.

> I fixed the problem with not being able to create files at the top
> level.  I wiped out and recreated your directory again so it is once
> again in a pristine state.

Thank you, I have now created that file.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Chris Sutcliffe]

I just used the new process to upload cppcheck and it worked like a charm!

Thanks for pulling this all together cgf!

Chris

-- 
Chris Sutcliffe
http://emergedesktop.org
http://www.google.com/profiles/ir0nh34d

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Andrew Schulman]

> I just used this command line to upload the time packages:
> 
> cd wherever
> lftp -c 'connect sftp://cygwin:@cygwin.com/; cd x86/release; mirror -eR time; put /dev/null -o !ready'

I'm sorry there's still no lftp for 64-bit - I'm having trouble compiling
it - a configure problem.  I'll post about this soon to see if someone can
help me to solve it.

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Christopher Faylor]

On Tue, Oct 22, 2013 at 12:40:39PM -0400, Chris Sutcliffe wrote:
>I just used the new process to upload cppcheck and it worked like a charm!
>
>Thanks for pulling this all together cgf!

You're welcome.  I'm very glad that you like it and that it is working
for you.

I've been using it myself (my own dog food and all).  Now I just have to
stop holding my breath every time I use it.  I can't afford more brain
damage.

Btw, if I added the capability to send the email to cygwin-announce
automatically (maybe via a !announce file) would that be useful?  That
would ensure that the announcement was sent only when the package is
actually in setup.ini so you wouldn't have to guess.

cgf

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Achim Gratz]

Christopher Faylor writes:
> You're welcome.  I'm very glad that you like it and that it is working
> for you.

Let me add that I like it also, it works well and it is very useful.

> Btw, if I added the capability to send the email to cygwin-announce
> automatically (maybe via a !announce file) would that be useful?  That
> would ensure that the announcement was sent only when the package is
> actually in setup.ini so you wouldn't have to guess.

Yes, that would be the icing on the cake.  The cherry on top would be a
!delete file — the content interpreted relative to its placement and
evaluated strictly in order.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Christopher Faylor]

On Tue, Oct 22, 2013 at 09:38:29PM +0200, Achim Gratz wrote:
>Christopher Faylor writes:
>> You're welcome.  I'm very glad that you like it and that it is working
>> for you.
>
>Let me add that I like it also, it works well and it is very useful.

Good to hear.

>> Btw, if I added the capability to send the email to cygwin-announce
>> automatically (maybe via a !announce file) would that be useful?  That
>> would ensure that the announcement was sent only when the package is
>> actually in setup.ini so you wouldn't have to guess.
>
>Yes, that would be the icing on the cake.  The cherry on top would be a
>!delete file ??? the content interpreted relative to its placement and
>evaluated strictly in order.

I don't see any reason for a delete file since only files referenced
by setup.hint should exist in the release directory.

cgf

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Achim Gratz]

Christopher Faylor writes:
> I don't see any reason for a delete file since only files referenced
> by setup.hint should exist in the release directory.

There are currently a handful of packages where an earlier version than
"previous" is kept (ostensibly so that this version can be installed by
hand when chosing the "previous" version of another package which
requires it).  I wouldn't mind a "keep" keyword in setup.hint for those
cases.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Christopher Faylor]

On Tue, Oct 22, 2013 at 10:18:45PM +0200, Achim Gratz wrote:
>Christopher Faylor writes:
>> I don't see any reason for a delete file since only files referenced
>> by setup.hint should exist in the release directory.
>
>There are currently a handful of packages where an earlier version than
>"previous" is kept (ostensibly so that this version can be installed by
>hand when chosing the "previous" version of another package which
>requires it).  I wouldn't mind a "keep" keyword in setup.hint for those
>cases.

If someone is intentionally doing this then this isn't supported
behavior.  If files aren't there for the purposes of setup.exe then they
really shouldn't be in the release directory at all.

cgf

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Yaakov (Cygwin/X)]

On 2013-10-22 12:59, Andrew Schulman wrote:
> I'm sorry there's still no lftp for 64-bit - I'm having trouble compiling
> it - a configure problem.  I'll post about this soon to see if someone can
> help me to solve it.

This builds and seems to be working:

http://cygwin-ports.git.sourceforge.net/git/gitweb.cgi?p=cygwin-ports/lftp


Yaakov

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Reini Urban]

On Wed, Oct 16, 2013 at 11:52 AM, Christopher Faylor wrote:
> On Sat, Oct 12, 2013 at 04:22:37PM -0400, Christopher Faylor wrote:
>>I think I now have a system set up which will allow maintainers to
>>upload their own packages to sourceware.  This system means that package
>>maintainers won't have to find a public web server to make their
>>packages available for the cygwin release.  Every package maintainer
>>will be able to upload their own stuff.  General login access to
>>sourceware will no longer be required to update the cygwin release.
>
> This system is now live.  Anyone who sent in their ssh key should be
> able to access sourceware via lftp/sftp.  Make sure that you use
> the user "cygwin" with no password and that you are using the same
> ssh key as the one that you specified previously.
>
> Also, please create a !mail file at the top level of your area with the
> email address where any upset errors should be sent.

Note: !mail

> Remember also that you need to create a !ready file somewhere above the
> directory holding the packages that you want to end up in the release.
> So, if you have uploaded x86 and x86_64 packages put the !ready at the
> "root" level.  If you have uploaded x86 and are working on x86_64 you
> can put the !ready in the x86 directory and then put another !ready in
> the x86_64 directory when you're done.  Putting the !ready directly in
> the package directory should work too.

Note: root !ready doesn't work yet.
You need a !ready in x86 and x86_64

> Packages won't be moved unless upset thinks everything is ok.  If there
> is something broken and your have an email address in the !email file
> then you should get email telling you what's wrong.  You'll get the
> email even if the problem isn't your fault so don't panic.

Note: !email

Question: !mail or !email ?
The original message only had !mail

> Refer to: http://cygwin.com/ml/cygwin-apps/2013-10/msg00117.html for more
> detail about what's going on.  Note that the sftp command used there should
> be something like:

-- 
Reini Urban
http://cpanel.net/   http://www.perl-compiler.org/

Re: The upload system is live (Re: Major changes coming to procedure for uploading to sourceware)

[Christopher Faylor]

On Wed, Oct 23, 2013 at 11:51:39AM -0500, Reini Urban wrote:
>On Wed, Oct 16, 2013 at 11:52 AM, Christopher Faylor wrote:
>> Remember also that you need to create a !ready file somewhere above the
>> directory holding the packages that you want to end up in the release.
>> So, if you have uploaded x86 and x86_64 packages put the !ready at the
>> "root" level.  If you have uploaded x86 and are working on x86_64 you
>> can put the !ready in the x86 directory and then put another !ready in
>> the x86_64 directory when you're done.  Putting the !ready directly in
>> the package directory should work too.
>
>Note: root !ready doesn't work yet.

It's not going to work.

>You need a !ready in x86 and x86_64
>
>> Packages won't be moved unless upset thinks everything is ok.  If there
>> is something broken and your have an email address in the !email file
>> then you should get email telling you what's wrong.  You'll get the
>> email even if the problem isn't your fault so don't panic.
>
>Note: !email
>
>Question: !mail or !email ?
>The original message only had !mail

Either works.

cgf