Cygport uploading different files to different arch directorys for noarch packages

Cygport uploading different files to different arch directorys for noarch packages

[Adam Dinwoodie]

I'm seeing what seems to be some very odd behaviour from Cygport when
uploading noarch packages: Cygport uploads all the packages for the
64-bit architecture, but only the main and source packages for 32-bit
architecture.

Mostly I'm looking to know whether other people experience the same
behaviour when using Cygport to upload noarch packages -- the behaviour
I'm seeing is entirely reproducible on my system.

I've done some digging, and I'm utterly baffled, so I'm going to post
this here in case anyone else has any cunning theories; arguably this is
into the realms of an lftp/cygport/bash bug, so if folk want me to take
it to the main list, I'm happy to do that too.

Here's what I'm seeing (having commented out the !ready line in
pkg_upload.cygpart to allow testing, although I've checked both ways and
that doesn't make a difference):

    $ cygport fzf.cygport upload
    >>> Uploading fzf-0.10.8-1.noarch
    >>> Running lftp sftp://cygwin:@cygwin.com
    cd ok, cwd=/x86/release
    Transferring file `fzf-0.10.8-1-src.tar.xz'
    Transferring file `fzf-0.10.8-1.tar.xz'
    Transferring file `setup.hint'
    cd ok, cwd=/x86_64/release
    New: 3 files, 0 symlinks
    116628 bytes transferred in 1 second (84.4 KiB/s)
    Transferring file `fzf-0.10.8-1-src.tar.xz'
    Transferring file `fzf-0.10.8-1.tar.xz'
    Transferring file `setup.hint'
    Making directory `fzf-bash'
    Transferring file `fzf-bash/fzf-bash-0.10.8-1.tar.xz'
    Transferring file `fzf-bash/setup.hint'
    Making directory `fzf-bash-completion'
    Transferring file `fzf-bash-completion/fzf-bash-completion-0.10.8-1.tar.xz'
    Transferring file `fzf-bash-completion/setup.hint'
    Making directory `fzf-fish'
    Transferring file `fzf-fish/fzf-fish-0.10.8-1.tar.xz'
    Transferring file `fzf-fish/setup.hint'
    Making directory `fzf-vim'
    Transferring file `fzf-vim/fzf-vim-0.10.8-1.tar.xz'
    Transferring file `fzf-vim/setup.hint'
    Making directory `fzf-zsh'
    Transferring file `fzf-zsh/fzf-zsh-0.10.8-1.tar.xz'
    Transferring file `fzf-zsh/setup.hint'
    Making directory `fzf-zsh-completion'
    Transferring file `fzf-zsh-completion/fzf-zsh-completion-0.10.8-1.tar.xz'
    Transferring file `fzf-zsh-completion/setup.hint'
    Total: 6 directories, 15 files, 0 symlinks
    New: 15 files, 0 symlinks
    129612 bytes transferred in 8 seconds (15.3 KiB/s)
    Upload complete.

Note only three files were uploaded before the cd to /x86_64/release.

I've looked at the cygport code, and I can see no way this could happen
-- the code for uploading to x86 and x86_64 is the same code on a loop
-- and yet it's happening reliably for me.

By way of further testing, I hacked the Cygport code to dump the
lftp commands to file, then emulated the lftp command from my Bash
shell:

    $ cat upload_commands
    set cmd:fail-exit on
    set cmd:interactive on
    set net:max-retries 1
    open sftp://cygwin:@cygwin.com
    cd /x86/release
    rm -f fzf/!ready || echo -n
    mirror -v -eR fzf
    cd /x86_64/release
    rm -f fzf/!ready || echo -n
    mirror -v -eR fzf

    $ lftp -f <(cat upload_commands)
    cd ok, cwd=/x86/release
    Making directory `fzf-bash'
    Transferring file `fzf-bash/fzf-bash-0.10.8-1.tar.xz'
    Transferring file `fzf-bash/setup.hint'
    Making directory `fzf-bash-completion'
    Transferring file `fzf-bash-completion/fzf-bash-completion-0.10.8-1.tar.xz'
    Transferring file `fzf-bash-completion/setup.hint'
    Making directory `fzf-fish'
    Transferring file `fzf-fish/fzf-fish-0.10.8-1.tar.xz'
    Transferring file `fzf-fish/setup.hint'
    Making directory `fzf-vim'
    Transferring file `fzf-vim/fzf-vim-0.10.8-1.tar.xz'
    Transferring file `fzf-vim/setup.hint'
    Making directory `fzf-zsh'
    Transferring file `fzf-zsh/fzf-zsh-0.10.8-1.tar.xz'
    Transferring file `fzf-zsh/setup.hint'
    Making directory `fzf-zsh-completion'
    Transferring file `fzf-zsh-completion/fzf-zsh-completion-0.10.8-1.tar.xz'
    Transferring file `fzf-zsh-completion/setup.hint'
    Total: 6 directories, 15 files, 0 symlinks
    New: 12 files, 0 symlinks
    12984 bytes transferred in 7 seconds (1.8 KiB/s)
    cd ok, cwd=/x86_64/release
    Total: 6 directories, 15 files, 0 symlinks

So everything works fine there: it correctly finishes the x86 upload I
started in Cygport and does nothing for the already completed x86_64
upload.

For further testing, I hacked Cygport up some more to write that
upload_commands file out, then immediately read from it using the same
command as above.  This produced the bugged behaviour again, which
implies it's definitely something about the Cygport environment that's
causing the odd behaviour.

As I say, I'm utterly baffled by this.  For now I'll just upload the
remaining x86 files manually, but I'd love to know what's going on.

Re: Cygport uploading different files to different arch directorys for noarch packages

[Andrew Schulman]

> I'm seeing what seems to be some very odd behaviour from Cygport when
> uploading noarch packages: Cygport uploads all the packages for the
> 64-bit architecture, but only the main and source packages for 32-bit
> architecture.

Thanks for the troubleshooting and detailed report.  This is just weird.
The best I can tell is that it's a bug in lftp.  For some reason the lftp
mirror command isn't working properly within the cygport environment, half
of the time. ??

So I suggest you call this problem to the attention of the lftp maintainer.
Uh... oops.

Re: Cygport uploading different files to different arch directorys for noarch packages

[Ken Brown]

On 11/5/2015 5:12 PM, Andrew Schulman wrote:
>> I'm seeing what seems to be some very odd behaviour from Cygport when
>> uploading noarch packages: Cygport uploads all the packages for the
>> 64-bit architecture, but only the main and source packages for 32-bit
>> architecture.
>
> Thanks for the troubleshooting and detailed report.  This is just weird.
> The best I can tell is that it's a bug in lftp.  For some reason the lftp
> mirror command isn't working properly within the cygport environment, half
> of the time. ??
>
> So I suggest you call this problem to the attention of the lftp maintainer.
> Uh... oops.

FWIW, I've seen occasional failures of the lftp mirror command outside 
of cygport.  But I never had a reproducible case.  Usually I could just 
repeat the mirror command and it would work the second time.

Ken