git and openssh issue

git and openssh issue

[Bill Hoffman]

I know this is an old issue.  I am wondering if I could help diagnose 
the problem, or if anyone is working on this issue.  The current fix 
seems to be to use plink and cygwin git.   But cygwin git and ssh just 
do not work together.

http://git.661346.n2.nabble.com/Error-when-cloning-gc-ed-repository-td4425832.html

Basically, it is this random error:
fatal: early EOFs:  47% (657/1396)


Seems like an issue with openssh and not git as using plink fixes the 
problem.   Anyway, with some direction I would be willing to help, I 
have a very reproducible case of this.

Thanks.

-Bill

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue

[Jeremy Bopp]

On 07/20/2010 11:35 AM, Bill Hoffman wrote:
> I know this is an old issue.  I am wondering if I could help diagnose
> the problem, or if anyone is working on this issue.  The current fix
> seems to be to use plink and cygwin git.   But cygwin git and ssh just
> do not work together.
> 
> http://git.661346.n2.nabble.com/Error-when-cloning-gc-ed-repository-td4425832.html
> 
> 
> Basically, it is this random error:
> fatal: early EOFs:  47% (657/1396)
> 
> 
> Seems like an issue with openssh and not git as using plink fixes the
> problem.   Anyway, with some direction I would be willing to help, I
> have a very reproducible case of this.

I haven't been able to reproduce this problem myself, but I was thinking
that using plink rather than ssh also eliminates a bit of Cygwin pipe
handling performed under ssh.  It might be useful to see if this can be
reproduced another way such as over something relatively basic like rsh.
 If the problem reproduces with rsh, that should point more directly to
a problem with the Cygwin DLL rather than ssh.

Do you have control over the server hosting the repository you're
cloning such that you can try that out?  If not, have you tried setting
up your own server with a complete clone of the repository to see if you
can reproduce the problem when cloning from that one?

-Jeremy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue

[Bill Hoffman]

On 7/21/2010 12:30 AM, Jeremy Bopp wrote:

>> http://git.661346.n2.nabble.com/Error-when-cloning-gc-ed-repository-td4425832.html
>>
>>
>> Basically, it is this random error:
>> fatal: early EOFs:  47% (657/1396)
>>
>>
>> Seems like an issue with openssh and not git as using plink fixes the
>> problem.   Anyway, with some direction I would be willing to help, I
>> have a very reproducible case of this.
>
> I haven't been able to reproduce this problem myself, but I was thinking
> that using plink rather than ssh also eliminates a bit of Cygwin pipe
> handling performed under ssh.  It might be useful to see if this can be
> reproduced another way such as over something relatively basic like rsh.
>   If the problem reproduces with rsh, that should point more directly to
> a problem with the Cygwin DLL rather than ssh.
>
> Do you have control over the server hosting the repository you're
> cloning such that you can try that out?  If not, have you tried setting
> up your own server with a complete clone of the repository to see if you
> can reproduce the problem when cloning from that one?
>

OK, I was able to test with rsh and it works.  So with the following 
install:
   cygwin                  1.7.5-1
   openssh                 5.5p1-2
   openssl                 0.9.8o-2
   rsh                     0.17-1

rsh works no problem.   ssh fails to clone with the EOF error.  Windows 
plink also works with no problem.


Are there any other experiments you would like to see?   Seems to happen 
with any reasonable size git repo.  So, if you were to say clone VTK or 
Qt, I am sure you could reproduce this as well.

-Bill



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue

[Jeremy Bopp]

On 7/22/2010 10:01 AM, Bill Hoffman wrote:
> On 7/21/2010 12:30 AM, Jeremy Bopp wrote:
> 
>>> http://git.661346.n2.nabble.com/Error-when-cloning-gc-ed-repository-td4425832.html
>>>
>>>
>>>
>>> Basically, it is this random error:
>>> fatal: early EOFs:  47% (657/1396)
>>>
>>>
>>> Seems like an issue with openssh and not git as using plink fixes the
>>> problem.   Anyway, with some direction I would be willing to help, I
>>> have a very reproducible case of this.
>>
>> I haven't been able to reproduce this problem myself, but I was thinking
>> that using plink rather than ssh also eliminates a bit of Cygwin pipe
>> handling performed under ssh.  It might be useful to see if this can be
>> reproduced another way such as over something relatively basic like rsh.
>>   If the problem reproduces with rsh, that should point more directly to
>> a problem with the Cygwin DLL rather than ssh.
>>
>> Do you have control over the server hosting the repository you're
>> cloning such that you can try that out?  If not, have you tried setting
>> up your own server with a complete clone of the repository to see if you
>> can reproduce the problem when cloning from that one?
>>
> 
> OK, I was able to test with rsh and it works.  So with the following
> install:
>   cygwin                  1.7.5-1
>   openssh                 5.5p1-2
>   openssl                 0.9.8o-2
>   rsh                     0.17-1
> 
> rsh works no problem.   ssh fails to clone with the EOF error.  Windows
> plink also works with no problem.
> 
> 
> Are there any other experiments you would like to see?   Seems to happen
> with any reasonable size git repo.  So, if you were to say clone VTK or
> Qt, I am sure you could reproduce this as well.

I'm game to try this out.  Can you send me the exact git command you use
to clone one or more of these repositories so that I can try to
reproduce what you see?

-Jeremy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue

[Bill Hoffman]

On 7/22/2010 11:25 AM, Jeremy Bopp wrote:

>>
>> Are there any other experiments you would like to see?   Seems to happen
>> with any reasonable size git repo.  So, if you were to say clone VTK or
>> Qt, I am sure you could reproduce this as well.
>
> I'm game to try this out.  Can you send me the exact git command you use
> to clone one or more of these repositories so that I can try to
> reproduce what you see?
>
> -Jeremy
>
We have reproduced this with the CMake git repo.  You can find out how 
to get a clone of that here:

http://www.cmake.org/Wiki/CMake/Git

So to clone CMake do this:
  git clone git://cmake.org/cmake.git CMake

Once you clone CMake to a machine, you can then clone from that machine 
like this:

Commands for ssh clone:

$ git clone user@machine:path/to/repo.git

Commands for rsh clone:

$ echo '#!/bin/sh
exec rsh -l user "$@"' > ~/ssh-rsh
$ chmod 750 ~/ssh-rsh
$ GIT_SSH=~/ssh-rsh git clone machine:path/to/repo.git

Commands for plink clone:

  http://my.afterdawn.com/agent_007/blog_entry.cfm/2909/git_and_cygwin


One thing that might be a good thing to try is to bisect the openssh 
source.  That might eliminate the cygwin dll as the source of the 
problem.  Or at least find out which change in openssh caused the problem.

-Bill


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue

[Jeremy Bopp]

On 7/22/2010 2:47 PM, Bill Hoffman wrote:
> On 7/22/2010 11:25 AM, Jeremy Bopp wrote:
> 
>>>
>>> Are there any other experiments you would like to see?   Seems to happen
>>> with any reasonable size git repo.  So, if you were to say clone VTK or
>>> Qt, I am sure you could reproduce this as well.
>>
>> I'm game to try this out.  Can you send me the exact git command you use
>> to clone one or more of these repositories so that I can try to
>> reproduce what you see?
>>
>> -Jeremy
>>
> We have reproduced this with the CMake git repo.  You can find out how
> to get a clone of that here:
> 
> http://www.cmake.org/Wiki/CMake/Git
> 
> So to clone CMake do this:
>  git clone git://cmake.org/cmake.git CMake
> 
> Once you clone CMake to a machine, you can then clone from that machine
> like this:
> 
> Commands for ssh clone:
> 
> $ git clone user@machine:path/to/repo.git
> 
> Commands for rsh clone:
> 
> $ echo '#!/bin/sh
> exec rsh -l user "$@"' > ~/ssh-rsh
> $ chmod 750 ~/ssh-rsh
> $ GIT_SSH=~/ssh-rsh git clone machine:path/to/repo.git
> 
> Commands for plink clone:
> 
>  http://my.afterdawn.com/agent_007/blog_entry.cfm/2909/git_and_cygwin
> 
> 
> One thing that might be a good thing to try is to bisect the openssh
> source.  That might eliminate the cygwin dll as the source of the
> problem.  Or at least find out which change in openssh caused the problem.

Just to clarify things, are you hosting your first clone under Cygwin
(from which you make another clone using Cygwin which fails), or are you
hosting it under a non-Windows system?  It might be good to try this
under the conditions of Cygwin client <-> Cygwin server, Cygwin client
<-> non-Windows server, and non-Windows client <-> Cygwin server.

Hopefully, we can localize the problem to just the client or server code
under Cygwin's ssh.  The bisect operation is not a bad idea, but I'm not
sure what revision of the openssh source was last known to work.  Given
that this is still not 100% reproducible even for you (if I understand
you correctly), the bisect tests may not be completely reliable anyway.

BTW, are you using text mode mounts under Cygwin for any of these
repository clones?

I'm going to try to make some time this evening to give this a quick test.

-Jeremy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue

[Bill Hoffman]

On 7/22/2010 6:44 PM, Jeremy Bopp wrote:

> Just to clarify things, are you hosting your first clone under Cygwin
> (from which you make another clone using Cygwin which fails), or are you
> hosting it under a non-Windows system?  It might be good to try this
> under the conditions of Cygwin client<->  Cygwin server, Cygwin client
> <->  non-Windows server, and non-Windows client<->  Cygwin server.
>
We are doing cygwin client to a Linux server that hosts git via ssh.
> Hopefully, we can localize the problem to just the client or server code
> under Cygwin's ssh.  The bisect operation is not a bad idea, but I'm not
> sure what revision of the openssh source was last known to work.  Given
> that this is still not 100% reproducible even for you (if I understand
> you correctly), the bisect tests may not be completely reliable anyway.
>
It is not that unreproducible... It fails maybe 5 out of 6 tries.  If 
you can get it to work 10 out of 10 times then I would say you don't 
have the problem.

According to this thread:
http://git.661346.n2.nabble.com/Error-when-cloning-gc-ed-repository-td4425832.html
 > I downgraded just the openssh  package in cygwin from 5.5p1-1 to 
5.4p1-1 and the
 > problem went away. This was easier than switching to putty/plink.

It seems that openssh 5.4p1-1 worked.  So, that would be a good place to 
start.

> BTW, are you using text mode mounts under Cygwin for any of these
> repository clones?
>
No, it is all binary mode mounts under Cygwin.
> I'm going to try to make some time this evening to give this a quick test.
>
Sounds good, thanks!



-Bill

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue

[Jeremy Bopp]

On 7/22/2010 8:19 PM, Bill Hoffman wrote:
> On 7/22/2010 6:44 PM, Jeremy Bopp wrote:
> 
> It is not that unreproducible... It fails maybe 5 out of 6 tries.  If
> you can get it to work 10 out of 10 times then I would say you don't
> have the problem.

I'm actually able to reproduce this early EOF error during cloning 100%
under the following scenario:

$ cd /tmp
$ git clone --bare git://cmake.org/cmake.git
$ git clone localhost:/tmp/cmake.git cmake-test

This is all under Cygwin using all binary mounts on Windows XP SP 3.
Cygwin, ssh, and git are all updated to current versions:

Cygwin Package Information
Package              Version        Status
cygwin               1.7.5-1        OK
git                  1.7.1-1        OK
openssh              5.5p1-2        OK


I haven't tried the other configurations I mentioned yet, but if there
is someone who has the ability to debug this in more depth, this looks
like a quick and readily reproducible test case.

-Jeremy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue (eblake?)

[Christopher Faylor]

On Fri, Jul 23, 2010 at 11:04:39AM -0500, Jeremy Bopp wrote:
>On 7/22/2010 8:19 PM, Bill Hoffman wrote:
>> On 7/22/2010 6:44 PM, Jeremy Bopp wrote:
>> 
>> It is not that unreproducible... It fails maybe 5 out of 6 tries.  If
>> you can get it to work 10 out of 10 times then I would say you don't
>> have the problem.
>
>I'm actually able to reproduce this early EOF error during cloning 100%
>under the following scenario:
>
>$ cd /tmp
>$ git clone --bare git://cmake.org/cmake.git
>$ git clone localhost:/tmp/cmake.git cmake-test
>
>This is all under Cygwin using all binary mounts on Windows XP SP 3.
>Cygwin, ssh, and git are all updated to current versions:
>
>Cygwin Package Information
>Package              Version        Status
>cygwin               1.7.5-1        OK
>git                  1.7.1-1        OK
>openssh              5.5p1-2        OK
>
>I haven't tried the other configurations I mentioned yet, but if there
>is someone who has the ability to debug this in more depth, this looks
>like a quick and readily reproducible test case.

I checked this out and it looks like something in git is overflowing
and scriblling on the stack.  I was hoping that it would be a cygwin
problem which I could fix (and I wouldn't be surprised if it eventually
turned out to be) but I didn't see much evidence of the DLL causing this
problem.  It looks like it is git itself doing something wrong.

Eric do you have the cycles to check this out?  It is pretty easy to
reproduce.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue (eblake?)

[Bill Hoffman]

On 7/23/2010 12:27 PM, Christopher Faylor wrote:

>> Cygwin Package Information
>> Package              Version        Status
>> cygwin               1.7.5-1        OK
>> git                  1.7.1-1        OK
>> openssh              5.5p1-2        OK
>>
>> I haven't tried the other configurations I mentioned yet, but if there
>> is someone who has the ability to debug this in more depth, this looks
>> like a quick and readily reproducible test case.
>
> I checked this out and it looks like something in git is overflowing
> and scriblling on the stack.  I was hoping that it would be a cygwin
> problem which I could fix (and I wouldn't be surprised if it eventually
> turned out to be) but I didn't see much evidence of the DLL causing this
> problem.  It looks like it is git itself doing something wrong.
>
> Eric do you have the cycles to check this out?  It is pretty easy to
> reproduce.
>

How did you come to the conclusion that it was a stack overwrite?  I 
would be willing to look at this a bit(not that I know much about git 
internals).  However, it would be helpful if I could see how you came to 
that conclusion?

Thanks.

-Bill

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue (eblake?)

[Bill Hoffman]

On 8/2/2010 10:08 AM, Bill Hoffman wrote:
> On 7/23/2010 12:27 PM, Christopher Faylor wrote:
>
>>> Cygwin Package Information
>>> Package Version Status
>>> cygwin 1.7.5-1 OK
>>> git 1.7.1-1 OK
>>> openssh 5.5p1-2 OK
>>>
>>> I haven't tried the other configurations I mentioned yet, but if there
>>> is someone who has the ability to debug this in more depth, this looks
>>> like a quick and readily reproducible test case.
>>
>> I checked this out and it looks like something in git is overflowing
>> and scriblling on the stack. I was hoping that it would be a cygwin
>> problem which I could fix (and I wouldn't be surprised if it eventually
>> turned out to be) but I didn't see much evidence of the DLL causing this
>> problem. It looks like it is git itself doing something wrong.
>>
>> Eric do you have the cycles to check this out? It is pretty easy to
>> reproduce.
>>
>
> How did you come to the conclusion that it was a stack overwrite? I
> would be willing to look at this a bit(not that I know much about git
> internals). However, it would be helpful if I could see how you came to
> that conclusion?
>
Is anyone looking at this?


Thanks.

-Bill

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: git and openssh issue (eblake?)

[Christopher Faylor]

On Fri, Jul 23, 2010 at 12:27:55PM -0400, Christopher Faylor wrote:
>On Fri, Jul 23, 2010 at 11:04:39AM -0500, Jeremy Bopp wrote:
>>On 7/22/2010 8:19 PM, Bill Hoffman wrote:
>>> On 7/22/2010 6:44 PM, Jeremy Bopp wrote:
>>> 
>>> It is not that unreproducible... It fails maybe 5 out of 6 tries.  If
>>> you can get it to work 10 out of 10 times then I would say you don't
>>> have the problem.
>>
>>I'm actually able to reproduce this early EOF error during cloning 100%
>>under the following scenario:
>>
>>$ cd /tmp
>>$ git clone --bare git://cmake.org/cmake.git
>>$ git clone localhost:/tmp/cmake.git cmake-test
>>
>>This is all under Cygwin using all binary mounts on Windows XP SP 3.
>>Cygwin, ssh, and git are all updated to current versions:
>>
>>Cygwin Package Information
>>Package              Version        Status
>>cygwin               1.7.5-1        OK
>>git                  1.7.1-1        OK
>>openssh              5.5p1-2        OK
>>
>>I haven't tried the other configurations I mentioned yet, but if there
>>is someone who has the ability to debug this in more depth, this looks
>>like a quick and readily reproducible test case.
>
>I checked this out and it looks like something in git is overflowing
>and scriblling on the stack.  I was hoping that it would be a cygwin
>problem which I could fix (and I wouldn't be surprised if it eventually
>turned out to be) but I didn't see much evidence of the DLL causing this
>problem.  It looks like it is git itself doing something wrong.

I've done some more testing this weekend and I'm coming to a different
conclusion.  I don't see git being implicated.  It really does seem to
be a Cygwin problem after all.  My current theory is that it's related
to Cygwin's pipe code.

Still investigating...

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple