Anonymous SVN read-only access problems

Anonymous SVN read-only access problems

[Diego Novillo]

We have local SVN checkouts that mirror several branches from
svn://gcc.gnu.org/svn/gcc. All these local checkouts use the anonymous
svn:// transport and get updated every hour or so.

Recently, one of the branches we mirror
(svn://gcc.gnu.org/svn/gcc/branches/google/main) started failing to
mirror. It seems that a very large commit is causing svn to fail
during the update. It goes something like this:

$ svn update --non-interactive --accept theirs-full -r 206484
D    libmudflap
U    libitm/ChangeLog
U    libitm/config/powerpc/sjlj.S
[ ... lots more ... ]
U    libsanitizer/tsan/tsan_interface_java.cc
A    libsanitizer/tsan/tsan_ignoreset.h
U    libsanitizer/tsan/tsan_rtl.h
svn: Malformed network data
svn: Network connection closed unexpectedly

If I try the exact same command on a personal SVN checkout, it works
fine. My suspicion is that the anonymous user is getting kicked out
because the operation takes too long, or uses more resources than its
allocation.

I am now trying with the http transport, which is considerably slower,
but it seems to be working (it's been going for almost 30 minutes,
however).

Are there any other options we could try?


Thanks.  Diego.

Re: Anonymous SVN read-only access problems

[Diego Novillo]

On Tue, Jan 14, 2014 at 11:43 AM, Diego Novillo <dnovillo@google.com> wrote:

> I am now trying with the http transport, which is considerably slower,
> but it seems to be working (it's been going for almost 30 minutes,
> however).

Correction. This also failed:

svn: REPORT of '/svn/gcc/!svn/vcc/default': Could not read response
body: Connection reset by peer (http://gcc.gnu.org)


Diego.

Re: Anonymous SVN read-only access problems

[Ian Lance Taylor]

Diego Novillo <dnovillo@google.com> writes:

> We have local SVN checkouts that mirror several branches from
> svn://gcc.gnu.org/svn/gcc. All these local checkouts use the anonymous
> svn:// transport and get updated every hour or so.
>
> Recently, one of the branches we mirror
> (svn://gcc.gnu.org/svn/gcc/branches/google/main) started failing to
> mirror. It seems that a very large commit is causing svn to fail
> during the update. It goes something like this:
>
> $ svn update --non-interactive --accept theirs-full -r 206484
> D    libmudflap
> U    libitm/ChangeLog
> U    libitm/config/powerpc/sjlj.S
> [ ... lots more ... ]
> U    libsanitizer/tsan/tsan_interface_java.cc
> A    libsanitizer/tsan/tsan_ignoreset.h
> U    libsanitizer/tsan/tsan_rtl.h
> svn: Malformed network data
> svn: Network connection closed unexpectedly
>
> If I try the exact same command on a personal SVN checkout, it works
> fine. My suspicion is that the anonymous user is getting kicked out
> because the operation takes too long, or uses more resources than its
> allocation.

It's possible.  The svn protocol is controlled by
/etc/xinetd.d/svnserve, which includes

        rlimit_as               = 2048M
        rlimit_cpu              = 1800


In /var/log/messages I see lots of

Jan 14 17:42:57 sourceware xinetd[1998]: EXIT: svn signal=6 pid=31214 duration=21(sec)

and a few

Jan 14 16:47:23 sourceware xinetd[1998]: EXIT: svn signal=13 pid=2665 duration=13200(sec)

The signal 6 messages are happening every minute or so.  The signal 13
messages are more like every 20 minutes.  Signal 6 is SIGABRT or SIGIOT.
Signal 13 is SIGPIPE.  I don't know how svn works internally, but it
seems at least possible that it opens a child process and that process
is being killed off, causing a SIGPIPE.

I increased rlimit_as to 3072m and sent a SIGHUP to xinetd.  See if that
makes any difference.

Ian

Re: Anonymous SVN read-only access problems

[Diego Novillo]

On Tue, Jan 14, 2014 at 12:53 PM, Ian Lance Taylor <ian@airs.com> wrote:

> I increased rlimit_as to 3072m and sent a SIGHUP to xinetd.  See if that
> makes any difference.

Thanks, Ian. Sadly, it didn't. I'm getting the exact same error.


Diego.

Re: Anonymous SVN read-only access problems

[Christopher Faylor]

On Tue, Jan 14, 2014 at 12:58:59PM -0500, Diego Novillo wrote:
>On Tue, Jan 14, 2014 at 12:53 PM, Ian Lance Taylor <ian@airs.com> wrote:
>
>> I increased rlimit_as to 3072m and sent a SIGHUP to xinetd.  See if that
>> makes any difference.
>
>Thanks, Ian. Sadly, it didn't. I'm getting the exact same error.

Diego,
Any chance you could pinpoint when this started happening?  I suspect that
a "yum update" pulled in a problematic package so it would help to know
when you started seeing this.

cgf

Re: Anonymous SVN read-only access problems

[Diego Novillo]

On Tue, Jan 14, 2014 at 1:15 PM, Christopher Faylor
<cgf-use-the-mailinglist-please@sourceware.org> wrote:
> On Tue, Jan 14, 2014 at 12:58:59PM -0500, Diego Novillo wrote:
>>On Tue, Jan 14, 2014 at 12:53 PM, Ian Lance Taylor <ian@airs.com> wrote:
>>
>>> I increased rlimit_as to 3072m and sent a SIGHUP to xinetd.  See if that
>>> makes any difference.
>>
>>Thanks, Ian. Sadly, it didn't. I'm getting the exact same error.
>
> Diego,
> Any chance you could pinpoint when this started happening?  I suspect that
> a "yum update" pulled in a problematic package so it would help to know
> when you started seeing this.

Yes. The first error occurred on 9/Jan at 21:21:59 UTC. This was the
first time that our mirror script tried to import SVN revision 206484
from svn://gcc.gnu.org/svn/gcc/branches/google/main.

This is a particularly large commit (a gigantic merge from trunk).
Since this works fine in my personal tree, I suspect that the limits
set on anonymous connections is what's causing the problem. The
puzzling thing is that we have done such large merges in the past.
This is the first time we run into issues with them.


Diego.

Re: Anonymous SVN read-only access problems

[Christopher Faylor]

On Tue, Jan 14, 2014 at 01:21:05PM -0500, Diego Novillo wrote:
>On Tue, Jan 14, 2014 at 1:15 PM, Christopher Faylor
><cgf-use-the-mailinglist-please@sourceware.org> wrote:
>> On Tue, Jan 14, 2014 at 12:58:59PM -0500, Diego Novillo wrote:
>>>On Tue, Jan 14, 2014 at 12:53 PM, Ian Lance Taylor <ian@airs.com> wrote:
>>>
>>>> I increased rlimit_as to 3072m and sent a SIGHUP to xinetd.  See if that
>>>> makes any difference.
>>>
>>>Thanks, Ian. Sadly, it didn't. I'm getting the exact same error.
>>
>> Diego,
>> Any chance you could pinpoint when this started happening?  I suspect that
>> a "yum update" pulled in a problematic package so it would help to know
>> when you started seeing this.
>
>Yes. The first error occurred on 9/Jan at 21:21:59 UTC. This was the
>first time that our mirror script tried to import SVN revision 206484
>from svn://gcc.gnu.org/svn/gcc/branches/google/main.

Here's a list of files updated since 9/Jan:

Jan 09 20:54:36 Updated: pcp-libs-3.8.9-1.el6.x86_64
Jan 09 20:54:37 Updated: python-pcp-3.8.9-1.el6.x86_64
Jan 09 20:54:37 Updated: perl-PCP-PMDA-3.8.9-1.el6.x86_64
Jan 09 20:54:40 Updated: openssl-1.0.1e-16.el6_5.4.x86_64
Jan 09 20:54:43 Updated: openssl-devel-1.0.1e-16.el6_5.4.x86_64
Jan 09 20:54:53 Updated: pcp-3.8.9-1.el6.x86_64
Jan 09 20:54:56 Updated: pcp-debuginfo-3.8.9-1.el6.x86_64
Jan 10 05:42:00 Installed: bc-1.06.95-1.el6.x86_64

I'd think that it could be openssl except svn doesn't seem to
use it.  It does use libk5crypto.so.3 though.

>This is a particularly large commit (a gigantic merge from trunk).
>Since this works fine in my personal tree, I suspect that the limits
>set on anonymous connections is what's causing the problem. The
>puzzling thing is that we have done such large merges in the past.
>This is the first time we run into issues with them.

I'm tempted to try a Microsoft solution to the problem since we've
had a kernel update since the last time we rebooted.

fche, iant, any objections to rebooting?

cgf

Re: Anonymous SVN read-only access problems

[Christopher Faylor]

On Tue, Jan 14, 2014 at 01:29:56PM -0500, Christopher Faylor wrote:
>fche, iant, any objections to rebooting?

The system is rebooted, Diego.  Any change?

cgf, the straw grasper

Re: Anonymous SVN read-only access problems

[Diego Novillo]

On Tue, Jan 14, 2014 at 1:58 PM, Christopher Faylor
<cgf-use-the-mailinglist-please@sourceware.org> wrote:
> On Tue, Jan 14, 2014 at 01:29:56PM -0500, Christopher Faylor wrote:
>>fche, iant, any objections to rebooting?
>
> The system is rebooted, Diego.  Any change?

Unfortunately, no. Failed with the same symptoms:

0114 19:08:30.712794: svn: Network connection closed unexpectedly

Re: Anonymous SVN read-only access problems

[Frank Ch. Eigler]

Hi -

> 0114 19:08:30.712794: svn: Network connection closed unexpectedly

Can you try again (and keep trying, if it is reoccurring)?

A systemtap script implicated a plain resource-limit type problem
(-lapr abort()ing after a memory allocation failure), so the limit was
bumped up.

- FChE

Re: Anonymous SVN read-only access problems

[Ian Lance Taylor]

Diego Novillo <dnovillo@google.com> writes:

> On Tue, Jan 14, 2014 at 1:58 PM, Christopher Faylor
> <cgf-use-the-mailinglist-please@sourceware.org> wrote:
>> On Tue, Jan 14, 2014 at 01:29:56PM -0500, Christopher Faylor wrote:
>>>fche, iant, any objections to rebooting?
>>
>> The system is rebooted, Diego.  Any change?
>
> Unfortunately, no. Failed with the same symptoms:
>
> 0114 19:08:30.712794: svn: Network connection closed unexpectedly

Comparing Diego running his script to the what xinetd puts in
/var/log/messages, it appears that svnserve is failing with signal 6
(SIGABRT, SIGIOT).

Ian

Re: Anonymous SVN read-only access problems

[Diego Novillo]

On Tue, Jan 14, 2014 at 2:51 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Hi -
>
>> 0114 19:08:30.712794: svn: Network connection closed unexpectedly
>
> Can you try again (and keep trying, if it is reoccurring)?
>
> A systemtap script implicated a plain resource-limit type problem
> (-lapr abort()ing after a memory allocation failure), so the limit was
> bumped up.

I just launched another one. This time it worked!

Thanks folks. I suppose this limit is only for anonymous connection?


Diego.