Re: Upstream cvs repository corruption?

Re: Upstream cvs repository corruption?

[Jim Meyering]

Dave Wysochanski <dwysocha@redhat.com> wrote:
> Is the upstream CVS repository corrupted?
>
> cvs checkout: Updating LVM2/test
> U LVM2/test/.gitignore
> U LVM2/test/Makefile.in
> cvs [checkout aborted]: head attribute does not match file for
> `/cvs/lvm2/LVM2/test/lvm-utils.sh,v'

Hi Dave,

I see the problem, and think I see how it happened, too.
The trouble is that the /cvs/lvm2/LVM2/test/CVS/fileattr file
(which normally records stuff about each file in its dir) is
out of sync.  When you look at the permissions on the containing
directory, you see why:

    sourceware$ pwd
    /cvs/lvm2/LVM2/test
    sourceware$ ls -ld CVS
    drwxrwxr-x  2 meyering automake 4096 Sep 18 14:00 CVS/

The "group" should be "lvm", not automake.
Trouble is that I'm a member of both, and automake is my default group,
because I contributed to that project long before I was added to the lvm
group.

For now, I've just remove the hosed "fileattr" file and corrected
the permissions on the offending directory.  That's enough so that
your checkout will now succeed.

The longer term fix is to make the group owner sticky (chmod g+s)
in all CVS trees, so that all added directories get the proper group.
FYI, this happened to "test/" because I just added that directory to lvm.

Would someone with root access please run the following to fix lvm2?

    chgrp -R lvm /cvs/lvm2/LVM2
    chmod -R g+s /cvs/lvm2/LVM2

I nearly Cc'd Frank about this (to thank him for helping to set up git
recently :-), but thought better of it, and am Cc'ing overseers, instead.

Of course, the real solution is to switch to git,
but lvm will have to wait a few weeks for that ;-)

Jim

Re: Upstream cvs repository corruption?

[Christopher Faylor]

On Mon, Sep 24, 2007 at 03:04:00PM +0200, Jim Meyering wrote:
>I nearly Cc'd Frank about this (to thank him for helping to set up git
>recently :-), but thought better of it, and am Cc'ing overseers, instead.
>
>Of course, the real solution is to switch to git,
>but lvm will have to wait a few weeks for that ;-)

In the meantime I've noticed quite a few long-running processes owned by
you running on sourceware.  Having long-running user-owned processes is
not really a good practice given the principle of "what happens if he's
hit by a bus?"

Should your git-related processes be owned by someone other than you?
Would that solve both problems?

cgf

Re: Upstream cvs repository corruption?

[Jim Meyering]

Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> wrote:
> On Mon, Sep 24, 2007 at 03:04:00PM +0200, Jim Meyering wrote:
>>I nearly Cc'd Frank about this (to thank him for helping to set up git
>>recently :-), but thought better of it, and am Cc'ing overseers, instead.
>>
>>Of course, the real solution is to switch to git,
>>but lvm will have to wait a few weeks for that ;-)
>
> In the meantime I've noticed quite a few long-running processes owned by
> you running on sourceware.  Having long-running user-owned processes is

As far as I know, there was one incident.  Frank told me you noticed some,
and I killed them right away.  Did I not kill them quickly enough? :-) 1/2
About 10 minutes elapsed on Saturday, between when I heard about the
problem and when they were all dead.

BTW, I don't know what made that happen (a defunct cvs process was
at the root of each of two process trees).  If it happens again, I'll
investigate.  Normal mirror-sync runs are very quick.

> not really a good practice given the principle of "what happens if he's
> hit by a bus?"

They weren't consuming significant resources, so
I don't see the problem with those few delinquent processes.

If you mean the mirroring service, I see that as a short-term
project, since Alasdair is interested in converting to git.
If the mirroring were expected to last more than a few weeks, I'd be
interested in moving the cron jobs to a role-based account.
If you'd prefer, I'll do that anyway, but it doesn't seem worthwhile
at this point.

> Should your git-related processes be owned by someone other than you?
> Would that solve both problems?

Re: Upstream cvs repository corruption?

[Christopher Faylor]

On Mon, Sep 24, 2007 at 04:27:56PM +0200, Jim Meyering wrote:
>Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> wrote:
>> On Mon, Sep 24, 2007 at 03:04:00PM +0200, Jim Meyering wrote:
>>>I nearly Cc'd Frank about this (to thank him for helping to set up git
>>>recently :-), but thought better of it, and am Cc'ing overseers, instead.
>>>
>>>Of course, the real solution is to switch to git,
>>>but lvm will have to wait a few weeks for that ;-)
>>
>> In the meantime I've noticed quite a few long-running processes owned by
>> you running on sourceware.  Having long-running user-owned processes is
>
>As far as I know, there was one incident.  Frank told me you noticed some,
>and I killed them right away.  Did I not kill them quickly enough? :-) 1/2

No, actually, I asked about them and, AFAIK, never got a response.  The
issue wasn't with the processes running.  Random users should not be
starting long running daemon processes on sourceware.

>About 10 minutes elapsed on Saturday, between when I heard about the
>problem and when they were all dead.
>
>BTW, I don't know what made that happen (a defunct cvs process was
>at the root of each of two process trees).  If it happens again, I'll
>investigate.  Normal mirror-sync runs are very quick.
>
>> not really a good practice given the principle of "what happens if he's
>> hit by a bus?"
>
>They weren't consuming significant resources, so
>I don't see the problem with those few delinquent processes.

I didn't say anything about resources.  User "meyring" should not be
running daemons or other long-running processes.  If we are going to be
running git then it should be run from a user-neutral account.  I was
trying to find out what that should actually be.  There is no need to
rationalize the need.  I just want to figure out how this should
actually be set up.  And, I thought that if we chose another account
with different access rights it might actually solve the problem.

cgf

Re: Upstream cvs repository corruption?

[Jim Meyering]

Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> wrote:

> On Mon, Sep 24, 2007 at 04:27:56PM +0200, Jim Meyering wrote:
>>Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> wrote:
>>> On Mon, Sep 24, 2007 at 03:04:00PM +0200, Jim Meyering wrote:
>>>>I nearly Cc'd Frank about this (to thank him for helping to set up git
>>>>recently :-), but thought better of it, and am Cc'ing overseers, instead.
>>>>
>>>>Of course, the real solution is to switch to git,
>>>>but lvm will have to wait a few weeks for that ;-)
>>>
>>> In the meantime I've noticed quite a few long-running processes owned by
>>> you running on sourceware.  Having long-running user-owned processes is
>>
>>As far as I know, there was one incident.  Frank told me you noticed some,
>>and I killed them right away.  Did I not kill them quickly enough? :-) 1/2
>
> No, actually, I asked about them and, AFAIK, never got a response.  The
> issue wasn't with the processes running.  Random users should not be
> starting long running daemon processes on sourceware.

For what it's worth, I'm _not_ running any daemon processes
from my account on sourceware.  The long-running processes you
spotted were the result of a bug somewhere, and not related to
the git "service".  The programs involved were converting
cvs deltas to git ones and importing them into a git repository.
Not actually _using_ the git server at all, but rather, adding to
a repo that is used by the git server.

>>About 10 minutes elapsed on Saturday, between when I heard about the
>>problem and when they were all dead.
>>
>>BTW, I don't know what made that happen (a defunct cvs process was
>>at the root of each of two process trees).  If it happens again, I'll
>>investigate.  Normal mirror-sync runs are very quick.
>>
>>> not really a good practice given the principle of "what happens if he's
>>> hit by a bus?"
>>
>>They weren't consuming significant resources, so
>>I don't see the problem with those few delinquent processes.
>
> I didn't say anything about resources.  User "meyring" should not be
> running daemons or other long-running processes.  If we are going to be
> running git then it should be run from a user-neutral account.  I was

Um... git is properly installed, and does not in any way rely on
my personal account.

> trying to find out what that should actually be.  There is no need to
> rationalize the need.  I just want to figure out how this should
> actually be set up.  And, I thought that if we chose another account
> with different access rights it might actually solve the problem.

Chris,

From the tone of your messages, I fear I may have done something to
offend you.  If so, please rest assured that whatever I did was not
intended to rustle any feathers.

Re: Upstream cvs repository corruption?

[Christopher Faylor]

On Mon, Sep 24, 2007 at 05:06:37PM +0200, Jim Meyering wrote:
>Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> wrote:
>>I was trying to find out what that should actually be.  There is no
>>need to rationalize the need.  I just want to figure out how this
>>should actually be set up.  And, I thought that if we chose another
>>account with different access rights it might actually solve the
>>problem.
>
>Chris,
>
From the tone of your messages, I fear I may have done something to
>offend you.  If so, please rest assured that whatever I did was not
>intended to rustle any feathers.

Please reread the above without any preconceptions about tone.  I'm not
"offended".  I want to solve a problem.

Here is my point:

SER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root     20017  0.0  0.0  6444 2688 ?        Ss   12:43   0:00 sshd: meyering@pts/0
meyering 20021  0.0  0.0  6768 2048 pts/0    Ss+  12:43   0:00 -zsh
meyering 20335  0.0  0.0  2184  884 ?        Ss   12:45   0:00 /bin/sh /home/meyering/bin/mirror-sw lvm
meyering 20343  0.0  0.0  2484  916 ?        S    12:45   0:00 /bin/sh /home/meyering/bin/mirror-cvs-to-git --rsync-url=/cvs/lvm2 --module=LVM2 --git-dir=/home/meyering/mirror-git-to-cvs/repo/LVM2 --state-dir=/home/meyering/mirror-git-to-cvs/.state/LVM2 --key-file=tools/lvm.c,v --user-map=/home/meyering/mirror-git-to-cvs/git-user-map/LVM2
meyering 20436  0.0  0.0  6908 5068 ?        S    12:45   0:00 /usr/bin/perl -w /usr/local/bin/git-cvsimport -A /home/meyering/mirror-git-to-cvs/.state/LVM2/user-map -d /home/meyering/mirror-git-to-cvs/.state/LVM2/cvsrepo/LVM2 -C /home/meyering/mirror-git-to-cvs/repo/LVM2 -p -z,120 LVM2
meyering 20466  0.0  0.0  2712  672 ?        S    12:45   0:00 cvs server
meyering 20472  0.0  0.0  6576 4364 ?        S    12:45   0:00 cvsps --norc -q -z 120 --cvs-direct -u -A --root /home/meyering/mirror-git-to-cvs/.state/LVM2/cvsrepo/LVM2 LVM2
meyering 20473  0.0  0.0  3664  900 ?        S    12:45   0:00 cvs server

Those processes have been running since 8:43AM EDT.  If we are going to
support git on sourceware, should we expect to see this type of thing on
a regular basis?  And, given the original email, which contained a
problem, is there some other way that git could be run to avoid the
problem?

I run a script on a regular basis which kills what seems to be hung
processes on sourceware.  I very nearly killed similar processes this
weekend.  I want to find out if I have to ignore these type of processes
in the future or if there is a better way to handle this so that it
doesn't look like user "meyering" (or some other git user) has lots of
long-running processes.

cgf

Re: Upstream cvs repository corruption?

[Jim Meyering]

Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> wrote:

> On Mon, Sep 24, 2007 at 05:06:37PM +0200, Jim Meyering wrote:
>>Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> wrote:
>>>I was trying to find out what that should actually be.  There is no
>>>need to rationalize the need.  I just want to figure out how this
>>>should actually be set up.  And, I thought that if we chose another
>>>account with different access rights it might actually solve the
>>>problem.
>>
>>Chris,
>>
>From the tone of your messages, I fear I may have done something to
>>offend you.  If so, please rest assured that whatever I did was not
>>intended to rustle any feathers.
>
> Please reread the above without any preconceptions about tone.  I'm not
> "offended".  I want to solve a problem.

I'm glad there was no perceived offense.

BTW, I have no preconceptions.
How could I have any, when I don't know you
and haven't heard anything to bias me?

> Here is my point:
>
> SER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
[more hung cvs-related processes]

The tools that seem to make cvs hang (mirror-* -> git-cvsimport -> cvsps -> cvs)
are solely to synchronize a mirrored git repository from the cvs one.
Thus, removing these tools would not affect whether sourceware.org
provides a git server.

Those same scripts work fine on at least three other systems (one of
which is also RHEL 4).  However, those systems all use cvs-1.12.13,
while sourceware is using cvs-1.11.15.  I've built 1.12.13 with -g
and installed it in my own ~/bin directory so the scripts will use that.
We'll see if that fixes the problem.  If not, I'll debug it, so one
way or another, this problem should last no longer than a day or so more.