htdig problems?

htdig problems?

[Christopher Faylor]

People are reporting that searches are taking a very long time and sometimes
reporting internal server errors.  Is there a problem with htdig?

cgf

Re: htdig problems? (PING)

[Christopher Faylor]

On Wed, Oct 30, 2002 at 04:10:00PM -0500, Christopher Faylor wrote:
>People are reporting that searches are taking a very long time and sometimes
>reporting internal server errors.  Is there a problem with htdig?

H-P do you have any idea why people are reporting internal server errors
when attempting to do searches?  I've seen the same myself.

Someone reported that doing a search for "setup.exe" in the cygwin archives
had this effect.

cgf

Re: htdig problems? (PING)

[Jason Molenda]

On Thu, Nov 07, 2002 at 10:36:24AM -0500, Christopher Faylor wrote:

> Someone reported that doing a search for "setup.exe" in the cygwin archives
> had this effect.

We could force a clean database rebuild this weekend.  If the database has
been corrupted in some way, that should clear things up.  It takes a _long_
time, though, so it's definitely something to put off until the weekend
when system load goes down..

J

Re: htdig problems? (PING)

[Hans-Peter Nilsson]

On Thu, 7 Nov 2002, Christopher Faylor wrote:
> On Wed, Oct 30, 2002 at 04:10:00PM -0500, Christopher Faylor wrote:
> >People are reporting that searches are taking a very long time and sometimes
> >reporting internal server errors.  Is there a problem with htdig?
>
> H-P do you have any idea why people are reporting internal server errors
> when attempting to do searches?  I've seen the same myself.

I've mentioned this before, FWIW.  If a search takes longer than
5min, htdig (or really: htsearch) catches (or actually, does not
catch) SIGALARM from its alarm(2) call and apache sees the
termination as a 5xx, which is arguably a correct interpretation.

You get this for searches that would return many hits, in big
databases such as sourceware's and gcc's.  A subsequent attempt
is often successful (finishes within 5min)  because the disk
cache is then a bit warmer.

I'm unsure if changing that alarm call would be for the better.
A database search run-wild or a DOS attack would quickly make
the machine unresponsive.

brgds, H-P

Re: htdig problems? (PING)

[Christopher Faylor]

On Thu, Nov 07, 2002 at 02:05:57PM -0500, Hans-Peter Nilsson wrote:
>On Thu, 7 Nov 2002, Christopher Faylor wrote:
>> On Wed, Oct 30, 2002 at 04:10:00PM -0500, Christopher Faylor wrote:
>> >People are reporting that searches are taking a very long time and sometimes
>> >reporting internal server errors.  Is there a problem with htdig?
>>
>> H-P do you have any idea why people are reporting internal server errors
>> when attempting to do searches?  I've seen the same myself.
>
>I've mentioned this before, FWIW.  If a search takes longer than
>5min, htdig (or really: htsearch) catches (or actually, does not
>catch) SIGALARM from its alarm(2) call and apache sees the
>termination as a 5xx, which is arguably a correct interpretation.

I've just been seeing this much more frequently recently so I'm wondering
if something is wrong somewhere.  Maybe sourceware is having disk problems.

cgf

Re: htdig problems? (PING)

[Hans-Peter Nilsson]

On Thu, 7 Nov 2002, Christopher Faylor wrote:
> On Thu, Nov 07, 2002 at 02:05:57PM -0500, Hans-Peter Nilsson wrote:
> >If a search takes longer than
> >5min, htdig (or really: htsearch) catches (or actually, does not
> >catch) SIGALARM from its alarm(2) call and apache sees the
> >termination as a 5xx, which is arguably a correct interpretation.
>
> I've just been seeing this much more frequently recently so I'm wondering
> if something is wrong somewhere.  Maybe sourceware is having disk problems.

Could be.  The logs from htdig db updates look fine, FWIW: no
spurius failures.  Speaking of htdig, one temporary file from
the gcc update is a bit too close to (1 << 31) bytes:
1905972789.  (Quite expected, though.)

brgds, H-P

Re: htdig problems? (PING)

[Angela Marie Thomas]

> I've just been seeing this much more frequently recently so I'm wondering
> if something is wrong somewhere.  Maybe sourceware is having disk problems.

I've started seeing longer than normal times backing up /sourceware/www.
The amount of data changing is roughly the same, so perhaps there is a
disk problem.  I took a quick peek at the logs, but only saw a few SCSI
error messages, not tons like I'd expect.

--Angela

Re: htdig problems? (PING)

[Christopher Faylor]

On Thu, Nov 07, 2002 at 12:27:22PM -0800, Angela Marie Thomas wrote:
>> I've just been seeing this much more frequently recently so I'm wondering
>> if something is wrong somewhere.  Maybe sourceware is having disk problems.
>
>I've started seeing longer than normal times backing up /sourceware/www.
>The amount of data changing is roughly the same, so perhaps there is a
>disk problem.  I took a quick peek at the logs, but only saw a few SCSI
>error messages, not tons like I'd expect.

It *seems* like load averages have been higher than usual lately, too.
I haven't been able to figure out what is causing it though.

cgf

Re: htdig problems? (PING)

[Angela Marie Thomas]

Forgot to mention... This seemed to start happening frequently around
10/22 but there are some normal time runs between then and now as well.
The times vary wildly.  Backups start at 02:23 US/Pacific and generally
end around 05:00 US/Pacific.  Lately they've been ending aroung 10:00
or even as late as 11:00.

--Angela

Re: htdig problems? (PING)

[Christopher Faylor]

On Thu, Nov 07, 2002 at 03:31:16PM -0500, Christopher Faylor wrote:
>On Thu, Nov 07, 2002 at 12:27:22PM -0800, Angela Marie Thomas wrote:
>>> I've just been seeing this much more frequently recently so I'm wondering
>>> if something is wrong somewhere.  Maybe sourceware is having disk problems.
>>
>>I've started seeing longer than normal times backing up /sourceware/www.
>>The amount of data changing is roughly the same, so perhaps there is a
>>disk problem.  I took a quick peek at the logs, but only saw a few SCSI
>>error messages, not tons like I'd expect.
>
>It *seems* like load averages have been higher than usual lately, too.
>I haven't been able to figure out what is causing it though.

Now that I've said that, the load average is currently at 1.29 of
course.

cgf

Re: htdig problems? (PING)

[Hans-Peter Nilsson]

On Thu, 7 Nov 2002, Christopher Faylor wrote:
> On Thu, Nov 07, 2002 at 03:31:16PM -0500, Christopher Faylor wrote:
> >It *seems* like load averages have been higher than usual lately, too.
> >I haven't been able to figure out what is causing it though.
>
> Now that I've said that, the load average is currently at 1.29 of
> course.

Which seems to coincide with the gcc htdig update finishing
12:12:27 PST, having started 00:17. I don't know why it took
this long.  It used to do that some months ago too, then dropped
back to a few hours.

brgds, H-P

Re: htdig problems? (PING)

[Christopher Faylor]

On Thu, Nov 07, 2002 at 04:05:46PM -0500, Hans-Peter Nilsson wrote:
>On Thu, 7 Nov 2002, Christopher Faylor wrote:
>> On Thu, Nov 07, 2002 at 03:31:16PM -0500, Christopher Faylor wrote:
>> >It *seems* like load averages have been higher than usual lately, too.
>> >I haven't been able to figure out what is causing it though.
>>
>> Now that I've said that, the load average is currently at 1.29 of
>> course.
>
>Which seems to coincide with the gcc htdig update finishing
>12:12:27 PST, having started 00:17. I don't know why it took
>this long.  It used to do that some months ago too, then dropped
>back to a few hours.

Right.  I remember when we added the CPU things got much faster.

cgf

Re: htdig problems? (PING)

[Hans-Peter Nilsson]

On Thu, 7 Nov 2002, Christopher Faylor wrote:
> On Thu, Nov 07, 2002 at 04:05:46PM -0500, Hans-Peter Nilsson wrote:
> >On Thu, 7 Nov 2002, Christopher Faylor wrote:
> >> Now that I've said that, the load average is currently at 1.29 of
> >> course.
> >
> >Which seems to coincide with the gcc htdig update finishing
> >12:12:27 PST, having started 00:17. I don't know why it took
> >this long.  It used to do that some months ago too, then dropped
> >back to a few hours.
>
> Right.  I remember when we added the CPU things got much faster.

Doh!  Yeah, I chose to remember just the effect or something.
Or is there another CPU to add?  Can it be done regularly? :-)

brgds, H-P