Shell access / mailing list mbox archives

Shell access / mailing list mbox archives

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

Hallo!

I'd like to get access to the mbox archives for sourceware.org /
gcc.gnu.org mailing list archives.  For some years, I've been subscribed
to a bunch of these (glibc, gcc, gdb, binutils), first in the course of
GNU/Hurd toolchain work, and later for CodeSourcery work, too.  Now, I
lost a bunch of my archives due to own stupidity -- and didn't preserve
most of the messages anyway, as they've been downloadable by anonymous
FTP, but are no more, as I had to figure out.  (And now, I found the
thread starting at
<http://sourceware.org/ml/overseers/2007-q3/msg00112.html>, continued at
<http://sourceware.org/ml/overseers/2007-q4/msg00013.html> to confirm
this.)

I very much prefer having the archives locally (and indexed / searchable
as I like it), instead of trying to extract what I'm occasionally looking
for from the web interface.  And no, I'm not going to sell email
addresses to spammers, etc.

I just had a chat with Ian L. Taylor in #overseers, and it seems that the
appropriate way to get hold of the archives is shell access to the
sourceware.org machine.

An option suggested by Ian was crawling the web list archives by HTTP
downloads (for the ``Raw text'' links), and re-constructing the original
messages (s%@% at% mangling, etc.) -- but I don't know if that was a
serious suggestions; it seems like unreasonable overhead to me (we're
talking about ten thousands of messages).

Likewise, I could talk to ezmlm to send me back each message; likely even
higher overhead.

I could use GMANE to export mbox files of gcc-patches starting in 2002,
for example.  Still, this seems illogical to me to rely on a third party
for this.

As I don't really want to bother you guys with this request too much, is
it possible to simply get shell access to the machine myself?


Grüße,
 Thomas

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: Shell access / mailing list mbox archives

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 2478 bytes --]

Hallo!

On Mon, 07 Feb 2011 23:45:30 +0100, I wrote:
> I'd like to get access to the mbox archives for sourceware.org /
> gcc.gnu.org mailing list archives.

No answer, so I take it this is not admissible.  To get what I'd like, I
shall have to write some scripting hackery for interfacing with ezmlm by
email -- luckily this is ezmlm-idx, which allows for obtaining for ``a
maximum of 100 [messages] per request''.  Yet, given the list archives'
sizes, this will be some thousand request / answer emails, several
hundred MiB worth of data to transfer; to which timeframe should I spread
my requests to not needlessly stress the poor sourceware.org machine?

> For some years, I've been subscribed
> to a bunch of these (glibc, gcc, gdb, binutils), first in the course of
> GNU/Hurd toolchain work, and later for CodeSourcery work, too.  Now, I
> lost a bunch of my archives due to own stupidity -- and didn't preserve
> most of the messages anyway, as they've been downloadable by anonymous
> FTP, but are no more, as I had to figure out.  (And now, I found the
> thread starting at
> <http://sourceware.org/ml/overseers/2007-q3/msg00112.html>, continued at
> <http://sourceware.org/ml/overseers/2007-q4/msg00013.html> to confirm
> this.)
> 
> I very much prefer having the archives locally (and indexed / searchable
> as I like it), instead of trying to extract what I'm occasionally looking
> for from the web interface.  And no, I'm not going to sell email
> addresses to spammers, etc.
> 
> I just had a chat with Ian L. Taylor in #overseers, and it seems that the
> appropriate way to get hold of the archives is shell access to the
> sourceware.org machine.
> 
> An option suggested by Ian was crawling the web list archives by HTTP
> downloads (for the ``Raw text'' links), and re-constructing the original
> messages (s%@% at% mangling, etc.) -- but I don't know if that was a
> serious suggestions; it seems like unreasonable overhead to me (we're
> talking about ten thousands of messages).
> 
> Likewise, I could talk to ezmlm to send me back each message; likely even
> higher overhead.
> 
> I could use GMANE to export mbox files of gcc-patches starting in 2002,
> for example.  Still, this seems illogical to me to rely on a third party
> for this.
> 
> As I don't really want to bother you guys with this request too much, is
> it possible to simply get shell access to the machine myself?


Grüße,
 Thomas

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: Shell access / mailing list mbox archives

[Christopher Faylor]

On Fri, Apr 01, 2011 at 11:01:00PM +0200, Thomas Schwinge wrote:
>Hallo!
>
>On Mon, 07 Feb 2011 23:45:30 +0100, I wrote:
>> I'd like to get access to the mbox archives for sourceware.org /
>> gcc.gnu.org mailing list archives.
>
>No answer, so I take it this is not admissible.

Sorry for the lack of response.  

No, it's not admissable.  We're not going to give out shell access to
the system to accommodate the needs of users who have personal reasons
to access the system.

We give out shell access on a per-project basis for people who need to
do administrative tasks and we try to be as stingy as possible when
doing even that.

cgf

Re: Shell access / mailing list mbox archives

[Gerald Pfeifer]

On Fri, 1 Apr 2011, Thomas Schwinge wrote:
>> I'd like to get access to the mbox archives for sourceware.org /
>> gcc.gnu.org mailing list archives.
> No answer, so I take it this is not admissible.  To get what I'd like, I 
> shall have to write some scripting hackery for interfacing with ezmlm by 
> email -- luckily this is ezmlm-idx, which allows for obtaining for ``a 
> maximum of 100 [messages] per request''.  Yet, given the list archives' 
> sizes, this will be some thousand request / answer emails, several 
> hundred MiB worth of data to transfer; to which timeframe should I 
> spread my requests to not needlessly stress the poor sourceware.org 
> machine?

Is there a more direct way to provide to Thomas what he wants?  I
could not find any mbox files on gcc.gnu.org any more, but perhaps
I've been missing them somewhere (or the could be recreated easily)?

Gerald

Re: Shell access / mailing list mbox archives

[Joseph S. Myers]

On Sat, 2 Apr 2011, Gerald Pfeifer wrote:

> On Fri, 1 Apr 2011, Thomas Schwinge wrote:
> >> I'd like to get access to the mbox archives for sourceware.org /
> >> gcc.gnu.org mailing list archives.
> > No answer, so I take it this is not admissible.  To get what I'd like, I 
> > shall have to write some scripting hackery for interfacing with ezmlm by 
> > email -- luckily this is ezmlm-idx, which allows for obtaining for ``a 
> > maximum of 100 [messages] per request''.  Yet, given the list archives' 
> > sizes, this will be some thousand request / answer emails, several 
> > hundred MiB worth of data to transfer; to which timeframe should I 
> > spread my requests to not needlessly stress the poor sourceware.org 
> > machine?
> 
> Is there a more direct way to provide to Thomas what he wants?  I
> could not find any mbox files on gcc.gnu.org any more, but perhaps
> I've been missing them somewhere (or the could be recreated easily)?

They're /sourceware/vault/old-ftp-archives/.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Shell access / mailing list mbox archives

[Gerald Pfeifer]

On Sat, 2 Apr 2011, Joseph S. Myers wrote:
> They're /sourceware/vault/old-ftp-archives/.

Thanks for the pointer, Joseph.  Sadly these end Sep 26  2007, but
if still you want, Thomas, contact me off-list.

Gerald

Re: Shell access / mailing list mbox archives

[Ian Lance Taylor]

Gerald Pfeifer <gerald@pfeifer.com> writes:

> On Sat, 2 Apr 2011, Joseph S. Myers wrote:
>> They're /sourceware/vault/old-ftp-archives/.
>
> Thanks for the pointer, Joseph.  Sadly these end Sep 26  2007, but
> if still you want, Thomas, contact me off-list.

There are more archives are in /qmail/*/*/archive.

Ian

Re: Shell access / mailing list mbox archives

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

Hallo!

On Sat, 02 Apr 2011 20:45:51 -0700, Ian Lance Taylor <ian@airs.com> wrote:
> Gerald Pfeifer <gerald@pfeifer.com> writes:
> > On Sat, 2 Apr 2011, Joseph S. Myers wrote:
> >> They're /sourceware/vault/old-ftp-archives/.
> >
> > Thanks for the pointer, Joseph.  Sadly these end Sep 26  2007, but
> > if still you want, Thomas, contact me off-list.
> 
> There are more archives are in /qmail/*/*/archive.

Sounds good!  How would I get access to these?  You send me a list of
those that are available, and I'll send back the list of those that I'm
interested in?


Grüße,
 Thomas

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: Shell access / mailing list mbox archives

[Gerald Pfeifer]

On Sun, 3 Apr 2011, Thomas Schwinge wrote:
> Sounds good!  How would I get access to these?  You send me a list of 
> those that are available, and I'll send back the list of those that I'm 
> interested in?

This is the internal qmail structure, one file per mail, 100 mails
per directory, some 280MB for the gcc@gcc.gnu.org archives tar-ed
up, 56MB for the same also bz2-ed.  I can provide this to you, if
you let me know a way to do so (offlist).  And one or two others.

Have you given http://blog.gmane.org/gmane.comp.gcc.patches et al
a try?  That should be able to offer you want via NNTP.

Gerald