public inbox for overseers@sourceware.org
 help / color / mirror / Atom feed
* inbox.sourceware.org experiment
@ 2022-08-13 14:14 Mark Wielaard
  2022-08-15 13:00 ` Mark Wielaard
  2022-08-16 21:36 ` Mark Wielaard
  0 siblings, 2 replies; 15+ messages in thread
From: Mark Wielaard @ 2022-08-13 14:14 UTC (permalink / raw)
  To: overseers; +Cc: Simon Marchi

Hi,

It looks like our public-inbox experiment at
https://inbox.sourceware.org/ is starting to work out.

Currently only I and Simon have access to the inbox account through
ssh, but I think we can automate it enough to not need any manual
intervention unless now lists are added. But please ask if you want to
help with the setup.

I have setup sourceware-vhost-inbox.conf with corresponding
letsencrypt certificate. And public-inbox-nntpd, public-inbox-imapd
and public-inbox-httpd through systemd socket and service files.

So you should be able to access the mailboxes through git mirroring,
https, mbox downloads, atom feeds, nntp and imap.

You can already look at the experimental setup per list, e.g.
web-archive:
  https://inbox.sourceware.org/elfutils-devel/
individual messages and mbox per thread instructions:
  https://inbox.sourceware.org/elfutils-devel/_/text/help/
git mirror instructions:
  https://inbox.sourceware.org/elfutils-devel/_/text/mirror/
atom feed:
  https://inbox.sourceware.org/elfutils-devel/new.atom
imap:
  imap://inbox.sourceware.org/ (readonly, port 143, any user/pass)
nntp:
  nntp://inbox.sourceware.org/ (readonly, port 119)

Note that nntp group names and imap folder names might still change.
All current mailboxes are imported/mirrored as public-inbox-v1-format
but for scalability we will want to import them into
public-inbox-v2-format (this also parallelizes xapian indexing and
uses an sqlite database).

It looks like the inbox user can access the original emails to the
lists before mailman mangles the headers, but it cannot easily see for
which domain (sourceware, gcc, cygwin, ecos, etc.) they are. It would
be nice if we could name the news groups/folders after the primary
domain e.g. inbox.sourceware.elfutils-devel, inbox.gcc.gcc-patches,
inbox.cygwin.cygwin-talk.

The inbox.sourceware.test group at https://inbox.sourceware.org/test
is a simple mirror of http://try.public-inbox.org/test/ and I will
remove it soon (plus the cronjob that does the mirroring).

Looking at the mailman2inbox.sh script I have a few suggestions (I can
make them to the script myself, but don't know if you are currently
editing/running it):

- public-inbox-init should probably use -V2 (see above). You can then
  also use -j JOBS to speed up the import.

- --indexlevel shuld be full to make the Xapian searching more useful
  (this is the default, so you can also not set it). Note that this
  also affects the incremental updating done by public-inbox-mda.

- You want to kill public-inbox-httpd using -SIGHUP so it just reloads
  the new config files. Yo also want to kill the other daemons,
  public-inbox-imapd and public-inbox-nntpd

- The --ng name should be based on the primary domain name (see
  above). I don't know how to determine that easily though. Maybe
  mailman knows, then we can also set the initial ADDRESS properly.

The formail -s public-inbox-mda seems to work well for batch
importing, but is it efficient enough for keeping the importing up to
date? It looks like the last .mbox file is just really big and new
messages are appended at the end, so we would be trying to import all
messages all the ime. And how do we make sure it is triggered when new
messages come in?

Cheers,

Mark

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2022-08-24 21:06 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-13 14:14 inbox.sourceware.org experiment Mark Wielaard
2022-08-15 13:00 ` Mark Wielaard
2022-08-16 21:36 ` Mark Wielaard
2022-08-16 22:10   ` Frank Ch. Eigler
2022-08-17 12:25     ` Mark Wielaard
2022-08-17 13:24       ` Frank Ch. Eigler
2022-08-17 21:18         ` Mark Wielaard
2022-08-17 21:33           ` Frank Ch. Eigler
2022-08-18 13:50             ` Mark Wielaard
2022-08-18 14:40               ` Simon Marchi
2022-08-21 17:41                 ` Mark Wielaard
2022-08-23 20:15                   ` Mark Wielaard
2022-08-23 22:08               ` Mark Wielaard
2022-08-24 10:05                 ` Mark Wielaard
2022-08-24 21:06                   ` Mark Wielaard

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).