Re: Moderation - Mailing list pgsql-www

Greetings,

* Adrian Klaver (adrian.klaver@aklaver.com) wrote:
> On 8/16/21 7:55 AM, Stephen Frost wrote:
> >* Adrian Klaver (adrian.klaver@aklaver.com) wrote:
> >>On 8/13/21 1:51 PM, Stephen Frost wrote:
> >>>>Am I right that there is no documentation currently?
> >>>
> >>>Yeah.. I'm afraid that's mostly the case, though maybe I'm forgetting
> >>>something.  Magnus?
> >>>
> >>>>Any pointers as to where to look for setup information e.g requirements.txt.
> >>>
> >>>Eh, not exactly the same as what you're asking for, but the package we
> >>>built for it has:
> >>>
> >>>Depends: pginfra-wrap-django, pginfra-wrap-eximserver, python-psycopg2,
> >>>     python3-psycopg2, pginfra-wrap-pgbouncer,
> >>>     pginfra-wrap-postgresql-11 | pginfra-wrap-postgresql-12,
> >>>     pginfra-wrap-exim-using-pgsql
> >>
> >>pginfra lives where?
> >
> >Not sure what you're asking here.
>
> I have yet to find a pginfra project so I am trying to figure out where the
> packages come from?

Those are just wrappers that we build that depend on regular packages
but add a bit of configuration and such, but nothing that is likely to
be interesting to this.  pgbouncer isn't required, certainly PG is but
it could be 11, 12, or probably others.  We use exim for most of our
email and we have a wrapper package to make exim wait for PG to start up
since our exim config queries the PG database for things (the actual
exim config is backed up but isn't part of the package since it's
different on our pglister system from other ones).

As much as I'd like this to all be automated and easy to deploy, it
isn't today.  One thing that might help is that the processes which need
to be started are:

/lib/systemd/system/pglister_bounceprocessor.service
/lib/systemd/system/pglister_cleanup.service
/lib/systemd/system/pglister_eximcleaner.service
/lib/systemd/system/pglister_mailprocessor.service
/lib/systemd/system/pglister_mailsender.service
/lib/systemd/system/pglister_moderationprocessor.service

Each of those is basically:

####
[Unit]
Description=pglister: bounce processor
After=postgresql.service

[Service]
ExecStart=/usr/local/pglister/bin/bounceprocessor.py
WorkingDirectory=/usr/local/pglister
Restart=always
RestartSec=30
User=list
Group=list

[Install]
WantedBy=multi-user.target
####

As mentioned, we use exim and so the way mail gets into the system is
through a pipe:

pglister_pipe:
  driver = pipe
  command = MAILINGLISTPIPESCRIPT
  #ignore_status
  temp_errors = *
  return_output = false
  freeze_exec_fail = true
  log_defer_output = true
  log_output = true
  headers_remove = List-Unsubscribe:List-Owner:List-Post:List-Subscribe:List-Help:List-Id:List-Archive
  #log_fail_output = true
  umask = 022
  environment =
  user = Debian-exim
  message_prefix =

where that command (MAILINGLISTPIPESCRIPT) is:

/usr/local/pglister/bin/inject.py -s $sender_address -d $local_part$local_part_suffix@$domain -m $header_message-id:

There's then some lookups we do from exim to check if we should be
sending things to pglister:

domainlist local_domains = @ : ${lookup pgsql{select domain from domains where domain='${quote_pgsql:$domain}'}
{$value}}
local_parts = mailer-daemon : ${lookup pgsql{select name from mailinglists WHERE name='${quote_pgsql:$local_part}' and
domain='${quote_pgsql:$domain}'}{$value}} 

though that's not strictly required, just nicer to bounce things earlier
than having them end up in the pglister inbound queue where they'll sit
until an admin deals with it.

Generally, email comes in via inject.py, which will decide if it's a
bounce or an incoming email and INSERT the email into the appropriate
table and call NOTIFY to wake up either the bouncer processor or the
mail processor.  Those will do some work and eventually may create
emails to go out and wake up the mailsender to process the queue of
mails to send.

And then there's a django interface to managing the lists, handling
moderation, etc.

Hopefully this helps give a bit more insight into how it works.

Thanks,

Stephen