How I run a small (discussion) mailing list

I used to run a fairly large (for two Ops people) mail system until 2014. I did it all, sendmail, MIMEDefang, my own milters, SPF, and a bunch of other accompanying acronyms and technologies. I’ve stopped doing that now. The big companies have both the money and the workforce to do it better. But I still run a small mailing list. It used to run on majordomo but then it started having issues with modern Perl and I moved it to Mailman. Which is kind of an overkill for just a list with 300 or so people on. So for a time I even run it manually using free GSuite (yes there was a time it was free) hosting.

Lately I wanted to really run it automated again. So I thought I should try something different that would also contribute to the general civility and high SNR of it. I’d been lurking around the picolisp mailing list for years and thought I should use it. Because it comes with a mailing list program:

#!bin/picolisp lib.l
# 19apr17abu
# (c) Software Lab. Alexander Burger

# Configuration
(setq
   *MailingList "foobar@example.com"
   *SpoolFile "/var/mail/foobar"
   *MailingDomain "server.example.com"
   *Mailings (make (in "/home/foobar/Mailings" (while (line T) (link @))))
   *SmtpHost "localhost"
   *SmtpPort 25 )

# Process mails
(loop
   (when (gt0 (car (info *SpoolFile)))
      (protect
         (in *SpoolFile
            (unless (= "From" (till " " T))
               (quit "Bad mbox file") )
            (char)
            (while (setq *From (lowc (till " " T)))
               (line)  # Skip rest of line and "\r\n"
               (off
                  *Name *Subject *Date *MessageID *InReplyTo *MimeVersion
                  *ContentType *ContentTransferEncoding *ContentDisposition *UserAgent )
               (while (trim (split (line) " "))
                  (let L @
                     (while (and (sub? (peek) " \t") (char))  # Skip WSP
                        (conc L (trim (split (line) " "))) )
                     (setq *Line (glue " " (cdr L)))
                     (case (pack (car L))
                        ("From:" (setq *Name *Line))
                        ("Subject:" (setq *Subject *Line))
                        ("Date:" (setq *Date *Line))
                        ("Message-ID:" (setq *MessageID *Line))
                        ("In-Reply-To:" (setq *InReplyTo *Line))
                        ("MIME-Version:" (setq *MimeVersion *Line))
                        ("Content-Type:" (setq *ContentType *Line))
                        ("Content-Transfer-Encoding:" (setq *ContentTransferEncoding *Line))
                        ("Content-Disposition:" (setq *ContentDisposition *Line))
                        ("User-Agent:" (setq *UserAgent *Line)) ) ) )
               (if (nor (member *From *Mailings) (= "subscribe" (lowc *Subject)))
                  (out "/dev/null" (echo "^JFrom ") (msg *From " discarded"))
                  (unless (setq *Sock (connect *SmtpHost *SmtpPort))
                     (quit "Can't connect to SMTP server") )
                  (unless
                     (and
                        (pre? "220 " (in *Sock (line T)))
                        (out *Sock (prinl "HELO " *MailingDomain "^M"))
                        (pre? "250 " (in *Sock (line T)))
                        (out *Sock (prinl "MAIL FROM:" *MailingList "^M"))
                        (pre? "250 " (in *Sock (line T))) )
                     (quit "Can't HELO") )
                  (when (= "subscribe" (lowc *Subject))
                     (push1 '*Mailings *From)
                     (out "Mailings" (mapc prinl *Mailings)) )
                  (for To *Mailings
                     (out *Sock (prinl "RCPT TO:" To "^M"))
                     (unless (pre? "250 " (in *Sock (line T)))
                        (msg T " can't mail") ) )
                  (when (and (out *Sock (prinl "DATA^M")) (pre? "354 " (in *Sock (line T))))
                     (out *Sock
                        (prinl "From: " (or *Name *From) "^M")
                        (prinl "Sender: " *MailingList "^M")
                        (prinl "Reply-To: " *MailingList "^M")
                        (prinl "To: " *MailingList "^M")
                        (prinl "Subject: " *Subject "^M")
                        (and *Date (prinl "Date: " @ "^M"))
                        (and *MessageID (prinl "Message-ID: " @ "^M"))
                        (and *InReplyTo (prinl "In-Reply-To: " @ "^M"))
                        (and *MimeVersion (prinl "MIME-Version: " @ "^M"))
                        (and *ContentType (prinl "Content-Type: " @ "^M"))
                        (and *ContentTransferEncoding (prinl "Content-Transfer-Encoding: " @ "^M"))
                        (and *ContentDisposition (prinl "Content-Disposition: " @ "^M"))
                        (and *UserAgent (prinl "User-Agent: " @ "^M"))
                        (prinl "^M")
                        (cond
                           ((= "subscribe" (lowc *Subject))
                              (prinl "Hello " (or *Name *From) " :-)^M")
                              (prinl "You are now subscribed^M")
                              (prinl "****^M^J^M") )
                           ((= "unsubscribe" (lowc *Subject))
                              (out "Mailings"
                                 (mapc prinl (del *From '*Mailings)) )
                              (prinl "Good bye " (or *Name *From) " :-(^M")
                              (prinl "You are now unsubscribed^M")
                              (prinl "****^M^J^M") ) )
                        (echo "^JFrom ")
                        (prinl "^J-- ^M")
                        (prinl "UNSUBSCRIBE: mailto:" *MailingList "?subject=Unsubscribe^M")
                        (prinl ".^M")
                        (prinl "QUIT^M") ) )
                  (close *Sock) ) ) )
         (out *SpoolFile (rewind)) ) )
   (call "fetchmail" "-as")
   (wait `(* 4 60 1000)) )

# vi:et:ts=3:sw=3

You do not have to understand a lot of Lisp to know how this is configured.
– Line 7 is where you put your mailing list’s address
– Line 8 is where incoming mail for the mailing list is saved. If you have a mail server and you run this there, just decide to run the list as a plain user and point to the user’s mailbox under /var/mail. Or, if you run fetchmail point to the file fetchmail saves incoming mail. See line 96 for that.
– Related to the above: Since I run this on my mail server, I delete line 96.
– Line 9 is my machine’s HELO/EHLO name.
– Line 10 is where the list membership is saved.
– Line 11 is the outgoing mail server.
– Line 12 is the outgoing’s mail server SMTP listening port.

Since my own incoming mail server is on a cloud provider and does not enjoy good sending reputation, I am making use of mailgun as a forwarding service. My mail server forwards to mailgun and mailgun delivers to the recipients. If you need to run your own small mail server there are tons of tutorials out there. While I am a die-hard sendmail person, I am running Postfix these days.

The final issue that remains is how to run the mailing list processor. mailing is a console program and you need to run it as a daemon somehow. Docker to the rescuse. I am building an image with the following Dockerfile:

FROM debian:buster
RUN apt-get update && apt-get install -y picolisp
WORKDIR /usr/src
COPY ./mailing .
CMD ./mailing

and I am executing this with:

docker run -d --name=foobar --restart=unless-stopped -v /var/mail/foobar:/var/mail/foobar -v /home/foobar/Mailings:/home/foobar/Mailings foobar_image

I can even inspect the logs with docker logs foobar and have made it resilient through reboots with one command.

There. I hope this gives you some ideas on how to run your own (small) mailing list. With some tinkering, you do not even need to run your own incoming mail server. It can be a mailbox in Gmail or elsewhere, you fetch mails with fetchmail locally, and submit to a relay service and done. Also note: Relay services require payment after some threshold.

addressing

I’ve started reading John Day’s Patterns in Network Architecture and during the first pages it makes strong references to Saltzer’s 1982 paper. Why would I bring this up? Well, I just heard Surprisingly Awesome‘s episode on Postal codes where they deal with two countries (Lebanon and Mongolia) with almost non-existent addressing plans. Here is what an addressing plan should give you:

  • a name identifies what you want,
  • an address identifies where it is, and
  • a route identifies a way to get there

Day makes the case that we usually use that address of a network element in the same way that we use its name also which is an error, since by moving an element elsewhere in the network, we need to change its name also.  You on the other hand do not change your name when you change your home address. You used to change your phone number, but even that has become equally portable.

In places where no stable addressing system exists the courier is required to build a mental representation of the routes in their area of delivery, based on landmarks, trees, neon signs, whatever can help to make the delivery. In Mongolia this is solved differently: When something arrives at the post office, they call you back and you go and pick it up.

Enter the NAC. What is it exactly? It is an effort to map longitude and latitude to a more memorable representation using the base 30 number system using digits and capital letters. Borrowing from Wikipedia, the NAC for the centre of the city of Brussels is HBV6R RG77T. Compact, accurate, but not quite memorable.

what3words seems to be a service set to solve this since with their solution a unique combination of just 3 words identifies a 3m x 3m square anywhere on the planet. For example, roughly the same place as above is described as october.donor.outlined. I admit, this is much easier to type in a GPS (or tell Siri).

However, I am still surprised that nobody ever thought of using IPv6 for this (maybe somebody has? Please tell me). Given the abundance that the 128bits give us, we could have indexed every square meter on the surface of the planet and make it addressable. Oh, the directories we could have built on top of that. But I have no fear. It is quite probable that much of the inhabited First World’s surface will be pingable in the foreseeable future. The IoT will make sure of that.

 

So now we get spammed via Github :)

So the following popped up in my mailbox:

It is nice and joyful to see your profile on https://github.com and i thought is beautiful to make you a friend,looking for a good friendship, mail me to my box, because i am not often here on xxx@yyy.zzz for more introduction and i will send you my picture.

I guess it was bound to happen. Anywhere you allow messaging to occur, spam will follow. I only wonder why it took so long to happen via a popular service like this.

A sure way to get into a spam blacklist

Well here is one: Trying to deceive me. No I do not believe the error message returned from your remove.php:

There was an error processing your request; please manually send an email to with Unsubscribe as its subject

And you know why is that I do not believe you? Because I never subscribed via this email address.  In your newsletter you say you want to disrupt the industry (which industry you leave it vague). So far you’re only annoying.

milter-greylist

After years of using graymilter (with a series of local hacks) I switched to milter-greylist.

After it run for a few days:

# Summary: 149173 records, 137182 greylisted, 11991 whitelisted, 0 tarpitted

and with only a few tweaks in its configuration:

racl whitelist domain google.com
racl whitelist domain googlemail.com
racl whitelist domain gmail.com
racl whitelist domain yahoo.com
racl whitelist domain hotmail.com
racl whitelist domain live.com
racl whitelist domain outlook.com
racl whitelist domain amazon.com
racl whitelist domain ebay.com
#racl whitelist domain gr

racl greylist default

You can apt-get install milter-greylist (which makes maintenance through OS upgrades manageble) and it has all the features that I would love to add in my series of hacks to graymilter but never got around to doing so.

(previous)