email will never die

These thoughts are triggered by the hey release, but they are otherwise unrelated.

Every now and then, an application comes along, grabs substantial userbase and claims to have solved the “issue” with email, texting and whatever memorabilia you want to share with your friends. Provided you’re part of their locked garden. Various pricing models are applied here. Math your heart out.

However, here is the thing: each one of them, requires your lock-in to that application. That’s how I ended up with 16 texting (or similar) applications on my phone. Because my contacts, personal and customers demand their favorite platform for my attention.

And that is why email will never die. It works most of the time and it does not matter who your provider is: Gmail, Yahoo, Hotmail, Protonmail, Fastmail, a local Exchange server, some postfix running on an RPi. It does not matter. Granted, when your message does not go through, there may be insurmountable hoops to jump, but in the end, email is still the ubiquitous text application that works for everyone, everywhere between big walled gardens and even your own backyard if you care to run it on your own. And that is why, even though email is not instant messaging, people (OK not my kids) still treat it like it being such.

I am still undecided whether to invest in Hey. The application looks good, the web client also, but the cost of switching from my current walled garden (that also offers some identity services) is big.

How I run a small (discussion) mailing list

I used to run a fairly large (for two Ops people) mail system until 2014. I did it all, sendmail, MIMEDefang, my own milters, SPF, and a bunch of other accompanying acronyms and technologies. I’ve stopped doing that now. The big companies have both the money and the workforce to do it better. But I still run a small mailing list. It used to run on majordomo but then it started having issues with modern Perl and I moved it to Mailman. Which is kind of an overkill for just a list with 300 or so people on. So for a time I even run it manually using free GSuite (yes there was a time it was free) hosting.

Lately I wanted to really run it automated again. So I thought I should try something different that would also contribute to the general civility and high SNR of it. I’d been lurking around the picolisp mailing list for years and thought I should use it. Because it comes with a mailing list program:

#!bin/picolisp lib.l
# 19apr17abu
# (c) Software Lab. Alexander Burger

# Configuration
(setq
   *MailingList "foobar@example.com"
   *SpoolFile "/var/mail/foobar"
   *MailingDomain "server.example.com"
   *Mailings (make (in "/home/foobar/Mailings" (while (line T) (link @))))
   *SmtpHost "localhost"
   *SmtpPort 25 )

# Process mails
(loop
   (when (gt0 (car (info *SpoolFile)))
      (protect
         (in *SpoolFile
            (unless (= "From" (till " " T))
               (quit "Bad mbox file") )
            (char)
            (while (setq *From (lowc (till " " T)))
               (line)  # Skip rest of line and "\r\n"
               (off
                  *Name *Subject *Date *MessageID *InReplyTo *MimeVersion
                  *ContentType *ContentTransferEncoding *ContentDisposition *UserAgent )
               (while (trim (split (line) " "))
                  (let L @
                     (while (and (sub? (peek) " \t") (char))  # Skip WSP
                        (conc L (trim (split (line) " "))) )
                     (setq *Line (glue " " (cdr L)))
                     (case (pack (car L))
                        ("From:" (setq *Name *Line))
                        ("Subject:" (setq *Subject *Line))
                        ("Date:" (setq *Date *Line))
                        ("Message-ID:" (setq *MessageID *Line))
                        ("In-Reply-To:" (setq *InReplyTo *Line))
                        ("MIME-Version:" (setq *MimeVersion *Line))
                        ("Content-Type:" (setq *ContentType *Line))
                        ("Content-Transfer-Encoding:" (setq *ContentTransferEncoding *Line))
                        ("Content-Disposition:" (setq *ContentDisposition *Line))
                        ("User-Agent:" (setq *UserAgent *Line)) ) ) )
               (if (nor (member *From *Mailings) (= "subscribe" (lowc *Subject)))
                  (out "/dev/null" (echo "^JFrom ") (msg *From " discarded"))
                  (unless (setq *Sock (connect *SmtpHost *SmtpPort))
                     (quit "Can't connect to SMTP server") )
                  (unless
                     (and
                        (pre? "220 " (in *Sock (line T)))
                        (out *Sock (prinl "HELO " *MailingDomain "^M"))
                        (pre? "250 " (in *Sock (line T)))
                        (out *Sock (prinl "MAIL FROM:" *MailingList "^M"))
                        (pre? "250 " (in *Sock (line T))) )
                     (quit "Can't HELO") )
                  (when (= "subscribe" (lowc *Subject))
                     (push1 '*Mailings *From)
                     (out "Mailings" (mapc prinl *Mailings)) )
                  (for To *Mailings
                     (out *Sock (prinl "RCPT TO:" To "^M"))
                     (unless (pre? "250 " (in *Sock (line T)))
                        (msg T " can't mail") ) )
                  (when (and (out *Sock (prinl "DATA^M")) (pre? "354 " (in *Sock (line T))))
                     (out *Sock
                        (prinl "From: " (or *Name *From) "^M")
                        (prinl "Sender: " *MailingList "^M")
                        (prinl "Reply-To: " *MailingList "^M")
                        (prinl "To: " *MailingList "^M")
                        (prinl "Subject: " *Subject "^M")
                        (and *Date (prinl "Date: " @ "^M"))
                        (and *MessageID (prinl "Message-ID: " @ "^M"))
                        (and *InReplyTo (prinl "In-Reply-To: " @ "^M"))
                        (and *MimeVersion (prinl "MIME-Version: " @ "^M"))
                        (and *ContentType (prinl "Content-Type: " @ "^M"))
                        (and *ContentTransferEncoding (prinl "Content-Transfer-Encoding: " @ "^M"))
                        (and *ContentDisposition (prinl "Content-Disposition: " @ "^M"))
                        (and *UserAgent (prinl "User-Agent: " @ "^M"))
                        (prinl "^M")
                        (cond
                           ((= "subscribe" (lowc *Subject))
                              (prinl "Hello " (or *Name *From) " :-)^M")
                              (prinl "You are now subscribed^M")
                              (prinl "****^M^J^M") )
                           ((= "unsubscribe" (lowc *Subject))
                              (out "Mailings"
                                 (mapc prinl (del *From '*Mailings)) )
                              (prinl "Good bye " (or *Name *From) " :-(^M")
                              (prinl "You are now unsubscribed^M")
                              (prinl "****^M^J^M") ) )
                        (echo "^JFrom ")
                        (prinl "^J-- ^M")
                        (prinl "UNSUBSCRIBE: mailto:" *MailingList "?subject=Unsubscribe^M")
                        (prinl ".^M")
                        (prinl "QUIT^M") ) )
                  (close *Sock) ) ) )
         (out *SpoolFile (rewind)) ) )
   (call "fetchmail" "-as")
   (wait `(* 4 60 1000)) )

# vi:et:ts=3:sw=3

You do not have to understand a lot of Lisp to know how this is configured.
– Line 7 is where you put your mailing list’s address
– Line 8 is where incoming mail for the mailing list is saved. If you have a mail server and you run this there, just decide to run the list as a plain user and point to the user’s mailbox under /var/mail. Or, if you run fetchmail point to the file fetchmail saves incoming mail. See line 96 for that.
– Related to the above: Since I run this on my mail server, I delete line 96.
– Line 9 is my machine’s HELO/EHLO name.
– Line 10 is where the list membership is saved.
– Line 11 is the outgoing mail server.
– Line 12 is the outgoing’s mail server SMTP listening port.

Since my own incoming mail server is on a cloud provider and does not enjoy good sending reputation, I am making use of mailgun as a forwarding service. My mail server forwards to mailgun and mailgun delivers to the recipients. If you need to run your own small mail server there are tons of tutorials out there. While I am a die-hard sendmail person, I am running Postfix these days.

The final issue that remains is how to run the mailing list processor. mailing is a console program and you need to run it as a daemon somehow. Docker to the rescuse. I am building an image with the following Dockerfile:

FROM debian:buster
RUN apt-get update && apt-get install -y picolisp
WORKDIR /usr/src
COPY ./mailing .
CMD ./mailing

and I am executing this with:

docker run -d --name=foobar --restart=unless-stopped -v /var/mail/foobar:/var/mail/foobar -v /home/foobar/Mailings:/home/foobar/Mailings foobar_image

I can even inspect the logs with docker logs foobar and have made it resilient through reboots with one command.

There. I hope this gives you some ideas on how to run your own (small) mailing list. With some tinkering, you do not even need to run your own incoming mail server. It can be a mailbox in Gmail or elsewhere, you fetch mails with fetchmail locally, and submit to a relay service and done. Also note: Relay services require payment after some threshold.

addressing

I’ve started reading John Day’s Patterns in Network Architecture and during the first pages it makes strong references to Saltzer’s 1982 paper. Why would I bring this up? Well, I just heard Surprisingly Awesome‘s episode on Postal codes where they deal with two countries (Lebanon and Mongolia) with almost non-existent addressing plans. Here is what an addressing plan should give you:

  • a name identifies what you want,
  • an address identifies where it is, and
  • a route identifies a way to get there

Day makes the case that we usually use that address of a network element in the same way that we use its name also which is an error, since by moving an element elsewhere in the network, we need to change its name also.  You on the other hand do not change your name when you change your home address. You used to change your phone number, but even that has become equally portable.

In places where no stable addressing system exists the courier is required to build a mental representation of the routes in their area of delivery, based on landmarks, trees, neon signs, whatever can help to make the delivery. In Mongolia this is solved differently: When something arrives at the post office, they call you back and you go and pick it up.

Enter the NAC. What is it exactly? It is an effort to map longitude and latitude to a more memorable representation using the base 30 number system using digits and capital letters. Borrowing from Wikipedia, the NAC for the centre of the city of Brussels is HBV6R RG77T. Compact, accurate, but not quite memorable.

what3words seems to be a service set to solve this since with their solution a unique combination of just 3 words identifies a 3m x 3m square anywhere on the planet. For example, roughly the same place as above is described as october.donor.outlined. I admit, this is much easier to type in a GPS (or tell Siri).

However, I am still surprised that nobody ever thought of using IPv6 for this (maybe somebody has? Please tell me). Given the abundance that the 128bits give us, we could have indexed every square meter on the surface of the planet and make it addressable. Oh, the directories we could have built on top of that. But I have no fear. It is quite probable that much of the inhabited First World’s surface will be pingable in the foreseeable future. The IoT will make sure of that.

 

So now we get spammed via Github :)

So the following popped up in my mailbox:

It is nice and joyful to see your profile on https://github.com and i thought is beautiful to make you a friend,looking for a good friendship, mail me to my box, because i am not often here on xxx@yyy.zzz for more introduction and i will send you my picture.

I guess it was bound to happen. Anywhere you allow messaging to occur, spam will follow. I only wonder why it took so long to happen via a popular service like this.

A sure way to get into a spam blacklist

Well here is one: Trying to deceive me. No I do not believe the error message returned from your remove.php:

There was an error processing your request; please manually send an email to with Unsubscribe as its subject

And you know why is that I do not believe you? Because I never subscribed via this email address.  In your newsletter you say you want to disrupt the industry (which industry you leave it vague). So far you’re only annoying.

milter-greylist

After years of using graymilter (with a series of local hacks) I switched to milter-greylist.

After it run for a few days:

# Summary: 149173 records, 137182 greylisted, 11991 whitelisted, 0 tarpitted

and with only a few tweaks in its configuration:

racl whitelist domain google.com
racl whitelist domain googlemail.com
racl whitelist domain gmail.com
racl whitelist domain yahoo.com
racl whitelist domain hotmail.com
racl whitelist domain live.com
racl whitelist domain outlook.com
racl whitelist domain amazon.com
racl whitelist domain ebay.com
#racl whitelist domain gr

racl greylist default

You can apt-get install milter-greylist (which makes maintenance through OS upgrades manageble) and it has all the features that I would love to add in my series of hacks to graymilter but never got around to doing so.

(previous)

So long and thanks for the fish

Mark Reed Crispin, inventor of IMAP, passed away on Friday, December 28, 2012 at Martha and Mary Healthcare Services in Poulsbo Washington. He was born on July 19, 1956 in Camden New Jersey and was 56 years of age.

Very few people have (almost single handledly) designed protocols and implemented (free) software that has facilitated communication among millions of people. Mark was one of them and thanks to his monumental achievement of IMAP (and the fact that the Net cannot “forget” his posts on comp.mail.imap) he will always be remembered and always there to teach those who want to hear him.

As someone who has had the privilege to exchange personal emails with him, I am deeply saddened by his departure. I think he is the first of my email heroes that leaves.

memcached and MIMEDefang – a cool combination

I like milter-ahead a lot. But in our particular deployment it is not a best fit for it assumes that all the useful information for deciding whether to accept or reject email resides not on the server that it runs on, but in the servers that it queries. This is not milter-ahead’s fault. Milters have no way of expanding aliases while checking the recipient address so the programmer has to use tricks like parsing the output of sendmail -bv user@address thus running a second sendmail process for the same delivery. The alternative would be to hack milter-ahead to check with the alias database the existence of recipient addresses, but doing so the way sendmail reads the alias database is overly complex. One could also write an external daemon to monitor the alias database and inject entries in the (Berkeley DB) database maintained by milter-ahead, but that database is locked exclusively. And yes, exceptions could be entered in the access database, but that would mean maintaining two files for a single (and not so frequent) change in the alias files.

As I’ve blogged before, one of the reasons that I like MIMEDefang is that it gives the Postmaster a full programming language to filter stuff. By simply using md_check_against_smtp_server() a poor man’s non-caching version of milter-ahead is possible. Adding support to read the alias database (be it the text file or the hash table) is also trivial.

But what about the case of busy mail systems? You do not want to hammer your mail servers all the time with queries for which the answer is going to be constant for long periods of time. You need a caching mechanism. At first I thought of implementing such a mechanism the way milter-ahead does: By using a Berkeley DB database and some expiration mechanism, either from within MIMEDefang (retrieve the key and if it should have been expired by now delete it, otherwise proceed as expected) or by an external “garbage collecting” daemon. But such an interface with a clean way to enter keys and values already exists and performs well: memcached. So by using Cache::Memcached within the mimedefang-filter mimicking basic milter-ahead behavior (with caching) was done.

But what about the local aliases in the mail server? After all this was all the fuss that prompted the switch anyway. I wrote a Perl script that opened the alias database using the BerkeleyDB package. Two details need caution here:

  • The first one is ignoring the invalid @:@ entry in the alias database. You do not see it in the alias text file, but you will see it when you run praliases. Sendmail uses this entry in order to know whether the database is up-to-date or not. See the bat book for a longer discussion of this.
  • The second detail is that since the alias database is written by a C program, all strings are NULL terminated. This is not the case with strings that are used as keys and values with Perl and the BerkeleyDB package. However the Perl BerkeleyDB package provides for filters to deal with this case. You need something like:
    $db->filter_fetch_key( sub { s/\0$// } );
    

And then there’s the issue of making such a script a daemon. One can go the traditional way, use a daemonizer on steroids or simply use Proc::Daemon::Init and be done with it.

memcached comes handy to storing key-value pairs in many system administration tasks and I think I’m going to use it a lot more in mail filtering stuff.