It turns out we do pay postage for our email

Well, not everybody, but some of us we do. Let me explain myself:

12 years ago after reading about TipJoy, a Y Combinator startup, I thought that this might be a scheme that could be used to force mass senders to pay something in order to ensure (via the investment cost to post to my inbox). I just thought of it kind the wrong way, and not in terms of snail-mail. I thought the recipient should be paid to read the email. But anyway TipJoy folded and life did its own thing and this was an idea left to collect dust.

It turns out that a whole industry focused on email delivery sprang in the meantime. IP reputation became a thing and the small server you operated from your house was part of a dialup or DSL pool that was used by spambots. No matter your intentions and your rigor in setting up your email server, your sender’s reputation was next to nothing. The same held true for your ISPs outgoing mail server too. The same is (still) true if you try to setup a VM to a cloud provider (if you’re allowed to send outgoing email at all). Once you needed to know about SMTP and POP3 only (IMAP too if you wanted to be fancier). Now you needed to learn new stuff too, SPF, DKIM, greylisting, RBL, DMARC, the list goes on. That’s how the sending industry was formed, providing guaranteed delivery, metrics like open-rate and more complex analytics.

Here I am now, some years later, and I am running a small mailing list for the Greek Database research community (I am not a database researcher, just a friend of the community). This mailing list always had reachability problems, even when I was running it on infrastructure I totally controlled when I was working on an ISP. Spam bots and inexperienced users always resulted in some sort of blacklisting that a single person postmaster team struggled to handle. There were delivery problems with a lot of personal time devoted to unblock them.

Since 2014 that I quit the postmaster business, I am running the list using Mailgun (It could have been any other M3AAWG member, I just picked them because of Bob Metcalfe mentioning them on twitter sometime). They used to provide a free service and the list was well within those limits, but they changed that. So there’s a monthly cost that varies from $1 to $10 depending the traffic. Delivery has been stellar ever since I switched to Mailgun and the few issues the mailing list had, were glitches from my side.

So it turns out that you do pay some postage to send your email after-all and the M3AAWG members are the couriers.

email will never die

These thoughts are triggered by the hey release, but they are otherwise unrelated.

Every now and then, an application comes along, grabs substantial userbase and claims to have solved the “issue” with email, texting and whatever memorabilia you want to share with your friends. Provided you’re part of their locked garden. Various pricing models are applied here. Math your heart out.

However, here is the thing: each one of them, requires your lock-in to that application. That’s how I ended up with 16 texting (or similar) applications on my phone. Because my contacts, personal and customers demand their favorite platform for my attention.

And that is why email will never die. It works most of the time and it does not matter who your provider is: Gmail, Yahoo, Hotmail, Protonmail, Fastmail, a local Exchange server, some postfix running on an RPi. It does not matter. Granted, when your message does not go through, there may be insurmountable hoops to jump, but in the end, email is still the ubiquitous text application that works for everyone, everywhere between big walled gardens and even your own backyard if you care to run it on your own. And that is why, even though email is not instant messaging, people (OK not my kids) still treat it like it being such.

I am still undecided whether to invest in Hey. The application looks good, the web client also, but the cost of switching from my current walled garden (that also offers some identity services) is big.

How I run a small (discussion) mailing list

I used to run a fairly large (for two Ops people) mail system until 2014. I did it all, sendmail, MIMEDefang, my own milters, SPF, and a bunch of other accompanying acronyms and technologies. I’ve stopped doing that now. The big companies have both the money and the workforce to do it better. But I still run a small mailing list. It used to run on majordomo but then it started having issues with modern Perl and I moved it to Mailman. Which is kind of an overkill for just a list with 300 or so people on. So for a time I even run it manually using free GSuite (yes there was a time it was free) hosting.

Lately I wanted to really run it automated again. So I thought I should try something different that would also contribute to the general civility and high SNR of it. I’d been lurking around the picolisp mailing list for years and thought I should use it. Because it comes with a mailing list program:

#!bin/picolisp lib.l
# 19apr17abu
# (c) Software Lab. Alexander Burger

# Configuration
(setq
   *MailingList "foobar@example.com"
   *SpoolFile "/var/mail/foobar"
   *MailingDomain "server.example.com"
   *Mailings (make (in "/home/foobar/Mailings" (while (line T) (link @))))
   *SmtpHost "localhost"
   *SmtpPort 25 )

# Process mails
(loop
   (when (gt0 (car (info *SpoolFile)))
      (protect
         (in *SpoolFile
            (unless (= "From" (till " " T))
               (quit "Bad mbox file") )
            (char)
            (while (setq *From (lowc (till " " T)))
               (line)  # Skip rest of line and "\r\n"
               (off
                  *Name *Subject *Date *MessageID *InReplyTo *MimeVersion
                  *ContentType *ContentTransferEncoding *ContentDisposition *UserAgent )
               (while (trim (split (line) " "))
                  (let L @
                     (while (and (sub? (peek) " \t") (char))  # Skip WSP
                        (conc L (trim (split (line) " "))) )
                     (setq *Line (glue " " (cdr L)))
                     (case (pack (car L))
                        ("From:" (setq *Name *Line))
                        ("Subject:" (setq *Subject *Line))
                        ("Date:" (setq *Date *Line))
                        ("Message-ID:" (setq *MessageID *Line))
                        ("In-Reply-To:" (setq *InReplyTo *Line))
                        ("MIME-Version:" (setq *MimeVersion *Line))
                        ("Content-Type:" (setq *ContentType *Line))
                        ("Content-Transfer-Encoding:" (setq *ContentTransferEncoding *Line))
                        ("Content-Disposition:" (setq *ContentDisposition *Line))
                        ("User-Agent:" (setq *UserAgent *Line)) ) ) )
               (if (nor (member *From *Mailings) (= "subscribe" (lowc *Subject)))
                  (out "/dev/null" (echo "^JFrom ") (msg *From " discarded"))
                  (unless (setq *Sock (connect *SmtpHost *SmtpPort))
                     (quit "Can't connect to SMTP server") )
                  (unless
                     (and
                        (pre? "220 " (in *Sock (line T)))
                        (out *Sock (prinl "HELO " *MailingDomain "^M"))
                        (pre? "250 " (in *Sock (line T)))
                        (out *Sock (prinl "MAIL FROM:" *MailingList "^M"))
                        (pre? "250 " (in *Sock (line T))) )
                     (quit "Can't HELO") )
                  (when (= "subscribe" (lowc *Subject))
                     (push1 '*Mailings *From)
                     (out "Mailings" (mapc prinl *Mailings)) )
                  (for To *Mailings
                     (out *Sock (prinl "RCPT TO:" To "^M"))
                     (unless (pre? "250 " (in *Sock (line T)))
                        (msg T " can't mail") ) )
                  (when (and (out *Sock (prinl "DATA^M")) (pre? "354 " (in *Sock (line T))))
                     (out *Sock
                        (prinl "From: " (or *Name *From) "^M")
                        (prinl "Sender: " *MailingList "^M")
                        (prinl "Reply-To: " *MailingList "^M")
                        (prinl "To: " *MailingList "^M")
                        (prinl "Subject: " *Subject "^M")
                        (and *Date (prinl "Date: " @ "^M"))
                        (and *MessageID (prinl "Message-ID: " @ "^M"))
                        (and *InReplyTo (prinl "In-Reply-To: " @ "^M"))
                        (and *MimeVersion (prinl "MIME-Version: " @ "^M"))
                        (and *ContentType (prinl "Content-Type: " @ "^M"))
                        (and *ContentTransferEncoding (prinl "Content-Transfer-Encoding: " @ "^M"))
                        (and *ContentDisposition (prinl "Content-Disposition: " @ "^M"))
                        (and *UserAgent (prinl "User-Agent: " @ "^M"))
                        (prinl "^M")
                        (cond
                           ((= "subscribe" (lowc *Subject))
                              (prinl "Hello " (or *Name *From) " :-)^M")
                              (prinl "You are now subscribed^M")
                              (prinl "****^M^J^M") )
                           ((= "unsubscribe" (lowc *Subject))
                              (out "Mailings"
                                 (mapc prinl (del *From '*Mailings)) )
                              (prinl "Good bye " (or *Name *From) " :-(^M")
                              (prinl "You are now unsubscribed^M")
                              (prinl "****^M^J^M") ) )
                        (echo "^JFrom ")
                        (prinl "^J-- ^M")
                        (prinl "UNSUBSCRIBE: mailto:" *MailingList "?subject=Unsubscribe^M")
                        (prinl ".^M")
                        (prinl "QUIT^M") ) )
                  (close *Sock) ) ) )
         (out *SpoolFile (rewind)) ) )
   (call "fetchmail" "-as")
   (wait `(* 4 60 1000)) )

# vi:et:ts=3:sw=3

You do not have to understand a lot of Lisp to know how this is configured.
– Line 7 is where you put your mailing list’s address
– Line 8 is where incoming mail for the mailing list is saved. If you have a mail server and you run this there, just decide to run the list as a plain user and point to the user’s mailbox under /var/mail. Or, if you run fetchmail point to the file fetchmail saves incoming mail. See line 96 for that.
– Related to the above: Since I run this on my mail server, I delete line 96.
– Line 9 is my machine’s HELO/EHLO name.
– Line 10 is where the list membership is saved.
– Line 11 is the outgoing mail server.
– Line 12 is the outgoing’s mail server SMTP listening port.

Since my own incoming mail server is on a cloud provider and does not enjoy good sending reputation, I am making use of mailgun as a forwarding service. My mail server forwards to mailgun and mailgun delivers to the recipients. If you need to run your own small mail server there are tons of tutorials out there. While I am a die-hard sendmail person, I am running Postfix these days.

The final issue that remains is how to run the mailing list processor. mailing is a console program and you need to run it as a daemon somehow. Docker to the rescuse. I am building an image with the following Dockerfile:

FROM debian:buster
RUN apt-get update && apt-get install -y picolisp
WORKDIR /usr/src
COPY ./mailing .
CMD ./mailing

and I am executing this with:

docker run -d --name=foobar --restart=unless-stopped -v /var/mail/foobar:/var/mail/foobar -v /home/foobar/Mailings:/home/foobar/Mailings foobar_image

I can even inspect the logs with docker logs foobar and have made it resilient through reboots with one command.

There. I hope this gives you some ideas on how to run your own (small) mailing list. With some tinkering, you do not even need to run your own incoming mail server. It can be a mailbox in Gmail or elsewhere, you fetch mails with fetchmail locally, and submit to a relay service and done. Also note: Relay services require payment after some threshold.

addressing

I’ve started reading John Day’s Patterns in Network Architecture and during the first pages it makes strong references to Saltzer’s 1982 paper. Why would I bring this up? Well, I just heard Surprisingly Awesome‘s episode on Postal codes where they deal with two countries (Lebanon and Mongolia) with almost non-existent addressing plans. Here is what an addressing plan should give you:

  • a name identifies what you want,
  • an address identifies where it is, and
  • a route identifies a way to get there

Day makes the case that we usually use that address of a network element in the same way that we use its name also which is an error, since by moving an element elsewhere in the network, we need to change its name also.  You on the other hand do not change your name when you change your home address. You used to change your phone number, but even that has become equally portable.

In places where no stable addressing system exists the courier is required to build a mental representation of the routes in their area of delivery, based on landmarks, trees, neon signs, whatever can help to make the delivery. In Mongolia this is solved differently: When something arrives at the post office, they call you back and you go and pick it up.

Enter the NAC. What is it exactly? It is an effort to map longitude and latitude to a more memorable representation using the base 30 number system using digits and capital letters. Borrowing from Wikipedia, the NAC for the centre of the city of Brussels is HBV6R RG77T. Compact, accurate, but not quite memorable.

what3words seems to be a service set to solve this since with their solution a unique combination of just 3 words identifies a 3m x 3m square anywhere on the planet. For example, roughly the same place as above is described as october.donor.outlined. I admit, this is much easier to type in a GPS (or tell Siri).

However, I am still surprised that nobody ever thought of using IPv6 for this (maybe somebody has? Please tell me). Given the abundance that the 128bits give us, we could have indexed every square meter on the surface of the planet and make it addressable. Oh, the directories we could have built on top of that. But I have no fear. It is quite probable that much of the inhabited First World’s surface will be pingable in the foreseeable future. The IoT will make sure of that.

 

So now we get spammed via Github :)

So the following popped up in my mailbox:

It is nice and joyful to see your profile on https://github.com and i thought is beautiful to make you a friend,looking for a good friendship, mail me to my box, because i am not often here on xxx@yyy.zzz for more introduction and i will send you my picture.

I guess it was bound to happen. Anywhere you allow messaging to occur, spam will follow. I only wonder why it took so long to happen via a popular service like this.

A sure way to get into a spam blacklist

Well here is one: Trying to deceive me. No I do not believe the error message returned from your remove.php:

There was an error processing your request; please manually send an email to with Unsubscribe as its subject

And you know why is that I do not believe you? Because I never subscribed via this email address.  In your newsletter you say you want to disrupt the industry (which industry you leave it vague). So far you’re only annoying.

milter-greylist

After years of using graymilter (with a series of local hacks) I switched to milter-greylist.

After it run for a few days:

# Summary: 149173 records, 137182 greylisted, 11991 whitelisted, 0 tarpitted

and with only a few tweaks in its configuration:

racl whitelist domain google.com
racl whitelist domain googlemail.com
racl whitelist domain gmail.com
racl whitelist domain yahoo.com
racl whitelist domain hotmail.com
racl whitelist domain live.com
racl whitelist domain outlook.com
racl whitelist domain amazon.com
racl whitelist domain ebay.com
#racl whitelist domain gr

racl greylist default

You can apt-get install milter-greylist (which makes maintenance through OS upgrades manageble) and it has all the features that I would love to add in my series of hacks to graymilter but never got around to doing so.

(previous)

So long and thanks for the fish

Mark Reed Crispin, inventor of IMAP, passed away on Friday, December 28, 2012 at Martha and Mary Healthcare Services in Poulsbo Washington. He was born on July 19, 1956 in Camden New Jersey and was 56 years of age.

Very few people have (almost single handledly) designed protocols and implemented (free) software that has facilitated communication among millions of people. Mark was one of them and thanks to his monumental achievement of IMAP (and the fact that the Net cannot “forget” his posts on comp.mail.imap) he will always be remembered and always there to teach those who want to hear him.

As someone who has had the privilege to exchange personal emails with him, I am deeply saddened by his departure. I think he is the first of my email heroes that leaves.