PORT is deprecated. Please use SCHEMA_REGISTRY_LISTENERS instead.

I was trying to launch a schema-registry within a kubernetes cluster and every time I wanted to expose the pod’s port through a service, I was greeted by the nice title message:

if [[ -n "${SCHEMA_REGISTRY_PORT-}" ]]
then
  echo "PORT is deprecated. Please use SCHEMA_REGISTRY_LISTENERS instead."
  exit 1
fi

This happened because I had named my service schema-registry also (which was kind of not negotiable at the time) and kubernetes happily sets the SCHEMA_REGISTRY_PORT environment variable to the value of the port you want to expose. But it turns out that this very named variable has special meaning within the container.

Fortunately, I was not the only one bitten by this error, albeit for a different variable name, but I also used the same ugly hack:

$ kubectl -n kafka-tests get deployment schema-registry -o yaml
:
    spec:
      containers:
      - command:
        - bash
        - -c
        - unset SCHEMA_REGISTRY_PORT; /etc/confluent/docker/run
        env:
        - name: SCHEMA_REGISTRY_LISTENERS
          value: http://0.0.0.0:8081/
:
Advertisements

How I run a small (discussion) mailing list

I used to run a fairly large (for two Ops people) mail system until 2014. I did it all, sendmail, MIMEDefang, my own milters, SPF, and a bunch of other accompanying acronyms and technologies. I’ve stopped doing that now. The big companies have both the money and the workforce to do it better. But I still run a small mailing list. It used to run on majordomo but then it started having issues with modern Perl and I moved it to Mailman. Which is kind of an overkill for just a list with 300 or so people on. So for a time I even run it manually using free GSuite (yes there was a time it was free) hosting.

Lately I wanted to really run it automated again. So I thought I should try something different that would also contribute to the general civility and high SNR of it. I’d been lurking around the picolisp mailing list for years and thought I should use it. Because it comes with a mailing list program:

#!bin/picolisp lib.l
# 19apr17abu
# (c) Software Lab. Alexander Burger

# Configuration
(setq
   *MailingList "foobar@example.com"
   *SpoolFile "/var/mail/foobar"
   *MailingDomain "server.example.com"
   *Mailings (make (in "/home/foobar/Mailings" (while (line T) (link @))))
   *SmtpHost "localhost"
   *SmtpPort 25 )

# Process mails
(loop
   (when (gt0 (car (info *SpoolFile)))
      (protect
         (in *SpoolFile
            (unless (= "From" (till " " T))
               (quit "Bad mbox file") )
            (char)
            (while (setq *From (lowc (till " " T)))
               (line)  # Skip rest of line and "\r\n"
               (off
                  *Name *Subject *Date *MessageID *InReplyTo *MimeVersion
                  *ContentType *ContentTransferEncoding *ContentDisposition *UserAgent )
               (while (trim (split (line) " "))
                  (let L @
                     (while (and (sub? (peek) " \t") (char))  # Skip WSP
                        (conc L (trim (split (line) " "))) )
                     (setq *Line (glue " " (cdr L)))
                     (case (pack (car L))
                        ("From:" (setq *Name *Line))
                        ("Subject:" (setq *Subject *Line))
                        ("Date:" (setq *Date *Line))
                        ("Message-ID:" (setq *MessageID *Line))
                        ("In-Reply-To:" (setq *InReplyTo *Line))
                        ("MIME-Version:" (setq *MimeVersion *Line))
                        ("Content-Type:" (setq *ContentType *Line))
                        ("Content-Transfer-Encoding:" (setq *ContentTransferEncoding *Line))
                        ("Content-Disposition:" (setq *ContentDisposition *Line))
                        ("User-Agent:" (setq *UserAgent *Line)) ) ) )
               (if (nor (member *From *Mailings) (= "subscribe" (lowc *Subject)))
                  (out "/dev/null" (echo "^JFrom ") (msg *From " discarded"))
                  (unless (setq *Sock (connect *SmtpHost *SmtpPort))
                     (quit "Can't connect to SMTP server") )
                  (unless
                     (and
                        (pre? "220 " (in *Sock (line T)))
                        (out *Sock (prinl "HELO " *MailingDomain "^M"))
                        (pre? "250 " (in *Sock (line T)))
                        (out *Sock (prinl "MAIL FROM:" *MailingList "^M"))
                        (pre? "250 " (in *Sock (line T))) )
                     (quit "Can't HELO") )
                  (when (= "subscribe" (lowc *Subject))
                     (push1 '*Mailings *From)
                     (out "Mailings" (mapc prinl *Mailings)) )
                  (for To *Mailings
                     (out *Sock (prinl "RCPT TO:" To "^M"))
                     (unless (pre? "250 " (in *Sock (line T)))
                        (msg T " can't mail") ) )
                  (when (and (out *Sock (prinl "DATA^M")) (pre? "354 " (in *Sock (line T))))
                     (out *Sock
                        (prinl "From: " (or *Name *From) "^M")
                        (prinl "Sender: " *MailingList "^M")
                        (prinl "Reply-To: " *MailingList "^M")
                        (prinl "To: " *MailingList "^M")
                        (prinl "Subject: " *Subject "^M")
                        (and *Date (prinl "Date: " @ "^M"))
                        (and *MessageID (prinl "Message-ID: " @ "^M"))
                        (and *InReplyTo (prinl "In-Reply-To: " @ "^M"))
                        (and *MimeVersion (prinl "MIME-Version: " @ "^M"))
                        (and *ContentType (prinl "Content-Type: " @ "^M"))
                        (and *ContentTransferEncoding (prinl "Content-Transfer-Encoding: " @ "^M"))
                        (and *ContentDisposition (prinl "Content-Disposition: " @ "^M"))
                        (and *UserAgent (prinl "User-Agent: " @ "^M"))
                        (prinl "^M")
                        (cond
                           ((= "subscribe" (lowc *Subject))
                              (prinl "Hello " (or *Name *From) " :-)^M")
                              (prinl "You are now subscribed^M")
                              (prinl "****^M^J^M") )
                           ((= "unsubscribe" (lowc *Subject))
                              (out "Mailings"
                                 (mapc prinl (del *From '*Mailings)) )
                              (prinl "Good bye " (or *Name *From) " :-(^M")
                              (prinl "You are now unsubscribed^M")
                              (prinl "****^M^J^M") ) )
                        (echo "^JFrom ")
                        (prinl "^J-- ^M")
                        (prinl "UNSUBSCRIBE: mailto:" *MailingList "?subject=Unsubscribe^M")
                        (prinl ".^M")
                        (prinl "QUIT^M") ) )
                  (close *Sock) ) ) )
         (out *SpoolFile (rewind)) ) )
   (call "fetchmail" "-as")
   (wait `(* 4 60 1000)) )

# vi:et:ts=3:sw=3

You do not have to understand a lot of Lisp to know how this is configured.
– Line 7 is where you put your mailing list’s address
– Line 8 is where incoming mail for the mailing list is saved. If you have a mail server and you run this there, just decide to run the list as a plain user and point to the user’s mailbox under /var/mail. Or, if you run fetchmail point to the file fetchmail saves incoming mail. See line 96 for that.
– Related to the above: Since I run this on my mail server, I delete line 96.
– Line 9 is my machine’s HELO/EHLO name.
– Line 10 is where the list membership is saved.
– Line 11 is the outgoing mail server.
– Line 12 is the outgoing’s mail server SMTP listening port.

Since my own incoming mail server is on a cloud provider and does not enjoy good sending reputation, I am making use of mailgun as a forwarding service. My mail server forwards to mailgun and mailgun delivers to the recipients. If you need to run your own small mail server there are tons of tutorials out there. While I am a die-hard sendmail person, I am running Postfix these days.

The final issue that remains is how to run the mailing list processor. mailing is a console program and you need to run it as a daemon somehow. Docker to the rescuse. I am building an image with the following Dockerfile:

FROM debian:buster
RUN apt-get update && apt-get install -y picolisp
WORKDIR /usr/src
COPY ./mailing .
CMD ./mailing

and I am executing this with:

docker run -d --name=foobar --restart=unless-stopped -v /var/mail/foobar:/var/mail/foobar -v /home/foobar/Mailings:/home/foobar/Mailings foobar_image

I can even inspect the logs with docker logs foobar and have made it resilient through reboots with one command.

There. I hope this gives you some ideas on how to run your own (small) mailing list. With some tinkering, you do not even need to run your own incoming mail server. It can be a mailbox in Gmail or elsewhere, you fetch mails with fetchmail locally, and submit to a relay service and done. Also note: Relay services require payment after some threshold.

ansible, timezone and JDK8

While one might think that they can change the timezone of a machine with ansible with:

  tasks:
  - name: set /etc/localtime
    timezone:
      name: UTC

with some JDK apps it is not enought, because the JVM looks at /etc/timezone. So you need to update that file too. Maybe in a more cleaner way than:

  - name: set /etc/timezone
    shell: echo UTC > /etc/timezone
    args:
      creates: /etc/timezone

Kafka, dotnet and SASL_SSL

This is similar to my previous post, only now the question is, how do you connect to a Kafka server using dotnet and SASL_SSL? This is how:

// based on https://github.com/confluentinc/confluent-kafka-dotnet/blob/v1.0.0/examples/Producer/Program.cs

using Confluent.Kafka;
using System;
using System.IO;
using System.Text;
using System.Threading.Tasks;
using System.Collections.Generic;


namespace Confluent.Kafka.Examples.ProducerExample
{
    public class Program
    {
        public static async Task Main(string[] args)
        {
            string topicName = "test-topic";

            var config = new ProducerConfig {
                BootstrapServers = "kafka-server.example.com:19094",
                SecurityProtocol = SecurityProtocol.SaslSsl,
                SslCaLocation = "ca-cert",
                SaslMechanism = SaslMechanism.Plain,
                SaslUsername = "USERNAME",
                SaslPassword = "PASSWORD",
                Acks = Acks.Leader,
                CompressionType = CompressionType.Lz4,
            };

            using (var producer = new ProducerBuilder<string, string>(config).Build())
            {
                for (int i = 0; i < 1000000; i++)
                {
                    var message = $"Event {i}";

                    try
                    {
                        // Note: Awaiting the asynchronous produce request
                        // below prevents flow of execution from proceeding
                        // until the acknowledgement from the broker is
                        // received (at the expense of low throughput).

                        var deliveryReport = await producer.ProduceAsync(topicName, new Message<string, string> { Key = null, Value = message } );
                        // Console.WriteLine($"delivered to: {deliveryReport.TopicPartitionOffset}");

                        // Let's not await then
                        // producer.ProduceAsync(topicName, new Message<string, string> { Key = null, Value = message } );
                        // Console.WriteLine($"Event {i} sent.");
                    }
                    catch (ProduceException<string, string> e)
                    {
                        Console.WriteLine($"failed to deliver message: {e.Message} [{e.Error.Code}]");
                    }
                }

                // producer.Flush(TimeSpan.FromSeconds(120));

                // Since we are producing synchronously, at this point there will be no messages
                // in-flight and no delivery reports waiting to be acknowledged, so there is no
                // need to call producer.Flush before disposing the producer.
            }
        }
    }
}

Since I am a total .NET newbie, I usually docker run -it --rm microsoft/dotnet and experiment from there.

Kafka, PHP and SASL_SSL

When you want to connect to a Kafka cluster from PHP there are numerous examples showing how to use php-rdkafka, but unauthenticated. But what happens when you need to let a customer connect to a Kafka setup and IP whitelisting is not enough? Not much easily locatable information is out there.

Why not correct this by combing through various web pages and the librdkafka source code:

<?php

$conf = new RdKafka\Conf();
$conf->set('security.protocol', 'SASL_SSL');
$conf->set('sasl.mechanisms', 'PLAIN');
$conf->set('sasl.username', 'USERNAME_HERE');
$conf->set('sasl.password', 'PASSWORD_HERE');
$conf->set('ssl.ca.location', '/usr/local/etc/ca-cert.pem');
$conf->set('ssl.cipher.suites', 'TLSv1.2');

$rk = new RdKafka\Producer($conf);
$rk->addBrokers("SASL_SSL://kafka-1.example.com:19094");
$rk->addBrokers("SASL_SSL://kafka-2.example.com:19094");
$rk->addBrokers("SASL_SSL://kafka-3.example.com:19094");

$topic = $rk->newTopic("kafka-test-topic");

for ($i = 0; $i < 10; $i++) {
    $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Message $i");
    $rk->poll(0);
}

while ($rk->getOutQLen() > 0) {
    $rk->poll(50);
}

?>

Still this may not be enough if it is the case that your Kafka server is on OpenSSL-1.0.2 (CentOS 7 for example) and your php client is on OpenSSL-1.1.0 (like the php:7.2-cli docker image). In such a case you need to alter your client’s openssl.cnf to comment out the following line:

;CipherString = DEFAULT@SECLEVEL=2

Wasting time with gawk while parsing lsof output

So I wanted to parse lsof, to see on what ports was a machine accepting connections. Normally one would write something like:

# lsof -Pn -i | grep LISTEN | awk '{print $9}' | cut -d: -f2 | sort -n | uniq
22
111
6066
7011
7015
7077
8080
10050
35735
37480
39118
44262
44444
52539

You get a sorted list of the open ports and are done with it. But why invoke four different programs to do extraction and sorting, when gawk is a complete programming language? Yes it is possible to do it with gawk in one go (and learn something in the process):

# lsof -Pn -i | awk '/LISTEN/ { split($9, a, ":"); b[a[2]] = 1; } END { n = asorti(b, c, "@ind_num_asc"); for (i = 1; i <= n; i++) { print c[i]; } }'
22
111
6066
7011
7015
7077
8080
10050
35735
37480
39118
44262
44444
52539

The /LISTEN/ effectively greps the lsof output for lines containing LISTEN and executes on them the code in curly braces to its right. Which splits the 9th column into an array using : as a delimiter. In awk arrays are indexed from 1 and the indices are strings (make a note of that).

END is a special match that executes the code in curly braces to its right after we’ve finished reading the input data. So, here is where the printing is done. Using the asorti() function we obtain a new array, indexed based on the values of the indices. We use @ind_num_asc to ensure that the order is 1, 5, 10, 15 and not 1, 10, 15, 5 as it would, should the indices be treated as strings. Finally, we can print the elements from the new array.

This would not be easily possible with awk / nawk, because as the gawk manual says:

In most awk implementations, sorting an array requires writing a sort() function. This can be educational for exploring different sorting algorithms, but usually that’s not the point of the program. gawk provides the built-in asort() and asorti() functions.

Somehow this reminds me of Knuth vs McIlory but of course I am neither.

fizzbuzz

Years ago, while browsing the original wiki site, I stumbled upon the fizzbuzz test:

“Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.”

Imagine my surprise when reading that it was designed to make most programming candidates fail the test. From time to time I have the fine opportunity to introduce people who have never coded before in their lives to Python. Within the first two hours they have managed to produce a fizzbuzz version that looks like this:

for i in range(1, 101):
  if i % 3 == 0 and i % 5 == 0:
    print("fizzbuzz")
  elif i % 3 == 0:
    print("fizz")
  elif i % 5 == 0:
    print("buzz")
  else:
    print(i)

I like the myth that 99.5% of candidates fail fizzbuzz. I tell students that they now can celebrate their achievement. But this is not where I stop with the test. I can now tell them about functions and then have them make one like:

def fizzbuzz(start, stop):
  :

where I have them modify their code above to make it a function. And afterwards they can learn about named parameters with default values:

def fizzbuzz(start=1, stop=100, three=3, five=5):
  :

Notice above how one can change the values of 3 and 5 from the original test and try a different pair. And yes, I leave the functions as exercises to the reader :)

But the best sparks in their eyes come when they remember that I had taught them certain properties of strings some three hours ago, like:

>>> 'fizz' + 'buzz'
'fizzbuzz'
>>> 'fizz' * 1
'fizz'
>>> 'fizz' * True
'fizz'
>>> 'fizz' * 0
''
>>> 'fizz' * False
''
>>> 'fizz' + ''
'fizz'
>>>

So their first observation that 'fizz' + 'buzz' equals 'fizzbuzz' is followed by what is summed up by the table:

String Sum of strings Expanded sum
‘fizzbuzz’ ‘fizz’ + ‘buzz’ ‘fizz’ * True + ‘buzz’ * True
‘fizz’ ‘fizz’ + ” ‘fizz’ * True + ‘buzz’ * False
‘buzz’ ” + ‘fizz’ ‘fizz’ * False + ‘buzz’ * True
” + ” ‘fizz’ * False + ‘buzz’ * False

Which makes them write something like that in the end:

def fizzbuzz(x, y):
  return 'fizz' * x + 'buzz' * y

for i in range(1, 101):
  print(fizzbuzz(i % 3 == 0, i % 5 == 0) or i)

How many things one can learn within a day starting from zero using the humble (and sometimes humbling) fizzbuzz.