Don’t disable IPv6, block the traffic instead

It is a somewhat common practice to disable IPv6 when you have to deal with no such traffic. However, doing so may have some unintended consequences, like when running rpcbind for NFS sharing:

Feb 20 11:27:12 server systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol

So, don’t disable IPv6 traffic. Aim to blocking it instead. Or if you must have it disabled, read here.

PS: This is not a post advocating not adopting IPv6. This is simply a workaround if the situation arises. You need to operate in an IPv6 world.

base64 and SASLprep: failed prohibited character check

I was trying something out on a minikube cluster the other day, and my python application could not connect to a mongo database that was deployed with helm. I had authentication enabled, the secret deployed in minikube and was greeted by the following error message:

SASLprep: failed prohibited character check

WTF? The password was definitely ascii printable characters. Could there be prohibited characters among the alphabet? I browsed through RFCs 4013 and 3454 but there was nothing I could immediately pinpoint there. However, this tweet by jpmens was circling my mind, and:

$ echo qwerty1234 | base64

$ printf qwerty1234 | base64

$ echo -n qwerty1234 | base64

Yes, the kubernetes secret had a \n attached to it. Because two things are hard in CS:
– naming things
– cache invalidation
– off by one errors

aws ssm describe-instance-information for quick ansible dynamic inventory

The aws ssm agent is very useful when working both with EC2 instances and with machinery outside AWS. Once you add an outside instance by installing and configuring the SSM agent, be it on-premises or a VM at another provider, you can tag it for further granularity with aws ssm add-tags-to-resource --resource-type ManagedInstance --resource-id mi-WXYZWXYZ --tags Key=onpremise,Value=true --region eu-west-1 where mi-WXYZWXYZ is the instance ID you see at the SSM’s managed instances list (alternatively you can get this list with aws ssm describe-instance-information along with lots of other information).

It may the case that sometimes you want to apply with ansible a certain change to those machines that live outside AWS. Yes you can run ansible workbooks via the SSM directly, but this requires ansible installed on said machines. If you need the simplest of dynamic inventories, to $ ansible -u user -i ./lala all -m ping here is the crudest version of ./lala, one that happily ignores the --list argument:

printf "%s%s%s" \
'{ "all": { "hosts": [' \
$(aws ssm describe-instance-information --region eu-west-1 --filter Key=tag:onpremise,Values=true --query "InstanceInformationList[].IPAddress" --output text | tr '[:blank:]' ',') \
'] } }'

You can go all the way scripting something like this for a proper solution though.

Why printf instead of echo above? Because jpmens suggested so.

Running monit inside Kubernetes

Sometimes you may want to run monit inside a Kubernetes cluster just to validate what you’re getting from your standard monitoring solution with a second monitor that does not require that much configuration or tinkering. In such cases the Dockerfile bellow might come handy:

FROM ubuntu:bionic
RUN apt-get update
RUN apt-get install monit bind9-host netcat fping -y
RUN ln -f -s /dev/fd/1 /var/log/monit.log
COPY monitrc /etc/monit
RUN chmod 0600 /etc/monit/monitrc
ENTRYPOINT [ "/usr/bin/monit" ]
CMD [ "-I", "-c", "/etc/monit/monitrc" ]

I connected to it via kubectl -n monit-test port-forward --address= pod/monit-XXXX-YYYY 2812:2812. Most people do not need --address=, but I run kubectl inside a VM for some degree of compartmentalization. Stringent, I know…

Why would you need something like this you ask? Well imagine the case where you have multiple pods running, no restarts, everything fine, but randomly you get connection timeouts to the clusterIP address:port pair. If you have no way of reproducing this, don’t you want an alert the exact moment it happens? That was the case for me.

And also the fun of using a tool in an unforeseen way.

Rancher’s cattle-cluster-agent and error 404

It may be the case that when you deploy a new Rancher2 Kubernetes cluster, all pods are working fine, with the exception of cattle-cluster-agent (whose scope is to connect to the Kubernetes API of Rancher Launched Kubernetes clusters) that enters a CrashLoopBackoff state (red state in your UI under the System project).

One common error you will see from View Logs of the agent’s pod is 404 due to a HTTP ping failing:

ERROR: is not accessible (The requested URL returned error: 404)

It is a DNS problem

The issue here is that if you watch the network traffic on your Rancher2 UI server, you will never see pings coming from the pod, yet the pod is sending traffic somewhere. Where?

Observe the contents of your pod’s /etc/resolv.conf:

search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Now if you happen to have a wildcard DNS A record in the HTTP ping in question becomes which happens to resolve to the A record of the wildcard (most likely not the A RR of the host where the Rancher UI runs). Hence if this machine runs a web server, you are at the mercy of what that web server responds.

One quick hack is to edit your Rancher2 cluster’s YAML and instruct the kubelet to start with a different resolv.conf that does not contain a search path with your domain with the wildcard record in it. The kubelet appends the search path line to the default and in this particular case you do not want that. So you tell your Rancher2 cluster the following:

      resolv-conf: /host/etc/resolv.rancher

resolv.rancher contains only nameserver entries in my case. The path is /host/etc/resolv.rancher because you have to remember that in Rancher2 clusters, the kubelet itself runs from within a container and access the host’s file system under /host.

Now I am pretty certain this can be dealt with, with some coredns configuration too, but did not have the time to pursue it.

The Uprising (Revolution Book 1)

The Uprising (Revolution Book 1)The Uprising by Konstantinos Christidis
My rating: 3 of 5 stars

This is the first book from the author and if I am not mistaken, it is self-published. It could use some more help with editing. That is why I did not give it more stars.

Think of this book as a prequel to the Expanse. The setting is similar, the Earth is governed by the UN and there is a terraforming project for Mars (and Venus) and at Callisto there is an installation that ships material and water (ice) to the terraforming projects.

Think also of the existence of the equivalent of the East India Company with its own private army, monopoly status and control over the judicial system and the government. What can go wrong when some theoretical physicist backed with VC money (to put it in today’s terms) threatens the status quo with faster than light travel?

This is what the book is about.

Could it have been written better? Yes. Does it matter that at times the author does not manage to keep the pace and be a bit boring? I don’t know. Maybe. It took me more than I anticipated to finish it.

Did I have a good time ultimately reading it? Sure.

View all my reviews

once again bitten by the MTU

At work we use Rancher2 clusters a lot. The UI makes some things easier I have to admit. Like sending logs from the cluster somewhere. I wanted to test sending such logs to an ElasticSearch and thus I setup a test installation with docker-compose:

version: "3.4"

    restart: always
    image: elasticsearch:7.5.1
    container_name: elasticsearch
      - "9200:9200"
      - ES_JAVA_OPTS=-Xmx16g
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - http.port=9200
      - xpack.monitoring.collection.enabled=true
      # ensure chown 1000:1000 /opt/elasticsearch/data please.
      - /opt/elasticsearch/data:/usr/share/elasticsearch/data

    restart: always
    image: kibana:7.5.1
      - "5601:5601"
    container_name: kibana
      - elasticsearch
      - /etc/docker/compose/kibana.yml:/usr/share/kibana/config/kibana.yml

Yes, this is a yellow cluster, but then again, it is a test cluster on a single machine.

This seemed to work for some days, and the it stopped. tcpdump showed packets arriving at the machine, but not really responding back after the three way handshake. So the old mantra kicked in:

It is a MTU problem.

Editing daemon.json to accommodate for that assumption:

  "mtu": 1400

and logging was back to normal.

I really hate fixes like this, but sometimes when pressed by other priorities they present a handy arsenal.