removing all containers via ssh

Assuming one wants to unconditionally remove all containers in a host, they would run:

$ docker rm -f $(docker ps -a -q)

Suppose now that you want to execute this in a remote host via ssh. The following won’t work:

$ ssh remote_user@remote_host "docker rm -f $(docker ps -a -q)"

because the $(docker ps -a -q) part is expanded locally on your machine and you ssh remotely after the expansion. So at best it would seek to remove your local containers over there :)

What would work though? xargs to the rescue:

$ ssh remote_user@remote_host "docker ps -a -q|xargs -n 1 docker rm -f "

Of course you can fine-tune further if you wish.

RUN –mount=type=ssh is not always easy

Let’s take a very barebones Jenkinsfile and use it to build a docker image that clones something from GitHub (and possibly does other stuff next):

pipeline {
  agent any

  environment {
    DOCKER_BUILDKIT=1
  }

  stages {
    stage('200ok') {
      steps {
        sshagent(["readonly-ssh-key-here"]) {
          script {
            sh 'docker build --ssh default -t adamo/200ok .'
          }
        }
      }
    }
  }
}

We are using the SSH Agent Plugin in order to allow a clone that happens in the Dockerfile:

# syntax=docker/dockerfile:experimental
FROM bitnami/git
RUN mkdir /root/.ssh && ssh-keyscan github.com >> /root/.ssh/known_hosts
RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git

This builds fine. But what if you need this to be some "rootless" container?

# syntax=docker/dockerfile:experimental
FROM bitnami/git
USER bitnami
WORKDIR /home/bitnami
RUN mkdir /home/bitnami/.ssh && ssh-keyscan github.com >> /home/bitnami/.ssh/known_hosts
RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git

This will fail with something like:

#14 [7/7] RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git
#14       digest: sha256:fb15ac6ca5703d056c7f9bf7dd61bf7ff70b32dea87acbb011e91152b4c78ad4
#14         name: "[7/7] RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git"
#14      started: 2021-12-17 12:00:22.859388318 +0000 UTC
#14 0.572 fatal: destination path '200ok' already exists and is not an empty directory.
#14    completed: 2021-12-17 12:00:23.508950696 +0000 UTC
#14     duration: 649.562378ms
#14        error: "executor failed running [/bin/sh -c git clone git@github.com:a-yiorgos/200ok.git]: exit code: 128"

rpc error: code = Unknown desc = executor failed running [/bin/sh -c git clone git@github.com:a-yiorgos/200ok.git]: exit code: 128

Why is that? Is not the SSH agent forwarding working? Well, kind of. Let’s add a couple of commands in the Dockerfile to see what might be the issue:

# syntax=docker/dockerfile:experimental
FROM bitnami/git
USER bitnami
WORKDIR /home/bitnami
RUN mkdir /home/bitnami/.ssh && ssh-keyscan github.com >> /home/bitnami/.ssh/known_hosts
RUN --mount=type=ssh env
RUN --mount=type=ssh ls -l ${SSH_AUTH_SOCK}
RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git

Then the build output gives us:

:
#13 [6/7] RUN --mount=type=ssh ls -l ${SSH_AUTH_SOCK}
#13       digest: sha256:ce8fcd7187eb813c16d84c13f8d318d21ac90945415b647aef9c753d0112a8a7
#13         name: "[6/7] RUN --mount=type=ssh ls -l ${SSH_AUTH_SOCK}"
#13      started: 2021-12-17 12:00:22.460172872 +0000 UTC
#13 0.320 srw------- 1 root root 0 Dec 17 12:00 /run/buildkit/ssh_agent.0
#13    completed: 2021-12-17 12:00:22.856049431 +0000 UTC
#13     duration: 395.876559ms
:

and subsequently fails to clone. This happens because the socket file /run/buildkit/ssh_agent.0 for the SSH agent forwarding is not accessible by user bitnami and thus no ssh identity is available to it.

I do not know whether it is possible to make use of RUN --mount=type=ssh in combination with USER where the user is not root. Please leave a comment if you know whether/how this can be accomplished.

Using systemd-nspawn to run an older Ubuntu version and Docker images

Work for this post was partly triggered by something I saw at work and partly because of a lightning talk I saw on Monday. There was a remark by a friend that even though we love and want to run latest and greatest supported versions, we cannot always upgrade in time. And as such, sometimes we end-up with unsupported OS versions for a significant time. Not unlike the law of welded systems.

But it got me thinking. Assuming, for example, that you have a number of Xenial systems that due to library dependencies you cannot upgrade, is there a middle ground that might be acceptable for a while until you push forward? Using a Xenial docker container and treating it as a lightweight VM somehow, could be a solution. But what if you need to run docker containers too? Maybe you need something more elaborate.

I fired up a VM running Ubuntu Focal and decided to figure out how to run a systemd-nspawn Xenial container.

Create the machine:

# debootstrap --arch=amd64 xenial /var/lib/machines/xenial1

Start the machine

# systemd-nspawn -D /var/lib/machines/xenial1
root@xenial1 # rm /etc/securetty
root@xenial1 # passwd root
root@xenial1 # apt-get update
root@xenial1 # apt-get install dbus resolvconf
root@xenial1 # systemctl enable systemd-resolved
root@xenial1 # cat > /etc/resolvconf/resolv.conf.d/base
nameserver 127.0.0.53
options edns0 trust-ad
search home
ctrl-D
root@xenial1 #

Yes, I know that it is best to fix securetty instead of removing it, but this is a PoC on my VM. You can now proceed to configure systemd to run the container:

# /etc/systemd/system/xenial1.service

[Unit]
Description=Xenial1 Container

[Service]
LimitNOFILE=100000
ExecStart=/usr/bin/systemd-nspawn --machine=xenial1 --directory=/var/lib/machines/xenial1/ --bind /var/run/docker.sock:/var/run/docker.sock --bind /mnt2:/mnt2 -b 
Restart=always

[Install]
Also=dbus.service

In your host system you can now:

root@focal # systemctl daemon-reload
root@focal # systemctl start xenial1

You can of course log into the machine with machinectl login xenial1. Notice above that the nspawn container mounts the docker socket, since we assume that you have docker installed in the Focal host and that you want to "run" docker containers from within the Xenial machine.

We only need to install the docker client in the Xenial container and as such:

root@xenial1 # cat > /etc/apt/sources.list.d/docker.list
deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu xenial stable
ctrl-D
root@xenial1 # apt-get update
root@xenial1 # apt-get install docker-ce-cli=5:18.09.7~3-0~ubuntu-xenial

Now you’re all set. The docker client inside Xenial, starts whatever container you want in Focal and does whatever you like.

Suppose that you want a non-root user like ubuntu in the Xenial machine to be able to run docker commands; what do you do? You add the /etc/group line for the docker group from Focal in Xenial and in Xenial you simply useradd -aG docker ubuntu.

ubuntu@xenial1 > docker run -d -p 8080:8080 bitnami/nginx
:
ubuntu@focal > curl http://127.0.0.1:8080/
:
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
:

Enjoy.

So is any string proper for a docker image tag?

There was a failure for a build in our system that looked like this:

docker build -t yiorgos/my-cool-application:service/1.1.2 .

invalid argument "yiorgos/my-cool-application:service/1.1.2" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.

Sometimes you get this error when you forget a space between the dot (the current directory where your Dockerfile usually lives and the tag). But this was not the case. Docker actually did not like the tag for the image.

My hunch was that it does not like slashes inside the actual tag part (right of the :).

And indeed by checking out the source code of podman we that this is indeed the case:

//	tag                             := /[\w][\w.-]{0,127}/

reference.go contains the full specification of a tag for anyone interested.

sometimes you need to change your docker network

I was working locally with some containers running over docker-compose. Everything was OK, except I could not access a specific service within our network. Here is the issue: docker by default assigns IP addresses to containers from the 172.17.0.0/16 pool and docker-compose from 172.18.0.0/16. It just so happened that what I needed to access lived on 172.18.0.0/16 space. So what to do to overcome the nuisance? Obviously you cannot renumber a whole network for some temporary IP overlapping. Let’s abuse reserved IP space instead. Here is the relevant part of my daemon.json now:

{
  "default-address-pools": [
    { "base": "192.0.2.0/24", "size": 28 },
    { "base": "198.51.100.0/24", "size": 28 },
    { "base": "203.0.113.0/24", "size": 28 }
  ]
}

According to RFC5737 are reserved for documentation purposes. I’d say local work is close enough to documentation to warrant the abuse, since we also adhere to its operational implications. Plus I wager that most of the people while they always remember classic RFC1918 addresses, seldom take into account TEST-NET-1 and friends.

Sometimes you need JDK8 in your docker image …

… and depending the base image this may not be available, even from the ppa repositories. Moreover, you cannot download it directly from Oracle without a username / password anymore, so this may break automation, if you do not host your own repositories. Have no fear, sdkman can come to the rescue and fetch for you a supported JDK8 version. You need to pay some extra care with JAVA_HOME and PATH though:

FROM python:3.6-slim
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y curl zip unzip
RUN curl -fsSL https://get.sdkman.io -o get-sdkman.sh
RUN sh ./get-sdkman.sh
RUN bash -c "source /root/.sdkman/bin/sdkman-init.sh && sdk install java 8.0.212-amzn"
ENV JAVA_HOME=/root/.sdkman/candidates/java/current
ENV PATH=$JAVA_HOME/bin:$PATH
CMD bash

Since curl | sudo bash is a topic of contention, do your homework and decide whether you’re going to use it as displayed in the Dockerfile above, or employ some extra verification mechanism of your choice.

minikube instead of docker-machine

docker-machine is a cool project that allows you to work with different docker environments, especially when your machine or OS (like Windows 10 Home) does not allow for native docker. My primary use case for docker-machine was spinning up a VirtualBox host and using it to run the docker service.

It seems that docker-machine is not in so much active development any more and I decided to switch to another project for my particular use case: Windows 10 Home with VirtualBox. One solution is minikube:

minikube start
minikube docker-env | Invoke-Expression # Windows PowerShell
eval $(minikube docker-env) # bash shell
docker pull python:3
docker images

and you’re set to go.

ansible, docker-compose, iptables and DOCKER-USER

NOTE: manipulating DOCKER-USER is beyond anyone’s sanity. The information bellow seems to work sometimes (like when I wrote the post) and others not. That is why you will find posts with similar advise on the Net that may or may not work for you. I plan to revisit this and figure out what is wrong, making the following information only temporarily correct.

When you want to run ZooNavigator, the recommendation to get you started is via this docker-compose.yml. However, Docker manages your iptables (unless you go the –iptables=false way) and certain ports will be left wide open. This may not be what you want to do. Docker provides the DOCKER-USER chain for user defined rules that are not affected by service restarts and this is where you want to work. Most of my googling resulted in recipes that did not work, because their final rule was to deny anything from 0.0.0.0/0 after having allowed whatever was to be whitelisted. I solved this in the following example playbook, and the rules worked like a charm. Others that may find themselves in the same situation may want to give it a shot:

---
- name: maintain the DOCKER-USER access list
  hosts: zoonavigators
  vars:
    - wl_hosts:
      - "172.31.0.1"
      - "172.31.0.2"
    - wl_ports:
      - "7070"
      - "7071"
  tasks:

  - name: check for iptables-services
    yum:
      name: iptables-services
      state: latest

  - name: enable iptables-services
    service:
      name: iptables
      enabled: yes
      state: started

  - name: flush DOCKER-USER
    iptables:
      chain: DOCKER-USER
      flush: true

  - name: whitelist for DOCKER-USER
    iptables:
      chain: DOCKER-USER
      protocol: tcp
      ctstate: NEW
      syn: match
      source: "{{ item[0] }}"
      destination_port: "{{ item[1] }}"
      jump: ACCEPT
    with_nested:
      - "{{ wl_hosts }}"
      - "{{ wl_ports }}"

  - name: drop non whitelisted connections to DOCKER-USER
    iptables:
      chain: DOCKER-USER
      protocol: tcp
      #source: "0.0.0.0/0"
      destination_port: "{{ item }}"
      jump: DROP
    with_items:
      - "{{ wl_ports }}"

  - name: save new iptables
    command:
      /usr/libexec/iptables/iptables.init save

Line 46 is the key. The obvious choice would have been source: "0.0.0.0/0" but this did not work for me.

[pastebin here]

vagrant, ansible local and docker

This is a minor annoyance to people who want to work with docker on their vagrant boxes and provision them with the ansible_local provisioner.

To have docker installed in your box, you simply need to enable the docker provisoner in your Vagrantfile:

config.vm.provision "docker", run: "once"

Since you’re using the ansible_local provisiner, you might skip this and write a task that installs docker from get.docker.com or wherever it suits you anyway, but I prefer this as vagrant knows how to best install docker onto itself.

Now obviously you can have the provisioner pull images for you, but for any crazy reason you want to pass most, if not all, of the provisioning to ansible. And thus you want to use among others the docker_image module. So you write something like:

- name: install python-docker
  become: true
  apt:
    name: python-docker
    state: present

- name: install docker images
  docker_image:
    name: busybox

Well this is going to greet you with an error message when you up the machine for the fist time:

Error message

TASK [install docker images] ***************************************************
fatal: [default]: FAILED! => {“changed”: false, “msg”: “Error connecting: Error while fetching server API version: (‘Connection aborted.’, error(13, ‘Permission denied’))”}
to retry, use: –limit @/vagrant/ansible.retry

Whereas when you happily run vagrant provision right away:

TASK [install docker images] ***************************************************
changed: [default]

Why does this happen? Because even though the installation of docker makes the vagrant user a member of group docker, this becomes effective with the next login.

The quickest way to bypass this is to make that part of your first run of ansible provisioning as super user:

- name: install docker images
  become: true  
  docker_image:
    name: busybox

I am using the docker_image module only as an example here for lack of a better example with other docker modules on a Saturday morning. Pulling images is something that is of course very easy to do with the vagrant docker provisioner itself.

default: Running ansible-playbook…

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [default]

TASK [install python-docker] ***************************************************
changed: [default]

TASK [install docker images] ***************************************************
changed: [default]

PLAY RECAP *********************************************************************
default : ok=3 changed=2 unreachable=0 failed=0