the case of the yellow cluster

org.elasticsearch.transport.ConnectTransportException or the case of the yellow cluster

Some time ago we needed to add two datanode in our ElasticSearch cluster which we happily ordered from our cloud provider. The first one joined OK and shards started moving around nicely. A happy green cluster. However upon adding a second node, the cluster started accepting shards but remained in yellow state. Consistently. Like hours. Even trying to empty the node in order to remove it was not working. Some shards would stay there forever.

Upon looking at the node logs, here is what caught our attention:

org.elasticsearch.transport.ConnectTransportException: [datanode-7][10.1.2.7:9300] connect_exception

A similar log entry was found in datanode-7‘s log file. What was going on here? Well, these two machines were assigned sequential IP addresses, 10.1.2.6 and 10.1.2.7. They could literally ping the whole internet but not find each other. To which fact the cloud provider’s support group replied:

in this case you need to configure a hostroute via the gw as our switch doesn't allow a direct communication.

Enter systemd territory then, and not wanting to make this yet another service, I defaulted to the oldest boot solution. Edit /etc/rc.local (in reality /etc/rc.d/rc.local) on both machines with appropriate routes:

ip route add 10.1.2.7/32 via 10.1.2.1 dev enp0s31f6

and enable it:

# chmod +x /etc/rc.d/rc.local
# systemctl enable rc-local

rc.local will never die. It is that loyal friend that will wait to be called upon when you need them most.

Sometimes n2n is good enough

You are not always in a position to use one of the big five cloud providers (BTW, I think Watson’s prediction was kind of true; serverless is making sure of that).

So when you’re working in different than usual public cloud environments, sometimes you miss features that are a given, like a VPC. A VPC is a pretty cool abstraction that allows you to have an isolated network of machines (and sometimes services) within your cloud provider and allows for easier management of things like security groups, routing traffic between machines and the like.

So what do you do when you do not have a VPC available? You need some kind of overlay networking. When deploying Kubernetes for example, you need to deploy an overlay network (there are many solutions to choose from) and you let it deal with iptables and routing hell. But, you may need to temporarily scale services that are not container orchestrated for whatever the reason (I, for example abhor running databases with Kubernetes). Still you may need an autoscaling solution like EC2 does. IPSec would be a cool solution, but deploying it in my current workplace would be too complex. Something simpler was needed. And I found it here, despite the shortcomings reported: N2N from ntop.org.

N2N was in a development hiatus, but now is back on active development. It utilizes TUN/TAP and allows you to build a VPC over the interface that the client you run on your machine creates. It comprises of two components: a supernode, which is actually a directory server that informs the members of the (let’s call it) VPC of the actual IP addresses of the members, and a client program (called edge) that creates the interface on each VM and contacts the supernode to register with it and query for needed routing information. The supernode itself is not routing any packets. It was a single point of failure, but current versions of N2N/edge support two supernodes, so your network is in a better position.

In my case I needed to autoscale a certain service for a limited amount of time, but did not have any prior knowledge of the IP addresses of the VMs that were to be created. So I had them spun off using an image that was starting edge, registering to the supernode and then routing a certain kind of traffic through a node that was also registered in the same network and acting as a NAT server. Hence, I simplified some of the iptables hell that I was dealing with until we deploy a better solution.

N2N supports encrypted traffic, and requires the equivalent of a username / password combination (common to all machines that are members of the VPC, but not known to supernode apriori).

So where else might you want to use N2N? Maybe you need a common IP address space between two cloud providers? You may be in a cloud provider that allows for VPCs but does not make it easy to route traffic from a VPC in one region to a VPC in another region? Or in cases when solutions like DMVPN are expensive and your own BGP solution an overkill? Stuff like that.

So how do the machines acquire an address in that VPC? You have two choices (a) DHCP and if that is not working (b) a static address. In the second case with you need to implement a poor man’s DHCP by having the machine assign an IP address to itself with a low probability of collision. To this end, let’s assign a /16 to that VPC and have the following entry in /etc/rc.local (yes I am still a fun of rc.local for limited usage) like:

edge -l supernode:port -c community-string -k password-string -s 255.255.0.0 -a static:172.31.$(shuf -i 1-251 -n 1).$(shuf -i 1-251 -n 1). 

The probability of a collision is (1/63001) so you get to decide at how many temporary instances you need to have a better hack at that. Plus you get to use 172.31.1.0-255 for static machinery within the VPC (like a NAT gateway for example).

Not a perfect solution, but definitely an easy, fast one.