Hello and happy SysAdmin day. The baby swing bellow is from 2011. While at rest it looks like a safe swing, it is not. The chains latch too close to the middle and it is very easy for the seat to revolve around a second horizontal axis while swinging. You can understand how I know.
It is SysAdmin day today. We make sure the chains latch properly so your software runs without extra revolutions.
We’re using unbound internally for DNS resolution. It works smoothly and allows for some DNS tricks when you want to implement some split-brain trickery, but not a complete split-brain deployment. The other day we needed to send out conditional replies based on the IP address of the querying machine. Unbound comes with a python module but it has some of the weirdest, unhelpful documentation ever. I am not alone in believing this.
It is very hard to figure out the source IP address of a DNS query using the unbound python library. My first pointer on how to do so was on ServerFault. I have uploaded my own version of an operate function at pastebin. The code in question that you need to consider is:
# Find out source IP address
rl = qstate.mesh_info.reply_list
q = rl.query_reply
rl = rl.next
# Careful with this conditional
try: addr = q.addr
except NameError: addr = None
The try … except handling is needed because I found out that sometimes the q.addr may not be defined and thus further down the line you may be bitten by an abnormal exit of your script.
Update: two friends have suggested that I change the while loop with a more Pythonic list comprehension:
q = next((x for x in qstate.mesh_info.reply_list if x.query_reply), None)
try: addr = q.query_reply.addr
except NameError: addr = None
A new colleague needed some help to setup a Graylog installation. He had never done this before, so he asked for assistance. What follows is a rehash of an email I sent him on how to proceed and build knowledge on the subject:
So initially I had zero knowledge of Graylog. What I did to accustom myself with it was to download an OVA file with a prepared virtual machine and run it via VMware Fusion. The same VM can also be imported to VirtualBox and even to AWS, although they also provide ready AMIs for deployment in AWS. Links:
Keep in mind that this is a full installation of what Graylog needs to work with and it also comes with a handly little script named “graylog-ctl” that manipulates a lot of configuration. The big catch is that graylog-ctl is not part of any standard Graylog deployment. It only comes with the OVA and the AMI images.
So after I had some fun with it on a VM on my workstation, reading the documentation and testing stuff, I had an initial deployment of the AMI image in AWS. But this is not an installation that can scale. Which brings us to the next steps:
For Graylog to work you need to provide it with a MongoDB and an ElasticSearch database. It is your choice whether these will be clustered for high availability or not, whether they will run in the same machine or not. You control the complete architecture. So in my case I made the following decisions:
I am running a MongoDB replica set using three VMs. This is a standard setup as it is described in the MongoDB online documentation. Since it is not password protected, it only accepts connections from the Graylog instance. I used AWS security groups for that.
I am using an ElasticSearch cluster with three VMs where the nodes are both data and masters. If you can, use 7 nodes, three masters (lower machines since they do not run queries and do not index any data) and four data nodes (higher end machines). Again, since this is not password protected, I used AWS security groups to allow access only from the Graylog instance.
I am running a single Graylog instance on a separate VM. Currently it only listens for syslog stuff. When the need arises, I will add a two more nodes to increase the availability. I think I changed as many as four or five lines in the main configuration file. Graylog uses MongoDB to store its configuration, which includes anything you configure via the web interface.
Pay extra attention to the versions of ElasticSearch and MongoDB that your Graylog version requires. Use exactly what is mentioned in the documentation. For example in my case I am not running ES 6.x but the latest 5.x.
Now it is time to up your game. Once you see that your installation is working you have to decide whether to password protect access to MongoDB and ElasticSearch and whether to encrypt traffic between all those instances or not. I say give it a go.
I’ve not even touched issues like database management for Mongo and Elastic, backing them up, restoring, deleting indices, etc because this is post from zero to your first week testing Graylog. There is plenty of stuff out there to take you to the next level, once you get used to the complexity of the software involved.
At work we try to manage as much as we can with terraform. This also includes Route53 for zones and records. In a certain situation we had about 14 zones and 1476 records managed in a single state file.
As it happened I needed a zone recreated (but not erased) and this affected about 409 records. Well deleting them with terraform apply took ages. To the point that the temporary STS token expired and botched the process. So after a little facepalming, I decided to cleanup the zone from the AWS console and then issue a batch of terraform state rm to reconcile the state. Happily, after that, applytook its time (but reasonably) and all was well.
I am thinking that next time I am faced with such a situation, to lock the state file in Dynamo, copy it over from S3, manipulate it locally, unlock and run a plan to see how it all plays out. Or, wherever I can, use a state per zone instead of a state file encompassing a set of zones.
Because WordPress.com does not always render MarkDown properly, you may need to read a copy of this post here.
One of the things I learned by reading AIM 239 is the Game of Life and Cellular Automata. One particular kind of one dimensional cellular automata, Rule 110 popped by my twitter stream the other day, so I thought I could try and code it with the minimal Haskell subset that I can handle.
Rule 110 is special because it is proven to be able to simulate a Turing machine. Head over to its Wikipedia page if you want to learn more about the proof and the interesting story around it.
Rule 110 starts with a string of zeros and ones and a transition table that decides the next state of the automaton. If you put each line of the strings after the other, interesting patterns can emerge. Let’s see the transition state:
New state for center cell
If you look closely, you can use a list of eight digits and its index in order to encode the above state transitions:
But what about the transitions of the leftmost and rightmost digit you might think. Let’s assume that their missing neighbor is zero. Therefore, given an initial state and a rule that governs the transitions, we may calculate the next state with:
nextState :: [Int] -> [Int] -> [Int]
nextState state rule =
[ rule !! x |
let t =  ++ state ++ ,
i <- [1..(length(t)-2)],
let x = (t !! (i-1)) * 4 + (t !! i) * 2 + (t !! (i+1))
-- construct an infinite sequence of next states
sequenceState :: [Int] -> [Int] -> [[Int]]
sequenceState state rule =
[state] ++ sequenceState (nextState state rule) rule
*Main> state = [0,1,1,0]
*Main> nextState state rule110
One of the most interesting patterns occurs when we begin with the right most digit being 1 and all the rest being zeros:
WordPress.com does not always render Markadown properly, so a copy of this post resides here.
I am coninuing my adventure in Haskell. In order to make it a bit more fun, I decided to code a simple yet very intriguing problem, I first heard of when I read AI Memo 239: The Collatz conjecture.
Construct a sequence of integers where given an arbitrary interger the value of the next is:
* If the number is even, divide it by two.
* If the number is odd, triple it and add one.
This can easily be coded in Haskell as follows:
collatz :: Int -> Int
collatz 1 = 1
collatz n =
if (even n)
then (n `div` 2)
else (3 * n + 1)
But how can one obtain a sequence of numbers from this? A very clever solution is here where the author implements a variation of takeWhile which includes also the first list item that fails the test the first time. So my question became, can it be done in another way? Yes it can:
collatzSequence :: Int -> [Int]
collatzSequence n =
if n == 1
else [n] ++ collatzSequence (collatz n)
You may have observed we only run it on positive integers. When we run it with negative integers, there are a few more cycles that we need to take into account. Here is the updated sequence function, written with guards:
collatzSequence :: Int -> [Int]
| n == 1 = 
| n == (-2) = [(-2)]
| n == (-5) = [(-5)]
| n == (-17) = [(-17)]
| otherwise = [n] ++ collatzSequence (collatz n)
Update: A friend posted me his own elegant version of the Collatz sequence:
collatz :: Int -> [Int]
collatz 1 = 
| even n = n : collatz (n `div` 2)
| odd n = n : collatz (n * 3 + 1)
main = do
putStrLn $ show $ collatz 1
putStrLn $ show $ collatz 6
putStrLn $ show $ collatz 23