I admit, I almost forgot. I got reminded that it is SysAdminDay via a text message from a friend.  I am in between jobs in a new country, trying to fix an agreement on a house, so I forgive myself.  But as always, I want to write a few lines. They might echo an older post but I cannot do better today. So I am going to revisit my Conway’s goals, but for DevOps:

  • Big Problem: I guess this is going to come my way thanks to my new employer. They hired my for a reason.
  • Workable Problem: There will always be a second problem to spend my day at. One that will progress easier than others and offer a sense of daily accomplishment. At least I am it to be this way.
  • Book problem: I have three of those lined up. I am not writing a book, but there are three that I aim to study thoroughly. In fact I am a year behind in one of them. However, with a lot of free time away from my family, I guess I do not have the excuse of non existent time.
  • Fun problem: I intend to learn Go. I’ve no problem with GNU-Go’s text interface.

Happy SysAdmin day everyone. May your puppets, salt minions and pod containers sail bug free and fail only within 9 to 5.

The Goal


Well if you’re doing DevOps (whatever that means for you) you’re supposed to have read The Phoenix Project (which I have). Inside the Phoenix there are constant references to Visible Ops and The Goal. Well, The Goal is a something of a gospel it seems so I went ahead a few months ago and read it.

It is a far better book to read than the Phoenix. Much more entertaining and insightful. I guess Goldratt is a better writer. The book does a hell of a good job to explain the basics of The Theory of Constraints which receives new attention in this age of software automation and pipelines and the perceived “clash” between DevOps, ITIL and any other related acronym. I see ToC posts pop up weekly now. The only thing that you may not like in the book is the technology from the 80s, but only if you’ve never used it.

My best part was the really nice simulation example it used to make clear a counter-intuitive result and his family crisis management. Granted, many things went the right way for the main hero, but hey, the author was trying to make a point and needed a vehicle. Fiction is not life.

I read it from the latest kindle edition. According to my Kindle I am still at 97% of the book and that is because right after the end of the book there is a paper explaining some of the concepts with real industry examples. The boredom to read this after the turmoil the hero has gone through is evident. It also explains why the author needed a novel in order to engage you in ToC.

Like a good friend said: “I really want to get in the book and help him out!”

We were interviewing someone for a DevOps position. They were working at a managed network / security services company and needed to change a life and career. Their work was pretty much summarized as “A client files a change request; we implement it at the network device and we’re done”.

No provisioning, no automation, no versioning of configurations was in place. The candidate was not in a position to write code also. Since they were clearly more junior than the position required, we were saddened and in a way gave them some subtle advice to improve on their skills in order to have better luck next time they knock on a door:

  • Learn to program in Python.
  • Do that because you can learn to code with paramiko and Netmiko.
  • Learn some basic git usage.
  • It does not matter that your employer does not require you to automate stuff and practice version control. You will write a program in Python that will ssh into the client’s equipment, backup the configuration in git and push the requested changes back.
  • You will have automated your work and will have more free minutes per day to read about stuff.
  • You will have more to talk about in your next interview.

When you’re not pushed by the environment, you need to begin from somewhere. Oh and understand some basic statistics, because you will need to understand what you graph. It should not only be pretty. It should be useful.

Beyond Blame


That went away fast. You can finish it in one shot. Especially if you are in one of the most thankless professions, with lots of responsibility and zero authority. The narrative, however thin, feels close to heart because this things happen. Or may have even happened to you or someone you know.


The Cynefin model

While this is no Phoenix Project, the first sixteen chapters serve to lay the playgound for implementing a proper postmortem process, where no one is afraid to withhold critical information. This along a very useful bibliography is presented in the last chapter.

As always, the hard thing is to make people who think that “rolling heads” improve morale and performance, to see the error of their ways.

Happy SysAdminDay


Since monitoring is at the heart of managing infrastructures, time-series and measurements also, I believe it is time to remember what the first derivative is. Because all those graphs we make with Graphite, Graphana and friends are not only eye-candy. They have some meaning too. “The Art of Monitoring” speaks of the rate of change in its first chapter, but this has a formal name.

Now go hug your SysAdmin.

DevOps in practice


This is a free report that you can download from O’Reilly’s website. It deals with how DevOps was adopted to facilitate Nordstrom and Texas.gov (with a glimpse of how the Texas.gov CISO inserted security provisioning in the development processes).

If you’ve read The Phoenix Project and felt that OK this is nice, but it is also a novel, the Nordstrom story is an indication of how to adapt this outside a novel concept and into the real world. It is worth your time, especially in cases when you need to appeal to real world examples.

systemd are not very compatible yet. But it is here to stay and one can hold back their machines for only so much. Inevitably, no matter how much you delay it, you will migrate to operating system versions that support it. So here is the path that I figured out for smooth coexistence until systemd grows on you:

Do most of if not everything that you would try to do with systemd with something like puppet or ansible by applying playbooks locally. Configuration management / orchestration software already knows how to handle systemd and they do it well. Take advantage of that then.