Home

Release It!

2017/09/23

I finally got to read Release It!. It would have served me better had I read it 10 years or so ago when I actually bought it. But never too late. There is a second edition that came out this month. While this first edition clearly shows its age and relics of another time, I guess the second edition is more hip, fresh, readable and with more experience to share.

Not a timeless book, but one that needs a refresh every decade or so. I tag it under “system administration” too, because a system is what you build, works and bring money in, not just what Ops runs and grumbles about.

Advertisements

Moses B. popped this in my timeline. Although the piece is about philosophy (of which I know nothing about) I thought I could entertain myself by paraphrasing the key points a bit. I am blatantly plagiarizing:

People like me, who have been trying to do Ops (system administration by a new name) for more than twenty years, do in due course learn, if they’re lucky, how to do what they’ve been trying to do: that is, they do learn how to do Ops. But although I’ve learned how to do Ops, nobody ever told me how do it, and, so far as I would guess, nobody will have told you how to do it, or is likely to tell you how to do it in the future.

So back in the early 90s why was I not taught how to do Ops? In parallelism with the philosophy article, here are some ideas:

  • System administration was free of methods.
  • The methods were kept quiet, perhaps because people thought “the methods of cannot be explicitly taught/said, but have to be shown via example/exemplars.”.
  • “those of us who have learned how to do it struggled so hard to get where we now are that…we think you, too, should suffer,” perhaps because “we’re now selfishly reluctant to give you some of the fruit of our struggle for free”.
  • “the methods of (early to middle) system administration were not unified”.
  • “it is a form of boundary policing. Under such a regime, outsiders are permanently mystified about the rules of the game because they don’t know what move to make to acquire standing”.

Let me see. When I started in this line of work there was no manual other than man and the technical documentation from the vendor (if you were lucky to have access to it).  Humble posts to the USENET groups and to lists like decstation-managers also paid off.

People lead by example. So their accumulated knowledge was not part of the technical documentation (I still find old vendor technical documentation far superior than today’s). Knowledge gained from experience, was passed the same way forward: By experience. “Do that” said the wizard in a stressful situation and afterwards they would share the war story that led to that. You were then supposed to generalize from what you were shown.

But sometimes, being shown was not enough. You had to struggle. Because the wizard as an apprentice was forced to learn how to swim by themselves, they retained the same belief for the next generation, or even for a peer (if you’re an equal, you can do it). The initiation process would require you to struggle, to earn the knowledge, to be able to read the packets in the wire like being the network driver yourself. The wizard was one with the machine, why should you benefit from this and not become one too?

How would you administer your systems? Via, SAM, SMIT, some other curses-based tool, if any, the shell, batch scripts doing rlogin? How? How would you do that in an era when infrastructure as code was not a thing and cfengine was a curiosity? How do you even do that now that you are hit by the paradox of choice? Did you ever hear of a book like this? If you did, did you read it?

Do you remember what a BOFH is? I know, because I was one. Before becoming one, I was told: “I do not understand why you’re asking this: You should either know what to do, and not ask, or you do not know what to do, and do not have to ask because you will not do it”, said the elder, then thought to be wise BOFH.

We are priests. Either you know how to read our hieroglyphs and you’re in, or you’re out. A great sense of empowerment came out of this. We take your crap and place it in production. It breaks. We tell you why and you still do not have a clue. We’ve explained the issue clearly, but not in your language. Your fault.

I will go with the most charitable explanation: We do not know how to pass the craft’s knowledge and everything else is a defense so as not to fall in your eyes.

I’m sorry.

PS: And then, DevOps came along

I admit, I almost forgot. I got reminded that it is SysAdminDay via a text message from a friend.  I am in between jobs in a new country, trying to fix an agreement on a house, so I forgive myself.  But as always, I want to write a few lines. They might echo an older post but I cannot do better today. So I am going to revisit my Conway’s goals, but for DevOps:

  • Big Problem: I guess this is going to come my way thanks to my new employer. They hired my for a reason.
  • Workable Problem: There will always be a second problem to spend my day at. One that will progress easier than others and offer a sense of daily accomplishment. At least I am it to be this way.
  • Book problem: I have three of those lined up. I am not writing a book, but there are three that I aim to study thoroughly. In fact I am a year behind in one of them. However, with a lot of free time away from my family, I guess I do not have the excuse of non existent time.
  • Fun problem: I intend to learn Go. I’ve no problem with GNU-Go’s text interface.

Happy SysAdmin day everyone. May your puppets, salt minions and pod containers sail bug free and fail only within 9 to 5.

The Goal

2017/05/09

Well if you’re doing DevOps (whatever that means for you) you’re supposed to have read The Phoenix Project (which I have). Inside the Phoenix there are constant references to Visible Ops and The Goal. Well, The Goal is a something of a gospel it seems so I went ahead a few months ago and read it.

It is a far better book to read than the Phoenix. Much more entertaining and insightful. I guess Goldratt is a better writer. The book does a hell of a good job to explain the basics of The Theory of Constraints which receives new attention in this age of software automation and pipelines and the perceived “clash” between DevOps, ITIL and any other related acronym. I see ToC posts pop up weekly now. The only thing that you may not like in the book is the technology from the 80s, but only if you’ve never used it.

My best part was the really nice simulation example it used to make clear a counter-intuitive result and his family crisis management. Granted, many things went the right way for the main hero, but hey, the author was trying to make a point and needed a vehicle. Fiction is not life.

I read it from the latest kindle edition. According to my Kindle I am still at 97% of the book and that is because right after the end of the book there is a paper explaining some of the concepts with real industry examples. The boredom to read this after the turmoil the hero has gone through is evident. It also explains why the author needed a novel in order to engage you in ToC.

Like a good friend said: “I really want to get in the book and help him out!”

We were interviewing someone for a DevOps position. They were working at a managed network / security services company and needed to change a life and career. Their work was pretty much summarized as “A client files a change request; we implement it at the network device and we’re done”.

No provisioning, no automation, no versioning of configurations was in place. The candidate was not in a position to write code also. Since they were clearly more junior than the position required, we were saddened and in a way gave them some subtle advice to improve on their skills in order to have better luck next time they knock on a door:

  • Learn to program in Python.
  • Do that because you can learn to code with paramiko and Netmiko.
  • Learn some basic git usage.
  • It does not matter that your employer does not require you to automate stuff and practice version control. You will write a program in Python that will ssh into the client’s equipment, backup the configuration in git and push the requested changes back.
  • You will have automated your work and will have more free minutes per day to read about stuff.
  • You will have more to talk about in your next interview.

When you’re not pushed by the environment, you need to begin from somewhere. Oh and understand some basic statistics, because you will need to understand what you graph. It should not only be pretty. It should be useful.

Beyond Blame

2017/01/05

That went away fast. You can finish it in one shot. Especially if you are in one of the most thankless professions, with lots of responsibility and zero authority. The narrative, however thin, feels close to heart because this things happen. Or may have even happened to you or someone you know.

cynefin_framework2c_february_2011_28229

The Cynefin model

While this is no Phoenix Project, the first sixteen chapters serve to lay the playgound for implementing a proper postmortem process, where no one is afraid to withhold critical information. This along a very useful bibliography is presented in the last chapter.

As always, the hard thing is to make people who think that “rolling heads” improve morale and performance, to see the error of their ways.

Happy SysAdminDay

2016/07/29

Since monitoring is at the heart of managing infrastructures, time-series and measurements also, I believe it is time to remember what the first derivative is. Because all those graphs we make with Graphite, Graphana and friends are not only eye-candy. They have some meaning too. “The Art of Monitoring” speaks of the rate of change in its first chapter, but this has a formal name.

Now go hug your SysAdmin.