Lesson learned … 13 years later

Now that I spend my days doing more tech support than coding, I’m finally coming to terms with a lesson that a coworker tried to teach me 13 years ago.

One of my first jobs out of school was a Webmaster position for Northrop Grumman. This was before I knew ColdFusion or Perl and was still doing all my web programming (CGI!) with C++. I wasn’t technically part of the IT department due to some political goings-on, but I was given the task of setting up the web server and maintaining it, which was a pretty heady experience for someone my age. A bunch of the guys I was working with had worked on the Apollo program, as in rockets not Adobe, and here I was fresh out of college!

One of the IT guys at the time, Bill Costley, watched as I chased my tail trying to keep up with Windows hotfixes, pre Windows Update, and recompiling Apache with each new patch and on and on and on. With each new hotfix or patch came the chance that something would break, so I was constantly backing things up and keeping a dozen old versions, just in case.

“Why do you do that?” he asked me one day.

I answered as a good, young sysadmin would: “Because they are security patches. What if someone exploits a hole to get into the system? There’s sensitive data on the web server!”

He smiled at me. “But it’s an internal web server.”

“So? It’s not like I know everyone that works for the company.”

We’d have variations of this conversation after each new patching frenzy, and each time he would just smile and nod and watch me sweat it out.

Frankly, I’m sure I thought he was a doddering old man. Smart as a whip, but crazy nonetheless.

Finally, one day I asked him: “Well, how often do you IT guys apply patches?”

“Only when we absolutely have to, and then only when it’s been out for at least six months.”

I’m sure I was probably agape at the horrid thought. So, I went on patching, and Bill went on smiling, and we agreed to disagree.

But here I am now, 13 years later, and I finally get it:

Get your system working, then leave it the hell alone.

Software developers can’t be trusted. I was one, still am one occasionally, and I feel that I can say that with authority. We, they, aren’t perfect. Occasionally, patches and hotfixes and new versions are going to break things. That’s just how it goes.

And, as a network admin now, I get the need to get everything to a working state and then tell the users to just leave everything alone. Don’t try to install anything new—just do their jobs. If they do something that breaks that peace, I’m inclined to tell them to figure out how to fix it themselves. Then I go and look for a way to keep them from breaking that same thing a second time.

If it works today, and you don’t change anything, then it will probably work tomorrow.

By Rick Osborne

I am a web geek who has been doing this sort of thing entirely too long. I rant, I muse, I whine. That is, I am not at all atypical for my breed.