The Dark Side of Puppet Forge (et. al.)

Let me preface this by saying I don’t think this is unique to Puppet Forge, and the problem isn’t even with Puppet Forge itself. Puppet Forge provides an excellent resource for the Puppet community, and I think it provides tremendous value.

But I think that a push towards “configuration management everywhere”, combined with reductions in resources leads folks to think that resources like PuppetForge are a panacea.

Sherman, set the Wayback machine….

A couple decades ago, as Visual Basic started to become popular, and various development platforms were created for simply making programming “attach this pre-made module to that pre-made module and plug in a couple values”, the industry (rightly) cried foul that people would confuse an ability to assemble the software-development version of Lego with actual programming. The difference in ability between “constructing a house from scratch” versus “sticking together some pre-fab materials” was an apt analogy.  Some people didn’t necessarily understand how their code worked, or even what it was doing, and still bandied around the title of “programmer”. To a large extent, we’ve dissuaded that sort of practice from continuing (near as I can tell).

In the system administration community, we need to be wary of falling into the same trap. There’s a world of difference between going to PuppetForge, grabbing a pre-made manifest for managing “OpenLDAP”, and that of installing OpenLDAP (even just from yum/apt), and then configuring it yourself. (Better still would be the level of knowledge imparted by compiling from source, but install/configure is a good middle-ground).

When sysadmin’ing a given package becomes “I grabbed the module from PF and installed it, and now it works”, a lot of the knowledge necessary for day to day maintenance is simply missing. What files were installed where? Why were they installed there? Configuration options in a Puppet module that seem benign might actually have much longer-lasting ripple effects than you can realize.

As I said, Puppet Forge is an excellent resource. But it is no substitute for understanding how to do the install in the first place. Puppet Forge should be used for ideas about “how to configure YOUR module,” as opposed to being “the module you use”.

While that seems like I’m saying “reinvent the wheel every time,” because of some weird theme of “not invented here” syndrome, I’m not. What it means is that after the application is installed, months down the road, there’s going to be some sort of problem with it. And if all you know about the configuration of the application is what was exposed to you (or worse, your predecessor) by the pre-made module that was downloaded, your ability to diagnose problems with that application is going to be substantially reduced. Having configured it yourself, from the ground up, and then built a Puppet module on your own (or by referencing existing modules) to recreate that config is most definitely the path to success.

As a profession, we need to be wary of falling into the trap of “Oh this is an easy and quick to solve this, and I’ve got so much other stuff to get done today before I go home.” This is the sort of problem which silently lurks below the surface, and wreaks untold damage when it goes foul.

We’re still early enough in the adoption of “config management everywhere” that it’s not too late to change the direction our collective mindset is heading; to ensure that we don’t end up in a realm of tech-skills disparity the way the programming industry did in the not-too-distant past.

The Ecology of the Goa’uld

(Non-sci-fi-nerds.. just turn away, this is the sort of thing you probably beat up kids in high school for talking about).

So I’ve been re-watching Stargate SG-1 (Wikipedia) (IMDb) on my morning and afternoon commute, and something has been bugging me.  I dug around on SG-1 fan sites and couldn’t find a satisfactory answer, hell even a discussion, of the topic.

Throughout the course of the series we encounter maybe a dozen Goa’uld – the System Lords. Which gives us the impression that Goa’ulds, in hosts, are uncommon. This would make sense as well, since the Goa’uld fool everyone into thinking they are gods, and there can’t be a lot of gods.

But we see literally hundreds of thousands of Jaffa throughout the show. The Jaffa carry, in their bellies, Goa’uld symbiotes that are maturing.

And so my question is:  Where the fuck are those symbiotes going when they mature?

Sure, some of them are dying – en masse – in various conflicts. But surely some of them are surviving, reaching maturity, and … then what?

At the rate symbiotes are growing inside Jaffa there should be planets full of matured, fully hosted, Goa’uld (just, presumably, ones who aren’t System Lords, and aren’t going around pretending to be gods, unless the faithful are supposed to believe in a pantheon of a billion all-powerful deities, which would strain credibility.

Why have we never even heard of this society in the course of the show? It seems to be a glaring plot-hole that I’m just surprised nobody has mentioned before. I feel like I must be missing something.

On Yahoo and PRISM, and the Art of Playing Chicken

I was reading the New York Times article which reveals that a secret intelligence court threatened to fine Yahoo! $250,000 a day for failing to turn over confidential customer/user data.

Now, one can hardly fault a publicly traded company for not wanting to incur nearly $2,000,000 a week in fines. That’s a hefty chunk of change to play chicken with.

But here’s a take-away for future companies in this position: PLAY CHICKEN. YOU WILL WIN.

First, the only way the government can collect is either by seizing the cash outright (which will expand the number of people who know there’s something going on between the NSA and the company in question).

Second, if they DO, remind them that — as a publicly traded company — you’re going to have to mention this fact in the next quarter’s SEC filings. That’s a material change to cash-flow that it would be a felony to conceal from shareholders, and which would show up in your next annual audit, the results of which are public information, anyway. “Of course, we won’t name you, we’ll simply say, ‘We are being fined based on court orders from an intelligence court which we can’t even confirm the existence of.'”  Let the NSA stew over how they’re going to react to the bad press on that.

Remember that the most important thing to the intelligence community is the cover of darkness. That’s one of the lessons of the Snowden disclosure. If you are willing to stand up to them, chances are, they are not going to take the chance that their bullying tactics, and the reason for those bullying tactics, will be exposed in the light of day.

Asking For Help From Your Customers

There’s a trap that a lot of companies fall into. In one way or another – whether it’s surveys, or forums, or focus groups, or whatever – companies ask their customers or users for feedback, suggestions, “ways to make things better”. This, in and of itself, is awesome. It’s how companies can best determine what their paying customers are looking for, direct feedback-loop closure from the people who pay the bills to make it all possible.

But too many companies – both ones I’ve worked for and ones I’ve been a customer of – will respond to a lot of suggestions with answers like:

  • “That’s just not feasible.”
  • “That can’t be done.”
  • “That doesn’t scale.”
  • “That makes things complicated.”
  • “We can’t do that.”

And all of those things may be true, but all of those statements, as written or spoken, are “shutting down the conversation” statements. They don’t brook any sort of follow-up dialogue. They tell your customer “that idea is SO bad, that I’m not even going to explain to you how bad it is and why.”

Contrast those with:

  • “That’s just not feasible, because the number of volunteers it would take to man those areas would be more than we have.”
  • “That can’t be done, because there’s a regulatory requirement to keep portions of that data private.”
  • “That doesn’t scale, because once you’ve got more than a couple hundred thousand rows in that table, your indices are going to look like shit.”
  • “That makes things complicated, because then we have to deal with two completely different products that go to the printers, two sets of inventory, etc.”
  • “We can’t do that, because the capital expenses of the widgets are too high.”

You can see how each of the second set leaves the door open to discussion. It says “Your idea is good, but we thought about that before, and we rejected it not out of hand because it’s just a bad idea, but for the following reason…,” leaving the possibility for the suggester to reply in a couple different ways:

  • “Ah, shit… I hadn’t thought of that, you’re right. Never mind.”
  • “That’s true, but maybe we don’t need that particular piece of data that’s regulatory-encumbered, we’ll just use all the non-encumbered data, and that’s actually enough.”
  • “Man, those indices are gonna suck. I wonder if there’s a way to make them easier to manage and be more efficient…?”
  • “You can probably get widgets as cheap as $0.whatever … is that more or less than what they were going for the last time you looked at this problem?”

Even if you don’t believe they are, and you rarely will — treat your customer as though they are at least as smart as you are. Yes, your company has been doing this for a long time. Yes, you’ve got really bright, really focused people working on these problems day in and day out. But you’re not the smartest people on the planet. There’s only one guy who is, and he’s definitely keeping a low profile these days it seems. Walk your customers and users through the reasons why you’ve considered that idea in the past and rejected it. Maybe they completely agree with you and just accept the answer. Maybe they point out some flaws in your internal logic, and a dialogue ensues, where it’s still a bad idea, and now you have another piece of data about why it’s a bad idea. But, maybe they have a completely novel way of solving a problem, which you haven’t thought of before. Taking that suggestion achieves two very valuable things:

  1. You’ve improved the product offering in a way that is directly valuable to your customer base. It was their idea, after all.
  2. You’ve demonstrated the willingness to do so in a very tangible, concrete fashion.

There’s certainly always going to be “vetoes”, but any time you can back up your veto with the “why”, it goes down much smoother with the folks who have to hear it.

Blog Version 3.0

With my departure from the land of Facebook, I decided to spend some time on a Sunday morning and re-vamp the blog a little bit, since it’ll probably be the new place for my sharing of thoughts. I got neck-deep in it pretty fast, and it became clear it was now a “whole new thing” as it were.

I don’t yet know what all will end up here, but I wanted to make it someplace I felt interested in again as opposed to a snapshot of what a web page looked like in 2011 or so when I moved this to WordPress.


Quitting Facebook

I deactivated my Facebook account last night.

This story on Slate does a pretty good job of summing up what I would call “the straw that broke the camel’s back” for me. Chuq von Rospach also summarizes things pretty well, also.

I guess this means I might start using this thing more often again. After all, I’m still going to have the occasional thing I want to “say”, but Google+ as a forum is a wasteland (and – frankly – Google is just as bad as Facebook, they’re just smart enough not to say out loud in public what they do).

I don’t harbor any belief that my act of defiance will change the world or anything. I just know that I can’t sit by and be a rat in their lab.

How I Got Where I Am, A Tribute To Steve

It’s 1981. ish. It’s kind of blurry now as I look back on it.

I’m in 5th grade. I still think that what’s passing for adult contemporary radio is cool, because my parents listen to it, and I’m not nearly hip enough to know that’s actually the kiss of death for coolness. I still spend a whole mess of time playing sports. I’ve got a pretty active circle of friends, and I’m not a social pariah at school. In short, I’m your typical early-80s 5th grader.

And then, for reasons that to this day I still don’t fully understand, I’m selected to be part of an experiment. Mrs. Miller has navigated some sort of grant- or aid-program and acquired a brand new Apple II computer. It’s sitting in one of the lower-grade classrooms, and she is the sole arbiter of who is permitted to touch this magical beast. There’s only two students who are allowed to touch it: her nephew [nepotism FTW!] and me.

To say that I was all over that like white on rice is an understatement that makes “epic proportions” seem small. The two of us are writing programs, playing games (Bobby Miller has apparently acquired an illicit copy of Castle Wolfenstein and we’ll play that from time to time when nobody’s looking).

The next year, the computer has moved to the library, and Bobby and I are put in charge of helping a bunch more, although still relatively few, of the students deemed “Gifted and Talented” by the school learn how to use the computer.  Tron has just come out, and I’m still young and naive enough not to realize that the fictionalized commands Flynn uses in the novelization don’t actually do shit, and I’m really disappointed when I type them into the Apple II when nobody’s around and find that precisely nothing has happened.

When I get to Junior High, I get my first taste of an actual “computer lab”. This is, apparently, what they’ve been prepping us for the past school-year and a half, to have a small core of students who really know what these things are and what to do with them. Over the next six years of Jr. and Sr. High School, it’s here that I’ll meet some of my life-long friends. It’s here that I’ll spend so much time that hanging out with my core “neighborhood friends” will basically go by the wayside, that I don’t really play sports much any more, and that I begin to show all the classic signs of becoming the social pariah that will later simply be called “Computer Geek”.

We can’t afford an Apple computer ourselves at home, so I end up buying a Commodore VIC-20 computer instead. It’s fun, and make no mistake, I have a lot of good times with that computer, and its various Commodore-made successors, but I still secretly wished I could have had an Apple.

It’s in high school that I start entering into computer-programming contests that the school used to run each year. I enter into it every single year (except the year I “went pro” because a computer store in Rhinebeck was having a similar contest the same day, but with cash prizes, baby!). And it’s in high school that I really decide, as any computer-oriented kid in the Hudson Valley in the 80s would, “I want to work for IBM some day,” not knowing that IBM’s own internal troubles are going to make that a pipe dream in about three more years.

When I get out of high-school, I go to college for computer science. But I’m a fuck-up, and basically get kicked out in a scene reminiscent of Animal House (“GPA… Zero. Point. Zero.”)  I then end up going through the usual post-high-school-no-college series of dead-end jobs until I finally end up working part-time for a tiny local Internet Service Provider. This was perfection – I got a free account since I worked there (and my day job wasn’t paying me enough to pay for one) and I got to really get back into computers “for a living”.

I got to enjoy the entire life-cycle of Apple computers, from hot upstart, to the time when I (and everyone else with any sense) abandoned them as completely uncool pieces of crap. Later, once Steve was back, I’d eventually become an “Apple bigot”, refusing to use any computer that didn’t have that familiar fruit-shaped logo on it, because I knew that (once again) it stood for quality hardware that was powerful, easy to use, and stable.

I would climb the entire ladder of IT management, starting off as a help-desk monkey, then working as a Perl programmer, web developer, Linux system administrator, all of which led me through varying levels of responsibility until I got to what I’ve spent the last six years doing, managing great teams of network and systems people at various organizations, a dream that started over thirty years ago.

A dream made possible by – heck, a dream carved out of whole cloth by – a pair of hippies in a garage who decided they should be aggressive about getting cheap Apple II computers into the hands of educators.

So, Steve… Thanks for giving this geeky kid a vision of what he wanted to do for a living, and providing the tools for thirty years (more or less) to help me do it.