Tag Archives: engineering

Ops is undervalued

I made the mistake of suggesting that there was a blog to be made from this tweet. This is that post.

People still underestimate the value in (Developer|Operator) Experience when building platforms and honestly it’s kinda shocking to me.

If you want to win mindshare you need to make your tools actually usable. If you don’t want to lose it you need the same.

First: I agree with the sentiment. Maybe not to the same extent as Danielle, but I fight the same battles in my day job. I wanted to say this now because, on reading the rest, you may think the opposite. What follows is an explanation of why this is a common situation. I don’t mean it as a justification.

In summary, making software work for ops teams is not a focus either for software companies or internal development teams because of at least three reasons:

  • It’s not a business driver
  • Ops are not the buyer
  • Engineering is run by developers

Naturally there are exceptions to these rules. Every company is different. These are observations, not rules.

Working at the “coal face” it’s easy to forget, but people don’t buy software. They buy a solution to a business problem1. These days, that solution is often software but you don’t buy a new product because it’s cool2.

A driver might be to get a report generated more quickly, or to provide a new service to paying customers, or to reduce the costs of some infrastructure3.

But, you argue, the ops team are the people who keep the system up and running. How can this new system generate reports or deliver a service if it’s not running reliably or has not been provisioned correctly?

You’d be right. Sadly, organisations are often not structured to recognise that fact. The IT teams tasked with making everything run smoothly are frequently in a completely different reporting hierarchy from “the business.” I put “the business” in quotes as I hear it described that way all the time, but it’s this us-versus-them philosophy that brings many issues, including how ops get sidelined4.

With “the business” and “ops” being in separate reporting structures, one or the other has to sign off on the purchase of new software (unless it goes way up the organisation) and that’s always going to be “the business.”

The buyers normally consult the ops team, but ultimately they’re going to put their own needs first. Objections that the ops team have will end up in the business case, but likely in an appendix that no one will ever read.

This makes no sense because, as we all know, most of the expense of a system occurs after go-live. But monitoring, management and deployment are not things at the top of mind of developers and business users.

But even in the IT organisation, the ops team frequently don’t get the attention they deserve either. Anecdotally, this is because IT leadership come from the development team. Their outlook on IT is therefore skewed towards making things rather than keeping them running. I see the same challenges for testing teams. Or, indeed, the lack of testing teams.

This lack of buy-in from the ops team is endemic. I see it in companies buying and using software. I see it in companies that make and sell software5.

At this point, what I would like to be able to do is say, “And the solution to this terrible problem is…” Sadly there is no easy answer. Saying “You should listen to your ops team” is both obvious and unhelpful. Making tools useful for the ops team to get mindshare is a good way to get your software on a shortlist but maybe not enough to clinch that sale.


  1. I’m talking here about software used in a business of course, but the difference between this and personal use isn’t as great as you might initially imagine. You buy a game to solve the “problem” of boredom. Very few people buy software because it’s software. ↩︎
  2. You might buy it because this, specific solution is cooler than the other options. ↩︎
  3. That is, even in the case where it’s the infrastructure that is being improved, the business case is not “make the ops team more efficient” but “make the cost of operations lower.” ↩︎
  4. There’s another blog in how toxic “the business” as a concept is. This isn’t that blog. ↩︎
  5. This is a chicken and egg problem. Do software companies fall into this trap because they’re run by developers? Or is it because they’re selling to companies who have already fallen into the trap? ↩︎

How to

“If you convert [your car] to run on copies of this book instead of gas, it will burn through 30,000 words per minute, several dozen times faster than the word consumption of a typical human.”

If you thought that “How to“, the follow-up to “What if…” would be more practical, then you’d be wrong.

Whether it’s chasing a tornado without getting up from your couch or moving your house with jet engines, Munroe takes another fun, inventive journey through science and maths. While it doesn’t quite hang together as well as “What if,” it still manages to amuse, educate1 and entertain.

There are so many good bits that it’s difficult to mention even a few highlights, but I think possibly my favourite part is where he fails to faze Colonel Chris Hadfield, even when asked how to land a space shuttle that’s attached to the carrier aircraft (response: “Easy peasy”).

If you’re at all interested in science or engineering, you should read this book (if you haven’t already). Just — please — don’t take the advice literally.


  1. I mean, not directly. You’re unlikely to have an exam where you need to know how to build a lava moat. But the thought-process in getting a serious answer to an absurd question absolutely has value. ↩︎

My delicious.com bookmarks for December 25th through January 9th

  • The Myth of Japan’s Failure – "Japan has succeeded in delivering an increasingly affluent lifestyle to its people despite the financial crash. In the fullness of time, it is likely that this era will be viewed as an outstanding success story."
  • Man Embraces Useless Machines, and Absurdity Ensues – Technology: making life simpler.
  • Merry – Sat here with my newborn son and wife, with all my family staying nearby, this post rang bells. It's sometimes important to realise what you have.

My delicious.com bookmarks for March 13th through March 14th

Communication

“Do you ever change this type of trade?”

I was sat on the trading floor discussing a new feature that I was implementing with the person who would be using it the most.

“No, never.”

This was one detail of the change that would have far-reaching consequences in the code. A “no” would mean a few days of development, a “yes” would indicate several weeks.

“Are you absolutely sure? You don’t change it even once a month?”

I knew they’d like the smaller estimate but, equally, I knew that I didn’t want to end up trying to implement several weeks worth of functionality in the smaller time frame. I’ve seen that kind of thing happen too often.

“John1, we don’t ever change these trades do we?”

John was the head trader. If anyone would know it would be him.

He rubs his head and leans back, thinking.

“Why would you want to? No, I can’t see us ever changing one.”

Of course, you can guess what happened thirty minutes into the first trading day with the new software.

So, what happened here? Why did we get the requirements so wrong? And why did we only find the mismatch after the system was moved into production?

Both the problem and the solution is the same thing: communication. Or more precisely, communicating using the right language.

Some developers are happy to only “speak” technical, proud that they are masters of their programming environment but ignorant of their users problems or how they really use the software.

Above I started out correctly, I was trying to understand the traders business and talking in terms of booking trades, positions, legal entities and a bunch of acronyms that would make even less sense out of context.

But I made one error: I used the word “change” without defining what I meant. I meant, well, any change. And so did they. Yet they didn’t consider moving a trade from one book to another to be a change and, unfortunately, I did.

You’d like to think that there were checks and balances in place to make sure that this kind of thing didn’t happen. And there were. In addition to informal and formal testing, there was over a week of “parallel running,” where the traders had to use the old and the new system together and check that the results were the same in both of them.

Were there any moved trades during this time? Of course. Why did the traders not notice? Well, it was about right; the differences, while present, were explicable and so not considered significant enough to mention even though I asked to hear about any problems at all.

So, again, communication. Or at least human nature. I wanted to hear about any differences but they tried to help me by only talking about differences that they couldn’t account for.

What’s the answer? Well, I’m not sure there’s an easy one. “Understanding your user” is a short, simple phrase but hides so much. If you spent the time to fully understood their job you probably wouldn’t have the time to do your own. But finding the balance is crucial.

  1. Names have been changed to protect the… well, they work for a bank so I hesitate to say “innocent” but you know what I mean. []