Tag Archives: engineering

Opinion

History

A few years ago I had a job where every new recruit would go through a long process of shock and gradual acclimatisation to the main software product.

What it did doesn’t matter as much as how it was built: it was an application developed on top of a proprietary programming language and user interface designer. The reaction was always the same. Why? Why?! Why would you reinvent Visual Basic on Unix? Why would you inflict a programming language even worse than Basic on developers?1

The answer, it turns out, is that the original developers were idiots.

No, of course that’s not true. But if that’s the case, then why did almost every developer start from that point of view when they first arrived at the company?

That brings us to Twitter and its new owner. One of his first public proclamations is to declare that there are too many micro-services running, and, worse, most of do nothing useful! The reply-guys all agree and, between them, argue that it’s entirely possible to rebuild Twitter from the ground-up in weeks, possibly even a weekend if given enough pizza and Blue Bottle.

Were the original developers of Twitter also idiots?

I don’t know as much about Twitter’s architecture, but I’d be willing to bet that, no, they were also not stupid.

If it’s not the original developers, what does it say about the critic? It says that they see the complexity but not the nuance. They see the current state but they do not see any of the decisions that lead up the current system. They see complexity, but without understanding the whole problem domain they don’t see why that complexity exists.

In the case of my job, the software predated Visual Basic, which is a pretty good reason for not using it. It also had to work on Unix and be editable on client sites without extra tooling. By the time I worked there, it may have been dated but it was in production at many clients. It worked. Sure, it’s not how you’d architect it now but the decisions that led to the design did make sense.

If it’s dated, then why not rewrite it? That has been covered many times before, but the short answer is that when you design it, you focus so much on the clean, new solution that you forget why you added the warts to the old system. The layers upon layers of fixes and enhancements represent real world experience. Those micro-services are there for a reason. Not understanding the reason doesn’t change that2.

This is not an argument against evolving the software, only that you should understand what you already have. Sometimes rewriting can be justified. Rationalising a bunch of micro-services isn’t always a ridiculous idea. But there’s an important difference between complex and complicated. Can you know which your inherited system is after a few days on the job?

  1. It was a stack-based language, along the lines of Forth and Postscript. Long time users could do amazing things with tiny amounts of code. I never quite got there. ↩︎

  2. Logical fallacy: argument from incredulity. ↩︎

Blog

Facts and Fallacies of Software Engineering

I’ll be honest: I wanted to like “Facts and Fallacies of Software Engineering” by Robert L. Glass more than I did. I’m not sure if it’s dated badly — it’s from 2002 — or I was in the wrong frame of mind, or something else, but it just didn’t work for me.

The book is structured as a list of facts grouped around areas such as “Management” and “Requirements.” For each fact, there is a discussion, the controversy, and then the sources and references. The writing aims to be friendly, but I found it a bit grating1.

Going back through the fifty-five facts and fallacies, I find that I agree with the majority of them2. Twenty years on from the original version, I suspect that some are less controversial than when it was originally published. There is some degree of (justified) scepticism about the newfangled “agile” process, which is now largely standard across the industry. Yet it’s also reassuring how little other things have changed. There is still little transfer between academia and industry. Quality continues to be a hot-button topic. And software maintenance rarely gets the love it deserves.

I’ve seen this book described as a classic, so maybe I’m missing something. There are many books I would suggest you read first. In the end, I didn’t like the writing, and I think that there are too few original insights in too many words.

  1. This is such a subjective thing. I think the objective in was modesty or self-deprecation, but it came across as boasting. ↩︎
  2. I’m resisting the temptation to argue against those that I don’t agree with. ↩︎
Opinion

Panic

The whole team got this email today. Okay, it wasn’t today and these are not the exact words, but it was something like this:

We have a serious regression in build 456. We have set the project back rather than taken it forward. We need the utmost focus and commitment on fixing it. We’ve broken it and we stay in the office until it’s fixed.

I’ve had a few of those messages over the years and while it’s intended to focus minds it often has the opposite effect. Let’s examine why.

Projects see the same mistakes made over and over and this email encompasses many of those sins; it’s one message but represents a microcosm of large part of my career.

Here are a few problems that I see immediately:

  • No problem definition
  • No person accountable
  • No next action
  • A deadline but no understanding of the work involved

This has a number of consequences.

There are studies that show if you have a heart attack in a crowded area you are less likely to receive life-saving CPR from a stranger than if you’re in an area with one other person.

In this case the passers by (the project team) don’t know that they’re needed. Without a problem definition I don’t know if the regression was caused by one of my changes or even if it affects my code. Without a person accountable everyone likely assumes that it’s someone else’s code. “Someone would have mentioned it if it was my code.” And with no “next action” it’s easy to assume that someone else will handle it.

Arguably the deadline is not really a deadline. What if the fix would take a week to implement? Instead it’s a target. You can’t take an estimate and reduce it to fit to an externally imposed date. It doesn’t work like that. You may hit your deadline if you’re lucky, but a good plan doesn’t need luck.

Even worse, the arbitrary deadline and lack of direction gives the entire project a sense of panic. I think the intention was urgency but urgency implies you know what you need to do and that you need to do it quickly. As we’ve seen, the task above is neither well defined nor assigned. The only clear things in the original email are the version that is broken and the deadline.

However, the biggest sin is questioning the commitment and competence of the people needed to resolve the issue. In my experience, this is rarely the case, yet asking the question can make it true. If you’re not trusted, why put in the extra work? Next time you need to make a change, are you going to do it the “right” way or the way with the absolute lowest risk? Putting in a lot of good work and then getting kicked for your efforts is not a good incentive for doing a job well.

Blog

The Art of Leadership

Before you ask, yes, it is weird that I’m reading a bunch of “management” books.

You can watch Michael Lopp’s career by following his various books. Start with “Being Geek,” the software developer’s career handbook. The move into management resulted in “Managing Humans.” And his promotion from manager to director and executive gets you “The Art of Leadership,” which is the book I recently finished.

My career has not followed the same trajectory. I continue to be an “individual contributor,” so why would I read this book?

Two basic reasons. First, Lopp is a great writer. He wraps the lessons around relatable stories, even if they don’t exactly mirror my experience. Secondly, to use a cliché, leadership comes from everywhere. I may not manage people, but I do have to lead. As a consultant, my whole job involves influence, persuasion and strategy.

In short, I don’t think you have to be a manager to get something out of this book, but if you like to sit in the corner and code all day, it’s unlikely to be your thing.

The book is structured into three sections, manager, director and then executive. Within each section, there are a bunch of small things that, done well, will result in great results (hence the sub-title). There are so many great parts that it would be easy to quote the whole book. I’ll refrain, but here are a few highlights and observations.

There are parallels with other books I’ve read recently. Chapter 15 “Saying the hard thing,” covers a lot of the same ground as “Radical Candor,” with many of the same positives.

The “faux zone” is very relatable. There are certainly times when I feel incredibly busy, but at the end of the day, I don’t feel like I got anything done. There are insights here that make me feel better about it.

A “Precious Hour” reminds us that being busy is not the same as being productive.

And, finally, this is so me: “I love to start new things, but I often lose interest when I can mentally see how the thing is going to finish, which might be weeks or months before the thing is actually done. Sorry. I’m getting better at this.” I remember I did one of those “type indicator” tests a long time ago, and one of the categories was “completer-finisher.” I immediately knew that I was the opposite of that.

As with all books like this, some of the suggestions I already do while others are not relevant; however, taken as a whole, there’s plenty of good advice.

Opinion

Ops is undervalued

I made the mistake of suggesting that there was a blog to be made from this tweet. This is that post.

People still underestimate the value in (Developer|Operator) Experience when building platforms and honestly it’s kinda shocking to me.

If you want to win mindshare you need to make your tools actually usable. If you don’t want to lose it you need the same.

First: I agree with the sentiment. Maybe not to the same extent as Danielle, but I fight the same battles in my day job. I wanted to say this now because, on reading the rest, you may think the opposite. What follows is an explanation of why this is a common situation. I don’t mean it as a justification.

In summary, making software work for ops teams is not a focus either for software companies or internal development teams because of at least three reasons:

  • It’s not a business driver
  • Ops are not the buyer
  • Engineering is run by developers

Naturally there are exceptions to these rules. Every company is different. These are observations, not rules.

Working at the “coal face” it’s easy to forget, but people don’t buy software. They buy a solution to a business problem1. These days, that solution is often software but you don’t buy a new product because it’s cool2.

A driver might be to get a report generated more quickly, or to provide a new service to paying customers, or to reduce the costs of some infrastructure3.

But, you argue, the ops team are the people who keep the system up and running. How can this new system generate reports or deliver a service if it’s not running reliably or has not been provisioned correctly?

You’d be right. Sadly, organisations are often not structured to recognise that fact. The IT teams tasked with making everything run smoothly are frequently in a completely different reporting hierarchy from “the business.” I put “the business” in quotes as I hear it described that way all the time, but it’s this us-versus-them philosophy that brings many issues, including how ops get sidelined4.

With “the business” and “ops” being in separate reporting structures, one or the other has to sign off on the purchase of new software (unless it goes way up the organisation) and that’s always going to be “the business.”

The buyers normally consult the ops team, but ultimately they’re going to put their own needs first. Objections that the ops team have will end up in the business case, but likely in an appendix that no one will ever read.

This makes no sense because, as we all know, most of the expense of a system occurs after go-live. But monitoring, management and deployment are not things at the top of mind of developers and business users.

But even in the IT organisation, the ops team frequently don’t get the attention they deserve either. Anecdotally, this is because IT leadership come from the development team. Their outlook on IT is therefore skewed towards making things rather than keeping them running. I see the same challenges for testing teams. Or, indeed, the lack of testing teams.

This lack of buy-in from the ops team is endemic. I see it in companies buying and using software. I see it in companies that make and sell software5.

At this point, what I would like to be able to do is say, “And the solution to this terrible problem is…” Sadly there is no easy answer. Saying “You should listen to your ops team” is both obvious and unhelpful. Making tools useful for the ops team to get mindshare is a good way to get your software on a shortlist but maybe not enough to clinch that sale.

  1. I’m talking here about software used in a business of course, but the difference between this and personal use isn’t as great as you might initially imagine. You buy a game to solve the “problem” of boredom. Very few people buy software because it’s software. ↩︎
  2. You might buy it because this, specific solution is cooler than the other options. ↩︎
  3. That is, even in the case where it’s the infrastructure that is being improved, the business case is not “make the ops team more efficient” but “make the cost of operations lower.” ↩︎
  4. There’s another blog in how toxic “the business” as a concept is. This isn’t that blog. ↩︎
  5. This is a chicken and egg problem. Do software companies fall into this trap because they’re run by developers? Or is it because they’re selling to companies who have already fallen into the trap? ↩︎
Blog

How to

“If you convert [your car] to run on copies of this book instead of gas, it will burn through 30,000 words per minute, several dozen times faster than the word consumption of a typical human.”

If you thought that “How to“, the follow-up to “What if…” would be more practical, then you’d be wrong.

Whether it’s chasing a tornado without getting up from your couch or moving your house with jet engines, Munroe takes another fun, inventive journey through science and maths. While it doesn’t quite hang together as well as “What if,” it still manages to amuse, educate1 and entertain.

There are so many good bits that it’s difficult to mention even a few highlights, but I think possibly my favourite part is where he fails to faze Colonel Chris Hadfield, even when asked how to land a space shuttle that’s attached to the carrier aircraft (response: “Easy peasy”).

If you’re at all interested in science or engineering, you should read this book (if you haven’t already). Just — please — don’t take the advice literally.

  1. I mean, not directly. You’re unlikely to have an exam where you need to know how to build a lava moat. But the thought-process in getting a serious answer to an absurd question absolutely has value. ↩︎