Tag Archives: Computing

History

A few years ago I had a job where every new recruit would go through a long process of shock and gradual acclimatisation to the main software product.

What it did doesn’t matter as much as how it was built: it was an application developed on top of a proprietary programming language and user interface designer. The reaction was always the same. Why? Why?! Why would you reinvent Visual Basic on Unix? Why would you inflict a programming language even worse than Basic on developers?1

The answer, it turns out, is that the original developers were idiots.

No, of course that’s not true. But if that’s the case, then why did almost every developer start from that point of view when they first arrived at the company?

That brings us to Twitter and its new owner. One of his first public proclamations is to declare that there are too many micro-services running, and, worse, most of do nothing useful! The reply-guys all agree and, between them, argue that it’s entirely possible to rebuild Twitter from the ground-up in weeks, possibly even a weekend if given enough pizza and Blue Bottle.

Were the original developers of Twitter also idiots?

I don’t know as much about Twitter’s architecture, but I’d be willing to bet that, no, they were also not stupid.

If it’s not the original developers, what does it say about the critic? It says that they see the complexity but not the nuance. They see the current state but they do not see any of the decisions that lead up the current system. They see complexity, but without understanding the whole problem domain they don’t see why that complexity exists.

In the case of my job, the software predated Visual Basic, which is a pretty good reason for not using it. It also had to work on Unix and be editable on client sites without extra tooling. By the time I worked there, it may have been dated but it was in production at many clients. It worked. Sure, it’s not how you’d architect it now but the decisions that led to the design did make sense.

If it’s dated, then why not rewrite it? That has been covered many times before, but the short answer is that when you design it, you focus so much on the clean, new solution that you forget why you added the warts to the old system. The layers upon layers of fixes and enhancements represent real world experience. Those micro-services are there for a reason. Not understanding the reason doesn’t change that2.

This is not an argument against evolving the software, only that you should understand what you already have. Sometimes rewriting can be justified. Rationalising a bunch of micro-services isn’t always a ridiculous idea. But there’s an important difference between complex and complicated. Can you know which your inherited system is after a few days on the job?


  1. It was a stack-based language, along the lines of Forth and Postscript. Long time users could do amazing things with tiny amounts of code. I never quite got there. ↩︎

  2. Logical fallacy: argument from incredulity. ↩︎

Panic

The whole team got this email today. Okay, it wasn’t today and these are not the exact words, but it was something like this:

We have a serious regression in build 456. We have set the project back rather than taken it forward. We need the utmost focus and commitment on fixing it. We’ve broken it and we stay in the office until it’s fixed.

I’ve had a few of those messages over the years and while it’s intended to focus minds it often has the opposite effect. Let’s examine why.

Projects see the same mistakes made over and over and this email encompasses many of those sins; it’s one message but represents a microcosm of large part of my career.

Here are a few problems that I see immediately:

  • No problem definition
  • No person accountable
  • No next action
  • A deadline but no understanding of the work involved

This has a number of consequences.

There are studies that show if you have a heart attack in a crowded area you are less likely to receive life-saving CPR from a stranger than if you’re in an area with one other person.

In this case the passers by (the project team) don’t know that they’re needed. Without a problem definition I don’t know if the regression was caused by one of my changes or even if it affects my code. Without a person accountable everyone likely assumes that it’s someone else’s code. “Someone would have mentioned it if it was my code.” And with no “next action” it’s easy to assume that someone else will handle it.

Arguably the deadline is not really a deadline. What if the fix would take a week to implement? Instead it’s a target. You can’t take an estimate and reduce it to fit to an externally imposed date. It doesn’t work like that. You may hit your deadline if you’re lucky, but a good plan doesn’t need luck.

Even worse, the arbitrary deadline and lack of direction gives the entire project a sense of panic. I think the intention was urgency but urgency implies you know what you need to do and that you need to do it quickly. As we’ve seen, the task above is neither well defined nor assigned. The only clear things in the original email are the version that is broken and the deadline.

However, the biggest sin is questioning the commitment and competence of the people needed to resolve the issue. In my experience, this is rarely the case, yet asking the question can make it true. If you’re not trusted, why put in the extra work? Next time you need to make a change, are you going to do it the “right” way or the way with the absolute lowest risk? Putting in a lot of good work and then getting kicked for your efforts is not a good incentive for doing a job well.

Project versus Product

With the fuss about the Log4Shell vulnerability finally dying down, it’s time to step back and take a good, long think about what happened and, more importantly, what can be done to stop it from happening again.

Sadly the prognosis is not good. The tl;dr is both simple and obvious: we simultaneously like free stuff and getting paid for our own work.

Most companies treat open source software exactly the same as commercial software but with a much lower purchase cost. When the software goes wrong, we want someone else to fix it for us. Unfortunately, sometimes we don’t even know where the software comes from. In the case of log4j, it’s run by volunteers. There is no 24/7 help desk with eager employees waiting to take your call.

But even if there is a company that backs the project, one that does have engineering, QA and support staff, is it reasonable to expect an immediate fix to a vulnerability?

Whether a company backs it or not, using open source software is more like being part of a community than being a customer. I came across this phrase about free software a few years ago: “If it breaks, then you get to keep both pieces.” It’s very apt here.

My small part in trying to fix this perception is by calling free software projects and commercially supported versions products or services.

The idea here is that the word “project” implies some degree of a work in progress, one that requires effort from all stakeholders.

In my Day Job1, I often see companies expecting to get commercial quality support for free because the software is free. By “commercial quality” I don’t mean “good.” The level of knowledge and support provided by most free projects is phenomenal. Instead, I mean that there are service-level agreements and guarantees of service within a particular time frame.

As it says in “Cloud Without Compromise”:

But there is something to the old adage that “You get what you pay for.” (… there’s a world of difference between building systems for pet projects versus designing for the needs of enterprise.)

Those service level agreements come with a cost. If you want a fix or an enhancement, you’re welcome to ask a project for it, but it might never come or, if it does, maybe not on a timescale that helps you. The rules change when you pay for the help.

My hope is that those companies that only ever take from open source projects and never contribute learned a lesson. By helping keep the project healthy, you don’t even need to be altruistic. Think of it as insurance.

If you use the software, there are many ways you can “pay it forward”: share enhancements and fixes, write documentation, share your knowledge and experience to bring in more users, or help other users with community support. But if there’s a mechanism to pay for it, the simplest way for most users is cold, hard cash.


  1. An open source database, so I have both skin in the game and bias. However, I think I’d be saying the same thing even if I worked for a vendor of closed source software. ↩︎

Security by Scapegoat

As is common these days, I was complaining about something on Twitter.

It’s easy to complain about security practices which, if I’m honest, is why I do it. But there is an important point, one that I included in a follow-up tweet:

The security team in many companies models itself on the DUP. Say no to everything. But – and this is the key – offer no alternative.

The tweet above is about passwords but I see it everywhere. Another common one is transferring files. I understand why sharing files can be problematic. Confidential data can be exported, either deliberately or accidentally. Viruses can be imported. Security defects can be exploited.

So yes, blocking OneDrive or DropBox is part of the job. What is missing is a legitimate alternative.

Security teams should be enabling staff to safely perform their jobs. Instead, they block the insecure methods and stop.

If I need to share a file or remember a complex password and you don’t provide suitable tools, you did not prevent a security problem. You forced people to write their password on a PostIt note and stick it on their monitor. You pushed someone to use some dodgy new file sharing service that you haven’t heard of.

In the attempts to make the system more secure, you, best case, prevented someone from doing their job. Worst case, you pushed someone into doing something insecure.

In either case, you effectively delegated security to everyone else.

The Art of Leadership

Before you ask, yes, it is weird that I’m reading a bunch of “management” books.

You can watch Michael Lopp’s career by following his various books. Start with “Being Geek,” the software developer’s career handbook. The move into management resulted in “Managing Humans.” And his promotion from manager to director and executive gets you “The Art of Leadership,” which is the book I recently finished.

My career has not followed the same trajectory. I continue to be an “individual contributor,” so why would I read this book?

Two basic reasons. First, Lopp is a great writer. He wraps the lessons around relatable stories, even if they don’t exactly mirror my experience. Secondly, to use a cliché, leadership comes from everywhere. I may not manage people, but I do have to lead. As a consultant, my whole job involves influence, persuasion and strategy.

In short, I don’t think you have to be a manager to get something out of this book, but if you like to sit in the corner and code all day, it’s unlikely to be your thing.

The book is structured into three sections, manager, director and then executive. Within each section, there are a bunch of small things that, done well, will result in great results (hence the sub-title). There are so many great parts that it would be easy to quote the whole book. I’ll refrain, but here are a few highlights and observations.

There are parallels with other books I’ve read recently. Chapter 15 “Saying the hard thing,” covers a lot of the same ground as “Radical Candor,” with many of the same positives.

The “faux zone” is very relatable. There are certainly times when I feel incredibly busy, but at the end of the day, I don’t feel like I got anything done. There are insights here that make me feel better about it.

A “Precious Hour” reminds us that being busy is not the same as being productive.

And, finally, this is so me: “I love to start new things, but I often lose interest when I can mentally see how the thing is going to finish, which might be weeks or months before the thing is actually done. Sorry. I’m getting better at this.” I remember I did one of those “type indicator” tests a long time ago, and one of the categories was “completer-finisher.” I immediately knew that I was the opposite of that.

As with all books like this, some of the suggestions I already do while others are not relevant; however, taken as a whole, there’s plenty of good advice.

Cloud Without Compromise

A couple of years ago I did a conference talk called “On Cloud Nine: How to be happy migrating your in-memory computing platform to the cloud.” I wish I’d had “Cloud Without Compromise” back then. It covers much of the same ground but, as you’d expect in a book rather than a forty minute conference talk, in much greater depth. More importantly, it puts some concepts into context much more clearly that I did, either by explaining it better or by giving it a good name.

One of the main takeaways of the book, something that is mentioned throughout, is the mantra “Cloud is a capability rather than a destination.” In my talk I hint strongly at that but never made it explicit. I always felt that “the cloud is just somebody else’s computer” didn’t fully encapsulate the magnitude of change but wasn’t able to articulate it concisely. Well, here’s the line to use.

This book took a long time to read. Not because it’s badly written or exceptionally long but because every few paragraphs I had had to stop and think about the implications, how the subject applied to my recent work or research a new tool that I had not come across previously.

There have not been many technical books I’ve read recently that have had this impact. Naturally not everything was new to me, but even then rephrasing a concept or putting it into context can be immensely useful.

It is also very approachable. There’s a nice appendix on some of the more technical aspects, there are some nice anecdotes and even a little humour.

“You’re the most unromantic person I know.”

If there’s a criticism, it’s that some parts read as an advertisement for IBM and RedHat. The last chapter in particular, which is about automation, could be a White Paper for Ansible. Sure, the focus of the book is on hybrid cloud, where the other big vendors are pushing their own agenda, but the OpenShift advocacy would be less signifiant were it not for the fact that a couple of the authors work for IBM. On the one hand you have to write what you know. On the other, shilling your employers products in what’s supposed to be a vendor neutral book sits awkwardly.

For what it’s worth, the advice does not appear to be incorrect but I wish there had been more discussion about competing products or they’d stayed clear of talking specifics at all.

Overall, though, “Cloud Without Compromise” comes highly recommended.