Tag Archives: Opinion

History

A few years ago I had a job where every new recruit would go through a long process of shock and gradual acclimatisation to the main software product.

What it did doesn’t matter as much as how it was built: it was an application developed on top of a proprietary programming language and user interface designer. The reaction was always the same. Why? Why?! Why would you reinvent Visual Basic on Unix? Why would you inflict a programming language even worse than Basic on developers?1

The answer, it turns out, is that the original developers were idiots.

No, of course that’s not true. But if that’s the case, then why did almost every developer start from that point of view when they first arrived at the company?

That brings us to Twitter and its new owner. One of his first public proclamations is to declare that there are too many micro-services running, and, worse, most of do nothing useful! The reply-guys all agree and, between them, argue that it’s entirely possible to rebuild Twitter from the ground-up in weeks, possibly even a weekend if given enough pizza and Blue Bottle.

Were the original developers of Twitter also idiots?

I don’t know as much about Twitter’s architecture, but I’d be willing to bet that, no, they were also not stupid.

If it’s not the original developers, what does it say about the critic? It says that they see the complexity but not the nuance. They see the current state but they do not see any of the decisions that lead up the current system. They see complexity, but without understanding the whole problem domain they don’t see why that complexity exists.

In the case of my job, the software predated Visual Basic, which is a pretty good reason for not using it. It also had to work on Unix and be editable on client sites without extra tooling. By the time I worked there, it may have been dated but it was in production at many clients. It worked. Sure, it’s not how you’d architect it now but the decisions that led to the design did make sense.

If it’s dated, then why not rewrite it? That has been covered many times before, but the short answer is that when you design it, you focus so much on the clean, new solution that you forget why you added the warts to the old system. The layers upon layers of fixes and enhancements represent real world experience. Those micro-services are there for a reason. Not understanding the reason doesn’t change that2.

This is not an argument against evolving the software, only that you should understand what you already have. Sometimes rewriting can be justified. Rationalising a bunch of micro-services isn’t always a ridiculous idea. But there’s an important difference between complex and complicated. Can you know which your inherited system is after a few days on the job?


  1. It was a stack-based language, along the lines of Forth and Postscript. Long time users could do amazing things with tiny amounts of code. I never quite got there. ↩︎

  2. Logical fallacy: argument from incredulity. ↩︎

In The Open

I recently shared a blog post entitled “The Most Successful Developers Share More Than They Take” with the comment:

I try to practice “public by default” though, because of my work, it’s often “on the internal wiki” rather than fully open.

Unfortunately the article spends a lot of time talking about blogging and podcasting which, perhaps, undermined the point I was trying to make. If you want to write blogs, speak on podcasts, and present at conferences, good luck to you1. Not everyone will want to do those things, and that’s fine. I’m not advocating for that. I think most people can do what I meant.

Here’s the key point: make your “content” as widely available as practicable. Allow people to pull when it’s convenient for them rather than you push the information you assume they’d be interested in.

In this context, “public” doesn’t have to mean on the internet or even visible to your entire company. Nor does it mean pushing it to everyone. Updates do not need to land in everyone’s inbox.

Here are a few examples.

I work on multiple projects with a number of different clients. When I make notes, or update the status, or write meeting minutes, I put them on the company wiki rather than keep them on my local machine. My manager might be interested in how often I’m meeting with a specific client. The product team might be interested to learn which clients are using Kubernetes. I wouldn’t share most of this outside the company, but internally it’s not confidental.

If I build a small demo for a client or play with some software, I push my toy project to GitHub. Depending on what it is, it might be limited only to my team, more widely to any of my colleagues or it might be public, but I’ll be as open with it as I can.

If I’m researching something, a new technology or how to implement a particular use case, I’ll put my notes on the wiki.

If I ask a question, I will typically ask it in a public Slack channel rather than as a direct message.

An important aspect of all of these things is that I was already typing the information. The only difference is that instead of keeping it on my local machine or sharing with individuals, it’s “public.”

It means that other people can see the current state of my projects without asking for it. This immediately benefits me because I’m lazy. But in a distributed environment, where timezones are significant, it can save everyone time.

Asking questions in public can get answers from unexpected sources. That new guy might have experience you didn’t know about. Someone in a nearby timezone might get you an answer hours earlier than you were expecting. The person you would have asked might not know or be on vacation.

There are downsides, of course. If you ask a stupid question in public, then everyone will see how dumb you are. Your notes might document a terrible, old technology that you shouldn’t be using at all, or your solution might fail horribly.

But here’s the thing: you’re not stupid. I bet other people have a similar misunderstanding. And the journey itself can be interesting. As Kepler noted:

“What matters to me is not merely to impart to the reader what I have to say, but above all to convey to him the reasons, subterfuges, and lucky hazards which led me to my discoveries.”

Those “lucky hazards” might help other avoid the same mistakes. Can we fix the documentation? Include it in the company induction? Is there a blog or a conference talk in it?2 These steps may require a little extra work but they have benefits for everyone, from future you, to your colleagues and your customers.

Someone is wrong on the internet.

The other thing is that it’s a good strategy for getting the right answer. People can be too busy to respond, right up to the point where they find that Someone On The Internet Was Wrong. People are more likely to offer to fix your work more readily than they will be to come up with a working solution from scratch.

What if no one looks up your status updates? What happens when your notes go unread? Well… nothing. You were already writing the notes and no one except you read them. Worst case, you’re exactly where you were.

In short, this is a terrible process if you want to be seen as being right all the time. However, if you value getting to the right answer and acknowledge that you’re a fallible human, if your ego can handle it, then I find it works well.

And, best of all, there is no need to speak on a podcast or to have a website.


  1. Again, possibly undermining my argument, I do write blogs — hello! — and have spoken at conferences. I’ve never appeared on a podcast, though! ↩︎

  2. I said I wasn’t advocating podcasting or blogging, but that doesn’t mean you shouldn’t if it’s the best way of sharing the information. ↩︎

Twitter

Sometimes it’s only when you start writing about a subject that you truly understand your opinion. That’s the approach I’m taking to answering the question: are you going to leave Twitter?

A few people have asked me in the last couple of months and the only response I have is that I’m not jumping ship and closing my account immediately.

But as the weeks have progressed, as I’ve written this piece, my thinking has evolved. It’s not that I’m going to immediately close my account but I can see The End approaching. Indeed, my usage of Twitter has dropped considerably.

When Twitter was delisted from the stock market, the concept at the top of my mind was this: can you judge a company on the person or entity that owns it?

Twitter has been badly managed or owned by incredibly rich people (or both) for a long time, but they still have millions of users. Is a change of rich person really that significant?

Is the fact that Musk isn’t terribly likeable a factor? Many people bought products from Apple even though Jobs was famous for pushing people to breaking point. You can appreciate the vision even if you couldn’t work for the individual.

To be clear, I’m not saying that no-one avoided Twitter and Apple for these reasons. I’m sure there are some, but not me and not millions of others. Is there a line that he could cross where I would leave immediately? Yes, and, in fairness, he’s got pretty close by allowing back some of the extremists who have been banned.

And, circling back to the management, Twitter has been a mess pretty much since the beginning. They seem to have difficulty shipping anything. They’ve largely eliminated the “fail whale” but what big, beneficial features have come since? The algorithmic timeline?1

Like it or not, maybe the company needs shaking up.

Though, starting on the “cons” side, shaking up the company like this likely isn’t what is needed. It is the tech equivalent of the Brexit “solution” to Britain’s problems. Needing change isn’t the same as supporting chaos.

I don’t understand what over seven thousand people do at Twitter, but neither did Musk, hence the call going out to some of those laid off, asking for them to come back. More slash and burn than measure twice, cut once.

And Twitter’s considered approach to changes is out, replaced by arbitrary deadlines and hunches. $20 for Twitter Blue? No, how about $8. Available on Monday. Or Tuesday. Could be next week.

One common reason that people have left Twitter previously is the volume of hate and harassment. While I don’t doubt their experience, it’s not something that I’ve seen personally. I stay in my little bubble with tech and jokes and a bit of politics.

But it doesn’t feel like we’re heading in the right direction. Musk’s naive views on free speech are perhaps the most worrying, not in the sense that they have the most direct, immediate effect but because they demonstrate that he doesn’t Get It.

My hope is that Musk quickly learns and pivots to a more sensible, nuanced position. But his recent tweets about American politics and abandoning putting warnings on COVID misinformation makes me think this isn’t likely. He seems to think that the problems at Twitter are about the technology, that removing a few micro-services and adding a few blade servers will make a difference. However, the problems are all about people, those who use the platform, those who advertise on it, and those who work there. Until he understands that, or defers to someone who does, things will continue to spiral.

In the end, as an end user, Twitter is all about the people I interact with every day. If they leave, it doesn’t matter whether it’s because of something that Musk said or did, or not. Their absence will make the site not worth visiting any more.

In short, I stay on Twitter despite the company that runs it and despite the person who owns it. I’m there for the geeky discussions, the dad jokes and despairing at the state of British politics. If that goes away, so do I. Find me here if that happens.


  1. Most long-time Twitter users think it’s terrible. While it does occasionally surface interesting Tweets, I do think I’d prefer the original reverse chronological timeline, too. ↩︎

Panic

The whole team got this email today. Okay, it wasn’t today and these are not the exact words, but it was something like this:

We have a serious regression in build 456. We have set the project back rather than taken it forward. We need the utmost focus and commitment on fixing it. We’ve broken it and we stay in the office until it’s fixed.

I’ve had a few of those messages over the years and while it’s intended to focus minds it often has the opposite effect. Let’s examine why.

Projects see the same mistakes made over and over and this email encompasses many of those sins; it’s one message but represents a microcosm of large part of my career.

Here are a few problems that I see immediately:

  • No problem definition
  • No person accountable
  • No next action
  • A deadline but no understanding of the work involved

This has a number of consequences.

There are studies that show if you have a heart attack in a crowded area you are less likely to receive life-saving CPR from a stranger than if you’re in an area with one other person.

In this case the passers by (the project team) don’t know that they’re needed. Without a problem definition I don’t know if the regression was caused by one of my changes or even if it affects my code. Without a person accountable everyone likely assumes that it’s someone else’s code. “Someone would have mentioned it if it was my code.” And with no “next action” it’s easy to assume that someone else will handle it.

Arguably the deadline is not really a deadline. What if the fix would take a week to implement? Instead it’s a target. You can’t take an estimate and reduce it to fit to an externally imposed date. It doesn’t work like that. You may hit your deadline if you’re lucky, but a good plan doesn’t need luck.

Even worse, the arbitrary deadline and lack of direction gives the entire project a sense of panic. I think the intention was urgency but urgency implies you know what you need to do and that you need to do it quickly. As we’ve seen, the task above is neither well defined nor assigned. The only clear things in the original email are the version that is broken and the deadline.

However, the biggest sin is questioning the commitment and competence of the people needed to resolve the issue. In my experience, this is rarely the case, yet asking the question can make it true. If you’re not trusted, why put in the extra work? Next time you need to make a change, are you going to do it the “right” way or the way with the absolute lowest risk? Putting in a lot of good work and then getting kicked for your efforts is not a good incentive for doing a job well.

Project versus Product

With the fuss about the Log4Shell vulnerability finally dying down, it’s time to step back and take a good, long think about what happened and, more importantly, what can be done to stop it from happening again.

Sadly the prognosis is not good. The tl;dr is both simple and obvious: we simultaneously like free stuff and getting paid for our own work.

Most companies treat open source software exactly the same as commercial software but with a much lower purchase cost. When the software goes wrong, we want someone else to fix it for us. Unfortunately, sometimes we don’t even know where the software comes from. In the case of log4j, it’s run by volunteers. There is no 24/7 help desk with eager employees waiting to take your call.

But even if there is a company that backs the project, one that does have engineering, QA and support staff, is it reasonable to expect an immediate fix to a vulnerability?

Whether a company backs it or not, using open source software is more like being part of a community than being a customer. I came across this phrase about free software a few years ago: “If it breaks, then you get to keep both pieces.” It’s very apt here.

My small part in trying to fix this perception is by calling free software projects and commercially supported versions products or services.

The idea here is that the word “project” implies some degree of a work in progress, one that requires effort from all stakeholders.

In my Day Job1, I often see companies expecting to get commercial quality support for free because the software is free. By “commercial quality” I don’t mean “good.” The level of knowledge and support provided by most free projects is phenomenal. Instead, I mean that there are service-level agreements and guarantees of service within a particular time frame.

As it says in “Cloud Without Compromise”:

But there is something to the old adage that “You get what you pay for.” (… there’s a world of difference between building systems for pet projects versus designing for the needs of enterprise.)

Those service level agreements come with a cost. If you want a fix or an enhancement, you’re welcome to ask a project for it, but it might never come or, if it does, maybe not on a timescale that helps you. The rules change when you pay for the help.

My hope is that those companies that only ever take from open source projects and never contribute learned a lesson. By helping keep the project healthy, you don’t even need to be altruistic. Think of it as insurance.

If you use the software, there are many ways you can “pay it forward”: share enhancements and fixes, write documentation, share your knowledge and experience to bring in more users, or help other users with community support. But if there’s a mechanism to pay for it, the simplest way for most users is cold, hard cash.


  1. An open source database, so I have both skin in the game and bias. However, I think I’d be saying the same thing even if I worked for a vendor of closed source software. ↩︎

Security by Scapegoat

As is common these days, I was complaining about something on Twitter.

https://twitter.com/sdarlington/status/1523588282986033152

It’s easy to complain about security practices which, if I’m honest, is why I do it. But there is an important point, one that I included in a follow-up tweet:

https://twitter.com/sdarlington/status/1523602044791115776?s=61&t=69wO28ER8NUpssCyeNkqJw

The security team in many companies models itself on the DUP. Say no to everything. But – and this is the key – offer no alternative.

The tweet above is about passwords but I see it everywhere. Another common one is transferring files. I understand why sharing files can be problematic. Confidential data can be exported, either deliberately or accidentally. Viruses can be imported. Security defects can be exploited.

So yes, blocking OneDrive or DropBox is part of the job. What is missing is a legitimate alternative.

Security teams should be enabling staff to safely perform their jobs. Instead, they block the insecure methods and stop.

If I need to share a file or remember a complex password and you don’t provide suitable tools, you did not prevent a security problem. You forced people to write their password on a PostIt note and stick it on their monitor. You pushed someone to use some dodgy new file sharing service that you haven’t heard of.

In the attempts to make the system more secure, you, best case, prevented someone from doing their job. Worst case, you pushed someone into doing something insecure.

In either case, you effectively delegated security to everyone else.